Made withvLLMTutorial

vLLM: Easily Deploying & Serving LLMs

N

NeuralNine

Creator on the vLLM channel

Watch on YouTube ↗

52K viewsSep 5, 202515:19 runtime

Learn how to easily deploy and serve large language models (LLMs) using vLLM, a fast and user-friendly Python library for LLM inference. This tutorial covers three key use cases: interacting with any Hugging Face model directly in your code, deploying a model as a local server…

Chapters

#vllm#tutorial#aillm

Introduction to vLLM

vvLLM is a high-throughput and memory-efficient Python library for Large Language Model (LLM) inference and serving. It simplifies deploying and interacting with LLMs.

Key capabilities covered:

Python code integration for local inference.
Serving models with an OpenAI-compatible API.
Deploying custom-tuned models (e.g., in GGUF format).

Installation

A virtual environment is recommended. Install vLLM using pip or another package manager like uv.

pip install vllm

Use Case 1: Local Inference in Python

Directly use any model from Hugging Face in your code with just a few lines. Use vllm.LLM to load the model and vllm.SamplingParams to control the generation process.

from vllm import LLM, SamplingParams

# Load the model and limit VRAM usage to 70%
llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", gpu_memory_utilization=0.7)

# Set sampling parameters
sampling_params = SamplingParams(max_tokens=128, temperature=0.7)

# Generate text from a prompt
prompts = ["What is special about the Python programming language?"]
outputs = llm.generate(prompts, sampling_params)

# Print the output
print(outputs[0].outputs[0].text)

Tip
To prevent out-of-memory errors on systems with limited VRAM, use the gpu_memory_utilization parameter to specify the maximum percentage of GPU memory vLLM should use.

Use Case 2: Serving a Model as an API

Deploy any Hugging Face model as a web server that exposes an OpenAI-compatible API endpoint using the vllm serve command.

Server Command:

# Serve a model, utilizing 70% of GPU memory and protecting with an API key
vllm serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --gpu-memory-utilization 0.7 \
    --api-key your-secret-key

Interacting with the Served API

Use the official openai Python package to interact with your self-hosted model. Configure the client with your api_key and the local base_url.

Important
The API endpoint is located at the /v1 path. Your base URL must be http://localhost:8000/v1.

Client Code Example:

from openai import OpenAI

client = OpenAI(
    api_key="your-secret-key",
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    messages=[{"role": "user", "content": "Summarize Docker in simple words"}]
)

print(response.choices[0].message.content)

Use Case 3: Serving a Custom Fine-Tuned GGUF Model

vLLM can serve locally saved, quantized models like GGUF. You must provide paths to the model file, the corresponding tokenizer directory, and the chat template.

GGUF Server Command:

vllm serve path/to/your/model.gguf \
    --tokenizer path/to/tokenizer/directory \
    --served-model-name "my-custom-model" \
    --chat-template path/to/chat_template.jinja \
    --api-key your-secret-key

Note
The --served-model-name argument defines the unique name you will use to call this specific model via the API. The client-side code remains nearly identical; you just update the model parameter in your client.chat.completions.create() call to match this name.