Made withvLLMClip

Install and Run Locally LLMs using vLLM library on Windows

A

Aleksandar Haber PhD

Creator on the vLLM channel

Watch on YouTube ↗

13K viewsNov 9, 202511:46 runtime

Learn how to install and run large language models (LLMs) locally on your Windows computer using the high-performance vLLM library. This comprehensive tutorial guides you through setting up the Windows Subsystem for Linux (WSL), installing the necessary NVIDIA CUDA Toolkit, and…

Chapters

#vllm#clip#aillm

Overview

This guide details how to install and run Large Language Models (LLMs) locally on a Windows machine using the vLLM library. The process leverages the Windows Subsystem for Linux (WSL) to create a compatible environment for vLLM and its dependencies.

Prerequisites

A Windows computer.
An NVIDIA GPU.

Step-by-Step Installation Guide

1. Install Windows Subsystem for Linux (WSL):

Open PowerShell as an administrator.
Run the command: wsl --install.
This installs the default Ubuntu distribution (e.g., 24.04).
After installation, create a UNIX username and password when prompted.

Important
A computer restart is recommended after installing WSL to ensure all changes take effect.

2. Install NVIDIA CUDA Toolkit:

Inside your Ubuntu WSL terminal, find the installation commands for WSL-Ubuntu on the official NVIDIA developer website.
The video uses version 12.6, but the process is similar for other versions.
Execute the series of wget, add-apt-repository, and apt-get install commands provided by NVIDIA.

3. Configure CUDA Path:

To make the CUDA compiler accessible from the command line, you must add it to the system's PATH.
Edit the .bashrc file in your home directory: nano ~/.bashrc.
Add the following line to the end of the file, adjusting the version number if you installed a different one:
```
export PATH=/usr/local/cuda-12.6/bin${PATH:+:${PATH}}
```
Save the file, restart the terminal session, and verify the installation with nvcc --version.

4. Set Up Python Environment with UV:

Update system packages: sudo apt update && sudo apt upgrade.
Install curl (sudo apt install curl) and build essentials.

Install UV, a fast Python package manager:

curl -LsSf https://astral.sh/uv/install.sh | sh

Create a project directory (mkdir test_llm), navigate into it, and create/activate a virtual environment:
```
uv venv
source .venv/bin/activate
```

5. Install vLLM with PyTorch:

Use uv to install vLLM and the specific PyTorch version for your CUDA installation.

For CUDA 12.6, the command is:

uv pip install vllm torch --extra-index-url https://download.pytorch.org/whl/cu126

Tip
The tag cu126 in the URL corresponds to CUDA 12.6. If you use a different version (e.g., 11.8), you would change this tag to cu118.

Running an LLM with a Python Script

1. Create the Script:

Create a Python file (e.g., test1.py) and import the necessary vLLM classes:
```
from vllm import LLM, SamplingParams
```

2. Define Prompts and Parameters:

Create a list of strings containing the questions for the model.
Initialize SamplingParams to control generation (e.g., temperature=0.8, top_p=0.95).

3. Load the Model & Generate Output:

Initialize the LLM class, specifying a model from Hugging Face.

The llm.generate() method takes your prompts and parameters to produce output.

# Specify model from Hugging Face
model_name = "TheBloke/Quant-GPT-HF"
llm = LLM(model=model_name)

# Generate responses
outputs = llm.generate(prompts, sampling_params)

4. Execute the Script:

Run the script from your activated virtual environment: python3 test1.py.

Note
The first time you run the script, vLLM will download and cache the model from Hugging Face. Subsequent runs will load the model directly from your local disk, making them significantly faster.

What is vLLM?

Hello everyone and welcome to machine learning and AI tutorials. In this tutorial, we explain how to install a VLLM library on a Windows computer. The VLLM library is a fast and easy to use library for large language model inference and serving. It's originally developed by one of the labs at UC Berkeley and over time it has evolved into a community-driven project with contributions from both academia and industry.

Step 1: Install Windows Subsystem for Linux (WSL)

Okay, let's start immediately. The first step is to install a Windows subsystem for Linux on your computer. Most likely you already have a Windows system for Linux. However, let's try to install it. Click here and search for PowerShell. Then do the right click and click on run as administrator.

Click on yes. And now you're in Windows PowerShell. And then to install Windows Subsystem for Linux, you just need to press this command. And this will download and install the default Ubuntu version, which in my case it's Ubuntu 24.04.

In some cases, you will have to restart your computer after running this command such that everything is being properly installed.

Step 2: Configure Your WSL User Account

After that, you need to set your account. You need to create a default unique user account. I'm going to use for example this one. And over here, you need to enter your password. Then you need to reset the password or basically retype the password. And that's it. Now Windows Subsystem for Linux is installed. In some cases, you might need to restart your computer for these changes to make effect. Consequently, my suggestion is to restart your computer.

After restarting the computer, let us test the Ubuntu installation. To do that, click here and search for Ubuntu. And over here, click on Ubuntu app. And this should start the default Ubuntu version. Let's see what is the version that we are currently running. To do that, simply type lsb_release -a and you should see the default Ubuntu version.

Step 3: Install the NVIDIA CUDA Toolkit

In this video tutorial, I assume that you have an Nvidia GPU and consequently to use Nvidia GPU together with VLLM, you need to have Nvidia CUDA toolkit installed on your system. To install Nvidia CUDA Toolkit, simply search or Google NVIDIA CUDA Toolkit. In my case, I'm going to install the version 12.6. However, you can also search with without the specific version number. And this will lead you to the newest version. However, I don't want the newest version. So, click here. Click on downloads. And the procedure is the same for any version. Click on Linux. Select your operating system or better to say architecture. And over here click on WSL Ubuntu click on version two and click on deb or network settings in order to get these installation commands.

So let's now run these commands. First of all copy this. Go back to your Ubuntu and run this thing. Then let's go with the next command. Run this command. Enter the password. And then finally, let's update everything.

This will take a while. And let's wait.

Step 4: Finalize CUDA Installation and Configure Path

And finally, let's run this command which will actually install Nvidia CUDA toolkit. And here we need to be patient. This will take a while.

The next step is to add the Nvidia CUDA toolkit compilers to the system path. For that purpose, we need to go to the home folder by typing this command and then we need to edit the file called bash rc. We can do it with this command. Then go all the way down to the end of this file. And over here at the end of the file, you need to type this line. Here's the line. The purpose of this line is to add the Nvidia CUDA compiler to the system path. And if you have chosen to install other CUDA version, for example, 12.8 or 12.2 or for example 13 here, you need to adjust this version number.

Okay. So now to save this file simply press Ctrl O press enter and then Ctrl X to exit and now we are out and we can start with installation of VLM. The next step is to verify that Nvidia CUDA compilers are properly installed. To do that we need to close the current Ubuntu session and we need to start a new session. Click here. Search for Ubuntu. Then click on the Ubuntu app. And then in the prompt, simply type nvcc --version and you should see your current version if Nvidia CUDA compilers are properly installed.

Step 5: Set Up the Python Environment with UV

Next, let's verify the current Python version. Type this. And in my case, you can see Python 3.12. Good. Next, let's update and let's upgrade everything by typing these two commands. Enter your password and be patient.

Good. Next, we need to install all the tools. Although most likely the tools are already installed on your system, it's still good idea to try to install them by running these commands. Good. Then let's run this command to make sure that we have all the compilers on our system. And that's it.

Next, we need to install curl. To install curl, we need to run this command. And let's verify the curl version by typing this. And you can see that curl is installed.

The next step is to install UV. UV will be be used to create Python virtual environments. To do that, we need to execute this command. Let me just first clear and then here's the command. And this will install UV. Next, let's create a workspace folder by going to the home folder. Then let's call the folder test LLM. And then let's navigate to this folder.

Good. Then let's create a UV virtual environment. To do that, we simply need to execute this command. And here it is. Next, let's activate this UV virtual environment by executing this command. And now you can see it's active.

Step 6: Install vLLM and PyTorch

The next step is to install VLLM and to install PyTorch together. For that purpose, we need to run this command uv pip install. Then we're installing VLLM as a torch back end we are using actually as a back end we're using PyTorch and over here we need to specify the CUDA version. In our case since we installed Nvidia CUDA toolkit 12.6 we need to write CU 12.6. On the other hand if you have installed some other version then you need to adjust the version number for example 11.8. In that case you'll write 118 or 12.8. In this case, you will write 12.8 or 128 over here.

And press enter. And be patient. It's going to take a while to install everything. In my case, since I have already cached everything on my system, and I did that by installing and reinstalling things often. Everything is immediately installed. In your case, it's going to take a while to install everything.

Step 7: Code a Python Script to Run an LLM

The next step is to explain how to use VLLM. For that purpose, we are going to create a Python script called test1.py. And over here, I'm going to paste the code. Let me explain what's happening over here.

Since VLLM is a Python library, we can simply type from vllm import LLM and SamplingParams. Then over here in this prompts list, we are defining questions. We are defining who are you? Or we specifying who are you? What is the capital of the US? and who is the smartest person on the planet. These are the questions that we want to forward to our model.

Then we need to specify the sampling parameters. We need to specify the temperature and top_p. Now here is one very important thing. By default VLLM pulls the models from the hugging face and we can also pull the models from model scope. Let me explain how to pull the model. Here I'm going to erase this model name and let's say that I want to for example use and download locally, that is to execute locally quant-gpt. I will simply go to the hugging face repository, I will copy this, I will go to my code, I will paste this over here and that's it. Simple as that. Then I will call llm function then I will specify the model name, GPU memory utilization, and the max model length. That's it.

Over here, I'm generating the output. I'm specifying the prompts I'm using and the sampling parameters. And over here in this for loop, I'm simply iterating through the responses. And I'm simply printing out the response on the computer screen.

Let's save this file by pressing Ctrl+O. Or uh no let me just cancel this just because I selected something. Okay, so you need to press Ctrl+O to save this. And then Ctrl+X to exit and that's it.

Step 8: Execute the Script and Get AI Responses

Then we can simply run this code by typing Python 3 test1.py and that's it. Press enter and this will execute this code. Let's see what's happening right now.

First of all, we are going to download the model and after we download the model, we are going to execute the model. Note that model download happens only once. That is the first time you're running the code. The second time you're running the code, the model will be loaded from your local computer. So, let's see what's happening now.

Let's see. Let's be patient. And now we are downloading the model. As you can see over here, the model is relatively small. And here are the answers. Prompt: Who are you? Generated text: Can you tell me what I did wrong? I always think I doing something. What is the capital of the US? The capital of US is Washington DC. Who is the smarter person on planet? This question is subjective and can be debated based on different perspectives. Note over here that the response is truncated because I limited the response size.