Introduction and system overview
There are literally hundreds of ways now to install and serve large language models locally. One of the oldest and probably one of the most stable tools is vllm. I have been covering vllm for a long time now and I think that this project has evolved a lot and yet it maintains its simplicity and quality. In this video I'm going to show you in a very easy step-by-step way how to install and then serve a model with vllm. So let's get started. I am Fahad Mirza and I welcome you to the channel. Before I start let me also give a huge shout out to Mass Compute for sponsoring the VM and GPU for this video. If you're looking to rent a GPU on very affordable prices you can find the link to their website in the video's description. Plus I'm also going to give you a coupon code of 50% for a range of GPUs. This is my Ubuntu system and the GPU which I'm going to use is Nvidia RTX 6000 with 48 GB of VRAM.
Create a Python virtual environment and install vLLM
First up, as usual let's start by creating a virtual environment. This is not a mandatory step but I believe it is always a good practice to do that so that everything remains nice, clean, and separate. And that should be done now. Next up we need to install vllm. For vllm I'm also going to install torch and optionally you can go with torchvision and torchaudio depending upon the model but as I'm just going to use the text model, even if you just install torch and then vllm and huggingface-hub you should be good to go. Hugging Face Hub: the reason why we are installing it is that because we are going to download the model from that website and we would need to log in there. And you would also need to grab a free read token from your Hugging Face profile. So, for that, all you need to do is to just go to huggingface.co, sign in, and then from your profile, grab your free read token. So, let's wait for these to get installed, and meanwhile, let me also introduce you to the sponsors of the video, who are Camel AI. Camel is an open-source community focused on building multi-agent infrastructures for finding the scaling laws with application and data generation, task automation, and world simulation.
Choosing and starting a model download
And everything is installed. Next up, we need to download the model and then serve it. Now, you can see that I'm just going to use this vllm-use-v1, which is going to use a previous version of vllm for compatibility issues. And then, I am serving this Code Llama 2.5 coder model. You can choose any model you like. First time when you run it, it is going to download the model. And you see that I haven't really logged in to Hugging Face. That is because if you go to this model page on Hugging Face, you would see that this model is not a gated model, so you really don't need to log in. But, if you see that like Gemma models or a few others where you have to log in, you would need to log in to the Hugging Face Hub through the Hugging Face CLI. Okay, so let's go back and run this. This is going to detect the GPU on our system and then download the model. And the model is being downloaded. The size is just over 3 gig.
vLLM features and optimization options
While it downloads, let's talk a bit more about vllm because the project is quite cool, in my opinion. It gives you really state-of-the-art throughput. So, even if you want to serve the model in your own local area network in your company, I will show you that you can just simply access it through an OpenAI-compatible API endpoint through URL, and you can secure it with some SSL certificate easily. You can serve it even globally if you want to do that. Also, you can enable continuous batching of incoming requests, so it makes it really easy. Model execution is really fast and you will see it shortly. And we are going to use the full model in full precision, but you can go with quantized models like we do with GPTQ, AWQ in four, in eight and FP8. Also, it supports flash attention and flash inference. You can also do some chunked prefill just to optimize the memory efficiency. So, there are a lot of features that you can check out on their GitHub repo and I will drop the link to it in the video's description.
Model loaded, vLLM engine running and accessing the server
And meanwhile, the model has been downloaded and also loaded onto our GPU. And now it is just checking the model and then it is shortly going to serve it, which is completed. So now the vLLM engine is now running. So, just don't cancel it. Just minimize the screen and then I'm just going to open another screen or another terminal, I mean, and this is where I'm going to access this vLLM. Okay, so first, let me show you the VRAM consumption for this model. So, you can see that this is how much it is consuming, just over 44.
VRAM troubleshooting and running inference code
Anyway, so let me clear the screen here. And the reason why it is showing 44 is that I believe I have two models serving at the moment because this is not really possible for it to serve that much. So, let me show you what I'm saying. So, if you run this curl command from your terminal, you see that this is a model which is being hosted at the moment. And then, I had another model which is also loaded onto the GPU. So, because I run multiple videos, I think this is what is happening here at the moment. And that is why that much VRAM. Anyway, we will worry about this VRAM later. So, let's try to run the inference on this one, and I will show you what is happening. So, all I'm doing here is I am going to run this code, which is going to do the inference. Before I run that, let me show you the code. This is a simple code in order to do the inference and use the model. We're importing OpenAI. The reason is because it is an OpenAI-compatible endpoint. And this is the endpoint at our localhost at port 8000 with v1, which is typical with OpenAI-compatible endpoints. For the base URL, just do any random value here. This is the model we are using. And then, with the help of this instantiated client of OpenAI, we are passing it the prompt from here. And then, it is generating the text on the basis of these parameters, which you can, of course, change as you like. And then, you can give it any prompt of your choice. So, I am asking the question from this model to write me a Python function to reverse a string. So, let's go here and run this.
Demo result, recap, and outro
There you go. You see, it is almost instantaneous. And it has given a perfectly well answer. So, if you think about it, you can simply serve your model from here, and then, maybe from your network or from wherever, you can simply point your applications to this endpoint. And then, by using this simple code, you can have your own local inference client-server networking. So, this is the beauty of vLLM and as I said, it does a lot of other things that you can use if you like. Not only that, you can use vision models with it and you can use a lot of other sorts of models including other modalities and I have done heaps of videos on it if you're interested, just search the channel. Anyway, if you're just starting out with vLLM, I think this should be good enough. If you still have any questions, please feel free to reach out. If you like the content, please consider subscribing to the channel. If you're already subscribed, please share it among your network as it helps. Thank you for watching.
