Made withvLLMClip

Deploy LLMs using Serverless vLLM on RunPod in 5 Minutes

A

AI Anytime

Creator on the vLLM channel

Watch on YouTube ↗

25K viewsJul 21, 202414:13 runtime

Learn to deploy Large Language Models (LLMs) efficiently and affordably by leveraging vLLM on RunPod's serverless platform. This guide demonstrates how to set up a high-throughput, memory-efficient inference engine for models like Llama 3 in just five minutes. Discover the power…

Chapters

#vllm#clip#aillm

Key Concepts: Deploying LLMs with vLLM and RunPod Serverless

This document summarizes how to quickly deploy a Large Language Model (LLM) using the vLLM inference engine on RunPod's serverless platform.

Why Use vLLM and RunPod?

vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs. Its performance is powered by technologies like paged attention.
RunPod: An affordable and secure cloud GPU provider. Its serverless offering simplifies deployment by managing the underlying infrastructure, allowing you to focus on your application.
Combined Power: Deploying vLLM on RunPod provides a fast, cost-effective, and scalable way to serve LLMs with an OpenAI-compatible API.

Deploying an LLM on RunPod Serverless

Navigate to Serverless: In the RunPod dashboard, select the "Serverless" option.
Select a Model: Choose an LLM from Hugging Face (e.g., meta-llama/Meta-Llama-3-8B-Instruct).
Provide Token: For gated models like Llama 3, provide a Hugging Face access token.
Configure Settings: Review optional settings like quantization (GPTQ, AWQ), KV cache, and custom chat templates.
Name the Endpoint: Give your endpoint a unique name (e.g., runpod-demo-llama-3).
Select GPU: Choose the appropriate GPU hardware for your model.
Deploy: Click "Deploy" to initialize the endpoint.

Tip
GPU Selection: A good rule of thumb is to select a GPU with VRAM at least 1.5 times the size of your model, especially for quantized models. For an 8B parameter model, a 48 GB GPU (like an L40) is a safe choice for unquantized inference.

Interacting with the Deployed Endpoint

RunPod UI: You can send initial test requests directly from the RunPod endpoint page to verify it's working.
Programmatic Access (Python): The endpoint is compatible with the OpenAI library. You can interact with it using your RunPod API key and the unique Endpoint ID.

# pip install openai
import os
from openai import OpenAI

# Set up the client
client = OpenAI(
    api_key=os.environ.get("RUNPOD_API_KEY"), # Use your RunPod API Key
    base_url=f"https://api.runpod.ai/v2/{YOUR_ENDPOINT_ID}" # Use your endpoint ID
)

# Make a chat completion request
chat_completion = client.chat.completions.create(
    model="instruct", # Can be any string
    messages=[
        {"role": "user", "content": "Write a FastAPI code to make an API call using the requests module."}
    ],
    temperature=0.7,
)

print(chat_completion.choices[0].message.content)

Important
The first request to a new endpoint will be slower as RunPod needs to download the model and initialize the container. Subsequent requests will be much faster.

Best Practices and Cleanup

Production: Using a managed service like RunPod with an efficient engine like vLLM is ideal for building production-ready applications, as it handles scaling and infrastructure complexity.
Resource Management: Always remember to delete your endpoint from the RunPod dashboard when you are finished to avoid incurring further costs.
Security: After your work is done, it's good practice to revoke or delete the API keys you used (e.g., your Hugging Face token) if they were exposed or are no longer needed.

Introduction to vLLM and RunPod

Hello everyone, welcome to AI Anytime channel. In this video, we are going to explore vLLM on RunPod. So, vLLM is a high-throughput and memory-efficient inference engine. So basically what it means is that you can inference LLMs faster. You can run large language models faster. And we're going to deploy this on RunPod in a very serverless manner. So if you don't know RunPod, it's a GPU provider, one of the most affordable GPU providers in the world right now. I think the most affordable and the secure as well. So there are other competitors like Vast.ai, PaperSpace, blah blah blah, but we're going to look at in this video how we can use this on RunPod.

Now if you look at here on my screen, I am on vLLM's GitHub repository. It has around 24k stars. "High-throughput and memory-efficient inference and serving engine for LLMs." One of the best innovations that has happened in the recent times when it comes to large language models. And you can also... you can do a lot of things. They are basically based on something called paged attention. So I'm not going to cover that in this video, but if you look at here it says "Efficient management of attention key and value memory with paged attention." But the focus is that how you can leverage vLLM on RunPod.

Getting Started with RunPod Serverless

So here I am on RunPod right now. You can see I have around $10 in my RunPod account. And if you log in for the first time, you have to come here in the left-hand side and search for this serverless. Now, what do you mean by serverless? That you do not have to worry about the infrastructure management or you do not have to create infrastructure manually and deploy these models. You just have to select large language models and a few of the settings that you want to, depending on the model that you select. And then you just deploy it, click on a button to deploy, start, whatever. And then you just use this deployed model as an LLM endpoint.

The good thing about this serverless vLLM is that it's OpenAI compatible. So you can just use the OpenAI library and their way of generating content using OpenAI. So they provide chat completions and a lot of other features in the OpenAI library, and you can just do it. So there are endpoints when you deploy it here; they are compatible with OpenAI.

Configuring the LLM Endpoint on RunPod

So let's start here. So what I'm going to do is start. When you say start, it asks for Hugging Face models. When you click on this, it takes you to all the model thingy. Now it suggests OpenChat 3.5, you can take any model. So you can see you can take Llama 3, you can take Mystral, Nemo, recent model. You can take Llama 3 8B Instruct. So for example if I take Llama 3 8B Instruct, let me just copy it here, come here and paste. So you can see I have just pasted my model name.

And if it's a gated repository, then you have to give your token. I'm going to select my token. I'm going to, of course, make this token expired on Hugging Face. Let me just do next. Now here are some LLM settings. For example, it also supports quantized models. So if you want to use GPTQ or GGUF, you can also use quantization. You can see it suggests three different quantization right now, GPTQ being one of the most used one. Then we have AWQ quantized and then we have SqueezeLLM and a lot of other things.

You have Max model length, maximum number of tokens for the engine to handle per... these are optional. Load format optional, base path, these are fine. You can also enable this if you want to do trust remote code, but I'll just keep this as it is. KV cache, we have some tokenizer settings if you want to bring up some tokenizer. You also have a custom chat template, which, for example, if you are using Llama 3 8B and if it accepts a specific chat template, you can also do that, but I'm not going to do it here. I'll just keep it as it is. I just want to show you how you can deploy and inference and use that as an endpoint. You have GPU and system settings, they are all fine. Streaming settings, these are all fine. OpenAI settings, these are all good. And serverless settings, so max concurrency, you can see, and these are all good. So let's do next.

GPU Selection and Model Deployment

The next is you're going to give an endpoint name. So let me just give it RunPod Demo Llama 3. And then you can select which GPU do you need. GPU selection depends on your experience as well, how much knowledge you have to select GPUs based on the LLMs. You should select 1.5x of GPU at least if you are taking a quantized model, if you want to run a quantized model on GPU. If you want to load this and run this as a default model without any quantization, then for this Llama 3 8B, if you look at here, let me just go up, you can see it's 8.03 billion parameter. I will go with at least 48 GB of GPU for this. So I'm going to select 48 GB GPU Pro. I think L4 provides how much, 24 GB if I'm not wrong. And then they have A800 here, 80GB, but I'm going to go with 48 GB GPU Pro. You can see it's L40. Yes. So L40, L40S, 6000 Ada, depending on the GPU that they have available, they will make it available for you. You can also work with multiple GPUs. You can select your GPU workers also, you can also select one more GPU if you want. But I'm happy with this. And then I'm just do next deploy. Once I do deploy it, it says successfully deployed endpoint.

First Inference and Endpoint Monitoring

And here, after you see this ready and stuff, you have to come here on the request, and you can see there is already a payload, you know, prompt "hello world". I'm just going to do something like, you know, "what is AI?" or something. Okay, and then you can just do run. When you do run, it will be in queue for the first time because it has to download the model and everything that has to do...

Let's go on the logs here. So when you click on logs, it will show you two logs: system logs, which we don't care about mainly in this right now at least as of now. What we care about is the container logs. So it says, "No container logs yet. This usually means the pod is still initializing." So you have to keep a check on the container logs. The reason being, because container logs, the first time when you do it, the model has to be downloaded from Hugging Face, it has to be up and running, started and everything. And you can see it's right now in queue. It takes a bit of time when you run it for the first time, so you have to be a bit patient for the first time, but after that it will be fast. Right? So what I'm going to do is pause the video and come back once this is done.

Using the OpenAI-Compatible API in Python

Alright, so guys, you can see that we got the output here. It took around 30 seconds for the first time. Don't worry if you don't get a long response, that is fine because we are not handling the inference parameter right now. We are going to do that in a bit. Let me show you something. As I said that this is OpenAI compatible, you have the OpenAI base URL that we're going to use. You also have a run that you see here, this run. So mainly if you don't want to use the OpenAI endpoint, like through OpenAI endpoint or through OpenAI libraries, then you can also do that without it as well.

Now, how we can do it? So you need an API key. So here in the settings, you can find out API keys. And once you have the API keys, you can come here on Colab. You can see I'm running on Colab. Let me just run this pip install OpenAI. I'm installing the OpenAI thing here. And from OpenAI... so let me just go back because I have to copy my... come here on serverless, you will see this is running. This is what your endpoint ID is. So let me just get this. Now I'm going to define my RunPod ID here. So let me just do that. And I already have my API key set up in here in the left-hand side in env secrets. So I'm just going to run this. And then I'm initializing this client, you can see through OpenAI. So let me just do it here.

Then I'm just giving a name. So you can give a name here. So for example, I just want to give this name called instructor or something. And then here, let's take the default question I got it from their documentation. I'm going to keep this 0.7, a bit creative. And then it takes your model name. It has a role user, like a pretty standard template. You can change the chat template, guys, that's up to you. Yeah, so you can see it says, "I apologize, but I'm not familiar with a platform called run sport." Yeah, because the knowledge cut-off has been there, it doesn't know. So that is fine. So what I'm going to do is probably ask a question, how to... or not how, "write a FastAPI code to make an API call using requests, using requests module." Okay, let me... is it request or requests? It's fine, I think it should understand at least that much. If it cannot, then it's not a model to be used, okay, to be honest. So, write a FastAPI code. So let's see if it's able to get this one. So you can see requests, by the way, it's able to get it. So it's my bad. Okay.

Now you can see such a beautiful output that we have got here. Here is an example of how you can use the requests module to make an API call from a FastAPI application. It brings up the imports, you initialize the app, uses just a GET method, and uses just async, asynchronous, one of the features in FastAPI. And then it explains the code. Fantastic. I love this output, guys. You can use max tokens, you can increase the max tokens as if you want depending on your need.

Conclusion: Production LLMs and Resource Cleanup

And this is how you can deploy any LLM that vLLM supports. So here is vLLM: Easy, fast, and cheap LLM serving for everyone. If you are someone who wants to do production level thing, because locally you cannot build, right? If you want to build it for a client, you have to deploy these LLMs through a high-throughput and memory-efficient inference. So you have Unsloth, you have vLLM, you know, you have TitanML coming, I don't know when they will come in, but they're not open if I'm not wrong, correct me if I'm wrong. And I love open source things. So vLLM is your choice. vLLM is the best when it comes to this. So you just deploy vLLM through RunPod, take a RunPod instance. You can always upscale, downscale, you can stop the service whenever you want.

So let me go here and you can delete the endpoint here. So let me just delete the endpoint. I'm just going to copy here, copy this, come here, and delete. So I deleted the endpoint. So you can see now there is nothing. I'll go to Hugging Face thingy. As you guys have seen my API key, so I have to delete it. So I'm just going to delete this. Or you can delete anything you want, and you can always create a key. That's not a big thing. So I deleted all my keys, so you can't do anything with the keys now. So this is done. RunPod is done. vLLM has been covered, how you can do it. So it's very easy, guys, not rocket science.

So let this services handle all the infrastructure management, all the complexity. You focus on building applications and projects using these LLMs. If you have any questions, thoughts, or feedbacks, let me know in the comment box. You can also reach out to me through my social media channels. You can also join our Discord community. I will give the link in the description. If you like the content, please hit the like icon. If you haven't subscribed the channel yet, please do subscribe the channel, guys. That motivates me to create more such videos in the future. That's all for this video, guys. Thank you so much for watching. See you in the next one.