The Challenge of Slow LLM Inference
So one of the key issues that we all face when working with large language models locally, or even serving them in the cloud ourselves rather than using an API, is just getting them to be of a speed that you can use them and that people and users can actually get them to feel like they're prepared to wait for the response, that kind of thing.
One of the challenges has been, if you're serving eight-bit models or 16-bit models from the Hugging Face platform, often they're just so slow that they're not practical for using, even if you're using an A100 GPU, et cetera. And there are a whole bunch of different reasons for this. And there are a lot of things around some of the sort of fast attention and some other mechanisms that people have been using. I think some of the MPT models have been really quite good at actively trying to speed up inference for this.
But unfortunately, a lot haven't. And while Hugging Face has basically created their own text generation inference library, it can be a real pain to use, especially on something like Colab or using it locally.
So this brings me to the topic of this video, which is a new serving library called vLLM.
Introducing vLLM and PagedAttention
And I've been using this over the past few weeks. I must say that it's very impressive at the speeds that you can get and also, just the ability to be able to run models that before were very slow and very tedious to run, and now being able to run them quite easily here.
Okay, so what is vLLM? So this is basically, they're calling this an easy, fast and cheap LLM serving with PagedAttention. And I'm not going to go hugely into PagedAttention. It basically is what is driving a lot of this speed up. And so you'll see, just first off, just by comparing the speed and how many tokens people are getting, you can see here that basically they're outperforming other things immensely compared to what's out there currently.
Now the way that they're doing this is they're using this thing called a PagedAttention. So, what they've looked at is that what's the key bottleneck. This is basically a whole thing to do with the cached memory that is often called the KV cache, in here. And what they're basically doing is trying to not use memory when they don't need to do it and to be able to basically efficiently use the memory much better than before.
And you can see when they do this well, it allows not only for a speed up of things, it also allows for better speeds on different kinds of sampling techniques, like beam search, that kind of thing, and for multiple outputs here. So this is their blog post. If you're really interested in it, go through it, have a good read of it and look at it.
Real-World Validation by LMSYS
Now, the thing that instead of impressed me and got me really interested in this in the start was that LMSYS are not just academics that are not actually using this stuff in production. They're actually the people who are running the Chatbot Arena, which I did a video on a couple months back. So obviously they're also aligned with the creators of the Vicuna model, as well.
So you can see that they've had a lot of interest in basically running a variety of models to compare and contrast the differences between them and allow people to be able to evaluate. To do that they've had to run a lot of models in the cloud and be able to serve those models to a lot of people. So this is where I think, what they've basically done is very interesting is that they talk about that initially they were using the Hugging Face backend for this. But then now this vLLM can actually get you about 30 times higher throughput than the initial Hugging Face back end here.
So, this is pretty amazing just to look at that, even as their traffic has increased, it seems like their compute needs are actually going down. So they've put up a Github, they basically have got all about it.
Key Features and OpenAPI-Compatible API
Currently, they're supporting a number of different models here including the LLaMA models, including the MPT models. And you can see, basically, this is some of the comparisons when they're comparing with Hugging Face text generation inference and just the straight up Hugging Face transformers libraries that they're able to do so much better.
Another thing that I really like with this, so they've got some nice documentation in here. Another thing that I think is really interesting idea and it's definitely useful to people in industry is that they actually have, so you can run this obviously in a notebook or something like that. you can also run your own API server for this, which is really good. And one of the cool things that they have on the API server is that they have this OpenAI compatible server.
So basically it means by running this, you can actually get it to be like a drop-in replacement for the OpenAI API protocol. So even if you were planning to use OpenAI for a lot of things, but you then want to be able to have a quick way to test out open source models or, you know, commercially available models, this will allow you to basically not have to change your current code that's using OpenAI, but just change where you're pointing the traffic at to a different URL and you can then basically serve one of these other models in the same way that you would be accessing OpenAI.
So this is a hugely useful thing If you're in an organization that's already using OpenAI out there. Okay let's jump in and have a look at some of the code and see how it turns out.
Code Demo: Setup and Fast Text Generation
So if we jump in here, you can see I'm going to be using the latest Vicuna 13B v1.3 model here, but you could really pick any model. So you've got a smaller GPU, try it out on one of the 7 billion parameter models, something like that. You can see the main thing I'm just bringing in here is just this install, vLLM there.
And we can basically just bring it in, set it up, and then start generating. So they've got some things, you know, just that's from their standard code there for generating. Now I've basically put in the system prompt for this, and set this up to basically do the sort of generation in here as well. And, I'm not going to go through the content output too much here. We're just going to look more at the times for this.
So, if you remember back to some of the previous videos I've made, things like this would often take, with maybe loading in eight bits with just the standard Hugging Face, could take up to two minutes for some of these generations. And you see here, this is basically generating in 13 seconds. The capital of England, generating in 2.34 seconds. Right? The one of the ones that I test a lot is, writing an email to Sam Altman. Here you can see the quality of the output is not changing compared to the model at all, but we're getting generation times of 13 seconds for 512 tokens here. And you see another version of this here doing, in under 13 seconds to basically generate out.
Code Demo: Specialized Tasks and JSON Extraction
So have a play with this yourself and see what you want it to do now. Where I think this really shines is not just in some of these normal sort of things, but where you've got a model and you want to use it for a very specific task. So one of the tasks we have is an example of this one, right? Convert the following to JSON. And I've got some information in there. And basically I'm just doing an extraction to get this out into JSON format. Now you can see here, it's done that. And it's done in 1.67 seconds, which is blazing fast for this.
So this allows you to do a lot of things with a decent quality model that you're maybe not going to use for everything. And maybe you're still going to use OpenAI and GPT-4 for certain things, that kind of thing. But for a lot of tasks, you can get them done with a sort of medium size or small size, local model and running with this in inference mode to get your results out.
Conclusion and Final Recommendations
You can see a final example is, you know, writing the three-day trip to London. Again, this is the quality of the output is being determined by the Vicuna model. So we could have picked a Wizard uncensored or any of those models that you might have a favorite for. But look at the generation times here. They're much faster than using the Hugging Face way or even using the text generation inference way for doing this.
Anyway, have a play with it. It's certainly something that you can use for production. Like I said, one of the things I really like about it is it has an API that you can just drop in as a replacement for OpenAI which makes it very easy if you've got a team working on this to test models out quickly and not have to change their backend code all the time for this kind of thing.
As always if you've got any questions please put them in the comments below. If you'd like the video please click like and subscribe. I will talk to you in the next video. Bye for now.
