Introduction & Prerequisites
We are now going to be setting up VLLM and we're going to be doing this locally. I'm going to be deploying this on the Quad 3090 rig. Again, 512 gigs RAM for 3090s. And also, we've added in some additional flash storage because I anticipate some big models coming down the pipeline pretty soon.
Following along with the guides that we've prior put out is something you need to do so that you have your open web UI up and running, which we will be using that for our chat interface. Of course, the first guide was O Lama and Open YBUI in an LXC container. The second guide was Llama C++. And in that guide, we created a backup that we're going to use today for really fast restoration. And this is actually one of the really cool parts about doing it the way that we have done it here.
Restoring & Configuring the LXC Container
So, the next thing that we're going to do is restore our image. Just click on the image. Go ahead, click on restore. I'm going to restore this to our much larger size. I've got two NVMEs over there. I'm expecting DeepSeek to drop their latest updates here really soon. So, V LLM is the host name we're going to use. I'm going to use 60 cores. The rest of it is going to prefill and you can just leave it as it is. Go ahead, click restore. This actually is pretty quick.
There will be two things that we are going to change after this image is up and running. The first is I'm going to give it some more hard drive space, which is allocated on the fly. Don't overallocate your space. That's actually one of the most constraining things aides from RAM in most home lab setups. The second thing that we're going to change is the IP address that we use. That'll be basically for the API. It's going to retain the prior IP address that we had statically set at 71. So I'll show you here really quick how to change those two things in our container.
Come over to network and double click and change that from 71 to 72 and click okay. That's how fast that one is. And then come over to resources. Click the disc. You can see that's 40 gigs. Now we're going to be using the smallest I think that I could find out there which is Janv14B. It's a 4B, but this should be applicable and demonstrate the exact same steps you would need to take for any of the models that you might find out there. It's just do you have enough VRAM? And 4B that should run on almost any graphics card out there. If you got 12 GB of VRAM, 16 GB definitely should get it, I think.
Going to volume actions and resize. I'm going to give it 20 extra gigabytes of space. And it's done. Just that fast.
Installing vLLM & Dependencies
So, come back over here to your VLM login. And the next thing you want to do is come over to digital spaceports updated guide for VLM. And I did get the hyphen hyphen thing figured out and working. So, hooray. All right. And so, we'll just copy and paste into here.
Now, the first thing that we're going to do is install Python 3 and pip. And we're going to use something called pip in, which is which is an environment manager. Very simplistic environment manager. You could use more complicated things like UV and other stuff, but I I think pipin is super duper basic. So, we're going to use that here. The next thing we're going to do is come over and scroll down a little bit. Next, we're going to install the VLM software, flash attention, transformers, and flash infer. So, copy that. Come over here. Paste it in and let it go. This one will take a little bit longer to install. for sure a little bit longer to install.
But at the end of this, we'll be ready to actually take our next step, which is pulling the model down. That's literally how fast it is. It's just launched. It's not yet out in a guff. You're waiting for Unsloth. You want to play with it. You can use VLM. You can use VLM for a lot of things. And as I have mentioned in all of these guides, none of these guides are deep dives on performance or how to do things. These are just the base level getting you up and running with a really solid, very competent system that you can use for rapidly creating and testing out things and have safe, secure backups. Basically building a better system is what this whole guide set has been around. And yeah, we've gone through four pieces of software. Well, five if you include Proxmox. But this has been a pretty decent I would guess hour and a half total time of information conveyed for me to you to get you up and running with the latest and greatest and the capabilities to run the latest and greatest that always almost is going to be coming out in safe tensor format on hugging face first.
Launching the vLLM Model Server
Okay, that definitely took some time but it did get done. I'll talk through it with you as we go through it here. Let me type reset really quick so you can get a less distracting view of it. If you we'll just start from the front and kind of work backwards. And we're going to be serving. So this is basically setting up and running a server API that we're going to be accessing for the JAN HQ JAN V14B. Like I mentioned, small model should run on most everything. Max model length, we've got that at 32K. The API key that we created is Nerdastic. Port 42069. Host 0.0.0.0. This will allow us to access it from the IP address of the machine which we just set to 72. Tensor at parallel I've got set to four because I've got four GPUs there. Uh, and we also have enable auto tool choice and tool call parser is set to Hermes.
So, as you run this, what it's going to do is reach out to hugging face. Well, first it's going to check its cache and see, do I have it? If it doesn't have it, it's going to reach out and pull down the model files. You may need to enter a hugging face API key for certain models out there if you're downloading them. This model in particular I knew was just going to work. So, I thought this was a good one to go to. We'll have a whole hugging face and transformers video. Yes, hugging face and transformers video will happen here at some point, which will go through all that and I'll show you how to set things up for downloads, specify a specific directory you want your models to go to, not just kind of the default. And you can see it pulling down and downloading them right now.
So this is pretty cool. This is it. Like this is literally it. As soon as we get a couple of green lines here at the end of all of this uh output, we'll be able to go to our open web UI instance and that'll be it. That's pretty simplistic. And that's what really all of this guide has been about is how to get you guys up and running with the most hasslefree, clear, clearly understandable to you and simplistic setup that you literally are going to have a hard time messing up. And if you do mess it up, you've got really easy backups and snapshots that you can go to.
Connecting vLLM to Open Web UI
All right, and we now have this up and running. Let's come over to our open web UI. Now I'm in the settings and this is a little bit confusing because there's multiple settings. But if you go to the lower left and click settings and then click admin settings and then connections, that's where we're at right now. I've already got this one kind of hooked up. So let me just go ahead and show you a couple things that you probably want to do. So the prefix ID, I've prefixed it with a VLM and a hyphen so that I know that it's coming from VLM. Any of the models that show up in the drop down from the selector for our key, we've got Nerdtastic. We've got the IP address and port and forward slashv1.
Hit the little verify connection. You should green up over there. Click save. And you can see that that's all you really got to do. I did also change the prefix ID for the Llama C++ instance as well, so that I know where they're coming from whenever they show up. I should actually at this point in time do this one for this uh oh, Llama. There we go. And yeah, that one's local. So we now have three backends connected to our open web UI interface. If we come over here, you can see Jan is loaded up there and I'll just uh drop it a quick hey Jan.
Now it's not going to display the tokens per second directly here. But you can see that it's actually quite fast. Ask it how to get started using Quinn effectively. And of course JAN is a fine tune of Quinn. So, I do expect that we will be using VLM hopefully in the pretty near-term future for running the latest DeepSseek, which I expect the V3 Instruct, which is usable by humans, to be out sometime probably in the next 24 hours. I mean, fingers crossed there. The base model, like I covered in the prior, hey, it's out. Uh, the base model is not usable uh for most intents and purposes, but there you have it. You are now up and running with VLM.
Recap, Snapshots, and Final Thoughts
So, in this guide, I showed you the very fast steps to restore your backup, change your IP address, provision some additional storage space, and download a model after you've installed VLM and its dependencies inside a shell. And this is not at all a comprehensive guide on pip or anything Python related, but the nature of the beast essentially is Python that we're dealing with here with VLLM snapshot.
And just take that snapshot there. That's it. Now I have a roll back to a known good spot and no further configuration would be needed. I think this is pretty cool. Let me know what you guys think in the comments below. Big shout out to our channel members and everybody who's been tossing me $5 or whatever you guys toss me. I don't even know these currencies, but thank you guys that are tossing me these currencies. Of course, our buy me a coffee people and our Patreons. Big hats off to you as well. Of course, you can find that on digitalspaceport.com.
This is really the culmination of a lot of effort over the past almost year that has led me to the clearest way to get some complex stuff uh up and running very simplistically for you. So, make sure to check out this video to get up and running with Open Web UI and O Lama, our first container in Proxmox for our Lexe guides and then our Llama C++ that is a prerequisite since we just restored that backup and did a little tinkering to it as we showed you how to create everything up to that point in the prior guide. So, those two work together to give you where you're at today with this one. Make sure to hit like, subscribe, and I look forward to everybody's comments down below.
