Introduction: Why self-host LLMs in your home lab
Hey everyone, Brandon Lee here with Virtualization HowTo and AI is everywhere these days and it's making its way into the home lab. If you're like me, you are likely using AI tools more and more, especially with things like Kubernetes and help with automation and your DevOps coding. However, if you've ever wanted to run your own version of ChatGBT or Gemini completely offline with full privacy and no cloud dependency, you can actually do this by self-hosting open-source AI tools. Today, I'm going to show you how to self-host large language models like GPT using Docker and Proxmox with tools like Open Web UI. So, let's dive in.
Benefits of self-hosting and core requirements
So, why go through the effort to self-host a GPT model? Isn't it just easier to consume these in the cloud? Well, like anything that you can self-host, you gain full control over your data when you self-host those resources. Your prompts and interactions stay local. You don't need an API key, and most importantly, there are no monthly subscription fees. It's also perfect for offline use and experimentation, which we all know and love about home labs. And what's even better, what you need to host these tools, you probably already have in your home lab anyway. What are those requirements and the infrastructure that you'll want to have? You'll probably want to have a Proxmox node or a Linux box running something like Docker or virtual machines. At the very least, you'll want to have 16 gigs of RAM. 32 gigs is even better for bigger models. A semi-modern Nvidia GPU with CUDA support. This is not absolutely required, but it's highly recommended. And I will be honest, you can self-host LLMs and run them on something like an Nvidia RTX 1060. That's right, something that old. Now, would something newer be better? Absolutely. But I just want to get the notion out of the way that you need the most cutting edge video card, the latest and greatest to be happy with performance. Also, you want to have Docker installed either natively or using something like Docker Desktop on Windows and around 10 to 20 gigs of disc space depending on the models you're testing. And the more models that you want to house locally, obviously you're going to need more disc space.
There are two main tools that make this possible. Those are Ollama and Open Web UI. First off, Ollama is the engine that actually downloads and runs the open-source LLMs locally. So, think of models like Gemini, Llama 3, Code Llama, Mistral, and Phi3. It also exposes an API that Open Web UI is able to connect to. And that's where you get the chat GPT style interface that runs on top of Ollama. And I'm being completely honest here, there are actually some things that I like even better about the web UI of Open Web UI than I do ChatGPT. It gives you a full browser-based UI to chat, manage models, switch users, and organize sessions. So, now let's talk about the setup. Using Docker makes this incredibly easy. You can actually house both as well as Open Web UI in a Docker container. So, you install something like Docker on Windows even. I recommend using Docker Desktop and WSL2 backend. And I've used that lately with quite a bit of my testing. On Linux, just follow the Docker docs to get that installed. You'll probably want to do this. It's something I like to do. I like to create a folder for all of my projects for Docker Compose and other Docker projects and put all of the relevant files in that directory. And that just helps keep things organized and project focused, project specific. Then you can either use docker run commands for each container or even better create a docker compose yaml file to bring up both solutions as a docker stack.
Docker run vs Docker Compose: planning your deployment
Now I'm going to show you how to do both of those methods either with the docker command line or running docker compose with this docker compose yaml file. Let's now take a look at spinning up those docker containers. Okay, so I'm just going to walk you guys through how to spin this up inside of Docker Desktop. If you are running a Windows system, Docker Desktop is free to download, free to use, just a great tool in general if you don't have access to a full-blown Linux system or a home server as of yet that you've spun up in your environment. So, I've got Docker Desktop open. I've just simply opened the terminal from the application window itself. And I'm just going to paste in the command that I want to walk you through what these various parameters are going to do for us. To spin up Ollama, I'm just entering the docker run command.
Running Ollama container: flags and ports
We are specifying -d for headless mode or daemon mode. We're naming it. We're setting it to restart always if it stops. Then we're also passing this port. And this port is important. This 11434 is the special API port that we're going to point Open Web UI to so that it can interact with Ollama, run models, download models, all of that interaction that's going to take place with our GPT chats. Now also an important parameter is the --gpus all in Docker Desktop or Docker command line for that matter. The --gpus all command tells our system to use any of the Nvidia GPUs that we have installed and available in our system so that we get that added performance benefit with our GPU performance. Also, we've got the -v for volume mount and then also we've got ollama/ollama for the container image itself that we're going to use to spin up this container. So, I'm just simply going to hit enter. And as we can see, we've spun up a container. And you probably saw it spin up as well in the background.
Running Open Web UI container and linking to Ollama
And now let's spin up the Open Web UI container from the docker command line. So I'm just going to paste this command in. I'm going to add the main tag here. So we've got docker run -d. The port that we are exposing externally is port 3000. And that forwards internally to 8080. Again, we're using this --gpus all for the performance benefit. And then also we're passing in the Ollama base URL. And if you notice, one of the interesting things that we are doing since we're housing both docker containers on the same host, my Windows host that I'm running Docker Desktop on, we're just simply using this special URL. And notice it's HTTP, not HTTPS. And we're passing along host.docker.internal. And that's that specialized FQDN that we can use in our Docker environment. This should look familiar to us as it is the port that we've exposed for Ollama 11434. Of course, our volume data, we're naming it, setting our restart policy, and then also our container image. So, I'm just going to hit enter. It's going to go out and download the image for this container. So, once this image downloads and we spin up that container, we will have two containers running on this Docker Desktop system. And they're both going to work in tandem. Of course, Ollama provides the API as well as the ability to interact with the models and then Open Web UI speaks to Ollama through that special API port 11434. With the containers running, your local LLM environment is now active and accessible.
Accessing Open Web UI and downloading models
Once your containers are up, you can navigate or browse out to in a web browser HTTP your server IP or FQDN port 3000 and that will allow you to access the Open Web UI. The first steps with this are fairly straightforward. You're going to create an admin user, go to admin settings, and then head over to the models section and configuration. And this is super intuitive, but you just simply enter the model name that you want to download and hit the download arrow that you see next to where you fill in the model you want to download. Once you hit download, you will see the progress begin in the download dialogue box and this will complete and add that model to your model selection. If you want better performance, you want to enable GPU acceleration. If you're running on Windows with Docker Desktop, it uses WSL2 and GPU support is actually built in. So you don't have to load the container package for Nvidia GPU support. You can actually just enable GPU access under Docker settings WSL integration. If you're on Linux, you'll need to install the Nvidia container toolkit if you're running this in something like a Proxmox LXC container. If you are using a virtual machine, you will need to install the Nvidia GPU driver just so that native virtual machine sees the Nvidia container that it's available as a resource.
GPU passthrough on Proxmox: overview and options
Now with Proxmox, we need to do the GPU passthrough method that allows us to take that physical card that's installed in the Proxmox host, pass that through to the virtual machine or the LXC container so that that GPU is natively seen by that virtual machine or that LXC container as a literal physical device that's installed in that environment. So, when you pass that through, you're able to then load the drivers for the Nvidia GPUs or the Nvidia container runtime. You're then able to take advantage of those physical resources and GPU resources that are available on that card for running your AI and LLM models. If you're running this from the Docker command line or Docker Compose, you'll want to make sure that the --gpus all flag is run or in your compose commands. And that basically is telling Docker that we're passing through the GPUs that are available for that container. Now, back to Proxmox. If you're using Proxmox, you've got two options. Option one, you can run an LXC container with nesting enabled. Uh, and this is super lightweight, super fast to provision, and it also supports GPU passthrough as a resource. Now, option two is the more traditional approach of full virtual machine. I in some ways think this is a more robust solution with arguably more resource requirements but much better compatibility perhaps with GPU passthrough. You can also use snapshots as you work your way through your configuration in Proxmox which I always say using a virtualized environment is great for learning due to those snapshots and those features it provides.
Enabling IOMMU and claiming the GPU in Proxmox
Let's take a look at my Proxmox configuration and I'll show you guys how I have the GPU passthrough to my Ollama virtual machine and a few of the configuration items that you'll want to check and make sure of there. So, I want to just in a high level show you guys a few of the steps that are required for GPU passthrough in Proxmox VE server. One of the first things that you need to do is to enable IOMMU. This is a technology that allows Proxmox to pass that GPU, the physical GPU into the virtual machine or the LXC container. So, I'm going to paste in a command that you can check and I'm running an AMD CPU. So, I'm going to uh check IOMMU from the AMD perspective. You can see here that we've got some output for IOMMU. And basically, it is enabled. It's showing that it's enabled. Got interrupt remapping, virtual APIC enabled, extended features. So, we've got a lot of indications that IOMMU is enabled. And this is something in most motherboard BIOSes that you're going to be able to go in on the CPU advanced features and you're going to be able to enable IOMMU. Now, the next step after you have IOMMU enabled is to identify your card, your physical Nvidia RTX card inside of your Proxmox host. I'm going to paste another command. It's lspci -nn. And we're going to grep for Nvidia. So, I'm just going to press enter. And as you can see, we have found the VGA compatible controller, Nvidia Corporation. So, we know it's our RTX card and it's a RTX 4060. And then you'll also see the audio device that also goes with your GPU. So this tells us that we can see this Nvidia RTX 4060 in our Proxmox host. And those entries help us to identify that that is the case. Now one other thing we need to do is claim the ids that we see here in another file. And this is vfio.conf. And as you can see I have done that. I've claimed the IDs and basically what this does is it claims these for use for passthrough and there are a few other steps such as blacklisting the drivers for our Nvidia GPU so that our Proxmox host cannot use those. And if you want the detailed step-by-step, I have a blog post that walks through all of the steps required for GPU passthrough and I'm going to have that in the description for the video. At the end of the process, what you're going to be able to see is in your virtual machine, you're going to be able to see that Nvidia RTX card as recognized from your virtual machine. So, you can see here I'm now at the terminal prompt for my virtual machine. Now, how did I get the connection made for the physical card on the Proxmox host over to my virtual machine? Well, under your hardware, you can see that I already have this entry PCI device. How did I get this? Well, if you go up to the add menu, go down to PCI device, and I'm going to click the raw device radio button, device dropdown, and as you can see here, we've got two Nvidia Corporation entries. Now, that equates to the video component and then the audio component of the GPU. Now all I had to do was select the first one and then if you check the box all functions it will then include the other devices that are part of that same IOMMU group which will include the audio device. So you'll get an error message if you check this box and then go through and add another device and check this box because it's included by default with this. I guess you could call it a root device for this IOMMU group. So I'm clicking all functions. Click add. I'm not going to click add because I've already done that. That's why we see PCI device. And then that is how we can see this inside of the virtual machine when we run the nvidia-smi command. Now I'm going to make no bones about this. This is a very convoluted process to pass through a GPU in Proxmox. I think this is definitely an area for improvement for the developers of Proxmox. I think this could be a lot easier, a lot better process and I think as they get more serious about targeting the enterprise as ones are certainly looking at AI this would be an area of focus making this process much, much easier for a virtual machine. In VMware this is a couple of checkboxes literally. I think Proxmox should follow suit. I think this would be a great area of improvement; however, it's not impossible to do. I'm just calling that out as it is very convoluted. So, I'm going to show you quickly what the performance looks like with this GPU passthrough. If I just click one of the default ones here that it's showing us, you're going to see how quick the inference is and then also how quickly we're getting responses. Response tokens 81.57, prompt tokens 153.97. So, the experience is really good. So, as you can see, it's just lightning quick. And this is with GPU passthrough obviously in our Proxmox virtual machine. So this is really nice. So, you can see this is the way that you definitely want to set up your environment for local AI processing. Now I have a few quick tips and best practices that I've learned from experience working with this in my home lab so that you have the best experience. Number one, you want to use models that will fit inside your GPU's VRAM. So if you only have a specific amount of VRAM, you don't want a model that will overrun the VRAM that you have available. Run everything off of NVMe for faster model loading. Snapshot your Proxmox VM or container often during the configuration so you have a place to roll back to. And if you want SSL certificates, you can spin up something easy like Nginx Proxy Manager. And that makes terminating SSL with something like Let's Encrypt certificates extremely easy. So, what can you do with a self-hosted GPT other than say you have one and just appear cool to your peers? Well, you can certainly use it as a private coding assistant. This is becoming more and more popular. You can use it as an offline chatbot for yourself or a website. You can connect it to apps like Obsidian or Home Assistant. And for me lately, one of the really cool projects I've got going on in the home lab is the recently released kubectl-ai tool that Google recently released for Kubernetes clusters. And that tool, which I hope to do a full video on in the very near future, allows you to point not only to cloud environments like ChatGPT, Gemini, you can also point it to self-hosted LLMs. And that really made things interesting for me. I can self-contain all of that AI processing and perhaps some sensitive information that I wouldn't necessarily want a cloud AI solution to have access to my Kubernetes cluster. But with the kubectl-ai command, you can point that to your Ollama instance and then any of your prompts can be processed locally in your own self-hosted AI environment, which I think is just really, really cool. Well, thanks to Ollama and Open Web UI, self-hosting a GPT-style AI model is easier than ever, and you don't need massive hardware to get started, as I've shown in this video. Whether you're experimenting or building a real AI assistant, this is one of the best HomeLab projects that you can try right now. Super hot technology, very interesting how quickly and rapidly things are developing. And if you're already self-hosting LLMs, drop a comment and let me know what tools or models you're using or what use cases that you have found very useful for self-hosting your own AI solution. I'd love to hear what you all are building. Be sure to like and subscribe for more home lab content, more AI content, Kubernetes content, and much, much more. Stay safe out there. Do keep on home labbing, and I will see you in the next video.
