Run Llama 3.1 Locally and Privately
I'm going to show you how to run the latest Llama model, Llama 3.1, the 8B model and the 70B model, locally on your computer. Technically, you could even use the best version of this model, which is called 405B, which is a massive, massive model, locally on your computer if you have the hardware to do it. I'm going to explain all that in this video and what kind of hardware you need to even run these models and the bigger one too. I couldn't run the bigger one on this computer.
And we're going to get this UI. This user interface is going to be completely locally running on our computer using the best model from Llama. And again, this is going to require no Wi-Fi, completely private. And I'll show you in five easy steps how to do this in about 10 minutes.
And if you want to dive deeper on our platform, Skill Leap, which is a library of courses, I have a dedicated course. It's about 60 to 70 minutes that shows you everything you need to know about running large language models, these open-source large language models like Llama, locally on your computer. The full process with some user guide, you could ask me any questions if you get stuck. And then I dive into the UI and all the different things you could do here, including interacting with your own document, creating your own knowledge bases, even creating images and connecting OpenAI's ChatGPT to it as well. I'll link that below, but right now, let's dive in and install the latest version of Llama 3 right on our computer.
Step 1: Download and Install Ollama
Okay, step one in the process is we need to download something to our computer. It's called Ollama. You could find it at ollama.com. This is just a free software, and if you press download, it's going to ask you what system you're on. So I'm on a Mac laptop over here, so I'm going to choose this option, but obviously Windows and Linux are available. And download it from here and just go ahead and install it like any other app. Move it to the application folder.
Okay, then I get this popup right here. Let me press next, and it says install a command line. Now this is going to give me the code to install Llama 3, and then I could install the later version of Llama 2 and any other large language model that's open source like Llama and available for free. So I could go ahead and get started with this. I'm just going to press finish right here. Okay, that's the very first step. You need to have Ollama and you need to copy that line of code, which is just three words here.
Step 2: Run Your First Model in Terminal
Now for the second step, we need to go to an app that a lot of people have never used before. It's called Terminal. And if you're on Windows, it's also called Terminal. If you have Windows 10 or 11, search for Terminal. This is what Terminal looks like. We actually don't have to know anything about using Terminal. All you have to do is paste that code, which is just three words: ollama run llama3. And typically this third word is the name of the large language model, so we'll install some other ones here. And just press enter and it's going to go ahead and install it for you. And this may take a few seconds here, depends on the speed of your computer how fast this is going to install. But I do at this point need to be connected to Wi-Fi because this does need to pull it from the web. And it's been installed. I accidentally pressed 'y' here, but now I could ask a question: what is your name? Okay, now I'm having a conversation with Llama 3. This is actually Llama 3 responding to me. But we want the later models. We want some other models here, and we want a better user experience. We don't want to really use this inside of Terminal. But I'll go ahead and close Terminal for a second, terminate.
Step 3: Choose Your Llama 3.1 Model Size
That takes us to the next step, which if we go to the Models tab right over here, this is where we could get access to other models. So by default right now, it just installed Llama 3. They may change that to the default model being Llama 3.1. But if you go to the Models tab, regardless of when you're watching this, the latest model always shows up on top of this models page, this library of models. And Llama 3.1 is what we want in this case. We just installed Llama 3. Now there are other ones that are really good too. So this Gemma 2, this recently came out, this is from Google. It's pretty good and it comes in two different sizes. Llama 3 comes in three different sizes. I'll explain sizes to you and what kind of computing power you need to use different sizes. Typically, the smaller the size, the easier it is to run on just about any type of computer could handle some of these smaller sizes. And some of these bigger sizes is pretty much impossible to run it on a home computer, so I'll mention that in a second. And all these other models are available. Mistral and Mixtral are from the same company. They've been a great model for a long time. So you could install as many as you want, but they do take hard drive space. Each one takes hard drive space. So let's go to Llama 3.1 and see what kind of space it's going to take on our computer.
Okay, so on this page right over here, we're going to be able to choose different models if that large language model has different sizes. Basically, this one does have 8B, that's 8 billion parameters, 70 billion parameters, 405 billion parameters. Those are the different sizes. And I could choose the different sizes over here. And as I mentioned, they do take up hard drive space on your computer. I'm just on a laptop here, so this is 4.7 GB of space. This one is 40 GB of space, the 70B model. And this 405B model, the best model that's available right now from any platform, 231 gigabytes of space. So that's huge, right? I mean, I have a, I think a one or two terabyte hard drive, but that's just a massive file for you to get this model and you're going to have computing power in order to run it. I'm going to try, but it's pretty much impossible to run this model on any kind of laptop. I'll mention how you could check that in a second.
Understanding Hardware Requirements
Right now, let's choose the 8B model. And then look over here, this right now shows you again, the same three words: ollama run the name of the model here. And the 8B model is what we want. So I'm going to copy this. I guess I closed Terminal. Let me open Terminal one more time and let's go ahead and paste this one, so we could have the new Llama 3. And it's going to go ahead and install it. And us usually it goes through this process. I think I had Llama 3 installed already, that's why it didn't do this download process. So when I was mentioning sometimes this takes a few minutes, I'm sure if you do the other one it takes hours. The 405B is a massive file, but it's going to go through the download process right here. So I'll let this finish up.
And while that's going, if you want to know what kind of computer you need to run these models, every model is going to be different. Right here under the model tab, you could click right here and this is going to give you a ton of technical stuff. But all I'm going to do is I'm going to copy everything. I'll just open up ChatGPT to ask it. So I'm just going to ask, what kind of hardware do I need to run this large language model? And I'm just going to paste pretty much everything that was on that page. I'll just ask ChatGPT exactly what's going on over here. And it says right here: CPU, at least a modern multicore processor. Pretty much most of us have this one. RAM, 32 gigs minimum. I have 64 on this laptop. GPU, 24 gigabytes of VRAM. Okay, so this is going to give you very specific things. Storage, your hard drive, it says recommended to have an SSD hard drive, 5 gigs available. Then the 70B model, it says 128 GB of regular RAM. I have only 64, but I'm on a Mac, I'm on a M3 chip Mac. It's about a $5,000 laptop, but that's still not enough over here based on what this is telling me. But I'm going to try to run this. Then it says VRAM, 80 gigs. I don't have that either. And SSD, 1 terabyte. So I have that one. But it's really based on GPU that's really what's going to cause speed issues and if your computer could handle it. And then you have your 400B model. This one, 500 gigabytes of RAM minimum. Well, I definitely don't have anywhere near that. And GPU, 400 gigabytes of VRAM. Nowhere near that, and all kinds of other additional things. So if you're more technical or if you want to learn which models can be run locally on your laptop or your desktop, it's really useful to just take that model's page, put all that stat into another AI like ChatGPT and see what it tells you.
Installing the 70B Llama 3.1 Model
Okay, so the download over here is done. So I'm going to say, hello, just to test it out. What model are you? Okay. It doesn't know what model it is.
Let's go back and let's use the 70B model and let's download this one too. So this is going to have a different line of code here. Go back to Terminal. I usually just start a fresh terminal and just paste it, go ahead and send this out and let this one install. Now this one, if you remember, if you click this, it's going to take 40 GB of hard drive space. Okay.
Step 4: Install Docker for Local Hosting
Step number four is going to be installing Docker. You could get it at docker.com. Again, we don't need to worry at all about how Docker works, but we do need this app and I'll show you a couple of steps that you need to add to it. But what you get with Docker is the ability to run this locally on your computer. So the previous step, make sure you have as many models as you want to install and then you could actually access it in the final step, which is step five. So I'm going to go ahead and install Docker. Again, you could get it for Mac, Windows, and Linux. Okay, the download is done. So let me go ahead and install that to my applications folder. Okay, this is what the Docker app looks like, but right now we don't have to do anything, but this has to be installed in the fourth step. So that's why we installed it here.
Step 5: Set Up Open WebUI with a Single Command
Now let's go to step number five. Now there's another website called Open WebUI. This is going to give you that ChatGPT-style user interface, but we're going to use the latest version of Llama and other open-source models inside of it. The user experience is going to be very similar to ChatGPT and other AI tools that you probably used before. So let's go ahead and open this up right here. Now, this takes you to GitHub, which is a website for developers. And this walks you through the step-by-step guide. Again, I don't really make videos dedicated for developers, more for entrepreneurs and creators. So let's go down a little bit here. This kind of walks you through step-by-step on how to do things, but we've already done a few of those steps. So starting Docker, we've done that. And we're going to come all the way down here. If Ollama is running on your computer, use this command. That is running on our computer. That was our very first step. So I'm going to copy this line of code. Now I'm going to open up Terminal again, again, a new copy of terminal, copy that line of code, enter. And this is going to do things. At this point, Docker does have to be installed on our computer, or Ollama has to be installed on our computer. That's why I went through this in those five steps. But we're now on step number five. Okay, now that is complete.
Using the 8B Model in Open WebUI Offline
So I did need to do that one step. And now if you go back to Docker, something suddenly appears over here, and it says running. And it's going to give us a link over here. So if I click this link, this port link, this should open up a brand new page on our computer, Local Host 3000. We're actually just locally running this on the browser, but we're not on any website. And I do need to sign up to Open WebUI. We do need to create ourselves an account. So go ahead and sign up. I already have one, so I'm going to log in.
Okay, now that is running. If I go back to the Docker app, by the way, anytime you want to stop it, this is typically running all the time on my computer, but you could always press the stop sign. And if you do that, it's going to stop that other page, which is running on our browser right now. But without Wi-Fi, it's running locally.
So let's go back here. This is Web UI. All the steps are complete. There's nothing else you need to do unless you want to go back to step three and install different models to your computer, which you would do in Terminal, and they will appear over here. But the very first thing you want to do is come up here inside of your Local Host 3000, this is the Web UI, and you want to choose a model. So as you could see, I have a bunch of them. It looks like from previous installs already on my computer. so phi3 was a good one. Now, Llama 3.1 is what we want. Or let's try this one, then we'll test out this other one just to kind of check the speed here. And I'm going to turn off my Wi-Fi just to make sure I'm running everything privately. And you could set a model to be your default model. So every time you don't have to pick a model if you set one as a default.
Let me send a prompt here. Tell me more about Llama 3. I'm using 8B. And okay, so I didn't cut anything just so you see the speed of how fast it's working. "I'm afraid I don't have information on Llama 3." Okay, so this is based on its training data because sometimes, pretty much every large language model has a delay. So they usually don't have training data that's so up to date. In this case, it would have no idea what Llama 3 was because that came out in 2024. Most likely this was trained with data from 2023 and before that.
Testing the 70B Model with Document Uploads
Okay, let me try this one: create a marketing plan for my e-learning platform. Wow, that was really fast. Here's a comprehensive marketing plan, marketing objective, target audience.
Now, I've done a complete test on Llama 3.1 in the previous video, so I'm not going to do a test here. I just want to kind of show you how to run this locally. So watch that video if you want to see Llama 3 in action with 10 different categories of prompts I give it, including some coding. But now look at the power you have over here that you wouldn't otherwise have. You actually have this option, this plus sign, where you could upload files and then use those files with Llama 3.1 locally on your computer.
So for that test, let's click over here and let's select the 70B model and let's see how that works out. I'll start a new chat, 70B. I'm going to give it a document that's code, and I'm going to say, "explain this to me." It took 60 seconds. I just took out a 60-second chunk over there where I was waiting for a response on my $5,000 M3 Mac laptop. 70B is completely struggling. Look at the speed, by the way, compared to the other smaller model, very slow, but it did do exactly what I want. This was a Python file for a game of checkers that I used in my testing in the previous video. He knows exactly what's in the code. Something like this is not possible right now with where you would normally use Llama 3.1, this model of it and the bigger model of it inside of meta.ai. And Meta AI doesn't have a way you would upload files. So this takes it to a whole new level. Now you could upload files. Let me make sure the 70B model could do the same thing. I'll give it some text file. All right, I'll try this document. Summarize this for me. All right, it took two seconds. A conversation revolving around a project called Claude. Again, this wouldn't know anything about what Claude is, so it has to read into this document because Claude probably is not based on its training data. It's newer than when its training data was. It starts with explaining how to use custom instructions without cloud projects. This is very new, so it's definitely using this to give me the summary. Very good.
Advanced Features and Final Recommendations
Now, if you go to the workspace tab here, they do have all the models that you have available. So if you want to modify or remove any of them, you could go ahead and do that from here. And they have an entire library of documents, your own knowledge base that you could go ahead and upload from here. And then if you press the plus sign and dump documents into it, you could then use the hashtag sign and bring those into a conversation and interact with your own knowledge base. This is all private, which is huge. And you have other more advanced things.
And in my complete course on using these private chatbots on your local computer, I go really deep in using this UI and all the other functions that it has. So I go through the installation process there too, which I covered in this video, but I really show you all the use cases, including knowledge base and some other things. You could even bring in OpenAI GPTs into this too, if you want and just pay for usage, but that's not going to be completely local. You can't do that without Wi-Fi because that needs to send information to OpenAI.
Now, since the 70B model was extremely slow, I'm assuming the 405B model, the biggest model Llama 3.1 has, is going to be pretty much unusable on my computer, on my laptop. So I'm most likely only going to use the 8B model because it was fast and so far it was pretty good. But again, test it out for yourself. And if you haven't watched it already, I did make a video showing you Llama 3.1 in action using the best model available on meta.ai. I ran it across 10 different prompt categories to show you what it could do. So that will be linked over here as well. Thanks so much for watching this one, and I will see you next time.
