What is Ollama?
In this short video, I'll teach you everything you need to know to get up and running with Ollama, which is a fantastic free open-source tool that allows you to manage and run LLMs locally rather than having to pay for ChatGPT or use these hosted services online. You can actually run all of these models locally on your own computer, so you get privacy, security, and best of all, they are completely free.
So with that in mind, let me show you how to set this up, get it running, and I'll also explain to you how you can utilize this through code because Ollama provides an HTTP server, which means you can call your models from really any type of application.
So first things first, we do need to install Ollama. So to do that you can go to the website which is ollama.com, I will link it in the description, and you can simply press on download and then select your operating system. In my case I'm using Windows, but of course, you have the command for Linux and then the installation for Mac. Now once that's downloaded, simply double-click it and install it, and then I'll show you the next step.
Installation and Verification
So once you've installed Ollama, there's a few different ways to run it. First of all, you can just open the desktop application. So if you're on Windows, you can go here to the search bar and just search for Ollama. If you're on Mac, you can simply go to the Spotlight search, same for Linux, and just run the application. Now when you do that, nothing's going to appear on your screen, and the reason for that is this just starts a backend server that's running the Ollama service.
Now, the other way to do this is to open up a command prompt or a terminal. So you can see I'm in command prompt here on Windows. And then to simply just type ollama. If you do this, you should get some kind of output, and if you see that, it means that you've installed Ollama correctly. At this point, I'll assume that you have this installed correctly and that this command gave you some kind of output.
Choosing an LLM & Hardware Needs
And now what we can do is start running models. So the first thing to look at is the different models that we have access to with Ollama. Now truthfully, you have access to pretty much any open-source model you want, and you can even write some custom configurations to use your own models or things you pull in from something like Hugging Face. Now if you go to the Ollama GitHub repository, which I will link down below, you can see some of the common models that you may want to download. Now keep in mind that since you are running these locally, you will need to download the entire model and you'll need to have enough space on your computer. You can see some of these are 43 GB, for example, and enough RAM to run and load the model depending on how large it is.
So you see if we look through here, it defines the number of parameters for these models. If we look at something like Llama 3.1, we have 231 GB and 405 billion parameters. And if you go down here to this note, it specifies how many gigabytes of RAM you should have based on the different model parameters. So even on my computer, which has 64 GB of RAM, it would be difficult to load the new Llama 3.1 model with the 405 billion parameters. So keep that in mind when you are choosing the models that you want to use. For now, I'm just going to go with the standard Llama 2 model because this is older and it's not as large, and I know that I can run it and most of you should be able to run it as well. So I'm going to show you how we can pull that. But if you're looking for a list of all of the models you have available, you can go to the Ollama library. So if you go to ollama.com/library and you can scroll through here, and you'll see there are hundreds of different models. You can sort them, you can filter, and you can find even multimodal models, accept things like video, photos, voice, etc.
Running Your First Local Model
So once you've decided on a model that you'd like to run, it's very simple to do so. All you need to do is type ollama run and then the identifier of that model. Now, in my case, I just want to run Llama 2. I know this is an outdated model, I'm just doing it because it's smaller. So I can simply type ollama run llama2 and if this model is not already installed on my system, then it will download it and install it for me. If it is already installed, it's just going to bring up a prompt where it allows me to actually start typing to the model and messaging with it. So notice here that it's just loading and it kind of gives me these three arrows and I can just start typing something to the model and get some kind of response. And you can see it's pretty much instant because there's no latency, it's running on my own machine.
Now again, if this wasn't already installed, it would start pulling the model for you and then you would have to wait for it to finish, it would install, then you can run the model and you can start using it. Now after some experimentation, it's told me that you can type /bye to get out of this. So if I type /bye, you can see that it will enclose this window.
And then if we want, we can type ollama list and we can list the different models that we have available on our system. In this case, you can see I have Llama 2, which is the latest version. If I had any other models, they would show up here. So that's the basics on running models using Ollama, but there's a lot more to show you, so make sure you stick around.
After a quick word from our sponsor.
Today's video is sponsored by SEO Writing, a tool that's transforming content creation across different niches and industries. Their new Brand Voice feature lets you generate content that matches your unique style, whether you're writing tutorials, reviews, or even industry analysis. One click generates a complete blog post with AI-generated images and relevant videos embedded automatically, potentially saving you hours of manual work.
What sets SEO Writing apart is their deep web research with built-in citations. When you need accurate, up-to-date information, the platform pulls from reliable sources and adds citations automatically. Their humanized text feature helps your AI-generated content stand out while their external linking feature intelligently connects to relevant resources. And for all you WordPress users out there, there's a game-changing feature that lets you connect your site and autopost content directly. This feature allows for consistent scheduling while focusing on other projects.
Now, if you're ready to try it for yourself, then use my code TWW T25 for a 25% discount. Click the link in the description and see how SEO Writing can fit into your content strategy.
Juggling Multiple Models
All right, so we are continuing here and I want to show you what happens when you pull multiple models. So again, if we go back to the library, we can start looking through different models that we may want to utilize. Maybe I can even just go back here to the GitHub if I want to find them a little bit easier, and maybe I want to use the Mistral model as well. If that's the case, I can just copy this command or the name Mistral. I can go back here, I can simply run the command ollama run mistral. It will then pull that manifest for me, pull the model. Once that's finished, I'll be able to use this and I'll show you how.
So looks like this has been downloaded and now I can start using the model. If I want, I can exit out of this and if I want to switch between the two different models, again I just type ollama run and then I can specify the model that I want to use. So if I want to go back to Llama 2, I use Llama 2. If I want to go back to Mistral, I just type mistral, and now I can start using Mistral.
So you can have as many models as you want. And again, you can list them by typing ollama list. And if you want all of the commands you can use, simply type ollama and then it will show you which ones you have access to. There's a lot of them. For example, you can also remove a model if you want to do that, copy a model. There's also customizations you can make to them which I'll show you in just one second.
The Ollama HTTP API Server
All right, so all of that is great, but we probably want to know how to utilize these models from something like code, from our applications. Sure, they're great to use here in the terminal, but a lot of times you want to integrate them with some kind of software, especially if you're a programmer and you watch this channel. So the interesting thing about Ollama is that it actually exposes an HTTP API on Localhost. That means that anything we just did here with commands, we can actually trigger through the API. So we can send requests to this from something like curl, Postman, something like Python code, really any code at all that can send some type of HTTP request.
Now, by default, if you're running Ollama, you should be able to see this if you're on Windows in kind of like the, I don't know what you would call this, Services bar wherever it's showing the running applications. And you can see I have this little Ollama logo. Now when Ollama is running as the desktop application, by default that port is going to be open so you'll be able to access the HTTP API. But if for some reason this isn't running, so for example, if I quit this, what I can do to trigger that to run is I can simply type ollama serve in my terminal. If I do this, it's now going to start running the HTTP API in this terminal instance and now I'll have access to it. And here it will also show us what port it's running on, although it should be standard. And you can see if we look through here, it gives us the exact port. So it's on 11,434. So if you wanted to, you can copy that and save it for later so that we can use it in our code.
Manual API Calls with Python
Regardless, now that the Ollama serve or the Ollama HTTP API is running, we're able to call it. And again just to clarify, if you're running this as the desktop application, it will already be running in the background. But if for some reason you want to manually invoke this to run, then you can run the command ollama serve where it will give you all of this output and you'll be able to view all of the requests to the HTTP server.
So now that the server is running, we can use something like the following Python code here to send a request to it. Now this is done manually, very intentionally. I'm going to show you an easier way to do this in one second, but it's just to illustrate that you do have kind of complete control over this if you want. So you can see here in Python I'm using the requests and the JSON module. Now, just by the way, if you want this to work on your machine, you will need to install the requests module, so you can say pip install requests or pip3 install requests. And I'm going to leave this code in the description, uh, linked in a GitHub repo in case you want to check it out.
Now, what we do is we define our base URL. This is the URL of the server and then /api/chat. There's a lot of other endpoints that you can use here and you can even control deleting models, adding models, etc. But in this case, we just want to chat with one of our models. Then we can define a payload. This is the model that we want to chat with. So in this case I've gone with Mistral and then we can define different messages. Here's a standard message with the role of a user. Next, we can send a post request here using requests.post to our URL with our JSON payload, which is this right here and enable the streaming mode, which allows us to grab all of the responses as they are typed. This way we can grab them in real time and we can show the model actually typing the response, rather than waiting for the entire response to be generated and then viewing it.
Now here's a little bit of code just to handle that streaming data for us. So we're going through all of the lines that are returned from this response and then we are simply kind of printing them out. Okay, so I'm going to show you what happens when I run this. So we already have requests installed and if I go python sample-request.py, just wait one second here, it will stream in all of the data and then print it out. So you can see that it's kind of printing it out line by line for us here as it gets it, and there you go. Python is a high level language, blah, blah, blah, gives us the answer. If we go back to the API, we can see that the request was sent here. It took 4.1 seconds to process and it returned to us that data.
The Official Ollama Python Library
Sweet. So there you go, that is how you utilize the API manually, but a lot of you probably don't want to write all of this code. So instead we can use a very simple module from Python called, you guessed it, Ollama. So if you're using Python or JavaScript, there are packages that will do this for you. So you can simply pip install ollama or pip3 install ollama in your systems that you have this module. And now you have access to the Ollama module. You can simply create a client, you can define your model, you can define some kind of prompt, and then you can use client.generate, specify the model and the prompt, and then you can grab the response. Okay, so I'm going to quickly show this to you. I can run this code with python package.py and you will see here in just one second that we should be able to get the response. Okay, and there you go, we get the response and it gives us the answer. So that is how you use the HTTP API. Now I'm going to show you how you can do some customizations to the models in Ollama.
Customizing Models with a Modelfile
Moving on, I'll show you a quick customization that you can make to any of the models that you can pull with Ollama. So you can see on the right hand side of my screen that I've created something called a Modelfile. Now, I've just put this in a directory that's on my desktop. You need to put the file in a location that you know and that you're able to access from your terminal. And for the model file, I've used this very simple syntax that I just took directly from the Ollama website. All you do is you specify FROM and then you have some kind of base model. So in this case, we're using Llama 3.2, but you can use any model that you want that's available with Ollama. You can do something like set the temperature of the model. You don't need to do this, but there's some other parameters you can set as well. And then you're able to pass something like a system message, which is essentially kind of instructing the model what it's supposed to be doing and how it should handle the upcoming messages. So in this case, they've just written, "You are Mario from Super Mario Bros. Answer as Mario the assistant only."
Okay, so we have this model file written. Notice I don't have any extension, it's literally just called Modelfile, no .txt or anything. And what I've done is I've put my terminal in the same directory where this file exists. Now, what I'm able to do is create a new model based on this Modelfile in Ollama and have one that's set up as Mario. So to do that, I can type ollama create. I can give this a name, in this case, I'll call it Mario. You can call it anything that you want. And then I'm going to specify -f, which stands for file, and then the location of my model file. Now in this case, it's just simply ./Modelfile. Okay, so I'm going to go ahead and create this and you'll see that it says success. That's because I've already pulled model Llama 3.2.
So now if I want to utilize this customized model, what I can do is type ollama run and then the name of the model, which is Mario. And now if I say hello, you'll see that it says, "It's-a me, Mario," and it kind of, you know, simulates like how Mario would reply. So if you want to set up some custom models where they have some system prompts, they have some different parameters set up with them, you want to tweak them somehow, you can do that using these model files. Then you can simply create them in Ollama.
Now, let's say you're done with this one and you want to remove it, you can say rm or sorry, ollama rm and then what is it, the name of this one, Mario, and it will remove that. So now if we type ollama list, you no longer see it. And also it's worth noting that these models like Mario, you can utilize them from code. So in my Python code here, I can just specify Mario once that's created and then I can use that.
Outro
Anyways guys, that is it. That's all I wanted to show you. I hope you found this valuable. If you did, make sure to leave a like, subscribe to the channel, and I will see you in the next one.
