Introduction to Ollama and Local LLMs
Welcome back everyone, Dan Vega here. And today we are going to be talking about Ollama. Now what is Ollama? Why should I care about it? And I found a really cool project/tool that will help us with the developer experience of running Ollama on our local machine.
So what is Ollama? Ollama allows us to get up and running with large language models on our local machines. You can run models like the newly open-sourced Llama 3.1 from Meta, Mistral, and many, many more.
So why would you want to run a large language model on your local machine? That's probably the first question, right? Two things come to mind: cost and security. When it comes to cost, you have to pay to use these large language models like OpenAI's GPT-4o, right? Google Gemini, etc. These things incur costs, maybe not during prototyping or MVP or just development, but soon enough, there is a cost that you have to incur there and you got to put a credit card on file. Maybe you don't want to do that.
When it comes to security, maybe you're working in an enterprise, an organization where you've been tasked with building something that includes private documentation. Maybe I have some documentation that is not something that's out in the public, and I want to see how I can incorporate that into an application or into something that is using one of these large language models. So those are two really big reasons for me.
So what we're going to do is we're going to go through and talk about how to get started with Ollama, and then I'm going to show you this cool tool that really kind of helped me in the developer experience area of using Ollama on my local machine.
Installing Ollama
Okay, so the first thing you need to do is you got to head over to ollama.com, that's o l l a m a.com. When you get there, it says download. That's the first thing you're going to want to do is download Ollama. This will run on Mac OS, Linux, or Windows. I have already downloaded this for macOS. I simply ran through a couple steps, answered a couple things, and it was installed.
Choosing an LLM Model
Now what this is going to install on Mac OS for me at least is a CLI, so I have a command line interface to talk to Ollama. Now what we need is we need one of these models to go ahead and run. So if I go over to models, I can see all the different models that are available to me, and here is where you're going to pick a model based on what you're trying to do.
Maybe there is a specific model based on the type of application that you're going to build. Uh so there are things like Mistral, um Command R is a large language model optimized for conversational interaction and long context tasks. Um you think see things like Llama, Gemma, so a whole bunch of different models. Uh we're going to work with Llama 3.1. As I said, this was just released by Meta and they came out with an 8B, a 70B, and a 405B model.
And if you dig in here, there's some information about this. There's also a reference to the Meta AI Llama 3.1 launch blog post. There's a lot of really good information in there, and even another link to Mark Zuckerberg's kind of points and focus on open source, which I thought was a really interesting read, so go ahead and dig through there if you have some time. So now we're at Llama 3.1 and if you click on this drop down, you'll see the different models that are available.
So there is the 8B, and I want you to pay attention to this number because this is the size of the download. We are actually downloading a file onto our system, uh so that is how much that file is going to take up space. So the 8B is the smallest model, it's going to take 4.7 GB. The 70B is going to take 40 GB and the 405B is going to take 231 GB.
Now it's also important to remember that um if even if you have 231 GB of free space, which I don't, uh you will need some hefty processing power to go ahead and run that. Um you probably want—I don't even know what the numbers are to run that one, so I'm not even going to guess, but they're more than this laptop can handle. And if you had some GPUs that would probably really help that out as well. But for these smaller models, the nice thing is that I don't need a ton of resources. I have, I think, 16 GB of RAM. Uh, I'm not using a GPU, um so this is this is really nice.
So I am actually going to use the 8B model. So when you select that, you'll see it gives you this command over here, and this is what we want to go ahead and copy.
Using the Ollama Command-Line Interface (CLI)
So now we can go over to our command line and basically run that. So when I run that, I've already installed Llama 3.1 8B, so it's going to brought me into a prompt here to basically send a message to the LLM. Uh this is really cool. Now if you haven't downloaded it yet, it's going to take a couple of minutes to go ahead and download that, again, 4.7 gigabyte file.
So what I can do now is send a message. And the way that I test all large language models is with some very serious prompting in the form of dad jokes, because that's where we're going to go. So, tell me a dad joke about computers. Uh, why did the computer go to the doctor? Because it had a virus. I don't know if you noticed that, but it was really fast, too. And again, all this is happening on my local machine. If I wanted to, I can go ahead and turn the Wi-Fi off. I'm not communicating with the outside world. I have this large language model running on my local machine.
Let's try it again. Uh I'm going to say something like, 'Hello, my name is Dan.' 'Nice to meet you, Dan. Uh, I've got a bit of a byte of info on you.' Um, sorry I couldn't resist another dad joke. So one thing that that's telling me there is that it does keep context. It knows how to uh keep the conversational history. So now if I say, uh what is my name? 'Your name is Dan, we've already established that.' So this is good. When I'm working with an LLM, conversational history is important because I may want to ask something and then refine my uh prompt for getting uh some information back. So this is pretty cool.
Introduction to Open Web UI
But what we are doing here is we are working on a command line. And while this is good just to kind of test out a couple things, I think we can improve the developer experience a little bit. And something that we're all used to using, I don't know, maybe not all of us, but if you've kind of been in the world of AI, uh there is a product out there called ChatGPT. And ChatGPT is the product that is sitting on top of the model like GPT-4o. So we have a model here, Llama 3.1, we need something to sit on top of that to give us a little bit of a better developer experience, and that is where this project is going to come in play: Open Web UI.
I found this, and this is a UI that can be used. So, Open Web UI is extensible, feature-rich, and user-friendly self-hosted web UI designed to operate entirely offline. It supports various LLM runners, including Ollama and OpenAI compatible APIs. So it works with Ollama, that is really cool. Now we'll get into it, I'll show you the UI, but if you've ever used anything like uh ChatGPT, this will feel very, very familiar. So if you have Docker Desktop installed, this is going to be very easy. So if Ollama is on your computer, it is, we just installed it, and we have Docker desktop, I can go ahead and copy this command.
What I'm going to do now is get out of here, and I'm going to run that Docker command and it's going to go through and do a couple things for me. And uh we should be up and running.
Exploring the Open Web UI
So now if we open Docker Desktop, we can see that we have this Open Web UI and we see that it's running. Here are the ports. So what I'm going to do uh is go to localhost:3000 and we can see that we are unauthorized. So we need to go ahead and authorize, and there we go. So now it drops us into this nice little user interface that again looks very familiar to us.
Now we don't have to go from the command line, and not that there's anything wrong with the the command line, especially for just simple text, but what if I want to look at some code examples? I'd like to see that code formatted nicely, uh be able to copy it. It's a little bit easier in a user interface like this.
Now with Ollama, we can run multiple models at the same time. We can download 3.1, we can download three, we can use something like Mistral. Um so if we have all these different models, this interface will support it because you can come up here and select a model. So I only have one model, which is Llama 3.1. I also want to go ahead and set this as a default. If I don't do this, every time I hit new chat, I'd have to pick a model and I keep forgetting to do that. So I want to go ahead and set this as a default. So let's look at an example of um how we can go ahead and use this.
So again, I can come in and say, uh tell me a dad joke about dogs. And we'll go ahead and get our answer. 'Why did the dog go to the vet? Because it was feeling a little ruff!' Sorry, I know, it's a positively terrible pun. I love that. I love the little emoji that we got back too. So, um there's a simple one.
Generating Code with Open Web UI
I'm going to start a new chat and now I want to do something with some code. So I am a Java developer, I'm a Spring developer. Um I want to say, um let's just say, 'What are the best ways to iterate over a list in Java?' Right? So that's one example. And we can see again, this is pretty fast, right? We we got it back. There's a way we can use the for-each loop, a traditional for loop, an iterator, we can use the Java 8 Streams API, uh we can use a lambda expression with the for-each. So some pretty good examples there. And again, this is running on my own local machine, which is uh pretty cool to see.
So I want to take a look at one more example. Um if I were to come in here and say, 'Give me a code example of how to use the Rest Client in Spring Boot 3.2.' So if it does this, uh it is not going to have the correct information to be able to talk to me about the Rest Client in Spring Boot 3.2 because at the time of this training, this wasn't out yet.
Enhancing Prompts with Custom Documents (RAG)
So what we can do is provide our own documents. So what I'm going to do is click on this 'more' button, I'm going to upload a file, and I am going to use the Spring Boot reference document PDF. Uh this is uh pretty up to date. It has information on the REST client. So we can go ahead and uh add that in. And now we have these additional docs. Again, if this these were docs private to your own company, you can use those here because we're not sending these out on the internet to one of these public uh cloud uh llms that are sitting out there.
So I'm going to take that uh same question, let's do that again. And now with Spring Boot reference, uh it should be able to give us a little bit better of an answer. So here is the REST client. This is the REST client I was looking for, and now we have an example of that. So this is really cool. This is a way that we can run uh Ollama. We got Ollama installed, we were able to use a model like Llama 3.1. We could go ahead and use the CLI to ask it questions, but being able to install Open Web UI gives us this nice interface. And we can do some things from here, especially me, I want to do like a bunch of code examples in here. Um so I get this nice interface. And I showed you, I told you we can go in and copy code in here, which is really nice. But for me, the the ability to go ahead and then add my own documents, uh is really what kind of brings this home for me.
Conclusion
So uh, that was my introduction to Ollama, using some of the models and this really great project, which is called Open Web UI. Hey friends, I hope you learned something new today. If you did, do me a big favor: leave me a thumbs up, subscribe to the channel, and as always, happy coding.
