Why run a local model — privacy, cost, offline use
That just took, from start to finish, 1 minute or so to get a local model running on my computer where I can now just have a chat with it like I would with ChatGPT, and it's 100% local.
Every single message stays entirely on my computer. It's full privacy.
Every time you type something into AI online, you're paying for it, either through your data, your money, or both.
But what if you could run an AI assistant that knew everything in your notes, could answer questions about your own work, and that entire conversation never left your computer?
This is what a local AI model gives you. It's your own private AI. It runs on your machine, it costs nothing, and no one else will ever see what you type. Let's get that set up.
Hi, my name is Callum, also known as Water Lootz, and welcome to today's video on running a local model on your computer using Ollama.
What this guide covers and why Ollama
I'm using Ollama for today's video, but the principles I'm talking about apply to all different kinds of local model frameworks.
I've just personally found Ollama to be the easiest to get set up and then connect to other tools like AI agents.
In today's video, I'll go through the three reasons why running a local model is worth it, how to install Ollama and get your first model running, and how to create a customized variant of that model, and why agentic AI tools like Hermes need something like that. They need that customization.
Anytime I talk about connecting to a local model, or using a local model, or running your own LLM, this is what I'm talking about.
If you find this video helpful, please like, hype, and subscribe as I really appreciate your support a lot. If you're looking for more ways to support me, please consider joining my YouTube or Patreon membership where I give more tips, insights, and kits on the world of AI and knowledge. Now, let's get your own private AI running.
Before we get started on actually running anything, installing something on your computer, I want to talk about the three reasons on why it's worth running your own local model so that you understand the rest of the video better.
The first one is privacy. When you type into a cloud model on a website or through an API, that conversation goes to a server, someone else's server. With a local model, nothing leaves your machine, ever.
That means you can feed it sensitive notes, client work, personal writing, whatever you want, and that conversation, all of that information and data, will only ever stay on your computer. It never leaves it.
So, when you're running a local model, you can be confident that no one else is looking at what you're putting into that chat.
The second is cost. Cloud AI runs on a subscription model or a per use billing per API call. A local model downloads once and runs forever.
The only cost is the electricity to run it, but compared to cloud services, that cost is negligible.
Third is that it works offline. Once the model is downloaded on your machine, on your computer, you can use it forever from anywhere even if you don't have internet access.
And I know all of that can feel a little abstract if you're not sure how you can actually begin using this local model. So, my personal favorite method, a great practical example, is through Obsidian or note-taking.
Install Ollama and check it's running
If you use a note-taking app like Obsidian as your personal knowledge management system, your second brain, you can connect a local model directly to your vault, ask questions across thousands of your own notes, build a personal wiki, summarize your research, and that entire conversation related to your notes always stays private and on your computer.
It's a way to use your own knowledge with AI and maintain privacy.
A great example of extending that beyond your own notes is through something called the LLM wiki, where you can use an AI agent to summarize information and build out your own personal Wikipedia of information that you're interested in. So, if you want to learn more about the LLM wiki and how you can use it to reduce information overload and connect models to Obsidian, I recommend checking out my videos on that.
So, that's the context on why you should care about running your own local model, but now let's actually build one. Let's get it set up.
The easiest in my experience has been Ollama. I've been enjoying that a lot. And there's two different ways you can do it. You can go to your terminal, you can copy this curl command and just put that in, and it will download and install it for you. Or, you can click download, and then you can download it for your local device. I'm using a Mac, so I'm going to install it like that.
Once you've downloaded Ollama, you can check to make sure that it has installed by typing in Ollama version. And we can see that it's currently not running, but it has version is 30.1. And that's it. We have the platform that's going to be able to run our local model for us. The next step that we need to do is download a model.
Choosing and pulling a model (Gemma variants)
So, if we go to Ollama again, we see models. There are a ton of models that you can go through. Many of them are good for different use cases, but a lot of them significantly depends on what type of hardware do you have? What is your computer running?
So, for most people with 8 to 16 GB of RAM, you're probably going to want to run something like Gemma 4. This is Google's latest open weight model. And if we scroll down a little bit, we can see how it's able to be run. You can use it in Cloud Code, CodeX, OpenClaw, Hermes, CodeX, OpenCode, whatever you want. I've personally been using it for Hermes a lot lately, so I'm going to talk about that a little bit more later.
But, what I really wanted to show you is if we scroll down to models here, we see all of the different variations of the models that we can run. We can also see, importantly, how big is the model, how much space is it going to take up on our hard drive, and what is the context window, how much short-term working memory does it have?
So, what's cool about Gemma 4 is that not only do we have the most powerful version, for example, 31 billion parameters, which would take up 20 gigs, and I would struggle to run it potentially on my 32 GB RAM computer, but we have something called the E4B and the E2B models. So, E stands for effective parameter.
So, basically what's happening with the effective parameters is that it uses something called per-layer embeddings, which lets a smaller number of parameters, less power, less thinking, that operates like it's a bigger model.
So, in real-world behavior, the E4B operates similar to the 12 billion parameter model. And how that applies to practical terms is something like an E2B, you probably want to run on a low-end laptop with only maybe 4 GB of RAM, or on a phone. This is good for running on a mobile device.
The E4B can handle something with 8 GB of RAM, which should be almost everyone's laptop. And then if you're getting into the 31, that's where you need a much bigger RAM.
So, I'm able to run the 12 billion parameter on my 32 GB RAM, but I figured since most people are going to be using the E4B, why don't we install that one today? So, how do we install it?
Well, that's where we get into the commands in Ollama. So, what we can do is we can copy what this one's called, the E4B, and we can go over to our terminal again and type in Ollama pull and then Gemma E4B. We click that, click enter, and it's pulling in the manifest, writing it, and installing. Great.
So, now what we can do is we can run Ollama list to make sure that it works properly. And we can see here we have the Gemma 4 E4B model downloaded. So, this is the one that I just installed 7 seconds ago. I also have the 12 billion parameter and I've got a couple others that I've installed previously.
And you'll notice too that there's something called Gemma 4 64K. And I'm going to explain that in a moment because it's really important for certain use cases with the agentic AI. So, now why don't we try getting this going?
Run the model and use the Ollama desktop app
To get it started, all you need to do is write Ollama run Gemma 4 E4B. And we can see it's thinking down in the bottom here. And there we go. It just says send a message. That's it.
We can see it's thinking, which is pretty cool. It knows already in its thinking process that it's Gemma. And here's the answer. I'm Gemma 4, a large language model developed by Google DeepMind. I'm an open weights model and my purpose is to assist you. How can I help you?
So, that's pretty cool. That just took from start to finish maybe 1 minute or so to get a local model running on my computer where I can now just have a chat with it like I would with ChatGPT. And it's 100% local. Every single message I send to Gemma 4 stays entirely on my computer. It's full privacy.
And just very quickly, if you're looking for an easier way to chat with your local models, you can also use the Ollama desktop app, which is what I downloaded to install the command line interface in terminal before, what we were using.
So, if you want to have conversations and then you go back and continue the conversation, you can always switch the model like we did here. And those conversations will stay on the side stored on your computer in the same way that you would use something like chatGPT or cloud code, but all of this is happening 100% locally if I've selected a model that I've downloaded here.
Creating variants and extending context (64K)
But that's just the beginning of using Ollama. Yeah, you can talk with it here in chat, but that's not really what I want to use it for. I want to be able to connect it to another tool so that it becomes more powerful and whenever I want my local agent, for example, something like my Hermes agent to be able to run and do things for me and connect locally to my running model in Ollama, I need to do a couple more things.
So the first thing we can do here is end the conversation by typing /bye. And next, what I want to show you is how we can go back to these models here and we can create what's called a variant.
So a variant is basically just a configuration, a modified version of the one that we already have here, but it doesn't duplicate it. It doesn't take up another 10 gigs on your hard drive. It's just a different way of launching the same underlying model.
So, for example, if I type in Ollama PS, we can see that we have Gemma 4 E4B. It's 3.3 gigs, operating 100% in my GPU, but its context window is 32,768.
So, that's where we start to get into potentially a problem depending on your use case. For a lot of things, this is totally fine, but for example, if I go over to Hermes and we take a look at running a local model inside of Hermes agent, we can see that a lot of Ollama models use a 2048 token context limitation and Hermes requires 64,000 to give your agent tools.
So, it even shows you exactly how to get this set up inside of Hermes, but I'm going to use something a little bit easier.
Basically, what we need to do is we need to create a model file that extends the context. So, we need to create a little temporary file, pull it from the Gemma model that we're using, establish the context that we want, and then get it to create the new model based on that modified one.
So, let's just start a new terminal here so we can paste in this string, which I will give you in the description or a link to an article that has it, and we are pulling in Gemma 4 E4B, which is the model that we just downloaded, and we're specifying that we want the context to be 64,000.
So, it's creating a little variant file. You can say, "Okay, here we go. Gathering model components using existing layer. Success." And now if we go Ollama list, we can see we have Gemma 4E 64K variant.
So, we just created this new model 8 seconds ago, and now if we run it, and then we type in Ollama PS, we can see that now it's pulling in this model, and we have the context of 64,000. Instead of 3.3 gigs, it's now 3.4, so it's using up a little bit more space, but we've doubled the amount of working memory that the model is able to use running inside of Ollama.
And you can also control the context window at the system level in Ollama. If we go up to settings here, and we can see that we have this context length. So, I can move this up to 64, and then whenever it launches a new model, it will always use the 64,000.
So, we don't have to then create that variant manually like I did, but this will force everyone of your models to run at 64,000. So, if you only want to change the one model, then you would do it using the method I showed you in the terminal. But this is a pretty easy way to get all of your Ollama models ready to go with AI agents.
Connecting Ollama to Hermes and agent endpoints
So, that becomes really important because now we can connect it to something like Hermes, and we're able to actually use it. Which if you remember here, it says that it requires 64,000. But now that we have the model in the format that we want, how do we actually connect it to a tool like Hermes or Claude Code or Codex?
This is where we get into what's called an endpoint.
So, I'm going to use Hermes just as an example here because this is what I've been exploring lately, but you can use this with any type of system you want. But basically what we need to do is, rather than for example pointing it to ChatGPT, we need to point Hermes or Claude Code or Codex to the local model that we have sitting on our computer right here.
So, this is where we get into what's called an endpoint, and this is what it looks like. It's just a string of numbers that is an address based on what's running or can run locally on our computer. This is an Ollama native endpoint, and then if we add a V1, it makes it compatible with what's called an OpenAI endpoint. So, that's what a lot of platforms use.
So, for example, if we go into Hermes, I'm going to go to my Gemma profile, and I can go to models, and I can go down to custom endpoint and click set up custom endpoint, and this is where, like I was talking about it, even includes an example that's very similar. All I have to do is type in that string, that address that I was showing you, and then we click connect. We now have a custom endpoint connected here.
If I go back down to the models, we now see that we have this model right here, Gemma 4e 64k latest. So, that's the same model that I just created the variant of right here.
So, now if I go and want to have a conversation and I type in, "Hi, who are you?" It might take a moment to run.
If you remember earlier when I clicked run on Ollama, it had that little spinning icon for a moment. So, right now what Hermes is doing is it's spawning a little agent inside of Ollama, a locally running model. And then once I get that started up, it's warmed up, I'll be able to have a conversation, and it can run for a few minutes before it goes back to sleep, which I'll explain in a moment.
Okay, there we go. So, the model just woke up. It's analyzing the request, and we can see it's thinking right here. But this time, rather than just saying, "Hi, I'm Gemma," instead, we're running it through Hermes. So, now it recognizes that it's a Hermes agent running the Gemma 4 model. So, that's pretty cool.
So, we have as many different options as we want. Once we have the models installed on our computer, we can then have different agents harness the power of that model. That's why this is called an agent harness. You could use Claude code, you could use Codex, you could use Open Claw, you could use Hermes. This is kind of the beginning of fully private, locally run AI.
And what's nice too is with an agent like this, you could have it running 24/7, but in order to do that, we would need to keep the model loaded, and this is where we can modify some more settings of Ollama so that it doesn't unload after a few minutes. So, you remember it took a few seconds there to get running. We could have it be always on ready to go.
And what's cool, too, is if you have a more powerful system, like a desktop computer with bigger RAM, for example, I can potentially run a bigger parameter model on my desktop and then connect to it from my laptop, so I can leverage the power of a local model running on one computer, but then access it from another one, which I have another video that explains how to do that with Hermes, if you're interested.
Like I mentioned earlier, one of the reasons I'm most excited to run a local model is because I can manage my information in something like Obsidian, and I can potentially have sensitive information sitting inside of my Obsidian vault, and I'm only running a local model that's accessing that information, so it's not being sent to a cloud provider. So, once you begin working with local models, it really opens the door to a lot of interesting workflows that maintain the privacy of your data.
And there are so many different models that you can use here. You can also run embedding models for setting up like a document retrieval system. You can run vision, thinking, tools. There's coding models. So, instead of paying for Claude Code or Codex all of the time, you could run a local model for coding. Qwen 3.6 is supposed to be incredible, and it has a few different sizes. So, this is a bigger model, but there's a lot that we can do here. Running local models really opens a lot of doors. So, I highly recommend exploring it. Experiment with it yourself.
And there you have it. You have your own personal private AI running locally on your computer with a context-tuned variant that can be plugged into different agentic AI tools.
To help you understand more of the practicality of this, how you can use this in real-world situations, I'm putting together an agentic AI playlist using Hermes, but other tools as well, that connect to local models, so you can autonomously run tasks that hopefully make your life a little bit easier. So, I recommend checking out that playlist if you want to go deeper into anything that I've talked about today or see how you can expand this system.
Also, let me know if you have any questions about what I talked about, as I know this can be confusing for new users to AI, and especially running local models on your computer. So, if you have questions or there's anything you'd like to see me work on in future videos, please let me know in the comments and I'm happy to help you out.
A reminder to please like and subscribe if you found this video helpful and consider sharing with a friend who's perhaps AI curious but has been wary of the privacy and the data and the cost because a local model is a great way to get people into using AI to make their lives easier without dealing with a lot of the potential issues that people face with these types of tools.
Thanks for watching and I will see you in the next video.
