Introduction: Why Run AI Locally for Privacy?
AI chatbots are taking over the world. But many of them are the opposite of private. Every time you use the most popular tools out there, your data is sent off to third-party servers—meaning companies can collect, store, and analyze it. You lose control over where your information goes, and who has access to it.
This is the 2nd video in our series about AI privacy, and it’s for people who want to use cutting-edge tech without worrying about who’s watching. In our previous video in this series we explored some of the most private cloud-based AI chatbots out there, but ultimately if you want to guarantee your privacy, running an LLM locally is going to be your best bet.
When you download an open-source model and run it on your own device, everything stays right there, under your control. No data gets sent out, and there’s no risk of companies using your personal info for their own purposes.
In this video I go over some AI terminology that will help you understand all the components involved in this process, and then I follow along with a tutorial made by The Hated One, showing you how to run AI locally yourself.
Understanding Large Language Models (LLMs)
Running AI locally is the most certain way to protect your privacy. Let’s get started by understanding how local AI chatbots work and going through some of the terms and concepts we’ll come across in our tutorial. If you don’t want an explainer and want to skip straight to the tutorial, now is a good time to skip ahead to this time code.
For the rest of us, a good term to start with is LLM. It stands for Large Language Model. Let’s break this down piece by piece.
In machine learning, a model is a mathematical representation that has learned to perform tasks by recognizing patterns in data. Sounds complicated, so you can think of a model as something like a computational program. It’s large because it’s been fed a huge amount of data, and contains a huge number of parameters which we’ll explain in a moment. And it’s a language model because the data it’s fed is text-based, and it’s trained to recognize patterns from that data about how words and sentences are structured.
But it’s more complicated than just being able to recognize basic word patterns -- it can learn patterns in complex ideas, concepts, and relationships between topics. The model uses what it’s learnt to make predictions for language-related tasks. So for text generation, it wouldn’t just consider the immediately preceding word in its predictions, but the entire context of a sentence or paragraph, as well as intricate patterns within the vast amounts of text its previously studied, in order to give the most relevant responses.
We mentioned that these models contain a huge number of what’s known as parameters, usually in the billions. These are like switches or knobs that get adjusted during training until the model's predictions closely match the actual data. The more parameters, the more complex the data patterns that the model is able to capture.
These patterns are encoded in the model's parameters, and used in mathematical computations. Again, this sounds complicated, but by setting up this vast network of parameters, a model becomes capable of making generalizations about its training data, and applying these generalizations to new, unseen data, to generate new content based on the patterns that it’s learned.
The result is pretty unexpected. When models reach a certain size and complexity, they begin to display abilities that weren't explicitly programmed, such as being able to translate languages, summarize text, or solve problems.
How Local Models Work on Your Computer
Now in this tutorial we’ll be hosting an LLM locally on our computer. Some people might mistakenly think that this means our computer needs to house a huge amount of data. But LLMs don’t actually store copies of the data.
The model we’ll download onto our computer is not a dataset, but a structure of parameters, or patterns and relationships, that has been learnt from the original training data. And when we run this model, it will be entirely local and offline, which means it will not need to access any training data or send prompts off your device. It will generate responses based on learnt patterns that are stored locally.
But even though we aren’t housing a giant dataset, we do need to host and run a computationally intensive program. The more parameters, the more computationally intense it is, so we have to understand what size model our machine can handle.
In our last video we talked about 3rd parties that host and run models for you, and the benefit of this is that these 3rd parties will have specialized equipment with large computational capacity, so they are capable of running larger models with more parameters. We dove into some privacy preserving platforms that you can use.
In this video, we’re talking about models that are still incredibly high powered, but small enough that they’ll be able to run on your laptop.
The Local AI Stack: Model, Engine, and UI
The final thing I want to explain is the difference between a model, an AI engine, and a user interface. Specifically you might hear terms like Llama, Ollama, and LlamaCPP. There are a lot of Llamas. These are just different layers. Um, and they're all called Llama for some reason. It’s Lamas all the way down. Exactly. It's a stack of lamas.
I chatted to Ryan Condron, who works on decentralized AI compute, and he explained that there are all kinds of LLMs trained on different data sets, and Llama is just one of them. The Llama model is an LLM model that was trained by AI engineers at Meta. They've released it as open source, and they give it away for free. The latest has over 400 billion parameters to it, so it's become very, very powerful. 400 billion parameters will be too large to run on your machine, but Llama comes in smaller sizes which we’ll cover later.
Now in order to interact with this model, you need something called an AI engine. There's an open source AI engine called Llama CPP that can take a LLM locally, and you can interact with it. Again, Llama CPP is just one example of an AI engine, and it can run all kinds of different models. But it’s a really bare-bones, technical tool, that requires you to set everything up yourself and manually manage how you interact with AI models.
To make the process easier and automate a lot of this, there’s another layer you can add. So on top of Llama, CPP there is another open source project called Ollama. Now, Ollama is a locally installed interface that gives you a better user experience on top of Llama CPP. Instead of having to manually configure everything, Ollama does the hard work for you, streamlining the process so you can get started with just a few clicks. Ollama is actually just one of many of these type of systems. FreedomGPT is another great one. There's a GPT4All. And these other platforms work in a similar way to Ollama, offering user-friendly setups for running models locally.
So to summarize what’s going on behind the scenes in your setup: You have a model, you'll have a AI engine on top of it, you'll have a UI or UX on top of that. And these all work together to allow you to run advanced AI models on your device, without sending your data to the cloud. Your prompts and conversations stay on your machine.
Tutorial Start: Installing Ollama
Now let’s start the tutorial part of this video, explaining how to set up a local AI chatbot. and we’re going to be following along with a video that The Hated One made for us.
To get started, you’ll only need to follow a few steps. My main goal is for you to be empowered so I am gonna guide you through the process and will be holding your hand along the way. Don’t worry, this will only take a few minutes, but the payoff is totally worth it.
Ok let’s do this. Running AI locally is only about two things: First, find a place where you download your favorite models from, then find a program you’ll use to interact with this model. Think about this no differently from when you want to watch a movie. First, you find a place to download the movie from, then you use a program to view the file so that you can watch it on your device. So take this as a form of encouragement that you can absolutely do this yourself.
Luckily, there is one quick solution that does both of these in a single shot and that is Ollama. We talked about Ollama already as an easy interface for interacting with a local LLM, But what we didn’t mention is that it's actually a comprehensive platform that allows you to browse, download, and interact with a variety of AI models—all within a single app. Once installed, all interactions with these models are kept private and local on your machine.
Ollama, is the quickest and easiest setup and it’s available on Mac, Windows and Linux. Step 1: head to Ollama.com and navigate to the download page. From there, select an option for your system. For Windows and Mac, this is a matter of downloading a single file. For Linux, you’ll need to run a simple command.
Once you’ve installed Ollama on your computer, congratulations! You’ve accomplished the first step as you now have an interface you can use to download and run models from. This interface will be through the command line. Once installed you may get an additional popup asking you to install Ollama in the command line. Click “install” to continue.
Using the Ollama Command Line Interface
To test whether Ollama is successfully installed, open the terminal or command prompt on your computer and simply type “Ollama” and hit enter. The command line allows you to interact with your operating system and apps using text. You might prefer to use a graphical interface instead of written commands, so don’t worry—we’ll explain how to set this up later in the video.
For now, here are some super basic commands you can use for interacting with Ollama. First by typing a program’s name, you’ll see an index of what you can do with that program. The name of our program is Ollama, so type that in. In the output, you’ll see the list of available commands. The ones you’ll be using the most are “pull” or “run” commands. You’ll be using these to download and run models from Ollama’s repository to and on your computer. Anytime you want Ollama to do something, you’ll type “Ollama”, hit space, then type in the desired command and then hit enter.
This is all you need to know to start running AI locally.
How to Choose the Right AI Model
Now you need to choose an AI model to use. To find a model you’d like to run on your computer, head back to Ollama.com and navigate to the Models page. This is Ollama’s official library of many, many AI models all of which can run locally. There is a lot of models here so how do you know which one to choose?
When you open up a model’s page, you’ll get a bunch of additional info about who created it and what the model is best suited for. This is what can help you determine whether the model is what you want for your use case.
There are also other ways to compare models. For example using the open source platform called Chatbot Arena, developed by researchers at UC Berkeley SkyLab and LM SYS. It is a crowdsourced ranking system, where models face off in head-to-head battles, and users evaluate which model performs better in a specific interaction. The model’s score adjusts based on whether a model wins or loses each battle. You can look through the rankings and see which models are currently scoring the highest.
But highest score isn’t your only consideration when choosing a model. Because this is something you need to be able to run on your own machine, The most relevant information for you is the model size. This is represented in billions of parameters and usually looks like this. The higher the number, the beefier the machine it’s gonna take.
For example, running Llama 3.1 with the 400 billion parameter model is not going to be feasible on a consumer laptop. To run something locally, you’ll need to choose a smaller model. Llama3, as an example, has two different sizes for its model – 8 billion parameters and 70 billion parameters. The 70 billion model will probably also be too much for many consumer grade laptops. Usually, single digit-parameter models are gonna run fine on a consumer-grade laptop. In this case, the 8 billion parameter model is gonna run fine on a dedicated GPU. With other models, you also have ultra-small options, and these shouldn’t have a problem running on any machine, even a phone.
Running Your First Model in the Terminal
Once you’ve chosen the model that you want to run, if you click on it from the Ollama website, you’ll get the exact command to copy into your command line. So if we were to choose, let’s say Aya 23, you want to select the 8 billion model and then copy the command “ollama run aya:8b” into your terminal. Hit enter to run the command and let it do its job. It’s gonna pull the model from the website, and when it’s ready, you’ll be able to chat with it inside the terminal.
Isn’t that neat? This setup is excellent for basic chatting, brainstorming and testing the models you downloaded this way. Anytime you want to chat with your model again, just type in ollama, run, and the name of the model . It’s super easy, and gets you used to using command line on your computer!
Introducing OpenWebUI for a Graphical Interface
But if you’d really prefer a graphical interface instead, you can use something called OpenWebUI. OpenWebUI is a mature user-friendly interface that will allow you to communicate and tinker with local models in a very powerful way. You definitely want this tool if you want to tweak your models, create custom characters or feed the models your confidential documents.
Basically OpenWebUI is A graphical user interface, or GUI, that you can use for Ollama instead of using command line. It’s a web-based interface, meaning that the interface is accessed through a web browser. It doesn’t connect to the internet or send data over the web, the application runs locally on your machine and just uses your browser to display the user interface.
Setting Up OpenWebUI with Docker
Think of the browser as the front end that you interact with, while on the backend you set up something called Docker to handle the processing. Docker is a platform that basically creates a container that includes all the libraries, tools, and configurations etc that OpenWebUI needs. It’s a single, self-contained package that simplifies the setup process.
Installing this is the first step to getting a graphical user interface for Ollama going on your machine. To get docker, head to the official website and navigate to the products page and Docker Desktop. Download Docker for your machine. For Windows and Mac, it’s a single file that will install Docker on your device. For Linux users, you’re gonna have to follow specific instructions for your distribution.
From there, we just follow the installation setup step by step until Docker is fully installed on our computer.
