Getting Started with Ollama
Today I'll be showing you how to make a python AI chatbot in just a few minutes, and the best part is this chatbot will run locally, meaning we don't need to pay for a subscription or connect to something like OpenAI.
Now the first step here is to download Ollama. So go to ollama.com. Simply press on download, install this application, and this is what we'll use to run LLMs locally on our machine. Once you've installed Ollama, the next step is to make sure the installation is working properly. To do that you're going to open up a terminal or a command prompt and you're going to type in the following command, which is simply 'ollama'. Now this should be working if Ollama is running on your computer. And assuming this command works you're ready to move on to the next step.
Now Llama is software that allows us to download and run open-source LLMs. I have a list of them on the right hand side of my screen and I'll link this in the description. You can see we can access Llama 3 with 8 billion parameters, 7 billion parameters. We have a lot of other models available to us as well.
Now notice that these models vary in size, and the larger the model is the more difficult it's going to be to run in terms of the type of hardware you need on your machine. Now if you scroll down here, you'll see there's some different specifications. You should have at least 8 GB of RAM to run the 7 billion parameter models, and then 16 to run 13 billion, and 32 to run the 33 billion models. So make sure that you meet that criteria before pulling these specific models.
Downloading and Testing the Local LLM
Now for our purposes, we're just going to use the Llama 3 model with 8 billion parameters, and in order to get that on our computer we're going to type the following command. This is ollama pull and then the name of the model that we want, and this is simply llama3. When we do that, it's going to download this model for us. This will take a bit of time depending on how fast your internet connection is, and then you'll be ready to use it. In my case it's already downloaded, so it happened instantly, but for you it will take a few minutes to download, and then I'll show you how we can test it.
Now that the model is downloaded we can test it out. And in order to do that, we can simply type ollama run and the name of the model that we want. It's worth noting that you can have multiple models here and you can use them by simply specifying their name. So I'm going to run Llama 3. I'll just go with something like 'hello world' and then you can see that it actually gives us a response. If you want to quit this, you can type /bye and then you will exit that prompt. So now that we have access to this LLM it's time to start using it from Python.
Now using LLMs like this in Python is actually surprisingly easy and you don't need to be an AI expert in order to work with them. However, I always find it interesting to learn about how they work at a deeper level and that's where our sponsor Brilliant can help.
Brilliant is where you learn by doing with thousands of interactive lessons in math, data analysis, programming, and AI. They use a first principles approach meaning you'll get the why behind everything. Each lesson is interactive and filled with hands-on problem solving, which is six times more effective than just watching lectures. The content is created by award-winning teachers, researchers and pros from place like MIT, Caltech, and Google. Brilliant focuses on improving your critical thinking skills through problem solving, not memorizing. While you're learning specific topics, you're also training your brain to think better.
Learning a bit every day is super important and Brilliant makes that easy. Their fun, bite-sized lessons fit into any schedule helping you gain real knowledge in just minutes a day. It's a great alternative to mindless scrolling. Brilliance even has an entire AI Workshop that deep dives into how LLMs work and teaches you about the importance of training data, how to tune an LLM, and more.
To try everything Brilliant has to offer for free for a full 30 days visit brilliant.org/techwithtim or click the link in the description. You'll also get 20% off an annual premium subscription.
Setting Up the Python Project Environment
Now the next step here is to create a virtual environment that we'll install a few different dependencies in that we need for this project. What I've done is open a folder in Visual Studio Code and the command I'm going to use now is python3 -m venv and then I'm going to give this virtual environment a name, which is chatbot. If you're on Mac or Linux this will be the command. If you're on Windows you can change this to python and this will create an environment for us that we can have some isolated dependencies inside of. You can see the chatbot folder has now been created and the next step is to activate the virtual environment and then install our packages.
Now if you're on Mac or Linux the command to activate this environment is the following. It is simply source the name of your environment, which in this case is chatbot, and then bin/activate. If you did this successfully you'll see that the name of your virtual environment will prefix your terminal. Now if you are on Windows the command can vary depending on the shell that you're using. One of the commands that you can attempt is the following \chatbot\Scripts\activate.bat. Executing this should initialize the virtual environment if you are running in command prompt. If you are running in Powershell then you can change this to say .ps1 and put a capital on activate. So try one of these commands to activate the virtual environment on Windows and again make sure you have that prefix before we move forward.
Now the next step is to install the packages that we need. I'm just going to make this a bit smaller so we can read all of them. We're going to install the LangChain module, the LangChain-Ollama module, and the Ollama module. So go ahead and hit enter. This will install it inside of our virtual environment and then we are good to go and start creating this application that will use our local LLM.
Basic Model Invocation with Python & LangChain
The next step here is to simply create a Python file, we can call this something like main.py, and now to start writing our code.
Now in order to interact with Ollama what we're going to do is say from langchain_ollama we're going to import the OllamaLLM. Now here we can connect to Ollama. So we can say that our model is equal to OllamaLLM and then all we need to do is specify the model that we want to work with. Now in this case it is llama3, but if you have a different model you can put that here. Now we can actually use this model. In order to use it we can say model.invoke and then we can pass to this function here a prompt that it will act upon. So you can see that I can pass some input, say input is equal to 'hello world'. We can store this in some kind of result and then simply print this out to make sure that it's working. So let's print out the result and execute our code.
So from my virtual environment I'm going to type python3 main.py and notice here that we're going to get some warning, you can ignore that for now, and you can see that we get the response which is 'hello there, it's nice to meet you' and we've invoked the model with this input. So that's the basics of interacting with the model but I'm going to show you a slightly more complicated script here that explains how we can pass some context to the model and how we can make this a little bit more user friendly and have a full chat window interface to interact with it.
Structuring Prompts with Templates and Chains
So in order to do that we're going to start by bringing in something known as a prompt template. So I'm going to say from langchain_core.prompts and we are going to import the ChatPromptTemplate. Now LangChain is something that allows us to more easily interact with LLMs and one of the things we can do is create a template that we will pass to the LLM that contains our specific query or prompt. And this way we can give it some more description and instruction on what it should do.
So I'm going to say template is equal to and then I'm going to do a multi-line string, which is simply three quotation marks, and I'm going to tell the model to 'Answer the question below' but you can give this any kind of detail that you want. Now I'm also going to provide to this some context. I'm going to say, 'Here is the conversation history' and I'm going to pass to this the context. Now whenever I want to embed a variable inside of a prompt that I'm passing to the model I can surround that in curly braces, which I'm doing here, and then I'm going to say 'Question' and I'm going to pass the question variable as well and then I'm going to have the answer, which I'm expecting the model to generate. So you can see I'm giving this a template for how it should behave or respond to my queries.
Now we have our model, the next thing we're going to do is create our prompt. So we're going to say the prompt is equal to the ChatPromptTemplate.from_template and then we're going to pass that template that we just wrote, which is called template. Now we have a prompt and we have a model and what we need to do is chain these together using LangChain. So what I can do is say chain is equal to prompt and then I can use this pipe operator and I can pass my model. What this will do is create a chain of these two operations. So the first thing we'll have is our prompt which we'll have the question and context embedded inside of. Then we will pass it to the model where it will be invoked automatically.
So in order to use the chain now we can change this slightly. So rather than model.invoke we're going to say chain.invoke, but this time what we need to do is pass to it the various variables that are inside of our prompt. So we're going to say the context is equal to, in this case we don't have any context so we'll leave it blank, and then we'll say the question is and then whatever our question is. And we can just say 'hey how are you'. Hopefully you get the idea here. Let me make this a bit bigger so we can read it. We're simply embedding these two variables inside of the prompt and then passing that prompt to the model where it will be invoked using LangChain. So now we can test this. python3 main.py and it says 'I'm doing well, thanks for asking.' Now that's great, but we want to be able to continually talk with the model and store a conversation history so let's write the code to handle that.
Building the Interactive Chat Loop
So what I'm going to do now is create a function called handle_conversation and this is where we'll kind of put all of our main code inside of here. I'm going to start by storing some context, which will just be an empty string, and I'm going to print some kind of welcome message. So I'll say 'Welcome to the AI chatbot' and then we'll just tell them that they can type something like 'exit' to quit so they have a way to get out of this.
On the next line we're going to make a while loop and we're going to say while True and inside of here we're going to collect some input from the user. So we're going to say user_input is equal to input and we'll just put 'You: ' and then a space that the user has the ability to type. We're going to say if the user_input.lower() so converting this to lowercase is equal to that 'exit' keyword, then we are going to break out of the loop so that we don't have an infinite loop.
Now next we're just going to generate a response. So we can take this right here and bring this up, so let's paste that inside of here and indent this properly. Okay, let me just get rid of this print. Now we're going to say the result is equal to chain.invoke, but this time we're going to pass the conversation context and we're going to pass the question the user asked, which is simply the user input. Now what we can do is we can print the response. So we can say 'The bot said' and then whatever the result was here. And then what I'm going to do is just store all of this in the context, so that this bot has kind of the conversation history and it can respond to previous things that have been said. So I'm going to say context += to and then we're going to use an f-string available in Python 3.6 and above. I'm going to put a new line character with the backslash n, I'm going to say user is equal to and I'm going to embed the user input. And then I'm going to do backslash n and I'm going to say the AI, and then this is equal to the result. So I'm just storing what the user asked and what the result was. So now every single time I use this prompt, I'm passing all of this context back into the bot so it knows what was said previously and can use that to respond.
So that is our function. Now all we need to do is call that function. So I'm going to say if __name__ == '__main__': this just means that we're directly executing this python file, and then I'm going to say handle_conversation. And now we are going to run this function which will simply ask the user to keep typing in things until they hit or enter 'exit' and it will store the conversation history and pass that back to our model so that it can respond based on what we said previously.
Final Chatbot Demo and Wrap-up
So let's test this out and make sure it works. python3 main.py. Welcome to the AI chatbot. We're going to say, 'Hey how are you doing?' It's going to give us some response. I'm going to say, 'What is your name?' Great, no personal name, that's fine. 'Can you help me understand history?' Let's see what it says. And it gives us this long response here, which I imagine is saying yes it can help us if we give it a good question. Let's try to exit with 'exit' and now we are gone.
And there you go. We have now created a program that we can actually interact with a local LLM. We don't need an OpenAI key, we don't need to pay any third party service, and we can run this open source model on our own computer and utilize it from our Python script. Obviously this is a very simple example. You can do some really cool things that are a lot more complicated, but that's what I wanted to show you in this video. Hopefully you found this helpful. If you did make sure you leave a like, subscribe to the channel, and I will see you in the next one.
