Made withAnythingLLMTutorial

Unlimited AI Agents running locally with Ollama & AnythingLLM

T

Tim Carambat

Creator on the AnythingLLM channel

Watch on YouTube ↗

238K viewsMay 9, 202415:21 runtime

Learn how to run unlimited, private AI agents locally using Ollama and AnythingLLM. This guide demonstrates how to give powerful agent capabilities, like web searching and file scraping, to any Ollama model, including Llama 3. Discover the importance of model quantization for…

Chapters

#anythingllm#tutorial#aillm

Summary

This video provides a comprehensive guide on how to transform any local Ollama LLM into a powerful, multi-skilled AI agent using the open-source AnythingLLM desktop application. It covers everything from foundational concepts to practical, hands-on demonstrations.

Core Concepts

Ollama: An application that allows you to run large language models (LLMs) like Llama 3 entirely on your own computer, ensuring 100% privacy.
Quantization: A compression technique that makes it possible to run massive LLMs on consumer-grade hardware (CPU or GPU). The level of quantization (e.g., Q4 vs. Q8) impacts a trade-off between model size and performance.
AI Agents: Unlike standard LLMs that only generate text, agents can perform actions. They use tools or skills (powered by function calling) to interact with external systems, such as searching the web, scraping websites, or managing files.

Setting Up Your Local Agent Environment

Choose the Right Model: For agentic tasks that require high reliability, it's better to use a less compressed model. The video recommends using the llama3:8b-instruct-q8_0 (Q8) version instead of the default Q4 version for better performance.

Important
Highly compressed small models (e.g., an 8B model at Q2 or Q3) can lead to poor performance, hallucinations, and unreliable outputs. For agents, a more robust model like a Q8 version is preferred.
Run the Ollama Server: After pulling the model, run oama serve to make it available over a local API endpoint. The video demonstrates using ngrok to tunnel this connection from a dedicated machine to another computer on the network.
Configure AnythingLLM:
- Download and install the AnythingLLM desktop app.
- During setup, connect to your Ollama instance by providing its base URL.
- Select your desired chat model (e.g., Llama 3 Q8).
- For maximum privacy, use the built-in embedder and vector database.

Key Agent Capabilities in AnythingLLM

AnythingLLM unlocks function-calling capabilities for any connected LLM, enabling a variety of built-in skills. To use them, you invoke the agent by starting your prompt with @agent.

Live Web Search: The agent can search the web to answer questions about current events or topics not in its training data.
Tip
You can get a free Google Programmable Search Engine API key that provides 100 queries per day, which is enough for most personal use cases.
Website Scraping: Instruct the agent to scrape a URL and extract specific information (e.g., "@agent can you scrape useanything.com and tell me the key features?").
Long-Term Memory: Ask the agent to remember the contents of a conversation for later retrieval. It summarizes the chat and saves it to a vector store.
Document Summarization: The agent can read an entire document from your workspace and provide a summary, a task that is difficult with standard RAG.
Retrieval-Augmented Generation (RAG): While RAG is a foundational feature, agents build upon it. The video first demonstrates resolving hallucinations by adding a relevant document to the workspace.

The Future of AnythingLLM

The current set of skills is just the beginning.
Future versions will allow users to define their own custom agents and tools, similar to frameworks like CrewAI.
AnythingLLM is open-source and free. The best way to support the project is by starring it on GitHub and providing feedback or feature requests.

Introduction to AI Agents with Ollama

Hey everyone, my name is Timothy Kbat, founder of Mlex Labs and Creator and maintainer of AnythingLLM. Today I'm actually going to showcase AnythingLLM, just kind of how it works, but then also show you something that makes Ollama models really powerful. We're actually going to give agent capabilities to any LLM available on Ollama where you can then search the web, save things to memory, scrape websites, do whatever you want, make charts even. And I'm going to show you how to unlock all of those abilities by just downloading AnythingLLM and connecting to Ollama. It'll be really simple. But first, I want to showcase a little bit of education about what Ollama is, quantization, and what agents even are.

What is Ollama and Quantization?

First, Ollama. If you found this video, you've definitely heard of Ollama because Ollama's in the title. Ollama is an application you can install for Mac, Windows, and Linux, and it allows you to run LLMs using your own computer's devices, no cloud, no anything like that. So it's totally private.

The way that this is possible, because Llama 3 is a massive model that takes dozens of GPUs to run, is through a process called quantization. Quantization is basically how we can get these models small enough to run on your CPU or your GPU. I'm not going to get into the weeds of how that works. In general, you should know how quantization works. It's basically compression of an LLM, and when we get into agents, I'll tell you why that's really important.

What is an AI Agent?

The next part of this really short lecture is what is an agent? So you have LLMs and they respond to you with text, right? They don't really do anything. An agent does something. It's an LLM that is able to execute what people call our tools or skills or there's a whole bunch of language, but it's an LLM that from your input doesn't just respond with text. It actually goes, runs some program or interface or API, gets that information, does that action, and then comes back to you with the result, the response, or your question answered with that tool's supplemented help. It's like RAG, but you're doing things instead of just chatting with a chunk of a document. And you can see that actually RAG is on the top part of this graph: short-term and long-term memory, which is kind of a common use case for retrieval augmented generation. Chat with your docs, all the same thing. What we're going to do is get this working for any LLM.

So you're probably familiar with cloud-based models like OpenAI or Anthropic's Claude or Perplexity, where you can say things to the model and sometimes it can go and do something like search the web, which is a very common use case. However, if you are using Ollama and you try to tell your model to search the web, it'll just tell you that it can't do that. Well, now with AnythingLLM, any LLM is an agent and can be an agent and can even search the web and do all of this for free on your computer with 100% privacy. So I'm going to show you how we're going to unlock that today.

Choosing the Right Quantized Model

So the first thing that we need to do is find a good model. As I said, any LLM will work with AnythingLLM and its agent capabilities. However, when we come back to quantization, there's one detail that people kind of overlook when it comes to Ollama. By default, Ollama will install a Q4 quantization. Now, that probably doesn't really mean anything to you, but here's kind of the go-by for it: Q1 is the most compressed version of that model, Q8 is the least compressed version of that model, but still compressed, not the raw model.

If you have a model that is 8 billion parameters and you compress it a lot to like two or three, you basically took something that's already small and then compressed it a lot. So now you have a pretty bad model and you'll get hallucinations, you'll get weird outputs, it'll just go crazy, not even respond to your questions. All of these become problems at smaller models being quantized very heavily. So what we're going to do today is intentionally download Llama 3 from Ollama but use the Q8 version so that it is more robust, the calls are more reliable, and the responses are just better. If we were messing with the 70 billion model, yeah, we probably wouldn't download the Q8, it'd be like 30 gigs. We'd use the Q4 and have a good time because 70 billion parameters is a lot. I know that sounds very technical, but hopefully you understand why quantization and picking the right model is a use case science, and it's something that you should understand if you're messing with LLMs at all.

Downloading the Llama 3 Q8 Model

If you go to Ollama and you go to Llama 3 and you scroll down, you'll see that the 8B tag and the latest tag, which is what downloads by default, are the same. But this tag is also matched to the instruct model, which is the same, and it is a Q4. So this is a pretty small model, and it's basically the middle of the road between size and performance. But we want really good performance because we're dealing with agents.

I'm going to go and find the Q8 version of this model, which you can do by just typing it, Q8, and you'll see that it's right here. It's 8 and a half gigs. I'm running on a MacBook Pro Intel. It's pretty bad for inferencing in general. I have a Windows computer in the other room, so I'm actually going to use Ollama on that computer, run AnythingLLM on this computer, all on my private network.

Setting up the Ollama Server with Ngrok

So here I am on my Windows computer, and I have Ollama installed. If I type in oama, we have it running. I need to pull in that Q8 model, and the easiest way to do that is oama and I already have this downloaded because I wasn't going to wait while making this video. And so you'll see it downloads all of the layers, we're good to go. So the only thing left is oama serve to make sure that the server is running. The server is already running, and as you can see, I have ngrok running and I'm tunneling my desktop computer in one room to the connection for in another room on my MacBook. This is where we can get into AnythingLLM.

First-Time Setup in AnythingLLM

AnythingLLM is an all-in-one AI agent and RAG tool that just runs in your desktop, fully locally, connects with pretty much anything that you care about. And it can work on Mac, Windows, and Linux. All you do is useanything.com/download and then click on the proper operating system and chip architecture.

And since I have AnythingLLM downloaded, we're going to boot it up. And because I've never run it before on this computer, it is going to basically just ask us what LLM do you want to use. That should be the first question. So here we are on onboarding, and it asks us, what LLM do you want? AnythingLLM actually ships with Ollama inside of it, so the whole setting up Ollama on my Windows computer completely extraneous if you have a GPU device. I am on an Intel MacBook, so it's really old, so I'm actually going to use the Ollama external connection. And all I'm going to do is paste in that address from ngrok and you'll see that my chat models are loaded. I want to use the Q8, and I know because I know about this model, it is a 8,192 context window. It's really annoying that they don't publish this information for every model, you have to go and Google it. It's just annoying, but anyway, we'll just continue. So you can see this is kind of a privacy overview. We're going to use AnythingLLM's built-in embedder so everything will embed on this device, and we're going to use the built-in vector database as well so that basically none of my chats are leaving my local network at all. All of my data is going to stay on premises, and it'll all just work very nicely. And of course, you can skip the survey, it's totally optional. Let's make a workspace and we're going to just call it 'sample' for now.

Testing RAG with a Local Document

The very first thing that people would want to do is just test to see does the model work. So let's just say hello. And what this is doing is sending a request to my Windows computer, and Ollama on that computer is going to stream it back. And you can see it works, it works about as well as you would expect it to, and it's fast. However, while it might be fast because I'm using a 4090 in the other room, it's still pretty dumb. And the reason that we can say that is because it doesn't know anything about what maybe I want it to know about. For example, AnythingLLM, while people love it and it's great and it's cool, it's not popular enough for an LLM to know about it. So if we were to ask the question, what is AnythingLLM, it's likely going to make something up. And it's going to say that AnythingLLM is a LLM, which is totally wrong and it's, yeah, this is all a hallucination. None of this is accurate. But what can we do to improve its ability to know about AnythingLLM? Well, the easiest way is RAG. So let's do that first.

So we're going to go and upload a document. I actually have AnythingLLM's GitHub README already downloaded as a PDF, so I'm just going to upload that and then move it over to the workspace so that when I am in this workspace chatting with Ollama, it will use this set of documents. And you can see that it was downloaded successfully, and so we can just close this window. Now let's reset the chat and ask it that same question again: what is AnythingLLM? What we would hope to see is to get a response back, which, wow, that was quick, and we get citations, and we can actually see what chunks exactly were relevant to my query that resulted in the LLM being able to complete this. And it says AnythingLLM is a full-stack application blah blah blah does all this stuff. That is accurate, this is actually factual information.

Enabling Agent Skills and Web Search

We can go into the workspace of settings and we can, you know, go to the vector database, we can increase the number of snippets per chat, change the way that documents are deemed relevant. But it's actually an easier way to just use LLMs and that is with agents. And as I had said before, this is not a capability built into Llama, it's not a capability built into Llama 3. This is actually something that we have been able to do to apply to any LLM that doesn't support function calling. Function calling is how all of this magic works and now you can unlock it when you use AnythingLLM with any LLM. So what we want to do is we want to use Ollama, we have Ollama, we have our model, that's it. Really don't want to use a worse model. I think we have Llama 3, but let's stick with the Q8 version.

And there are some default skills that exist, of course, RAG and long-term memory, we already saw that that's built into AnythingLLM. We should have the ability to look at the documents in our workspace, modify them, summarize them, commit new information to long-term memory just from chatting, and all of that. We should be able to summarize these documents, we should be able to scrape websites, that's a feature just built into AnythingLLM. We should be able or can generate charts. I will admit this one is a little model dependent. Some models just aren't great with like, you know, you could paste in a CSV and say make a bar chart. Some models kill it. Llama 3 honestly, isn't that great. Generate and save files to a browser. So if we're talking to it and we say, hey can you save that contact information to, you know, tim.txt, it'll download it and save it on your desktop on this device. And then of course, live web search and browsing. This makes any LLM that you download running locally basically on par with Perplexity. And actually, you can do it for free. I'm sure you were like, 'Ah, but I need an API key.' You do, but Google actually offers this service totally for free. You can just click on this link that we provide, and it opens up this new programmable search engine stuff. You get 100 queries a day, which is honestly pretty good. We do support other like search engine results providers, but this one's totally free and anybody with a Google account can sign up. So let's connect mine and so we can get web browsing. Okay, so I have that information put in, I'm going to click update, and now everything is saved. Let's go back to the chat window.

Using the Web Scraper and Memory Agent

Now keep in mind, we had information about AnythingLLM already stored in here, so let's remove it. So we're just going to go remove that right now. If we are to reset the chat and say, um, 'what is AnythingLLM', we should again get a made-up response that has nothing to do with the actual tool. We can actually get agent into the loop on this. And the way you can do that is by typing '@agent' or you can click this and we tell you about how agents work. Um but if you click this, you can actually see agent is how you would invoke this. You would say '@agent can you scrape useanything.com, which is our website, and tell me the key features?' Let's just call it that. And what we should hope to see is this model go to useanything.com, scrape that, compile all that information, specifically the key features, and hopefully give us back a pretty good text response.

And you can see that we actually get what I would consider a pretty decent response. But keep in mind, this is not in long-term memory. So let's ask the model to remember that for later. So let's say, 'Thank you. Can you remember that information for later?' And what we should hope to see is the model recognize this as an available function and it say, 'Oh, yes, of course.' I will take the chat as it is right now, summarize it, and then save that for later so that when we ask in regular chat, it would work. And you can see that it's done that.

Summarizing Documents with an Agent

But now let's look at summarization. Summarization is one of the most asked and used features of AnythingLLM. It's not how RAG works. It's actually a pretty big misunderstanding that people think that you can just upload a document in a vector database and say, 'summarize my document.' It's just not how vector databases work. But with AnythingLLM you can do it.

And so I'm going to open up a new workspace. So we'll just call it 'AnythingLLM' and we're going to upload that same README document because I've already embedded in another workspace, embedding is instant. And now, with no other kind of inferencing or leading or anything like that, let's just ask the agent, 'Can you summarize readme.pdf,' which is the name of the file in the workspace. And you can see it looks at the available documents, found a document called readme.pdf, and then begins to summarize it. Again, this is all running locally within my network because I'm using my Windows computer, but it is summarizing. And you can see that it says it summarized it, blah blah blah, did all this stuff, mentions it's MIT licensed. That is kind of the quick preview of what agents can do for any LLM when you put them in AnythingLLM.

The Future of AnythingLLM & How to Contribute

And while I do recognize that this list of default skills is pretty limited right now, I do want to really, really emphasize that this is just the beginning for AnythingLLM. We're actually going to have the ability for you to define your own agents like you would in tools like CrewAI and any other kind of agent builder that you know is already out there. That'll just exist in AnythingLLM. AnythingLLM plus Ollama can be your go-to for not only RAG but also AI agents that can do things for you. We have a lot more cooking on this front and so I'm really excited to show you this even in its current state. And I also do want to remind everybody that AnythingLLM is open source. You can use the app that I just showed you right now today for free with no ifs, ands, or buts. You just download it and get it running. And the easiest way to support us is actually by starring us on GitHub. We would really appreciate that. More so, I'd also appreciate feedback, suggestions on new tools that you would like to see agents accomplish. We'd love to know what you're working on and how AnythingLLM fits into that flow. So that's it for this short video. I really appreciate your time. Thank you.