Introduction to AnythingLLM
Okay, so today we're going to be looking at AnythingLLM, which is an open-source software that lets you interact with different LLMs from different providers, including open-weight LLMs using both Ollama and LM Studio. And since all of this is running on top of Llama CVP, it's very well optimized for RTX GPUs, which are consumer-grade GPUs from Nvidia.
And with the help of AnythingLLM, you can run powerful local agents completely privately on your local machine. So in this video, I'm going to show you how to set up AnythingLLM locally, then we'll see how to use these custom purpose-built agents to interact with your environment, all of this completely locally within your own environment.
Initial Setup and Workspace Configuration
So to get started, you first need to download the desktop app for AnythingLLM. It's going to look something like this. AnythingLLM uses the concept of a workspace, so first you need to create a new workspace if you want to work within AnythingLLM.
Now, if you click on this settings and then go to chat settings, you're going to see a list of different API providers that you can use. In this list, you have Ollama as well as LM Studio and even LocalAI. You can use these to run local models that you have downloaded or you can directly provide a Hugging Face ID. This will download the model. Even AnythingLLM can run some models by default, but that will require those models to be downloaded.
Configuring and Chatting with an Ollama Model
Okay, so first I'm going to show you how you can run Ollama models with AnythingLLM, and then I'll show you how you can configure the same models using LM Studio. Now, later in the video, I'll show you how you can use some of the agents that they have built, but in order to run an agent, you need to make sure that you select a model that is actually able to run the agent.
Now in my case, first we're going to select Ollama as our model provider. Then these are all the models that are downloaded on my local machine. So you will first need to start one of these models in your terminal, and then you can start interacting with that model. So let's say I'm going to select Llama 3 8 billion, and in my terminal, you can see that I've already started the Llama 3 8 billion model. Okay, once you select the model, you need to click on update workspace and you need to wait for this message. So the workspace has been updated. Now you can start interacting with this model like any other LLM that you will interact with. So let's say if I type hi. So here's the response, "Hello, it's nice to meet you. Is there something I can help you with or would you like to chat?" Right? So this is how you would normally interact with the model.
Using LM Studio as an LLM Provider
Now the second way to use a model is using the LM studio. So in this case, I am loading the Mistral Small Instruct model in 8-bit floating point precision. And the reason I'm doing this is that for agentic workflows, you want to use higher quantization. Four-bit quantization for smaller models probably is not going to work. So I selected this model and after that, I'm going to start the server. So this started the server and now LM studio is listening for incoming traffic.
So now for that, I'm going to create another workspace. Let's call it agent two. Now within agent two, I will go back to my default system and here in the LLM providers, I'm going to select LM Studio. Now if the server is running, it will automatically detect which model you're currently running. So I'm using the Mistral small instruct model. I will update my space. Okay, now I can go back and here we can just type in something like hi. All right, so here's the response that we got, "Hello, how can I assist you today?" And on the back end, you can also see here's the system prompt and then this is the user input or message and then the corresponding response from the model.
Okay, next I'm going to show you that beyond just interacting with the model, how can you use different custom-built agents.
Configuring Agent Skills and Providers
So first we're going to go to settings again. You want to go to agent configuration and in here again, you want to select the LLM provider for your agent. So I'm going to select Studio again because that is running the Mistral small model for us and we are going to update the workspace. Now if you go to configure agent skills, here there are a number of skills that are already predefined in here. You can enable or disable some of them. So for example, RAG and long-term memory, then View and summarize documents, scrape websites. So these are default skills that are enabled, but within this list, there are some other skills that I enabled. So generate and save files to browser, you can generate charts, or a web search. You can also connect your databases and then it will be able to interact with those databases using SQL. There are some custom skills which I'm going to show you later in the video. So this is a custom skill that will do Archive search on all the papers that are available within Archive.
So first let's look at the web search. Now in here, there are a number of different options that you can choose from. I am using Pry Search because this lets you make 1,000 queries per month for free. You will need to sign up on Pry, then just get the API key, provide the API key here, and then you can start using this web search as an agent skill. You can also generate plots, but let me first show you how you can use an agent and then we're going to look at some of these options.
Example: Invoking the Agent for Web Search
Okay, so in order to use a agent skill you need to use the @agent and if you look at the @ command, this is basically the list of different skills, which is a RAG search if you embed documents. We're going to look at an example. Web scraping, web browsing, save file to browser, or you can list all the documents that are available or summarize the documents or generate charts, right?
So the way this works is if you say @agent, then it will invoke the agent and will try to use one of these skills if it thinks that is going to be helpful. So let's see, I'm going to ask it, who won the 2024 USA elections? So it should be able to pull this information from the internet. So first it invoked the agent, then it decided to use the web browsing tool, and within the web browsing it has access to Pry as a search engine tool. So it searched for this specific phrase, who won the 2024 USA elections. It found five results and then based on those results it came up with this answer. Based on the most recent results, Donald Trump has won the 2024 US presidential election, defeating Kamala Harris to retake the White House. This marks a historical comeback for Trump, making him the 47th President of the United States. So based on the query and the LLM that you choose, it will be able to pick a specific agent skill in order to perform certain operations.
Example: Using the Agent for Web Scraping
Okay, so here's another example. I asked it, can you summarize the contents of this webpage? So then it decided to use the web scraping tool and it scraped the data from that website and after that, it provided us with a quick summary. Apart from this, there are some tools that are available on their community hub. So if you go to explore the community hub tools, here you can see there are some tools, for example Open Street Map, or you can use the Jina Reader. Jina AI is doing some really amazing stuff, especially with their late chunking techniques and segmented API. So it's definitely worth checking out.
Discovering Custom and Community Agent Skills
And then I think there are a couple of ones which is Home Assistant automation or you can look at your calendar events as well. Now, the one that I imported was this Archive Search tool. So you can select any one of these, click on import and then you will need to review those and append those or import those. So we just want to make sure that whatever skills you're using are published by someone you actually trust. In this case, this specific skill is from the AnythingLLM team, so we can just click on import agent skills. Okay, so this is successfully imported and if you look at my agent skills, this specific skill is listed in the custom skills. So I can enable this if I want. And now just like the way I was interacting with the rest of the agent skills, we can use the same @agent command. And let's say, can you find papers related to, let's say, skin cancer? And let's see what it's going to come up with. So here it pulled up some papers that are related to skin cancer risks, right? So pretty cool that you have an agentic system that is using completely local models and are able to perform different actions on behalf of the user.
Conclusion and Final Thoughts
If you are a developer, you can also contribute some agentic skills on the AnythingLLM Community Hub. And if you are somebody who has an RTX GPU, you will be able to run most of these models locally. Now, AnythingLLM is a really interesting project which goes beyond just chatting with an LLM. It is actually able to do a lot more using local models, so it's definitely worth checking out. It's a completely open-source project, so do check out their GitHub repo and give them a star if you like the work they're doing. Do let me know if you want me to create more content on AnythingLLM. I hope you found this video useful. Thanks for watching and as always, see you in the next one.
