The Rise of Local AI
This video is sponsored by Nvidia and shares how you can use the next evolution of AI with AI agents and how you can do it fastest and locally on your RTX AI PC using AnythingLLM.
AI is clearly reshaping the way that we approach tasks, the way that we work, and the way that we solve problems, right? And as developers, we've sort of gotten a front row seat to how these things have significantly improved our workflow. And I don't know about you, but I use generative AI almost every single day at work while I'm writing code.
Cloud vs. Local AI: Privacy, Speed & Cost
Now the AI tools that most people are familiar with like GPT run on the cloud, right? And that's because to run an LLM like GP4, you need an insane amount of computing power. Now I'm talking about clusters of GPUs working together in parallel. With that being said, there has been a growing trend of running AI locally on your personal machines or inside of your own local network. And that's because running AI locally one, gives you greater control of your privacy; two, it's much faster; and three, it can be significantly lower cost than using cloud solutions.
Running LLMs Locally with Ollama
And with tools like Ollama, we have access to an open-source LLM that we can download and run locally right on our personal machines. Now Ollama does have a 400 plus billion parameter model, but luckily Ollama does have a compressed version of their flagship LLM that's compressed down to just 8 billion parameters, so we can run it locally right on our machines.
And while it's entirely possible to run this LLM locally on our machine, even a compressed LLM still requires a decent amount of processing power. LLMs must perform massive matrix multiplication and process highly complex data sets efficiently.
Why Nvidia RTX GPUs are Essential for Local AI
So if you or I plan on running an LLM locally, you'll certainly want a fast GPU. And the perfect GPU to run LLMs locally are RTX GPUs right from Nvidia. And the reason why Nvidia GPUs are so good is because Nvidia's Tensor Cores that are found directly in RTX GPUs are specifically designed to handle the complex tasks that AI requires. The hardware that is found in the RTX GPUs like the RTX 40 series, which is exactly what I have, is tailored for Cutting Edge AI applications and processes.
Now I've tried running AI locally on my MacBook and then on my PC with my Nvidia GeForce RTX 4080 GPU, and I'm telling you it's just a night and day difference. So the bottom line is if you want LLMs to efficiently run locally on your own personal machine, it's best to have a good GPU like the GeForce or the Nvidia RTX. But you will especially need a good GPU to run AI agents locally on your machine, which is exactly what this video is about: the next, in my opinion, coolest evolution in AI.
What Are AI Agents?
So what are AI agents exactly? So you're probably familiar with generative AI chatbots like ChatGPT. Well, if a chatbot is generative AI that answers a question once you prompt it, AI agents can sort of be thought of as little workers that go do something with that information generated. For example, let's say theoretically if I wanted to learn about a black hole for a theoretical research paper that I'm doing, generative AI can give me a pretty good overview of what a black hole is, but agentic AI can sort of take it to the next level by taking my prompt and doing something like scraping the web from accredited space websites to provide me with a data set on something like the latest black hole discovered. It can then take that scraped data, store that in a vector database so I can access it later, and then write the paper for me and save that paper locally to my machine as a text document.
So again, you can think of agentic AI as little workers or little agents that do things for you. And if you use a little bit of creativity and think about running AI locally on your machines and AI agents locally, the possibilities are endless.
Introducing AnythingLLM for Local AI Agents
So how can you take advantage of AI agents? Well, the first thing you're going to need is a solid PC with a Nvidia GeForce RTX GPU, but after that, you can utilize this completely free tool called AnythingLLM. So AnythingLLM lets you easily download and run an LLM with essentially zero additional setup or programs. It's completely free, it doesn't require an account to run, and everything is designed to stay local to your machine by default, meaning all your documents, all your chats, and everything you do will be stored locally to your machine with no account associated to AnythingLLM.
On top of that, AnythingLLM has recently built a community hub that allows developers to build, customize, and deploy specific AI agent skills. And because AnythingLLM is built on top of llama.cpp, it's also optimized for use with GeForce RTX and Nvidia RTX GPUs. So inside of the AnythingLLM Community Hub, there's a growing list of possible agent skills that you can add locally to your agentic AI, and this is really just the start. These skills were recently added about two weeks ago, and developers in the community are already working on building even more AI agent features.
Setup: Connecting AnythingLLM to a Local Model
All right, so let's go ahead and take a step back and walk through setting up an LLM, an AI agent, to run locally right on our machine with AnythingLLM. So you can start by downloading AnythingLLM from anythingllm.com right here. Now I've already downloaded this, but I promise you the setup is easy, and it doesn't really warrant a walk-through. So once that's downloaded and you walk through the setup, you want to start by hooking into your own LLM. So just like how ChatGPT is a UI interface for the GPT models, AnythingLLM can also be used as a chatting UI interface, but to interface with whatever model you want. So you can select the model you want AnythingLLM to use by clicking this settings wrench right here at the bottom of the screen and then clicking the LLM option in the tab bar.
From here, you want to select your LLM provider. And because I want to use a locally ran LLM for privacy reasons, efficiency, and then cost-saving measures like I described in the beginning of the video, I'll simply select the Ollama LLM provider that I already downloaded. Then for the actual model, I'll select the Llama 3.1 8B model. I'll click save changes. From here, I'll go back to my workspaces, I'll select new workspace, I'll name this Kenny's workspace, and just like that we are now connected to our Ollama local instance, and I can now ask questions directly through the AnythingLLM UI interface.
Demonstrating the Base LLM's Limitations
So I'll ask, "Hello, who made you?" And you'll see it told us it was made by Meta, which is true, this model was in fact made by Meta. So pretty standard LLM chatting experience, but the cool thing is is that this is running locally on my machine for completely free.
So what about agents? Well, if I were to do something like ask this chatbot to do a task like scrape a website for data, it wouldn't be able to do it, right? So if I asked it, "Can you scrape the https://thestateofhealth.com? Tell me what the website is about." It will say that it can't because our local LLM can't connect to the internet to scrape websites because it just doesn't have that capability. And this is exactly where AI agents come into play.
Configuring AI Agent Skills in AnythingLLM
So with AnythingLLM, we can go back into our workspace settings by clicking this cog icon right here, navigate to this agent configuration tab. From here we can specify our agent LLM provider. So I'll just pick the Ollama one that I already have running. I'll click this update agent button. Then from here, I will click configure agent skills, and right out of the box, you'll notice that AnythingLLM has these default agent skills baked into it, such as RAG, long-term memory, and then scraping websites. But you might notice the custom skills right down here, and these are the plugins that I, in fact, downloaded from the community Hub that I showcased earlier on in this video. Anyways, I'll turn on the web scraping agent skill by clicking this web search item here and then clicking the toggle. Now this does require a Google API key, but it's completely free and you can generate it using this link here. Aside from that, I will also toggle this custom skill, which is Gmail draft. So again, this is an agent skill that I downloaded directly from the AnythingLLM Community Hub.
Task 1: Web Scraping and Long-Term Memory
So now my AI agent has the capabilities to search and scrape the web for data, store things in long-term memory, and then create Gmail drafts for me. So I'm going to save these changes and then go back to my chat. Remember how we prompted our AI to scrape data but it wasn't able to? Well, if we invoke the AI agent with the @agent command like this and then paste in the same exact prompt, the agent will now scrape the website for data, and that data will be processed by my local Ollama instance.
And just to show you that this information is correct, here is the website thestateofhealth.com, which is simply just a landing page to promote my personal fitness tracking app. And if we compare the response from the LLM to the actual website data, you're able to tell that this information is correct. Now aside from scraping data, my agent can also save this information for later. So I can invoke my agent again and say something like: @agent Thank you. Please remember this information about stateofhealth.com so I can review it later. So as you see, it's storing it in our vector database, and now the agent has stored this information so I can access it later.
Real-World Use Case: Generating a Client Proposal
So the question is, why is this useful? Well, this is a very generalized example for the sake of simplicity, but let's go ahead and actually walk through a real-world scenario. So let's go ahead and imagine for a moment that I wasn't the creator of this thestateofhealth.com website, and instead picture me as a freelancer. And as a freelancer, I'm hired to gather insights about a client's software, and in this case, let's assume the client is thestateofhealth.com. By using an AI agent, I could scrape this website's data, store it in a database, which we've already done, and then analyze that data to craft a proposal for possible feature enhancements. And I could do that by typing something like: With the information you know about thestateofhealth.com, come up with additional features that could make this application better and write a proposal to the client of how I could, as a freelance software developer, improve their application.
So this will write out a very generic proposal, and of course, using additional AI agent skills like RAG, you could upload your own personal documents to make sure proposals were tailored to your actual freelance skills.
Task 2: Drafting a Client Email with an Agent
So with the proposal written, I can then use the agentic Gmail draft agent skill that I added directly from the community Hub and type in something like: @agent draft. Create a Gmail draft link with the subject line 'State of Health App Improvement.' The recipient should be [email protected].
And as you see, this will invoke the agent skill that we added to create a Gmail draft link. So I can take this link, plug it into a web browser, and boom, I have the draft email prepared for me. And then obviously, in this theoretical example, you would tweak this proposal before sending it to a client, but hopefully that gives you the gist of how you can use AI agents to significantly improve your workflow.
Conclusion and Next Steps
Anyways, I definitely think you should keep up to speed with AI agents as they're only going to get better and the skill capabilities will become much more complex. Also, just to mention, if you're interested in coding your own AI agent skill, definitely check out the AnythingLLM Community Discord and GitHub to figure out how you can do that.
I hope this video got you as excited as I am about AI agents, as I think they're extremely cool. And if you are running LLMs and AI agents locally on your own machine, make sure to go ahead and grab yourself a GeForce or Nvidia RTX GPU. You can check out the link in my description for that. Thank you to Nvidia for sponsoring this video, and I will see you all in the next one. Peace.
