Introduction to AnythingLLM & NVIDIA NIM
Hey everyone, Timothy, creator of AnythingLLM. And actually today I want to show a very special video, something we have been working so hard on. I honestly think for all of you who have GPUs in your computer right now, you're going to love this.
If you're not familiar with AnythingLLM, we build a desktop app that just does the things that you want AI to be able to do: chat with documents, run agents, create your own agents without code, but all 100% locally on your device. That's what we do, and that's what we continue to do. And today's announcement is actually more of that, but even more powerful.
So for those of you who have used AnythingLLM, you know that we have 20 or more LLM providers, whether that be running locally on your device or running in a cloud somewhere, or however else you want to run an AI model. We just make it all usable, all the same. And today I want to talk about AnythingLLM's partnership with NVIDIA to bring NVIDIA NIM onto your RTX GPU. Now you might be wondering, what is a NIM? A NIM stands for NVIDIA Inference Microservice.
Now, you can think of this as an LLM in a tiny container that just comes with all of the industry-standard tools and software dependencies baked inside of it. Now that may sound very complicated, and it is, but using it is not complicated, as I'm going to show you today. One of the benefits of using an NVIDIA NIM, if you've ever tried to run a model pulled off Hugging Face of any kind and run that on your Windows machine that has an RTX GPU, then you already know how complicated this is. You need to make sure you have the right version of CUDA, you need to make sure that first off you have the right version of Python. You also have to make sure you have the right version of the Transformers library, which also has dependencies on all these other things that also might not be compatible. It is a nightmare.
When you have an NVIDIA NIM, you get to have all of that power as fast as possible running on your RTX GPU with none of that overhead. You just get to use it. And that is one of the things that I'm going to show you today. We're going to run some pretty powerful models, and they're going to run so fast, and I think you're really going to like what we have in store.
Hardware Requirements & Available NIM Models
If you caught Nvidia's keynote at CES 2025, you actually would have heard about the NVIDIA NIM whole service. You're going to need a GPU, and you're not going to be able to get away with some of the smaller or older series cards, but you will be able to do this on some of the mid and late 40 series and of course the 50 series GPUs that just came out. Today I'm going to demo running a NIM on a 4090. You do not need a 50 series, although I do happen to have one, but as you've noticed, this is not plugged into a computer. And I just want to show you that you don't need to go wait outside of a Micro Center for 24 hours in a tent to be able to run a NIM. You can run a NIM on the card that is in your machine today, and that's what we're going to show you.
I'm doing this on my RTX 4090, and I do want to show you that we're not limited really here with a lot of the models. Now, of course, this isn't every model on Hugging Face, but these are all the models that you definitely have heard of. So if you go to build.nvidia.com, you can actually view these, and of course all the models that you're expecting to run. And actually two of the ones we're going to run in this demo are available. So DeepSeek R1, Qwen 2 and 2. We also have some very specific science models, we have classification models. Here's Llama 3.1 405B; we need multiple 5090s to run that. We're looking for more attainable models. So for efficiency and speed reasons, we're going to run the 8B today. They also have Gemma, we have Mistral, we have NeMo, we have a whole bunch of models here including Microsoft Phi. So a lot of the more capable and common models you've heard of are available as NIMs today. And we're going to run two of them. I'm going to run DeepSeek R1, a distillation of Llama, and then we're also going to run Llama 3.1 8B. And I just want to show you that these models are just as capable, extremely, I mean stunningly fast on an RTX GPU, and they just work.
System Specs & Initial AnythingLLM Setup
First, let me just show you what we're working with. So in this machine, I actually have a good amount of RAM in here, this is like a whole workhorse machine. So we've got 128 gigs of RAM, went crazy with it. But you can see that the GPU is indeed an RTX 4090. So nothing crazy there, no trickery, no chaining, no nothing, just a regular consumer RTX GPU 4090 you can get right off the shelf today. And we're going to really use this thing.
So first things first is I'm just going to start up AnythingLLM. Haven't done anything yet, haven't really like set anything up, just regular standard install of AnythingLLM. And you can actually see my default LLM right now is using our built-in LLM engine and it's running Llama 3.2 3B, which is a small model and it runs really fast on this machine with this GPU. But I want to run something even better because three billion is really small and, you know, it's not great for large context or agent tools or anything like that. So what we're going to do is we're going to run the NVIDIA NIM.
Installing the NVIDIA NIM Engine
And the way we do that is you just can look it up or click on it. And we've had this integration in AnythingLLM for a while now, but you'll notice that this blue container is different. So in order to run NVIDIA NIMs in a very simple format, we actually utilize WSL, which if you don't know what that is, this installer takes care of it for you. This installer was written by NVIDIA, so everything is up to spec and right and done the way it needs to be done. And all you have to do in AnythingLLM is just click it and you'll see it comes up with the authorization window. Just click Yes, and then what we're going to do is we're just going to step through this agreement and install it step-by-step. This process is pretty quick so I'll just let it go. And so that was it, the installer is done. But you'll notice that after the installer is run, it says swap to managed mode. This is the mode you really want to be in to get the most bang for your buck. If you want to set up everything manually, do everything yourself, you can still connect to your NIM however you like. For simplicity purposes, you probably want to use the managed mode.
Downloading Your First NIM Models
So you'll notice right when we swap over, there's the ability to import a NIM from NVIDIA, but then also show the current ones we've downloaded. So if you click on this, you'll notice as I said, we have Llama 3.1 8B ready to go. Just click it and download. But we also have DeepSeek R1 Llama 8B distillation. This is not the full DeepSeek R1, there's a lot of confusion about that. We don't really need to get into it, but it is a very capable reasoning version of DeepSeek that you can run locally on a consumer RTX GPU.
Also, if you don't like these two models that are suggested, you can always go to the link on the site and you can also click and run any of these models. If you can run them, if you have enough VRAM, if you have enough resources to do it, you would just click on the model and then could actually just copy paste this string and paste it into AnythingLLM and it would import this specific model if you're not interested in running either of these two suggested.
So what I'm going to do now is I'm actually going to pull in both of these images. They're not that large, they're both around 16 gigs each. And I do want to again reiterate, you may immediately think 16 gigs, that's crazy. When I download it from wherever, it's like 4 gigs. You are running the full model, no quantization, no compression, no cuts anywhere. This is the full version of the model. It is going to be a little bit bigger. So let me go ahead and download both of these so that I can show you them both running and also just kind of just play with them for a little bit. So first, let's get to Llama 3.1 8B. So we click on this, there are of course terms to agree to. These are the standard Llama terms that you already agree to when you use the model anywhere else. They're the same terms, there's nothing fancy or different here. And you notice that while it's downloading, we get a little notification and we have a way to watch it download so that we can know when it's done. So we're going to let this run for both Llama and DeepSeek and then I'll pick up back after they are both installed.
Running Llama 3.1 8B on an RTX 4090
So we now have downloaded both the Llama 3.1 8B full model as well as the DeepSeek R1 Llama 8B distillation. They both will function exactly the way that you expect them to function. But first, we should just start one of these containers and just save our changes and then just take it for a spin and let's just see how it works in AnythingLLM.
Since AnythingLLM is basically agnostic to how or what model you run, you'll still be able to chat with documents, scrape websites with agent skills, and also run your own agent skills. Everything inside of AnythingLLM will work exactly the same, but you're going to be running a way more powerful model on your RTX GPU. So to start one of these is as simple as just clicking Start. It's going to ask you just to make sure, because it will allocate some VRAM so that you can run the model. And I might as well open up performance while we're doing this, and you'll notice that as the model begins to load, we will see dedicated GPU memory begin to rise. And so that is what we will see. And there is a way to actually go and see the model boot up in progress. This is just great for those of you who are curious, but also good for debugging. And so once the model is fully booted, we'll pick up from there.
Okay, so after about literally 10 seconds, the model is now ready to run. And so we can close this. In fact, actually, if we open this, you'll see that now we are running a pretty hefty model, using a lot of the GPU memory, but still much more to go. Anyway, let's get to the good stuff. So let's save our changes. And so now we're using this as our LLM in AnythingLLM. And now the first thing to do is to just ask it a question. And you can see that that happened so fast it almost seemed like it didn't stream. But let's also just, you know, ask it to say, like, tell me a short story or something. And you'll see that this thing is just absolutely cruising right now, getting 132 tokens per second on that response.
Agent skills are also all perfectly capable to be run here. So let's go look at some that we have on. And you can see let's just turn on web scraping so that we have that available. We can turn on summarization of documents if we want to. We can turn on long-term memory. So now let's just try and run an agent skill. We're going to just scrape a website and ask it, ask us to tell us about it, right? So let's just ask it about anythingllm.com and you can see that it's working the exact way that you expect many of your other models to work. And that was insane. That was that was so fast. And there was no streaming there, that just popped up. And so you can see that it went to the website, you know, obviously it scraped the content of AnythingLLM, told us all about the core features, told us about the desktop application, self-hosted, and also the cloud hosted version we offer. So it has everything there. This works just the way that you expect these things to work.
Demonstrating Agent Flows with NIM
And so let's go and now run something maybe a little more specialized, right? So here we actually have Agent Flows. Now Agent Flows are really new. You may actually not even know about them yet. They are something that we haven't made a video about. But an Agent Flow is actually, I let me just show you, it's so awesome. So we're going to edit a flow and you'll see that the LLM that's running on our GPU right now is going to interpret these instructions as if you wrote it with code. So what this does is this actually goes to the website to scrape, gets all the text content. But also we can call our own APIs. We can make the LLM do stuff with the content to pre or post-process it. We can scrape the web, as we're already doing here, but let's have it run a custom tool now instead of just running the regular web scraping skill. So in order to do that, we just turn it on. And so here I'm just going to ask it if it can check the NVIDIA GTC ticket prices. And there we go, it ran our custom tool this time instead of just scraping the regular website. And yeah, that's basically it.
Switching to and Running DeepSeek
As you can see, running a NIM on your RTX GPU is just a vastly different experience than running LLMs locally in the way that you're probably familiar with running them. This is just another level. If you have the resources, you should be running local models like this. I really can't emphasize that enough. Speaking of which, let me show you a reasoning model, since this is just regular Llama.
So in order to do this, all we need to do is first stop the NIM because obviously we want to free those resources so we can boot up DeepSeek. And then we just want to start DeepSeek. So it's going to ask us the same question, like, "Hey, give me a second." And then we can just click on this, click Save, and now we are running DeepSeek. It really is that simple. So now let's run some prompts with it and see what the thinking output looks like. We do show that in AnythingLLM so you can always see the reasoning for models.
Also, I did want to highlight that we're running the DeepSeek Llama 8B distillation. I have to be sure to say that, because people will get confused. You'll see that it actually utilizes a lot less memory. So if you are on a lower VRAM RTX GPU card, this model actually may be better for you. You can go and look at the build.nvidia site I showed earlier in the video and you really just want to kind of play around with the best model for your use case and your resources and really just what you can afford to play around with. No matter what, the experience will be way better than running the model through any other local provider. So this is definitely recommended to at least try. Let's do a reasoning question now, like something very standard and just see what DeepSeek says to us. So we can say something like, can you explain the Tower of Hanoi problem? And you'll see we're getting reasoning output. This is again running fully locally on our RTX GPU and this thing is absolutely insanely fast. I mean I'm getting 144 tokens a second here. And you can see that we've gotten very sufficient, complex reasoning, and we've also gotten a regular output answer. We could go all day with these reasoning models. People have very different use cases for them, so I don't want to get too in the weeds because this video is actually about running a NIM on your RTX GPU, not showing you the infinite possibilities that you probably are already aware of.
Conclusion: The Power of Local AI Unleashed
And again, I do want to emphasize that while we are running these models fully locally on device, none of the tooling here is unavailable that you're already familiar with in AnythingLLM. You still have the ability to upload documents, you still have the ability to go to data connectors, pull in this information. You still have the ability to scrape websites, chat, talk with PDFs, all of the things that you love to do in AnythingLLM, you can still do, but faster, and that's awesome.
And so that, everyone, is just a small taste of the power that you unlock when you use NVIDIA NIM plus AnythingLLM on your RTX GPU. And as you can see, we did basically a ton of work, you probably aren't even familiar with how much work we just did, because all we did was click some buttons and then we had a full parameter model running on our RTX GPU. The amount of effort that that would have taken to do with our own Python script or even just a copy-paste script from Hugging Face themselves wouldn't have been that easy. So this is a really, really awesome tool and a great integration to add into AnythingLLM because at the end of the day, at AnythingLLM, we believe the future of AI is local and you should be able to run those models as fast as possible on hardware that you already own. And that is what we are doing today with this integration.
I again want to thank NVIDIA for this partnership and the ability for us to be one of the launch partners. Many of you who watch this channel already know that we've been working on this for a long time and I just generally am excited about any time we can add an unlock to the everyday person's hardware to run AI locally. And so I really also am thankful for you who are watching the video because y'all support us, y'all support AnythingLLM, and I'm forever grateful for that. So hopefully you get value out of this and running a NIM on your RTX GPU. I really, really do know that you're going to enjoy that experience. And you can download AnythingLLM today from anythingllm.com and start doing this. All you need is just a basically a Windows machine with an NVIDIA RTX GPU in it, and then our app, and that's it, and the whole thing's done for you. So let me know what your experience is, let me know if you have any comments or questions. I'm happy to answer them and I look forward to hearing from every one of you.
