Introduction to a Free, Local AI Stack
Hey there. My name is Timothy Carat, founder of Implex Labs and creator of AnythingLLM. And today, I actually want to show you possibly the easiest way to get a very extremely capable, locally running, fully RAG-like talk to anything with any LLM application running on honestly your laptop, a desktop. If you have something with a GPU, this will be a way better experience. If all you have is a CPU, this is still possible. And we're going to use two tools, both of which are a single-click installable application. And one of them is LM Studio and the other is of course AnythingLLM desktop.
Right now I'm on lmstudio.ai. They have three different operating systems they support. We're going to use the Windows one today because that's the machine that I have a GPU for. And I'll show you how to set it up, how the chat normally works, and then how to connect it to AnythingLLM to really unlock a lot of its capabilities.
If you aren't familiar with AnythingLLM, AnythingLLM is an all-in-one chat with anything desktop application. It's fully private. It can connect to pretty much anything and you get a whole lot for actually free. AnythingLLM is also fully open source, so if you are capable of programming or have an integration you want to add, you can actually do it here and we're happy to accept contributions. So what we're going to do now is we're going to switch over to my Windows machine and I'm going to show you how to use LM Studio with AnythingLLM and walking through both of the products so that you can really get honestly like the most comprehensive LLM experience and pay nothing for it. Okay, so here we are on my Windows desktop.
Installation and First Steps
And of course the first thing we're going to want to do is click LM Studio for Windows. This is version 0.2.16. Whatever version you might be on, things may change a little bit, but in general this tutorial should be accurate. You're going to want to go to useanything.com, go to download AnythingLLM for desktop, and select your appropriate operating system. Once you have these two programs installed, you are actually 50% done with the entire process. That's how quick this was. Let me get LM Studio installed and running and we'll show you what that looks like.
Exploring Models in LM Studio
So you've probably installed LM Studio by now. You click the icon on your desktop and you usually get dropped on this screen. I don't work for LM Studio so I'm just going to show you kind of some of the capabilities that are relevant to this integration and really unlocking any LLM you use. They kind of land you on this exploring page. And this exploring page is great. It shows you basically some of the more popular models that exist, like Google's Gemma just dropped and it's already live. That's really awesome. If you go down here into if you click on the bottom, you'll see I've actually already downloaded some models, because this takes time. Downloading the models will probably take you the longest time out of this entire operation. I went ahead and downloaded the Mistral 7B Instruct, the Q4, means 4-bit quantized model. Now, I'm using a Q4 model. Honestly, Q4 is kind of the lowest end you should really go for. Q5 is really, really great. Q8 if you want to... if you actually go and look up any model on LM Studio, like for example let's look up Mistral.
GPU Offloading and Model Selection
As you can see, there's a whole bunch of models here for Mistral. There's a whole bunch of different types. These are all coming from the Hugging Face repository, and there's a whole bunch of different types that you can find here published by a bunch of different people. You can see that you know how many times this one has been downloaded. This is a very popular model. And once you click on it, you'll likely get some options. Now, LM Studio will tell you if the model is compatible with your GPU or your system. This is pretty accurate. I've found that sometimes it doesn't quite work. One thing you'll be interested in is full GPU offloading, exactly what it sounds like, using the GPU as much as you can. You'll get way faster tokens, something honestly on the speed level of a ChatGPT if you're working with a small enough model or have a big enough graphics card. I have 12 gigs of VRAM available, and you can see there's all these Q4 models. Again, you probably want to stick with the Q5 models at least for the best experience versus size. As you can see the Q8 is quite hefty, 7.7 gigs, which even if you have fast internet won't matter because it takes forever to download something from Hugging Face. If you want to get working on this in the day, you might want to start the download now. For the sake of this video, I've already downloaded a model.
Testing the Basic Chat in LM Studio
So now that we have a model downloaded, we're going to want to try to chat with it. LM Studio actually comes with a chat client inside of it. It's very, very simplistic though and it's really just for experimenting with models. We're going to want to go to this chat bubble icon and you can see that we have a thread already started. And I'm going to want to pick the one model that I have available, and you'll see this loading bar continue. There are some system prompts that you can pre-set for the model. I have GPU offloading enabled and I've set it to Max already and as you can see I have Nvidia Cuda already going. There are some tools, there are some other things that you can mess with but in general that's really all you need to do. So let's test the chat And let's just say, 'Hello, how are you?' and you get the pretty standard response from any AI model. And you even get some really cool metrics down here like time to first token was 1.21 seconds. I mean really really kind of cool showing the GPU layers that are there. However you really can't get much out of this right here. If you wanted to add a document you'd have to copy paste it into the entire user prompt. There's really just a lot more that can be done here to leverage the power of this local LLM that I have running even though it's a quite small one.
Starting the LM Studio Inference Server
So to really kind of express how powerful these models can be for your own local use, we're going to use AnythingLLM. Now, I've already downloaded AnythingLLM. Let me show you how to get that running and how to get LM Studio to work with AnythingLLM. Just booted up AnythingLLM after installing it, and you'll usually land on a screen like this. Let's get started. We already know who we're looking for here: LM Studio. And you'll see it asks for two pieces of information: a token context window, which is a property of your model that you'd already be familiar with, and then the LM Studio base URL. If we open up LM Studio and go to this local server tab on the side, this is a really, really cool part of LM Studio. This doesn't work with multimodel support, so once you have a model selected, that's the model that you are going to be using. So here we're going to select the exact same model, but we're going to start a server to run completions against this model. So the way that we do that is we can configure the server port. Usually it's 1234, but you can change it to whatever you want. You probably want to turn off CORS, allow request queuing so you can keep sending requests over and over and they don't just fail. You want to enable logging and prompt formatting. These are all just kind of debugging tools. On the right side, you are going to still want to make sure that you have GPU offloading allowed, if that is appropriate. But other than that, you just click Start Server, and you'll see that we get some logs saved here.
Connecting AnythingLLM to the Local Server
Now to connect the LM Studio inference server to AnythingLLM, you just want to copy this string right here up to the V1 part, and then you're going to want to open AnythingLLM, paste that into here. I know that my model's max token window is 4096. I'll click next. Embedding preference, we don't really even need one. We can just use the AnythingLLM built-in embedder, which is free and private, same for the vector database. All of this is going to be running on machines that I own. And then of course, we can skip the survey. And let's make our first workspace and we'll just call it AnythingLLM. We don't have any documents or anything like that. So if we were to send a chat asking the model about AnythingLLM, we'll either get a refusal response or it will just make something up.
Demonstrating RAG with AnythingLLM
So let's ask, 'What is AnythingLLM?' And if you go to LM Studio during any part, you can actually see that we sent the requests to the model and it is now streaming the response. First token has been generated, continuing to stream. When AnythingLLM does receive that first token stream, this is when we will start to show it on our side. And you can see that we get a response. It just kind of pops up instantly, which was very quick, but it is totally wrong. And it is wrong because we actually don't have any context to give the model on what AnythingLLM actually is. Now we can augment the LLM's ability to know about our private documents by clicking and adding them here or I can just go and scrape a website. So I'm going to go and scrape the useanythingllm.com homepage because that should give us enough information. And you'll see that we've scraped the page. So now it's time to embed it and we'll just run that embedding.
Achieving Accurate, Cited Responses
And now our LLM should be smarter. So let's ask the same question again, but this time knowing that it has information that could be useful. And now you can see that we've again just been given a response that says AnythingLLM is an AI business intelligence tool to form human-like text messages based on a prompt. It offers LLM support as well as a variety of enterprise models. This is definitely much more accurate, but we also tell you where this information came from. And you can see that it cited the useanythingllm.com website. This is what the actual chunks that were used to formulate this response. And so now actually we have a very coherent machine. We can embed and modify, create different threads. We can do a whole bunch of stuff from within AnythingLLM, but the core piece of infrastructure, the LLM itself, we have running on LM Studio on a machine that we own. So now we have a fully private end-to-end kind of system for chatting with documents privately using the latest and greatest models that are open source and available on Hugging Face.
Conclusion and Final Recommendations
So hopefully this tutorial for how to integrate LM Studio and AnythingLLM desktop was helpful for you and unlocks probably a whole bunch of potential for your local LLM usage. Tools like LM Studio, Ollama, and Local AI make running a local LLM no longer a very technical task. And you can see that with tools that provide an interface like LM Studio, pair that with another more powerful tool built for chatting exclusively like AnythingLLM on your desktop, and now you can have this entire experience and not have to pay OpenAI 20 bucks a month. And again, I do want to iterate that the model that you use will determine ultimately your experience with chatting. Now, there are more capable models, there are more niche models for programming. So be careful and know about the model that you're choosing, or just choose some of the ones that are more popular like Llama 2 or Mistral and you'll honestly be great. Hopefully LM Studio plus AnythingLLM desktop just become a core part of your local LLM stack and we're happy to be a part of it and hear your feedback. We'll put the links in the description and have fun.
