Introduction to AnythingLLM
Hey everybody, my name is Timothy Carat, founder of Mintplex Labs, creator of AnythingLLM among some other things. And what I am going to show you today is basically what we've built that allows you to run a ChatGPT equivalent. It's not just a chatbot, it's literally equivalent to ChatGPT, and you can use whatever you want. You can use Mistral, Llama, OpenAI, Azure, Anthropic if you want to. And then when it comes to vector store or retrieval, which was just actually added by default into ChatGPT, we've been supporting it, but we also allow private vector storage. You can use Pinecone, Chroma, Weaviate, Qdrant, whatever you want.
You also get multi-user authentication right out of the box, so you can have multiple people using the same instance. It is ChatGPT equivalent plus a bunch of extra stuff, and in the future, it'll be doing so much, much more. But I had to turn around and make this video today because we just added local LLM support. So now you can run any LLM you want inside of AnythingLLM.
Using LM Studio for Local, Uncensored LLMs
So first, let's just say you can get a hosted version of AnythingLLM at use.anythingllm.com, but we deal with open source, and so we are open source. You can find us on GitHub, I'll link it in the description. You can spin this up in Docker, just pretty much a single command, and then be off to the races. The part that we're going to focus on today is actually using a local LLM model. I'm going to run Mistral because I want to run some uncensored prompts.
The way to, easiest way to do that, and what we support currently (we will be supporting more) is using an app called LM Studio. This is an awesome app. If you have a Mac laptop and you have an M1 or M2 chip you can use it. I run this on my Windows machine, so we're actually going to cut to how I do this on my Windows machine soon, and then eventually they'll have Discord support. But using this tool, every single model on Hugging Face becomes available to you for chatting. So whenever the latest and greatest model comes out on Twitter, it's already on Hugging Face, so you can use it in AnythingLLM to start chatting.
And as I said, today we're going to focus on Mistral's AI, the Mistral 7B. And the cool thing about Mistral is stuff that would get you outright banned on GPT is totally fine for Mistral. So if you want to use it to help you with stuff that is either extremely personal or private and you don't want OpenAI to see, or maybe just stuff that might be unsavory in the eyes of OpenAI, you can use this kind of model plus our interface and pretty much get ChatGPT for you.
AnythingLLM Onboarding Process
So right here, I have AnythingLLM running on Localhost. I can run this in Docker or you can use our cloud instance, it all works the exact same. When you load up this program for the first time, you're going to be brought to an onboarding screen and the very first question is what LLM do you want to use? Before we finish this step, I'm going to go get LM Studio running on my Windows machine, so join me over there.
Configuring the LM Studio Inference Server
Now I've already installed LM Studio and when you do, you'll open up and you'll get a screen that looks like this. You will see some recommended models. I actually went and already pre-installed the Mistral 7B Instruct model and actually downloaded the quantization model that is smaller but faster. Now that being said though, if I wanted to install Code Llama because I know that most of my chats will be coding related, I could do that here. Also pretty much every single model that you can think of, like Vicuna, I think it is. Vicuna, yeah, Vicuna is available, Diablo. And if you explore this UI some more, you'll see that you can search for more models.
There is actually a chat inside of here, although it is quite limited. All you can do is send chats here, there's really nothing more that you can do. And then you have this. This is where all of the power of LM Studio comes from, and also how you can make it work with AnythingLLM. What I'm going to do here is I'm going to do a local inference server. This comes pre-installed with LM Studio, all you need to do is just pretty much click start server. What this is going to do is make a Localhost endpoint available for you that then we can send requests to, and it's in OpenAI format if you want to do a custom integration. What I'm going to do is just click Start Server.
Exposing the Local Server with Ngrok
And now we have a server running the Mistral Instruct 7B model at Localhost Port 1234. However, on my MacBook back outside of this room, I need to be able to reach that. So how do I do that?
I actually have ngrok installed already. And so what I'm going to do for the resources and purposes of this video is temporarily create a HTTP tunnel from this port to where I can reach it from an outside network. Now, I don't recommend you do this in real life and for long terms because it's not necessarily secure, but for the purposes of this video, I just wanted to show you that this is possible. If I wanted to, I could run LM Studio on this same machine as AnythingLLM. And so what this has done is I am now tunneling, and this URL if I were to visit in a browser now shows a unexpected method GET. That is okay, that's actually expected. So what I'm going to do now is we're going to go back to my MacBook and we're going to put this URL into AnythingLLM.
Connecting AnythingLLM to LM Studio
Okay, so now we are back on my MacBook that's running AnythingLLM and I am just going to take that URL that we got from my Windows machine, which is just how my setup is. You can run both of these things on the same machine, that's just because of my limitations on hardware. But we're going to select LM Studio and for our base URL, we're going to put in our URL and then we're going to add a /v1. The Mistral model max token window size is 4,096 tokens, just like GPT-3.5.
We can now click continue. The language learning model that we chose only does text completion, so how do we do embeddings, which is how we do document retrieval, similarity search, and all of that? Right now we have OpenAI and Azure supported for embedding models. We are going to add open-source support here, it's coming soon, it's actually the last thing to do, but that's okay, that's coming soon. So what I'm going to do is put my OpenAI API key in here and just go to the next step.
Now we get to choose our vector database. If you're going to be embedding files, you need a vector database. We have every single option you could need. The easiest one is actually LanceDB. It's zero setup, runs on your AnythingLLM instance and your data will never leave it, so everything happens on instance. So we're going to do that for fully private LLM and chatting and embeddings.
Finalizing Setup and Data Privacy
Now here you can add a custom logo if you want to, if you want to personalize your instance, we're going to skip. Next, we get to choose how we're going to use this. You can change these at any time, but I know it's just going to be me on this instance, so a simple one password is all I need. If you want to have a team or multi-users where each user has their own thread in a workspace and everybody has their own account, you can do team. I'm going to do just me. I don't need a password because we're running this locally.
The next step is actually an alert that we do that lets you know how your data is being handled depending on whatever you selected. So for our LLM selection, we're using LM Studio, so the model and chats are only accessible on the server running LM Studio. If you use a closed or a isolated vector database like LanceDB, our text and vectors are also only accessible by this AnythingLLM instance. So we can continue. Now we need to create a workspace. A workspace is similar to a thread on ChatGPT. We're going to name this one Mistral because that's what the model name we're using is.
Your First Private Chat with Mistral
So this is what it looks like when you've already set up your instance and you come back to it and all of that. So the first thing we're going to do is click on our workspace. Now we can just send a chat. There is no limit to prompt windows on AnythingLLM, we actually will always figure out how to get your prompt dependent on the size and the window of your LLM. We'll get it through and all your information will make it to your LLM. So what we're going to do is, I am going to, let's just send a chat. Let's just say, 'Hello, what model are you?'
And it responds that it is Mistral 7B v0.1, a large language model trained by Mistral AI, which is incredible because that's accurate, that's 100% correct. We can send now any chat, we can chat with Mistral as if it was just any other LLM or that we were using ChatGPT, but all of our data is private. And since I haven't even used any embeddings or done any embeddings, I actually haven't even sent any information outside of the servers that I maintain to make this work.
Document Retrieval (RAG) with a Custom Model
Now just to go over some other features, there is a chat mode and a query mode if you're working with documents. There is all of the settings where you can define, for example, the prompt if you want to have a very specific system prompt. Maybe you wanted to respond as a pirate all the time, maybe you always want JSON format or CSVs. That's where you can put these kind of instructions. And then you can have the name, the temperature, and also some other configurable settings that just make your life a little bit easier.
Now I have already gone and uploaded two text documents. One is a very short one called anythingllm.txt and it's just a short description on what is AnythingLLM. So let's try and do a RAG, or retrieval augmented generation, run using a custom document on Mistral. So I'm just going to click my file, move it over, click save and embed. Now this didn't cost me anything because I've actually already embedded this file before because it had this cache key, so you don't have to pay again. So that's nice because you can then embed the same file in multiple workspaces and only pay once.
Now let's send Mistral a chat that it probably doesn't know anything about, and it'll use our document to get information and form a better answer. So let's ask it, 'What is AnythingLLM?'
And it responds, 'AnythingLLM is the best way to take tons of notes, PDFs, and other source materials and turn them into a chatbot without privacy concerns.' And it says that we even do citations, and we do. You can see the document, click on it, and even see the exact chunk of text that was given to the model to help it form its answer.
Behind the Scenes: Server Logs and Streaming
Now I do want to be clear that we don't have streaming mode enabled yet, it is coming. So responses, especially if you're using a custom LLM or running it on a server elsewhere, can sometimes take longer than what you would expect because the full answer has to be calculated before we can send it back to you. But that problem is going to be fixed probably by the time you install AnythingLLM.
I did just want to take a moment actually to show what an output looks like coming from LM Studio. And you can actually see I passed in a message at this point where you can see the history and it says 'Hello, what model are you?' and it said 'I am Mistral'. And then I said 'What is AnythingLLM?' You can actually see every single step that the LLM takes to formulate your answer. When streaming is enabled, you won't even need to look at this, but I did just want to show you 'cause I do think it actually is an incredible thing to watch in the server logs as it happens. Anyway, I'm going to turn off the server now and we can go back to the video.
API, Community, and Conclusion
And also just to be clear, there's more to do within AnythingLLM. We also offer a fully featured API so that you can do custom integrations so that AnythingLLM can just be the middleware of whatever it is that you're building. And this comes complete with full developer documentation. Hopefully this gives you a small glimpse into the crazy amount of power that AnythingLLM gives you to use as backend or middleware for whatever it is you're building: a totally private chatbot looking at a custom model that maybe you have even built, and also just allows you to have full control over a ChatGPT experience without literally any privacy concerns at all.
Every month AnythingLLM gets better and better and it's because of contributors in our open source project and our core team that make this all possible. If you need to reach out to us to talk about how you can use AnythingLLM or just questions about how it all works in general, you can visit this Discord down here or go to contact Mintplex Labs. You can comment on this video and I will respond to you and we will get in touch and we can talk about your needs. But either way, this tool's open source, it's built for you, and we want to see what you build with it. Thank you.
