Introduction to AnythingLLM
Hey there, my name is Timothy Karamat and the tool that I'm going to show you today, AnythingLLM, is going to change the way that you set up and chat with any kind of document using an LLM.
This could be PDFs, it could be Word docs, it could be anything. AnythingLLM is the easiest way to unlock that ability without having to worry about technical overhead, having a huge, you know, amount of RAM on your machine. This tool can run passively in the background of your computer and you won't even notice it's there.
With that being said, let's talk about how it works. Just a quick overview. This is a tool that is open source, it's MIT license. You can do whatever you want with it, mess around with it, doesn't matter. This tool primarily functions off of two commercially available, very easy to get, and pretty much nothing costs—nothing. You can just use the OpenAI API. It will cost you very little to run this tool and also a free pinecone.io for the vectorizing in the vector database. These two things are just hosted on the cloud. You don't have to worry about them, they don't take up any space, and this tool manages it all for you. You just put in the API Keys, it's super simple.
Now, I do just want to take a second to mention you've probably heard of these things like Pinecone, Vector DBs, Private GPT, Local GPT—all of these things I'm sure you've heard of and those are also all like very good reference points for you to have stepping into this project. I just want to highlight some other projects and what the difference between AnythingLLM is and some of these other more well-known projects.
So Private GPT is great. It works in a very similar fashion however it is only a command line tool, so you don't really get a nice user interface like AnythingLLM. That being said, it also requires you to run a local LLM on your machine which if you're like me, you probably don't have a super powerful machine. Local GPT which was inspired by Private GPT also requires you to run a local language model, which of course requires lots of RAM. It does run on your CPU but it does require a lot of overhead to run, you're not really running this back in the background of your computer. And then of course, GPT for all. Of course, all the projects that I just talked about all have one similarity where they all run on a private LLM on your machine. And that's great, but I wanted to build a tool that is more accessible and also frankly more powerful by just using the tools that we already know how to use. Also, it can run in the background, that's kind of my main benefit here. And that is why I wanted to showcase today AnythingLLM, which will be on GitHub. It's MIT licensed and the link to it will be in the description of this video.
Prerequisites and The Collector Utility
Now if you are a GitHub user, you probably only need these three things and you probably already have them installed. You need access to a terminal—if you're on Mac that's already there—you need Python at least 3.8 if you're going to be doing any kind of data collection which I'll showcase in a little bit, and then there's just yarn or npm and node.js. Those are the only three things you need to run this.
First, let's start with what I'm looking at here is the collector folder. The collector folder is just utilities, it's scripts, it just makes your life easy. So let's show what collector does.
So here we are in the collector folder of the project. I've pulled it in from GitHub and I have installed the proper dependencies. Just you know, this is all in the Readme of this folder. Now there are two main things that the collector utility can do, and keep in mind this is a utility, it's just supposed to help you collect data. So the first script is just called, as you would expect, main. Can do a couple things right out the box. It is by far the easiest way to collect information. You see that we can collect a YouTube channel, an entire YouTube channel, it will grab the transcriptions for you automatically. Substack, give it the author of a substack and it will go and find and collect all of their publicly readable articles. This won't work for subscription-based articles. Medium, same case with mediums; find a publisher, put in their link, and it will go and collect all of their recent public documents. Article or blog links: just come across a random article or a random blog that you just want to pull in, give it a link here and it will automatically pull that data in. And then of course GitBooks; for those of you who are looking to automate or simplify documentation, this will go and scrape an entire GitBook regardless of whatever it is, and it will vectorize and get ready for that information to be processed so you can just chat with it.
That being said, I did promise that you can do this with local files, so let me show you how that's done. The second script is called watch, and the way that it works is there is a directory in this repository that when you run this script—it is called hotdir, hot directory—any file you drop in here will automatically get detected and will also get processed and converted into essentially it being ready to be consumed by a language learning model like ChatGPT or something like that. And the UI, which I'm about to show you next, will actually handle all of that for you.
Running the Server, UI, and Initial Setup
So the next step is, well how do I boot up the actual app? This is the thing that can run in the background of your computer and you don't really have to give it a second thought. So the first thing is starting up the server. That's just yarn dev:server. Now a couple things to know about this. The database for this is actually using sqlite, so it's stored locally on your machine. This isn't talking to a third-party database and you have complete control over this. And you can see that it opens up a port on 5000. This is a Node app, it's using express.js if you cared to wonder.
The next bit is the front end. So before we show you the front end, let me also just show you some screenshots just to kind of warm things up. So this is what the home screen looks like. It just looks like a simple chat interface. And from here you can go and find and locate documents that you have collected using the main.py collector script or the watch.py collector script. And then of course, you can chat because of course what's the point of all of this if you can't chat?
And then also a way to set up and make sure that you have installed all the proper environment variables. There's a file in the repo again, there is a Readme, tells you exactly how to set this up. You only need two keys: an OpenAI key and a Pinecone DB key. That's it. Everything else is just your specifications. If right here I'm using GPT 3 and a half turbo in this image, if you have access to GPT-4, GPT-4 8k, whatever you want doesn't matter throw it in there.
And then of course I have a free Pinecone index running already. All I did was sign up, I don't even think I had to put down a credit card and I was able to make an index and I just called it socials-to-chatbot, that was the name I was working with at the time. And if you look I have no vectors right now. Keep in mind AnythingLLM will manage this for you. You just need to get the key, it will do everything else. That's how simple it is.
So now let's get started and let's boot up the front end. You just run yarn dev:frontend and here we are, we have our UI. And of course if you'd like to open an issue on GitHub because maybe something isn't supported or you encountered a bug, feel free.
Workspaces and Efficient Document Caching
The first thing that I want to drive home is that AnythingLLM works by essentially containerizing documents. So the way that this works is a workspace has access to documents, but multiple workspaces can share the same documents but not talk to each other. And I'll just showcase that because this will make more sense when we're actually doing it.
Let's create a workspace, I'm just gonna call this one Workspace 1. You can call this anything you like. So here we are, we're given Workspace 1, we of course have no chat history. So we're going to want to open up the settings icon where we are then greeted with all of the documents I have collected. I have some YouTube channels, I have some Substack, some mediums, and some custom documents.
Custom documents were from the watch.py script that I mentioned earlier in the video. There are a couple documents in here, but there's one here called Certificate of Incorporation. This is actually a legal document that when you create a business you submit to the state that you're registering in. There's a lot of important information on this, like how many shares are in your company.
You'll notice there's a lightning bolt symbol next to it. And that is because when you use AnythingLLM, we actually will efficiently cache all of the information that you vectorize, which in fancy words, we save you a ton of money. And the reason that we do that is so if you have one massive document, let's say it's an entire textbook because you're trying to pass Econ 480 or whatever, you can vectorize the entire textbook, but then have multiple workspaces share that textbook but not vectorize it each time. The reason that you want to do that is because imagine a book is 5,000 pages, there's a lot of words in those pages, you have to pay to embed each word essentially. So this will save you a lot of money. So once you embed it once, it's embedded and you're good to go. Of course, you can manage this cache and delete it if you have to update a document or anything like that.
Live Demo: Chatting with a Local Document
So let's just do something simple and incorporate and vectorize my Articles of Incorporation. But to show you that it works, let's first ask it a piece of information that's on this document that ChatGPT by itself, by default probably has no idea about. So here I asked a question: What are the total amount of shares of Ramp Labs Inc? That's what the document for the Articles of Incorporation talks about. And it obviously responds, what the hell are you talking about? I have no idea what Ramp Labs Inc is.
Oh, okay. Well, let's change that. So here I just click on the file itself. We know it's already cached, so this won't cost me anything, and even if it did, it'd be less than a penny. This is a 16 page document and now the document has been embedded. We're back in the chatting. Let's just copy paste the same exact question. It should now be able to reference that document and find the information. And as you can see, there are 15 million authorized shares, 14 million common, a million of preferred. And that's interesting to know, but what's more interesting is the citation of this. Now because this is a local document, we can't link it because of browser security problems, but if this was a URL, like let's say it was a research paper that you grabbed from a URL or a Medium article or a YouTube video, clicking this citation would actually open the source of that. So it doesn't work for local documents, but it does work for public.
Of course, this is a work in progress and the video you're watching right now is v0.0.1, so there's a lot more work to do but this is by far the simplest way to just talk to your documents. You can run it privately and it's just using off-the-shelf stuff. You don't have to worry about running an LLM, it's just, it's so easy. Let's just do the easy thing.
Conclusion and Call for Contributors
So I hope you enjoyed this demo of AnythingLLM. And obviously, looking for contributors, would love for you to contribute. Of course, just make an issue and we can talk it out. There's a Discord Channel you can join just to chat. It's just me in there, it's just one channel, so you know, let's chat. Hope you liked it and have fun!
