Made withAnythingLLMClip

How to Implement RAG locally using LM Studio and AnythingLLM

F

Fahd Mirza

Creator on the AnythingLLM channel

Watch on YouTube ↗

22K viewsMay 29, 202410:15 runtime

Learn how to implement a fully local Retrieval-Augmented Generation (RAG) pipeline from scratch. This guide walks you through using LM Studio to download and serve a large language model like Llama 3 on your own machine. You'll then connect it to AnythingLLM, a user-friendly…

Chapters

#anythingllm#clip#aillm

Local RAG with LM Studio & AnythingLLM

This document outlines the key concepts and steps to build a completely local Retrieval-Augmented Generation (RAG) system. By combining LM Studio for model serving and AnythingLLM for the user interface, you can create a private chatbot that answers questions based on your own documents.

Core Concepts

Retrieval-Augmented Generation (RAG): A technique that enhances a Large Language Model's (LLM) responses by first retrieving relevant information from a private knowledge base (like your documents) and then using that context to generate a more accurate and informed answer.

Tools Overview

LM Studio: An application that simplifies downloading, managing, and running LLMs locally. It can also create a local server endpoint, making the model accessible to other applications like AnythingLLM.
AnythingLLM: A desktop application that provides an end-to-end RAG solution. It manages document ingestion, embedding, and provides a chat interface to interact with your LLM and your data.

Step-by-Step Implementation Guide

Follow these steps to set up your local RAG pipeline:

1. Serve a Model with LM Studio

Download a Model: Search for and download a GGUF-format model in LM Studio (e.g., a Llama 3 variant).
Start the Server: Navigate to the 'Local Server' tab, load your downloaded model, and click 'Start Server'.

Important
Your local model will be served at http://localhost:1234 by default. This URL is required to connect AnythingLLM.

2. Install and Run AnythingLLM

Download: Get the appropriate desktop version (e.g., the .AppImage file for Linux) from the AnythingLLM website.
Make it Executable (Linux): In the terminal, run chmod u+x ./AnythingLLM-Desktop.AppImage.
Run: Execute the application with ./AnythingLLM-Desktop.AppImage.

3. Connect AnythingLLM to LM Studio

In the AnythingLLM setup wizard, select LM Studio as the LLM provider.
The base URL should auto-populate. AnythingLLM will then detect the model being served by LM Studio.
Choose your preferred Embedding Model (the default is usually fine for local use).
Select LanceDB as the vector database for a fully local setup.

4. Create a Workspace and Add Documents

Create a new workspace and give it a name (e.g., "My RAG").
Inside the workspace, use the upload feature to add your own files (PDFs, text files, etc.).
Click 'Save and embed'. This processes your document, splits it into chunks, converts them into numerical vectors (embeddings), and stores them in the LanceDB vector database.

5. Chat with Your Data

Once embedding is complete, you can start a chat in the workspace.
Ask questions related to the content of your uploaded document. The system will now use the information from your file to answer, demonstrating that the RAG pipeline is working.

Key Benefits

Privacy: All components—the model, the data, and the applications—run locally on your machine. No data is sent to external services.
Offline Capability: Once set up, the entire system can operate without an internet connection.
Ease of Use: The combination of LM Studio and AnythingLLM provides a user-friendly way to implement a powerful RAG system without extensive coding.

Introduction to Local RAG

Hello guys. In this video, I will be showing you how you can implement Retrieval Augmented Generation, or RAG, locally by using LM Studio and AnythingLLM.

LM Studio is probably one of the easiest ways to run your large language models locally in a simple and easy way. AnythingLLM is another technology that enables you to use these LLMs with even more ease, and you can create a chatbot very, very easily. I have covered both of them separately in various videos, so just search AnythingLLM or LM Studio in the channel and you should be able to find a lot of videos.

But for the purpose of this video, I'm going to simply use both of these tools to first download a model and then implement RAG by using both of them.

Before I proceed, I wanted to give a huge shout out to Mast Compute who are sponsoring this virtual machine (VM) where I'm running Ubuntu 22.04 on an A6000. If you are looking to rent a GPU, then I will drop the link in the video's description with a discount coupon, which if you use, you will be getting a 50% discount. And I would highly recommend them because it is not only performant but very, very flexible.

Okay, so let's go back to our RAG thing which we are going to implement.

What is Retrieval-Augmented Generation (RAG)?

And if you don't know what RAG is, RAG stands for Retrieval-Augmented Generation. It is a technique used in artificial intelligence to improve the ability of models to generate text or other content. It involves an AI model retrieving a set of relevant documents or pieces of text from a large dataset and then using this information to generate new text that is more accurate and informative.

This approach is particularly useful for tasks such as answering questions, summarizing text, and generating long-form content, as it allows the AI model to draw on a large body of existing knowledge and generate more comprehensive and accurate responses. In simple terms, RAG means like having a super smart research assistant that can quickly find the most relevant information and use it to generate high-quality content.

And by the way, this is not the only way of implementing RAG, but this is one of the ways because a lot of people either use LM Studio or AnythingLLM. If you combine both of them, great things happen very, very easily.

Serving a Model with LM Studio

Okay, so that said, let's first go to my LM Studio. This is my LM Studio. Just download any model of your choice. I'm just going to download this Llama 3 for this purpose. So I have just typed Llama 3 in LM Studio, pressed enter, and there are heaps of cont levels available. I believe I already have downloaded this one. If you click on this folder icon, you will see I have gone with this Quant Factory Dolphin 2.9 Llama 3 8 billion GGUF.

Once you have that, you can either load it to chat or just click on this local server icon on the left-hand side. From there, you can simply click on 'Start Model'... sorry, 'Start Server'. It says select the model to load, so just click on the middle and from here just load this Dolphin 2.9. It won't take too long, and that is done. And you see that the server is also started from here. Make sure that you have got this server name, which is http://localhost, running on Port 1234. Okay, so that is done now, and now we know that our server is running, model is being served here.

Installing the AnythingLLM Desktop App on Linux

Now let's go to our AnythingLLM. And by the way, if you don't want to watch the other video of installation of LM Studio, all you need to do for installation is just click here, download LM studio for Linux, Windows, and stuff, and then follow the prompts, and you should be able to run it.

Similarly for AnythingLLM, you can just simply click on 'Download Anything for LLM'. If you want to go with 'Download Anything for Linux'. So I'm installing it on Linux, as you can see it is appearing here at the top. That is done, and I'm just going to show it in the folder for you only, if it would show me. Let me open the folder here. There you go. So this is the file which it has downloaded, and it is in the downloads folder.

So let me open the terminal. This is my LM Studio terminal. and so I have, I'm going to open another terminal here, you can see. So in one terminal, my LM Studio is running, and here I'm just going to go to my downloads directory. If I do ls, you will see that there is AnythingLLM desktop. So I'm just going to go and do u+x just to make the permissions executable on this Anything. That is done. And now let's run it. And in order to run it, just do ./ and it is going to run it. So let's wait for it, shouldn't take too long. It is loading the workspaces. So let's wait for it to load.

Configuring AnythingLLM to Connect to LM Studio

And then you see that it is loaded, and then next, just click on 'Get started'. From this list, just scroll down, select LM Studio. You see LM Studio selected, and it has picked up its LM Studio base URL, which is this. Chat model selection, click here, it is waiting for the URL, which is correct. So now go to your LM Studio and you can either type this one... I'm just going to go and type this one here. Or you can grab it from your LM Studio. So as soon as you will type this URL, it has automatically populated here, Quant Factory, the one which we downloaded.

So let's click on 'Next'. It says four... so yeah, just go with... I think it is up to you that how much context window you want. Normally I just take the default 4096. This is how much spotlight it is going to use. Now in the next one, I normally just go with this one, embedding preference, which is good. Vector database, LanceDB. This is all local, so I'm just going to go with this one.

Now we can skip the survey for now, and you can just call it 'My RAG'. Click 'Next', and you see your RAG is implemented.

Creating a Workspace and Embedding Documents

Now what has happened here is that you have very easily implemented this, and now it is even telling you what you need to do here. How good is that? So what we have done is we have used LM Studio to load our model, and we have put it in our AnythingLLM to create a RAG. Now in order to use your own stuff, just click on this 'My RAG'.

You see that now we are in our own workspace. If you want to upload your own document through which you want to chat, just click on this upward arrow. And from here, just upload the file. I'm just going to go with maybe Downloads. From here, I have just my PDF file, and this contains all of my own personal information. Or you can even fetch your website. So I could give my own website, which is... I could simply go with that, but I'm not going to go there. Just click here on my PDF, 'Move to workspace', and now it is in my workspace. Click on 'Save and embed'.

What 'Save and embed' does is it converts your PDF file or whatever data is there into a numerical representation, which is called as embedding, and it puts it into the vector database. In our case, it will be LanceDB. Now that is done. That is awesome.

Testing the RAG Pipeline by Chatting with Data

And now you just click here, and then you can chat with it. So you can just ask it anything, like, 'Who is Fahd Mirza?' You can even upload the document from here, by the way. So if I click here, you will see my document is already there. So I'm just going to ask it, 'Who is Fahd Mirza?' Send, enter here. There you go. Because otherwise, that model, Dolphin, doesn't know who I am. You see? How good is that!

You can upload any document here. You can use anything. And this is, once you have downloaded this stuff, you can disconnect the internet, and it will be fully remote. Also, you can serve it to other people. You can just install it on other servers. You just point it to LM Studio. You can put it in your whole network within your own company. How good is that?

So that's it, guys. This is how easy it is to use both LM Studio and download Anything to get a RAG pipeline or RAG system or RAG application.

Conclusion and Support

If you still are having issues, let me know, happy to help out. If you like the content, please consider subscribing to the channel. And if you want to support the channel, you know, there is a 'Buy me a coffee' link, or you could also use my affiliate link to rent a GPU from them. So that's it. Have a good day, and I'll see you in the next video. Thank you.