Introduction to LlamaIndex
Today I'll show you how to implement a chat with your document system in just four lines of code. I recently started exploring LlamaIndex, which is an alternative to LangChain. Just like LangChain, LlamaIndex gives us the ability to build powerful applications based on large language models. These applications include document Q&A, data augmented chat bots, knowledge agents. The great part is you can connect different types of data sources including structured, unstructured, and semi-structured data sources.
So using LlamaIndex, you can quickly build LLM-based applications just like LangChain. The main reason for me of exploring LlamaIndex was the ability to fine-tune embedding models to improve the performance of document Q&A systems using large language models. So basically that means you can not only fine-tune the large language model on your data set, but you can also fine tune the embedding model, and that will improve the performance of your retrieval system. I'll cover this topic in a future video.
In this first video, I'll show you how to build a chat with your document system using LlamaIndex.
The RAG System Architecture
You probably have seen this diagram before. So there are different components. The first component is basically loading your documents. Then you want to divide your documents into smaller chunks with a predefined chunk size. So each of the chunks, you compute different embeddings. These embeddings are numerical representations of the text contained in a chunk. Then you create a semantic index. So basically this is your vector store.
After that, you can actually use this vector store to chat with your documents. Now in order to chat with your document, the system takes your question, computes the same embedding as the embedding it computed for the chunks, then performs a semantic search on your knowledge base. So as a result of this search, it will return a certain number of chunks that are relevant to your question. So let's say by default, LlamaIndex returns only two chunks out of all the chunks available in the knowledge base. And these return chunks will be used as a context to the LLM of your choosing.
So the LLM gets the question from the user along with the context that was returned by the embedding model during the semantic search, and the LLM will generate a response, and that is going to be shown to the user. As you can see, this whole process can be divided into four different steps, and I'll show you how to implement this whole process using just four lines of codes within LlamaIndex.
Initial Setup and Loading Documents
Okay, so let's start looking at the code. So I'm using Google Colab in order to run the notebook and the code. First we're installing all the required packages. So we are installing LlamaIndex, then OpenAI. So for the initial example, I will show you how to implement the document Q&A using OpenAI LLM and OpenAI embeddings, but I'll also show you how to replace the OpenAI LLM with an open source LLM from Hugging Face. And for that, we will need Transformers package to access the Hugging Face-based LLMs and the Accelerate package in order to accelerate the processing or speed of running local LLMs.
Now for the initial example, we will need the OpenAI API key in order to access OpenAI's embeddings and LLM. So you'll need to provide those. Okay, next we need to import the required packages. So we are importing OpenAI from LlamaIndex, then we are inputting two auxiliary functions or objects, one is VectorStoreIndex and the other one is a SimpleDirectoryReader. The use will become clearer when we look at the code. And then we are also importing some packages just to properly format the output from the model.
Now in order to chat with your documents, you will first need access to those documents. So the way I'm going to do it is I will create a folder called data and within this folder, I will upload this file. Now this specific essay is titled "What I Worked On" and this is basically an essay written by Paul Graham when he was working with Y Combinator. So we're going to be using this as an example data set, but you can upload any type of documents, for example, text, Word documents or PDFs.
Creating the Vector Index and Query Engine
Now going back to the architecture, first we will need to load the documents. Now LlamaIndex provides a very simple functionality to do that. So there's this class called SimpleDirectoryReader. You simply need to provide the name of the folder or path of the folder. So in this case, our folder is called data and that's why I provided the name data. Then we need to call the load_data function on top of that, and that will load our documents. Now the great thing about this is you can have different type of documents within the folder and this class will define which loader to use with those documents. And we can also look at it. This is basically the text file and the whole text file is loaded in here.
Now the next step in our architecture is to create a vector store. This involves dividing your documents into chunks, computing their embeddings, and then storing both the embeddings as well as the chunks in a vector store. Again, LlamaIndex makes this very easy. We're going to use the VectorStoreIndex class. Then we are going to be creating the vector store from documents. We'll pass on our documents in here and this will create a vector store index for us. Okay, so we just created our index.
Now next we are going to look at how to customize different options, for example, the chunk size, the type of embeddings, and everything. But before that, we just want to ask questions. So for that, we need to create a query engine. Now I find this to be very streamlined in LlamaIndex compared to LangChain. So here, basically you can create a query engine on the index by using the as_query_engine. Now if you want to chat with your documents and want to have memory, so you were going to use a function called as_chatbot and that will basically enable memory, but as a query engine, you don't have the memory component in here. Now in order to use this query engine to get a response based on the user question, so it has to go through this process that it needs to compute embeddings for the question, then do the semantic search, and use the returned chunks as context with the LLM along with the question to get to generate an answer. All of this is implemented using a single line of code. But basically what you do is you use the query engine, then call this function called query, and you pass on your question to this function. For example, the question that we are asking is, "What did the author do growing up?" and we will get a response in here.
Now the response that you get has a lot of different components in here, but we're simply interested in the answer. I'll show you what all this text means in a bit, but first let me show you how to get the answer from the LLM. For that, we will use the display function from IPython and put the response in markdown and also bold it. So here's the answer from the model: The author worked on writing and programming outside of the school. Before college, they wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. Right, so this is the answer that was generated by the LLM using the context provided by the embedding model. As you can see, apart from the imports, we were able to build a document Q&A system using just four lines of codes, and this implementation is very clean.
Persisting and Inspecting the Vector Store
Next, I want to show you how to customize different parts of this diagram for example, how do you define the chunk size, chunk overlap, or how do you change the LLM and so on and so forth. But before that, let's look at some very important concepts when it comes to the vector stores. So right now the index that we created is just in memory, but you can persist this by calling this function persist on the index that you created. Now the way it's going to work is if we run this, so by default it's going to create a folder called storage and within this you will see multiple JSON files. Now this functionality is critical because it enables you to create the index once, store it on your disk, and then use it in your future runs. Right.
Now in order to load a vector store that was stored on the disk, we're going to be using the StorageContext along with the load_index_from_storage class. Now the way it works is that you will use a storage context to read the content of the index and then from the storage context that you just created, you are going to recreate the index, right. And then you can use it exactly the way we did it before. Now I hope this is clear. Another question that comes in mind is what exactly is in this vector store?
Now in order to show you, I'm going to open all these four files and the two important files that we want to look at is the docstore and the vectorstore. So the vector store as you can see here is basically the embeddings that we computed for each of the chunk and depending on the embedding model that you choose, you are going to get different number of embeddings per chunk. Now similar to the vector store for embeddings, there is a doc store. So here you will see different chunks from the document. So basically this is text that is split into different chunks. And the third most important thing is index store. Basically this has the hash or address for different chunks, and using this index store it determines which embeddings in here belongs to which chunk.
Now when you create a vector store, it's going to store both the embeddings that it computed as well as the original chunks from the documents. Now during the retrieval process the LLM is going to have access to both the embeddings as well as the chunks that are retrieved based on the embedding model. I hope this is clear.
Customizing LLMs and Chunk Size
Now let's look at some of the customizations that you can make. The first customization that we are going to be looking at is how to change the LLM. Now in order to change the default values, we are going to be using the concept of ServiceContext. So this defines all the default values of different parameters within LlamaIndex. Now by default, I think the LLM LlamaIndex uses is the davinci-003 model. However, if you want to change it to, let's say, gpt-3.5-turbo, so we basically use the OpenAI function within LlamaIndex, create an LLM based on that. So you can define the temperature, the max number of tokens that is going to generate, right, and then pass that LLM to this ServiceContext to change the defaults. So here we're changing the default LLM to GPT-3.5 and then pass that LLM to the VectorStoreIndex that we just created. And then you can use the vector store index exactly the way we used above.
Now if you want to use another LLM such as, let's say, Google PaLM, so again you can simply import the PaLM LLM from the LlamaIndex LLM class, then pass that to the LLM within the ServiceContext. That will basically change the default value. And then again you can pass it exactly like here to the VectorStoreIndex and then it will start using the PaLM LLM in this case. Again, you will need to set the API key for PaLM, but the process is going to be very similar to what we did for OpenAI model.
Now another parameter which is very important when it comes to these chatbots for your documents is the chunk size. So in order to change the chunk size, we can follow the same pattern. So here you can simply pass on and define two other parameters. So the first is the chunk size, the second one is chunk overlap. So here I'm using a thousand tokens for example with an overlap of 20 tokens. So I would recommend everybody to look at the documentations and see what the default values are and how you can change them in here.
Now another way of doing this is you can set the global ServiceContext. So in that case you don't have to pass it to the vector store index that you create. You simply set the global service context using the service context that you just created and then you can use those as defaults in the rest of your code.
Using Open Source Hugging Face LLMs
Now in this last example, I'll show you how to use an open-source LLM from Hugging Face. And the way you do it is we're importing the HuggingFaceLLM class from LlamaIndex LLMs. Then you can also import PromptTemplate from the prompts class. This is very similar to what you probably have seen in LangChain, right? So here is basically the system prompt because some of the open source LLMs need a system prompt. But this system prompt is specifically for the StableLM models. Enclose the user query within some special tokens, right?
Now in order to use the LLM, it's basically using llama-cpp in the background. So we create an LLM based on different parameters. So for example, the context window, you can pass this on, maximum new tokens that you want to generate, right, and other parameters such as the temperature that you want to set, you want to do sampling or not, right? Then you need to pass the model name as well as the tokenizer name, the system prompt if the model supports one, then the prompt template with the user query, right? If you have multiple GPUs and you want to use all of them, you can set the device map to auto. Now in some certain cases, there are certain specific tokens that are going to be used as stopping IDs. You can pass on those depending on the model that you're using because different models are going to be using different tokenizers a right? So these are different parameters you can set. Essentially this will create an LLM class.
Now you can simply pass that LLM class to the service context that we created, right, along with the chunk size that we want to use. This will set the default LLM to this Hugging Face LLM and then you can do a query on your vector store and this LLM is going to be used for generating the responses.
Conclusion and Future Topics
Now this was a quick overview of how to build a document Q&A system using LlamaIndex. Now LlamaIndex is extremely powerful and it has some really cool features. For example, one of the applications that I'm currently looking at is how to fine-tune embedding models along with fine-tuning the LLMs. So this is going to be the first video in the series on LlamaIndex. I will be creating more advanced tutorials on the different topics and different features that LlamaIndex have. If you like what you see on the channel and would like to support it, check out my Patreon.
Now if you want to build a Q&A system and want to use LangChain instead of LlamaIndex, I would recommend you to check out this video. As always, if you found the content helpful, consider liking the video and subscribe to the channel. Thanks for watching and see you in the next one.
