Introduction to Retrieval-Augmented Generation (RAG)
Hi, today we're going to explore LangChain and LlamaIndex, especially how to perform retrieval-augmented generation with both frameworks. And we'll cover the basics and, of course, then the differences of these frameworks.
If you don't know what RAG is, here are the basics. When you want an LLM to generate answers on data that it was not trained on, you can directly pass the data to the LLM in the prompt. But you probably don't want to pass your complete data set to the LLM, since first, it's a lot of unrelated data for the answer, so-called noise. And second, many models are not able to handle this amount of data. This is the so-called context window limitation.
This is why you want to store your data in a database, query the required data, and pass this small piece of data to the LLM. So this is the exact workflow: you first have multiple data sets, let's say text files, JSON files, or websites, and you load that data into memory. Then you don't put the complete data set into the vector store, but you first split the data in so-called chunks, or in LlamaIndex it's called nodes. But you make smaller pieces of that very large document.
Then you create embeddings. Embeddings are vector representations of your text, of the semantic meaning of the text, and then you can compare the similarity later of that vectors. And you store the vectors inside that vector store. So that's the indexation step.
After the indexing step, you can make retrieval. So you've got a question and you embed that question. You now compare your question to the vectors inside the vector store, and you retrieve the most similar documents from that vector store. You take that small amount of documents, pass it to a prompt, and then pass the complete prompt and the documents to the LLM, which will generate a final answer from that combination. So that's the generation step in retrieval-augmented generation.
Project Setup and Data Loading
Okay, so this is how it works in theory. Let's now have a look at how this works with LlamaIndex and LangChain in practice. Okay, I'm in VS Code and here on the left, you can see we've got two files: the LangChain iPython notebook and the LlamaIndex iPython notebook. So these perform similar tasks, but yet different in syntax and a little bit in the approach, of course.
Then we've got this data directory which contains a text.txt file, and this is a text of a fictional restaurant where we've got this Q&A format. So what makes your pizza unique? And then you've got the answer. So this is what we want to answer with our RAG system, this is the data which was not trained into the model.
So before we can start, let's first install the packages. So I've created this requirements.txt file, so you can run pip install -r requirements.txt file. This will install LangChain and also LlamaIndex into our virtual environment. And after that, we use OpenAI. So we're going to load our OpenAI environment variable. So you have to put it inside here, your OpenAI API key, and we will use it here in the LlamaIndex and LangChain iPython notebook.
Okay, so the first step is to load the data inside our memory. So we've got this approach from LangChain. So we've got a DirectoryLoader, and we've got multiple other data loaders like text loaders, PDF loaders, and so on, which are specialized to perform a loading task. And I use the DirectoryLoader, pass in the data path, and we only filter for .txt files. So this will be loaded inside memory. And after performing that, let's have a look at the documents variable. And as you can see, we've got a list, and this contains an instance of a Document class, which has got this page_content attribute. And also if we scroll to the right, we can see that this also has got a meta... this is a very long text, I will not scroll. Ah, here it is. Okay, so sorry, it has got this page_content attribute and this metadata attribute. In this metadata attribute, we've got source, and this is the path to the file where we loaded the data from. So this will automatically be stored inside this metadata object. And we can access the page content here. We access the first object of our list and access the page_content attribute. So this is how you load data in LangChain.
In LlamaIndex, it works pretty similar. You've got also a DirectoryLoader where you pass in the name of the directory and directly can pass in the load_data method. So this looks pretty similar. And yeah, we also got a list with a Document class, but this document class contains a little bit more information. So every document has got a special ID. We don't have any embeddings stored inside there. And we also got this metadata, but inside the metadata attribute, we've got the file path, the file name, file type, size, creation date, last modified, and so on. So this default contains a lot more information. So the drawback of this is that it makes, of course, the object a little bit larger since you store more information in memory. So that's a drawback, but you get more information.
Text Splitting: Chunks vs. Nodes
Now we want to create multiple smaller documents, so-called chunks, from our larger document. Let's start with LangChain. Here, we've got multiple text splitters, which are in the text_splitter module of LangChain. And we choose the CharacterTextSplitter, which we instantiate where we pass in a chunk size, chunk overlap, and some other stuff.
So let's create that. And after creating our text splitter, we use the split_documents method where we pass in the list of documents, which only contains a single document. So now we've got chunks. And as you can see, again, we've got this Document instance, multiple Document instances now, with page_content. And also here the Q&A and so on. So here we can see that we've got the metadata, and it is the same like before. So we've got metadata and the source to the original text file.
In LlamaIndex, it's a little bit different. Here, you also import a text splitter, instantiate it like with the same attributes like in LangChain, but then you create so-called nodes. So this is a different class than the original Document class. So let's have a look at the nodes class. Again, the nodes class contains more information. So we've got two nodes here. So important here, you can see this is a different class than this class. This is a Document class, and this is a TextNode class. In LangChain, this is not the case. We've got again documents, but not a TextNode. Also, so there is a different class.
We can see that this node means that it always contains data which was already splitted via a text splitter. And what's very nice, as you can see, this contains a lot of information, and it also contains the relationship. I think this is a very nice information which can be used in some advanced retrieval topics. So we've got a relationship, and here we've got the related node information. So we've got this node ID, and we can see that this node ID is the same node ID like here. So we know that these nodes originally belonged together. I think that's some valuable information when you do advanced retrieval.
Indexing and Retrieval Comparison
Okay, let's go to the next step, which is indexing. So to create an index, we need an embedding function, and we also need a vector database. So in our case, we will use Chroma as our vector database and OpenAI embeddings to create our embeddings from our chunks. So we use the from_documents method from the Chroma class and pass in the chunks. So this is the data we want to embed and the embedding function. So this will create an index. And now we've got our vector store.
We can also turn the vector store into a retriever by using the as_retriever method. So this provides a standard interface for retrieving data from the vector store. So with the retriever, you don't put data in the vector store. This is the vector store which is responsible for that. To retrieve data, you use a standardized retriever. This retriever has got a method called get_relevant_documents, and here you pass in your question. This will be embedded, and then you make a cosine similarity search against the other documents inside the vector store, and per default, you get the four most relevant documents back from the vector store. So this is how you make retrieval in LangChain. So let's do that now in LlamaIndex.
So again, we have to import a vector store. So we use the ChromaVectorStore again, but in this case, it's a little bit more complex to set this up. So we need a client, and we also need to create a collection. So we can call it however we want. I would just call it like this. And then we use the ChromaVectorStore class and pass in the Chroma collection. So this will create our vector store. But that's not enough. In LlamaIndex, you have to create a StorageContext. And you use the StorageContext class and use the class method from_defaults where you pass in the vector store. So now we've got a storage context, and we can now use another class called VectorStoreIndex, which has got a class method from_documents. So it looks very similar to LangChain. We pass in our documents, we pass in the storage context, and we pass in the embedding model, and this will create our final index now.
As you can see here, I passed the documents but not the nodes. To pass nodes, so the chunks, we have to make it a little bit different. We just instantiate it like this, where we pass in the nodes as a named argument, pass in the storage context, and also pass in the embedding model. So in my opinion, this is the preferred way to create this index. And again, like in LangChain, we can run the as_retriever method to convert our index into a retriever, and now we can run the retrieve method. "How long does it take to prepare a pizza?" And this will now return the most similar nodes. So we only created two nodes, we only get back the two most relevant nodes from our vector store or our retriever.
Generating Responses with LangChain Expression Language (LCEL)
Okay, the next step in this process is to make an LLM call with the retrieved documents. So we need to create a chain to do so. So let's first go to LangChain. And in LangChain, there exists two methods to do so. So there is a chain interface, which is the legacy interface in LangChain, and there is the new LangChain Expression Language. This is the preferred way to create a chain. So we will use the preferred way, the LangChain Expression Language way, to create a chain.
So the first step is to create a template. This is just a string with variables. So this is a normal RAG template where we pass in the context. The context are the documents which were retrieved from the vector store. And then we've got the question, which also has to be part of the template. After that, we instantiate our prompt with the from_template method where we pass in the string. This will create an instance of the ChatPromptTemplate. We also pass in a chat model, which will be GPT-3.5 Turbo. And this is how we create our chain.
So in the LangChain Expression Language, you can use the pipe operator to pipe output from the first step of our chain to the second, and then to the third, and then pipe it to the output parser. So this is actually achieved with operator overloading. So there is a lot of complexity going on inside this little pipes, but as a user of LangChain, you don't really have to care about the implementation of that. And you just have to use it. That's why you use a framework.
And what's going on here in this chain is that we pass in the question with this item getter, pass it to the retriever. So we make the query to our retriever, so this actually runs get_relevant_documents. So we store the documents inside the dictionary; so this is the key and this is the value. After that, we do the same for the question, but for the question we don't do any processing, we only take the question as it is. And the output will be passed to the prompt, which contains here the context variable and the question variable. So after that, we pass the complete prompt to the model and pass the output of the model to the output parser. So this is our chain which we want to execute or invoke.
And now we've got this invoke method where we pass in a dictionary. So we only pass in a single argument, here you can see this is only question, and the question is, "How long does it take to prepare a pizza?" So this is our complete chain, and all of the information will be retrieved from our vector store. So on average, it takes 15 to 20 minutes. And if we have a look at the text, we should find 15 to 20 minutes. Here we can see this is the correct answer which was retrieved from our vector store. So this is how we do that in LangChain.
Generating Responses and Customizing Prompts in LlamaIndex
Now let's have a look at how this works in LlamaIndex. So again, we've got our retriever, but in LlamaIndex, you do that a little bit different. As you can see, we instantiate an LLM like the ChatOpenAI class in LangChain, but here it's only the OpenAI class. And we use the as_query_engine method, and we pass in the LLM. So this adds a little bit more abstraction on top of our chain. So the LangChain Expression Language is a little bit more low-level.
Another alternative is to use the Settings object. So this is a singleton which, I think, is a very nice solution to handle ground truth values inside your application. So you can always know if you use Settings, then the LLM which will be used, for example, like this. So we don't have to explicitly pass it. This will always be the attribute of that settings object. I think that's very nice if your application grows bigger. But in this case, we will just comment that out. So we will instantiate OpenAI, convert the index to a query engine, pass a single argument, and then query the query engine, "How long does it take to prepare a pizza?" So let's run that. And you can see we get the response: on average, it takes about 20- 15 to 20 minutes. So this is the correct answer.
It works the same way like here, but in LangChain, you have to know a little bit more what's actually going on by constructing that all on your own. Here, you've got this single method to do that. So what's the difference here? So in this example, we created a prompt on our own. In LlamaIndex, we did not have to do that. So what if we want to have a custom prompt? To do that, we can use the query engine and run the get_prompts method on that. And here you can see this is our prompt. This is of type ResponseSynthesizer, and we can access it like this. So we can use this syntax here and override our prompt by using that key.
But the first step is actually to create a new prompt. So we use the PromptTemplate class. And what we're going to do here is a little bit different than before because we want to see that we actually override our prompt. So we want that the bot always says, "Hello my friend" at the beginning of an answer. Then we've got again the context and also the query. And now we create our new prompt template. So this looks very similar to LangChain, but now we want to update our prompt of the query engine with our custom prompt. So this is the key we can access, and we run the update_prompts method on that. We pass in the key and here we pass in our new prompt. So this will now update the prompt. And if we have a look at our prompt again, we can see that this is now of prompt type custom. So let's run the query engine again, and now we should see a difference in the behavior. "Hello my friend. On average, it takes about 15 to 20 minutes."
Yeah, so that's it. This is how you can perform retrieval with LlamaIndex.
Key Takeaways and Final Comparison
So the key takeaways for me are that in my opinion, both frameworks work pretty similar and are straightforward to use. One key difference is that the default use of LangChain is more low-level and higher-level in LlamaIndex. LangChain had a high-level chain interface which was deprecated because many people were actually fighting the framework to customize prompts, for example. LlamaIndex also provides a lower-level API if you want more control. In my opinion, LlamaIndex seems to be a little bit easier to learn since LangChain is the bigger framework. But if you learn one, you can easily switch from one framework to the other.
Okay, that's it. Thanks for watching. See you, bye-bye.
