Introduction and Project Setup
Hello guys. So in this video we're going to be coding out the first part of the RAG pipeline called the injection pipeline. Right? So in the previous video we looked at a bunch of theory regarding RAGs. In this video we are going to be coding it out. So first thing is let us quickly go ahead and set up our development environment. So what I've done is I've just created a folder called rag for beginners in my computer. Opened it using my Visual Studio Code. So the first thing that I'm going to do is I'm going to create a file called injection_pipeline.py. Okay. So this is where we are going to write out all the code where we are going to load all the source documents and then we are going to chunk it up. Okay. If you remember we chunked it up, right? And then we are going to embed each of those chunks and then we are going to store it in the vector database. That is going to be the part one of the RAG system. Right?
Create and Activate Virtual Environment, Install Dependencies
And for this file, we need a couple of dependencies. Uh, and I want to go with virtual environment this time. But if you're going to be working with larger projects, then you ideally want to go with poetry. So I'm going to go ahead and open the terminal. And then I'm going to say Python 3. Okay. So I'm using a Mac, so I'm going to say Python 3. But if you've got a Windows machine, then you can just say Python. And I want to use a particular module inside of Python. And the module that I want to use is the venv module. And I want to create a folder right here to save all of my dependencies. And I want to name that folder as venv as well. So if I go ahead and press enter, you should be able to see the venv folder created right here. Now the next step is we have to activate this particular venv. Meaning we want the terminal to point to this venv. Right now it is pointing to global. We want it to point to venv. So in Mac the command is going to be source. Okay. I'm going to go inside of venv and I'm going to go inside of bin and inside of bin we have the activate file. I'm going to press enter. So right now you should be able to see that the terminal is now pointing to the virtual environment right here. For Windows the command is a little different but it is well documented on the internet. you can go look it up. So the first thing that I'm going to do is I'm going to install a few dependencies. Uh pretty basic dependencies. We are going to be using LangChain classes because they just make life a lot easier when working with RAGs and LLMs. So I'm going to go ahead and install the langchain package and then langchain community and we're also going to install the langchain text splitter because if you remember we have to do chunking right. So this particular package allows for splitting text. So langchain text splitters and then we need langchain openai because we need to use the embedding model of OpenAI and then we need langchain chroma and this chroma is going to be the vector database. If you remember we need some database where we should be able to store all of the vectors right. So we're going to be using the Chroma vector store and then finally we also need python-dotenv. So let's go ahead and press enter. Okay. So it is taking a couple of seconds and perfect, it is done.
Imports and Main Function Skeleton
And now I'm going to go ahead and paste a few imports. And let's go through it line by line. So the first thing is from the langchain community document loaders module. We are importing two different classes. Both these classes are going to help us read text files or ppts or docx files from a particular directory. And the next thing is we need the CharacterTextSplitter because if you remember once we do load all the source documents we need to chunk it up right. So that is why we need this particular class. And next thing is we need the embedding model right. So we need this embedding model to convert the chunks to vector embeddings. So we are going to be making use of LangChain's OpenAI wrapper. So after the embedding process right here we have all of the vector embeddings. Now we have to store it somewhere. So we need a vector database. So the vector database that we are going to be using today is the Chroma vector database. So the main reason why I go with ChromaDB is because we can host it locally. It makes things a whole lot easier. And then finally we are loading all of the environment variables. Uh so if at all we in the future create the environment file and we have some API keys right here which is something that we will do in the case of OpenAI's API we are going to have this available inside of this particular file right here. All right. So the next thing is I'm going to go ahead and define the main function like this. Okay. So I'm just going to at this moment I'm just going to print main function. And to test this out I'm just going to come here and then run this Python file. And as you can see we have the main function printed right here. Perfect. So it is inside of this particular main function where we are going to write sub functions which are going to take care of every single step in this diagram that we saw right. So the first thing is we will have to load all the files and then we have to chunk the files and then we have to embed and store it in the vector database. Okay.
Preparing and Inspecting Source Documents
So first off let's actually go ahead and prepare our source documents. So as a first step I'm going to go ahead and copy paste a folder called docs. Okay. So this is going to be our source documents. So what I've essentially done is I have just gone to Wikipedia and then I have downloaded the entire contents of five different companies. So we have companies about we have Google, Microsoft, Nvidia, SpaceX and Tesla. So if I go inside of here, it would basically resemble the same text as the Wikipedia page. So you can see these are huge text documents, right? Okay, it's you can see it's actually pretty huge. And the same thing for Microsoft, Tesla, right? So these are pretty huge text files and this is what we're going to be working with in this tutorial. So let's as a first step let's go ahead and load the files and uh don't worry regarding the code I'll be providing you the repo link in the comments below. Okay so let's look at the first one. The first one is we will have to load all the source documents into our file right here so that we can go ahead and chunk it. Right. So to do this, I'm going to go ahead and paste the load_files method right here.
Loading Files with DirectoryLoader
All right. So let me go ahead and make this a little smaller so that you can see everything. Okay. And then we can come down here and then we can invoke this particular this thing. So we can call it docs because that is going to be the name of the folder right here. Okay. So let's look at what this load_documents is doing. So basically this is going to load all the text files from the docs directory. Uh we're first checking if the docs directory exists even. Okay. So if that does not exist right here then we're just throwing an error saying that the directory does not exist. So now we have to go ahead and load all the text files from the docs directory and for this I'm going to use the DirectoryLoader class from LangChain. So the first thing is we will have to provide the docs path right here. So we've pointed it to this particular docs folder and then we are also saying that the glob is going to be *.txt. So what this means is that we're basically telling the class okay only look for txt files. Okay, if there are any ppt files or docx files or pdf files don't worry about all of that only focus on *.txt. We can always extend it later. And then as a third keyword parameter we have the loader. Okay, so since we're dealing with text files, we're going to be using the LangChain TextLoader class. But if you want to load PDFs or CSV files, we've got separate classes for that. Okay, so we've got PDFLoader, we've got WebsiteLoader, we've got a bunch of other loaders. But since we are working with text files, this is what we going to be using. And now the loader has been configured. Now we can go ahead and invoke the load method. And that is going to give you a list of LangChain documents. This is important. Okay, so this is going to have a list of five different LangChain documents and the LangChain documents are going to look something like this. So let me actually go ahead and paste an example of what it looks like. All right, so this is how it looks like. So this is going to be a list of five different LangChain documents and each LangChain document is going to have the page_content attribute and the metadata attribute. So since we are dealing with the first let's say google.txt could be the first file. So this first item is going to have the entire contents of the google.txt inside of this page_content. So this is going to have the entire content of the entire text file. And then we are also going to have the metadata which is going to be auto populated. So inside of this we have the source. Uh this docs is something that we have specified and it's just going to take the name of the txt and the same thing for Microsoft and Nvidia and SpaceX. All right. So this is what we are going to get right here. So we're just saying okay if the documents are if the length of the documents is zero in that case okay no txt files were found but in our case when we run it we are actually going to see five different elements in the documents list all right and uh and then we're just going to return it and I'm just going to print something so that we can actually see uh everything in action okay just to make sure that everything is working fine so let's go ahead and run this Python file.
Inspecting Loaded Documents in Terminal
And perfect. Let's expand this. Uh the document one is this is going to be the name of the source file. We've got content length. This is a lot of characters. Okay. It chose to do the Tesla first, right? And we can also see the content preview. Okay. It starts with Tesla Inc. And uh it how to pronounce Tesla. So if we go inside of this particular file, you can see this is exactly what we have. Okay. So we have the entire content right here. I've just truncated it because we don't have the space to show everything in the terminal. So, uh yeah, we also have source as well. And perfect, that is it for the first step. So, let's come down. So, the first loading the files is done. Let us now go back to our diagram and let's see what is next. So, now that we've loaded all of the documents, now the next step is we have to chunk it.
Chunking Documents with CharacterTextSplitter
So, let's actually go ahead. I'm going to go ahead and paste the chunking method right here and paste it over here. Okay. And I'm also going to invoke it down here like this. Okay. So, we are passing in all the LangChain documents that we have. And we're passing it inside of this method right here called split_documents. And if you remember this output of the split_documents, I mean the chunking method is going to be a list of smaller chunks. So that is exactly what we get right here. So this method you can see it splits documents into smaller chunks with overlap. It takes in the chunk size. Okay, in this case we are going to set the default value to 800 characters. Okay, we're not dealing with tokens right now. We're dealing with 800 characters. The chunk overlap right now is zero. We'll get to what that means in a minute. So this is the major main method that we are going to be using this class CharacterTextSplitter. Okay, this is the most basic text splitting class that exists in LangChain. Okay. So later on there are more advanced splitting methods we look at. But right now we're just like learning to crawl, we learning to walk, right? So this is what we're going to be starting with. So now we are going to say okay go ahead and split all of the documents. And now we have all these chunks right here. And that's it. Okay, we're just going to return it and in the middle I'm going to print something so that we can actually see how it has been chunked and you know how many chunks are there. So I can go ahead and run this file by saying python3 injection_pipeline.py. Okay. So also note that in the prints I've limited it to the first five chunks right here. Okay. So we should be able to see the first five chunks.
Embed Chunks and Persist in ChromaDB
So right here you can actually see the it started with tesla.txt and then let's actually go to the file right here and let's see where the chunk starts and ends. So Tesla Inc. And if I do this. Oh, sorry. Um let's go all the way up again. And all right. So it ends at the first chunk ends at related products and services. Related products and services, right? Let's look at the second chunk where it starts. Tesla was incorporated in July 2003. Okay, it starts here and then it ends at Tesla was A. Okay, so it ends at Tesla A. And the third chunk is going to start probably here. Okay. So, Gigafactory taxes and it ends at company type public. So, it ends right here. Okay. So, that is exactly what we have. Okay. So, we have five different chunks. But actually, if you come all the way down, you can see we've got 792 more chunks. So, that's a lot of different chunks that we have right here. And at the end of the method, we're also returning the chunks as well. Okay. So, we have all of the chunks ready right here. The next step, the third step is to send all of these chunks through the embedding model and convert it into the vector embedding and also store it in the vector DB. Okay, so that is exactly what we are going to be implementing. So let me come up here and we are going to be pasting the third method right here. You can see it looks pretty simple. I'm just going to invoke this right here. Pass in all of the chunks and then we are just going to have the vector store right here because it returns the vector store. So let's go ahead and see what is inside of this method. So here we are passing in all of the chunks and then also the location where we want the vector database to be created locally. Okay. So we are going to create and persist the Chroma vector store. So the first thing that we're doing is we are going to initialize the embedding model. So in the previous video we saw that there are a couple of different OpenAI embedding models, right? So we saw text-embedding-3-small which had like a default dimension size, text-embedding-3-large as well. So that was 3,072 right? So in this case we are going to be going with the small model. Okay. So I hope that makes sense. And now we are going to go ahead and create the vector store. Okay. So this particular method is going to do two things right. It is going to take all of the chunks. It's going to convert it into the embedding the vector embedding versions and then it is going to store it as well. So that is why it takes in all the chunks. Basically these are LangChain documents only but chunked smaller versions right so that is so if I come up here this is also going to return the LangChain documents except they are the page_content is going to be much more smaller okay it's only going to contain like you know 500 or thousand or whatever we set it so we are going to have the LangChain documents come in here in the form of chunks we are also specifying the embedding model to use for the vector DB and then we're also saying okay this is where you need to store the vector database. Okay, so locally we are saying okay store it right here and then finally we have collection metadata and right here we are specifying the algorithm to be cosine similarity. Okay, so this is something that we will circle back to in the fourth or the fifth video in this series but this is very important. Okay, so this essentially just talks about you know what is the algorithm that the system is going to use to compare all these different chunks and retrieve the top quality chunks. Okay, so we are going to be using something called cosine similarity. But for now, ignore this. All right, so once this is done, we're just printing saying that you know vector store has been created and we are going to return the vector store. Okay, so let's go ahead and run this file and hopefully Okay, so the reason why this is not working is the OpenAI API key environment key is not been added to the environment file. Okay, so um it's going to be very simple. Just go to platform.openai.com openai.com and if you go to settings and then click on API keys and then we can go ahead and create a key right here saying rag tutorial. Okay, I'm just going to copy it and come to the environment file. Okay, so this is the variable that we need and then I'm just going to copy it, put it over here. Okay, that's it. And one more thing, uh, if you have not paid for it, uh, it's just going to be costing you a minimum of $5. Okay? Just go to billing right here. Add that $5. If you're based out of India, it cost you some 400, 300 rupees. But, uh, I have to assure you that it would very well last you for at least like four or five or six months of learning. Okay. So, this is a worthwhile investment. Later on uh once you understand you know how things work then we can go for open source models and you know learn how to you know pull it from llama and run it locally but since we're just starting to learn it is a worthwhile investment all right perfect so let's try it one more time so before I run it I'm just going to go ahead and comment out all the prints so our terminal looks cleaner all right so let's run the file all right uh so it is taking some time right here so you can see it has started creating embeddings and storing in Chroma. You can also see a new folder has been created. Right now it is empty. All right guys, you can see the vector store has been successfully created and it has been stored in db/chromadb which is right here. Okay, you can see all the data has been populated. So if you come back to the diagram, we have completely done this entire flow, right? Right, we have chunked it and now we have embedded all of the chunks and then we've stored it in the vector database as well. Okay, so if at all you want this code, it's all going to be available in the comments. I'll provide the repo right there. And in the next video, we are going to be tackling this retrieval pipeline. So I'm pretty excited. I will see you next video.
