The Problem with In-Memory Vector Stores
The ability to upload files like PDFs and text files is one of the major strengths of using LangChain and Flowise. In the previous video, we had a look at creating a document chatbot using Flowise, and this allowed us to upload our file using the text document uploader. And we then broke that file up into chunks and we stored it in our in-memory Vector store.
If you are unfamiliar with Vector stores and document loaders, I recommend having a look at my previous video first and then coming back to this video. This solution is perfect for very simple use cases where we want to upload a text file ad hoc and then ask the AI model questions related to the content of that file. We can also easily replace the text file with a PDF document loader or anything else that we want.
However, there is a limitation with the solution. Whenever we interact with our application by asking it a question, this entire flow is executed, which means we will upload the file, we will then chunk it, store it into a vector store, and the model will then be able to access that information in the vector store. This might be ideal when we upload small files. In our example, this text file is only about 500 words, but if we wanted to upload large files or multiple files, the process of uploading this data, chunking it, and installing it in the database could take a very long time.
The Production Solution: Two Separate Chat Flows
Also, we do not want to execute this each and every time we upload our application. Ideally, we want to upload this data once, store it in a database, and then our application should simply read the data from the vector store without having to rewrite everything each and every time. So therefore, we will now have a look at the correct way of building production-ready document chatbots.
This process basically involves two steps or two chat flows. For the very first step, we will simply upload our documentation and then store it in a vector store. This process is called upserting or ingesting data into the database. We will then create a separate chat flow that is solely responsible for querying the database, and that will be the main chatbot that we'll be interacting with.
Using a Persistent Database: Pinecone
There is another limitation to our current solution. At the moment, we are storing our data in an in-memory Vector store. These in-memory Vector stores will get lost whenever we reset our server. This means that if I had to close our server, all the data stored in this in-memory Vector store will get lost, which is also not ideal in a production system. We want the data to be persisted somewhere in a proper database.
So in this video, we will also have a look at storing our data in a Pinecone database. So by the end of this video, we will have a chat flow that is responsible for uploading documents and then storing the content in a Pinecone database, and we will then create a second chat flow that we can use to chat to our data. So let's get started.
Setting Up Your Pinecone Index
The first thing we need to do is to create our account with Pinecone. So go over to pinecone.io and then click on sign up for free. After logging in, go ahead and create a new index. Give your index a name. I will call mine flowwise and in Dimensions enter 1536. This is the dimensions required for an openAI embedding. For pod type select S1. We will stick with the starter plan and then click on create index.
Our index is now initializing and this will be ready after a minute or two. And after about a minute we can see that this index is ready. When we click on this index we can see that at the moment there are zero vectors in this database. Vectors are basically the amount of records available in our database.
Building the Ingestion Flow in Flowise
Let's switch back to Flowise. We will now create the chat flow that will be responsible for uploading the documents and then storing these documents in our database. Let's click on Add New. Let's give our project a name by saving and let's call it Ingest Documents.
As with any chat flow, we need either an agent or a chain. For this, we'll go to chains and then I will use the Conversational Retrieval QA chain. We can now assign our language model by going to Chat Models and we can then select the Chat OpenAI model. Please note that if you want, you can also use the OpenAI model instead. At this point, it's really down to preference. We can now paste in our OpenAI API key and then connect this to our chain.
We now also need to add our Vector Store node. So we can click on nodes, we can go down to Vector Stores and we'll select the Pinecone Upsert Document node and we can now connect this to our chain. So in this case, we are now hooking our Pinecone instead of a local vector store. This Pinecone node requires a Pinecone API key, an environment value, as well as the index name. We called our index flowwise so we can go ahead and enter flowwise as the index name. Then for the API key, we need to go to Pinecone and then click on API keys. On this screen, we can click on this button to copy the key value, and we can then paste that into the node. Then for the environment, we can simply copy whatever value we stored here and then paste that into this field.
Adding Documents, Splitters, and Embeddings
Now that we have Pinecone set up in our project, we need to feed it the documents. So let's add a document loader by clicking on nodes. We can then go to document loader and then select one of these loaders from this list. I'll just keep it simple and select Text File, as I will be using the same file from the previous video. And I'll go ahead and connect this node to the document attribute on the Pinecone node.
And as with the previous video, we'll also add a text splitter. So we'll click on nodes, we'll go to text splitters, and then we'll select the Recursive Character Text Splitter node and we'll connect that as well. I'm going to change the chunk size to 200 to 200 characters with the chunk overlap of 20 characters. Finally, we also have to hook up an embedding node. So under nodes, we'll go to the embeddings and I'll add this OpenAI embedding node, and this also requires the OpenAI API key, which I'll paste in like so.
Executing the Ingestion and Verifying in Pinecone
So far, this is very similar to what we saw in the previous video. The only change is that instead of using the local vector store which we now know will get refreshed whenever we restart our application, the only difference now is that we are using a Pinecone vector store instead of the local vector store which we now know gets lost when our server restarts. I'm going to select my file and I'm going to save my project.
But what I also want to show you is what happens to our vector store when we run our ingestion chat flow. At the moment, we've got zero records in this database, but watch what happens when we chat to this application. And quite honestly it doesn't matter what we ask the AI at this point. All we want to do is run this chat flow so that our document is uploaded and then stored in the Pinecone database. So I'm just going to ask the AI to summarize the story and you will notice that this does take a bit of time, and that is because Flowise is first uploading this document, chunking it, running embeddings on each of the chunks, and installing those embeddings in the database. This process does take quite a bit of time, and this is ideally why we only want to do this once. But by asking this question, we now know that it's definitely summarized the file that we uploaded. And if we go back to Pinecone, we can now see that there are 20 records stored in the database now.
Building the Dedicated Query Flow
So now that we have a flow that we can run whenever we want to upload information to the database, we now want to create a separate flow that will simply interact with our database so that we can ask questions on it. Let's do that now.
Let's go back to the Flowise dashboard and let's create a new chat flow and let's call this Document QA. The name is up to you. As with any chat flow, we need either an agent or a chain. Again, I'll go with the Conversational Retrieval QA chain. Our chain needs an LLM, and for the LLM I'll use the Chat model. Again the choice of LLM is really up to you. I personally love you using these chat LLMs as I really like the behavior of these ChatGPT-like models. For the LLM we need to provide the OpenAI API key. We'll then connect this to our chain.
Now for the vector store, we'll do the following. In nodes, go to vector stores, and instead of selecting the Pinecone upsert document node, we'll go with this one instead. This Pinecone load existing index node. And we can go ahead and connect this to our chain. You will now see that this node looks slightly different to the absurd node. We no longer have to provide documents as input. We still need to provide an embeddings node, but this is simply so that our chain can convert our question into an embedding and that will allow Pinecone to do a similarity search on the database. Let's add our Pinecone API key, environment, and index name. The index we called flowwise. For the API key, I'll just grab it from API keys and I'll do the same for the environment, like so. For embeddings, we'll go to embeddings, and we'll select the OpenAI embeddings node, and we'll connect that to our embeddings. Let's also add our OpenAI API key, like so. And believe it or not, that's it.
Testing the Query-Only Chat Flow
We can save our chat flow and test it out. In fact, let's go back to our ingest document chat flow and open up the chat. I just want to have a look at the summary of our story, and we can see that the main character of our story is called Emily, and she is a talented architect. Let's now see if our other chat flow has a view of this information. Let's go back to our Document QA chat flow and let's test this out. Let's ask the question, who is Emily?
And take note that at this point, we are not feeding this information in via a text file, but we are expecting our model to fetch this information straight from the database. And as we can see, the response is correct. Emily is indeed a young woman who is a talented architect living in the city of Everdale.
Conclusion and Key Takeaways
So this now means that whenever we want to update our information or add additional information, we can simply run our ingest chat flow and we only have to do that once. Whenever we want to interact with our data, all we have to do is run our document QA chat flow. And lastly, because we are now storing our data in a third-party production-ready database, our data will be persisted and there's no chance of losing this data when our server resets.
If you found this content useful, please consider subscribing to my channel. Please like this video and share it. If you are fascinated by Flowise, I highly recommend checking out my series on LangChain as well, where we dive into the guts of using LangChain. Thank you very much for the support. I'll see you in the next one. Bye bye.
