Introduction to a Document Q&A System
PDF, Excel files, document files are everywhere. Getting exact answers for the queries that you have can be a difficult process. Well, what if I tell you there is a solution out there?
In today's video, I'm going to unveil a superpower that you can utilize for asking questions to your PDF and other documents in natural language. What will I be using in this entire video? Well, I'll use tools like LlamaIndex, the amazing Llama 2 model, which is open source, Apache Cassandra, Gradient LLMs, and Python. So stick around and watch the video till the end to witness magic. Let's begin.
Understanding the LlamaIndex Framework
Before I jump and show you the actual implementation, I want to break this video down into the multiple technologies and tools that I'll be using to create the solution. So I'll kickstart the activity by starting the discussion on LlamaIndex. What exactly is LlamaIndex?
Well, Lama Index, which was previously known as GPT Index, is a data framework for large language model applications which provides tools to ingest, structure and access private or domain-specific data. It's basically available in Python and supports a variety of data sources including unstructured data like documents, PDFs, videos, and images, etc. Lama Index basically provides a high-level API that makes it easy to get started with data ingestion and query, as well as lower-level APIs that allow advanced users to customize and extend any modules to fit their needs. So it basically gives you an entire framework where you can kind of have advanced retrieval and query interfaces over the data, which kind of allows you to feed in any LLM input prompt and get back retrieved context and knowledge documented output.
So this is the entire block diagram of how Lama Index functions. So you have documents like PDFs, Word files, and text files. All of that is converted into something called as an index. This index is stored into a vector database. Once a query arrives, there is a retriever that kind of retrieves relevant nodes based on the query. The query engine then decides which particular index to kind of display and then you have the final response. So I'll kind of try to explain this even further.
So think of a document like a PDF document that is loaded into LlamaIndex. The document are parsed or broken down into chunks, and these chunks are basically called as node objects. The node objects are split into overlapping chunks and a semantic vector representation is computed for each chunk using a large language model. The semantic vector representations are then stored in a data structure called as a vector store. The vector store is used to build an index. So this is the entire indexing process that goes behind the scene when when you kind of start using Lama Index. So once you have the query in place, there is something called as a retriever that comes in handy. So the retriever is a tool for extracting and gathering relevant information based on a user's query. So this plays a very vital role in terms of extracting the relevant answers for the query.
So we've looked at most of the modules, but one piece that was still remaining is the query engine. So the query engine is built on the top of the index creator module and the retriever module. The query engine goes through the various results that have been kind of extracted by the retriever, and then it kind of selects based on similarity which is the most probable output and then gives out the response. So this is how the entire LlamaIndex functions.
Introducing the Llama 2 Large Language Model
The next piece of discussion that I want to have is on the open source large language model that I'll be using for this entire activity, which is Lama 2.
So Lama 2 has been open sourced by Meta and it's basically an auto-regressive model. So it kind of released on 18th of July, 2023 in partnership with Microsoft, and this is basically an open source model. That is something that I've already specified. Just to give you some context, Llama 2 has been pre-trained on two trillion tokens, and we also had Llama 1, but the context length of Llama 2 is twice as large when you compare it to Llama 1.
There are three models which are there. So you have the smallest model with 7 billion parameters, then you have 30 billion parameter based model, and then you have a 20 billion parameter based model. So just to kind of highlight the improvements of Llama 2 over Llama 1, the first point is increased training on tokens. Lama 2 is trained on 40% more tokens as compared to Lama 1. Longer context length. So with a longer context, so Lama 2 has a context length of 4K tokens. The current version of Lama 2 has been fine-tuned and aimed at being optimized for for dialogue applications using RLHF. So this is the uniqueness of Lama 2, given you can kind of run this entire piece locally as well, which is where I'm building the solution using Llama index and Lama 2, and I'll kind of show you the vector databases that I'm using.
Setting Up the Apache Cassandra Vector Database
For vector databases, I'm using Apache Cassandra, which is what I'll discuss next. For using a vector database, I'm basically using DataStax's implementation of Apache Cassandra. I've already discussed the advantages of Apache Cassandra in my previous video. I'll add the link to my previous video so that you have a better understanding of why Apache Cassandra is so widely used.
So you can kickstart the journey of using Apache Cassandra on DataStax by going to astra.datastax.com. Once you reach the website and once you sign up, you will have a prompt to create a database. So what you have to do is you have to select vector database, which is something that we'll require in this entire activity. Enter the database name, a keyspace name, select the cloud provider as Google Cloud—you can choose the other cloud providers as well—and then you can select the country. Once you go through the entire process, you will have an active vector database with you.
All you have to do in order to use the database is you have to go to the connect section. So once you reach the connect section, there are two things that you require. The first thing is the application token. So if you look at the selection that I've made, you have to select database administrator and then click on generate token. So it will kind of generate a JSON file for you. The other piece that you require is a secure connect bundle. So for that you just have to click on bundle and it will download a zip file. Once this is up and running, you've kind of done the heavy lifting with respect to the vector database.
Accessing Llama 2 with Gradient
Now the other piece remaining is how do you access Lama 2 model, which is where I'll show you the next solution. So there are many ways wherein you can access Lama 2, but one solution that I really liked, and this is something that I've started using on a daily basis, is Gradient's LLMs.
Gradient makes it very easy for you to personalize and build on top of the open source LLMs through a simple fine-tuning and inference web API. There are various open source models which are available like Bloom 560, Lama 2, Code Llama, etc. Then there are various embedding models which are available, and Gradient also has integration with LangChain and Lama Index, which is where I'm kind of using Gradient to kind of do the entire heavy lifting for us.
In order for you to start using Gradient's LLMs, what you require is Gradient's access token and Gradient's workspace ID, and once you click on create workspace, it will kind of create a workspace ID for you. So you can find the access token in the profile section once you create your own profile and paste it into an environment variable, which I'll show in in the coding section. So this is something that you require before you kind of start using the Gradient LLMs as well.
Initial Project Setup in Google Colab
Now that all the heavy lifting is done, let's actually move to the implementation piece wherein I'll show you how you can chat with your PDF using Llama index, Astra DB, and Gradient's open source models. So let's begin.
Before I begin, just some basics that I want to reiterate again. I have already created a Google Colab session, and I'm currently using a CPU-based session. I've already uploaded two files here. One of them is my secret JSON file, which is used to connect to the vector database, and the other is the secure connect zip file. So these are the two files that I've already uploaded. Now, I've also uploaded two PDF files in a folder called as documents. So I've uploaded the attention paper, so Attention Is All You Need is something that revolutionized the entire LLM ecosystem. And the other PDF that I've uploaded is the Apache Cassandra white paper. So these are the two PDFs that I've uploaded. I'll ask questions to these two PDFs and I'll kind of show you the results as well. So this is something that I wanted to state before I actually move on to the code.
Installing Python Dependencies
So let's begin. So I'll kickstart the activity with the installation. So as you can clearly see, I require LlamaIndex, I require PyPDF, Cassandra, and the other libraries that are there. Before I start using them, I'll have to install them, which is what I'll do right now. So I'll quickly run the cell.
So I have the installation in place. Let's move forward.
Configuring Gradient Environment Variables
I'll kickstart the activity by importing the OS and the JSON module. And if you go back to the earlier part of the video wherein I had specified for using Gradient's LLMs you require the access token and the workspace ID. So I have my access tokens and workspace ID in a unique new feature by Google Colab called as secrets. So I've kind of saved my secrets here in Google Colab. So I have my gradient access token here in this particular secret, and I have my gradient workspace ID as a secret here. So both my secrets are here, so I don't have to reveal them while kind of going through the code. So this is something that I wanted to share. So I'll quickly go forward and run the cell.
So basically the gradient access token and workspace IDs are added as environment variables through the OS module. So I'll quickly run this cell. So this piece of code has been executed. Now I'll move forward.
Connecting to the Apache Cassandra Vector Database
Now here I'll import functions from Lama index and Cassandra. So I'll quickly run the cell. Just to check which version of Cassandra am I using. So I'll quickly run this piece of code as well. So I have Cassandra 3.28.0. So this is the version that I'm currently on.
Now in order for you to ask questions to your PDF, the first thing that you require is a vector database to store the indexes, which is where the first thing that we'll do is we'll connect to the vector database. So this is a readily available code which is available in Astra DB's website. So once you create the database, if you scroll down, you would see this particular piece of code which is something that you can kind of easily kind of copy from there and paste it here. This can kind of help you connect to the Astra DB or the vector database that's available. All this is basically doing is it's kind of extracting the client ID and the client secret from the token that you had supplied, which is the JSON file, and it's kind of then authenticating it based on the client ID and client secret that you've supplied. Once all of that is done, it kind of creates a session on Apache Cassandra cluster, and then it kind of executes a simple command called as select release version from system.local. If everything works fine, then it will kind of give out the system release version for you. If there are issues, then it will kind of give you an error message. So let's execute this cell.
So it gives me a proper release version, which is where I have established a successful connection to my vector database. So we are good in that aspect. Let's move forward.
Instantiating the LLM and Embedding Models via Gradient
One of the reasons why I prefer using Gradient's interface for accessing open source models such as Llama 2 is because the overall abstraction that they have done is simply commendable and you just have to write very few lines of code, mostly one line of code to create an LLM instance. So I'll show that to you as well.
So I'll quickly unhide the cell. Whatever you're seeing online right now is all the amount of code that you have to write in order to create an instance of the Llama 2 model. All you have to do is you have to create an instance of the class GradientBaseModelLLM, supply the base model slug which is like a unique identifier for this particular model, specify the max number of tokens, and then save it into a variable called as llm. That's all that's required. So I'll quickly move forward.
The other piece that we'll require in order to generate the index are embeddings and here is where again, I'll use Gradient's approach of creating embeddings as well. So all you have to do is you have to call the GradientEmbeddings function, pass in the access token, workspace ID, and the Gradient model ID or slug which is used for generating embeddings. And once you do that, save everything into a variable called as embed_model. So I'll quickly do that as well.
Configuring LlamaIndex and Loading Documents
Now what we have so far is a valid connection to a vector database, which in our case is Apache Cassandra hosted on DataStax. We have an active embedding model with us. We have an active large language model, which is Llama 2 with us using Gradient APIs. What we now have to work on is executing the Lama Index part of the entire solution.
So which is where what I'm doing is from LlamaIndex I'm kind of importing the function ServiceContext.from_defaults and I'm specifying the LLM, I'm specifying the embedding model, and I'm specifying the chunking size. So for index generation, what you have to do is you have to recursively split the text that's part of the PDF. Once the splitting has happened, you have to convert the text or the chunks into indexes, which is where this particular function comes in. And you also have to supply the large language model that you're kind of utilizing for the embedding generation and also the embedding model. So this is something that I'm doing through this piece of code, and I set the global service context as this particular service context. So I'll quickly run the cell.
Almost all of our heavy lifting work is done in terms of code. Now I'll quickly move forward and I'll load the PDFs and I'll kind of point LlamaIndex to where my documents or data resides. So I simply create a variable called as documents and I call the function SimpleDirectoryReader wherein I pass in the directory and I call the function load_data. Once all the data resides in documents, I want to find out how much documents have we generated in the entire process. Okay. So I'll quickly run the cell and the output here tells me that you have loaded 17 documents. The attention paper had 11 pages; the Apache Cassandra white paper had six pages. So what this particular function does is it kind of splits the PDF into single pages and saves all of them as individual documents. That's how LlamaIndex is working behind the scene here. Now if this idea is clear, we have the documents ready in the document variable.
Indexing, Querying, and Final Results
All we have to do is we have to set up the index, the query engine, and then ask questions to our documents. Okay. So here is where we are generating indexes for our documents and saving it into our vector database. So which is where VectorStoreIndex.from_documents, and I pass in the documents and I pass in the service context. For this particular instance variable that I've just created, I want to create a query engine for this, which is where I call the function as_query_engine and save it into a variable called as query_engine.
So with this heavy lifting done, what I'll do next is I'll kind of run the cell. Now is the point where you will see magic happen. Okay. So I have the instance of the variable query_engine with me and I call the query function and I pass in a question here. So let's start with something related to what is Cassandra? And let's wait for the response. So here is the output. It says Cassandra is a distributed storage system. So this is the result it's generating from the PDF. I'll ask the next question. How does Facebook use Cassandra according to the paper? Look at the output. According to the paper, one of the applications in the Facebook platform that uses Cassandra is for storing data. So it kind of highlights the specific section in terms of how the entire platform uses Cassandra.
So amazing, right? I have a PDF, I don't have to go through the entire PDF in order to understand the output. All I have to do is I can kind of ask questions to my PDF in natural language. Amazing, isn't it?
Now, I'll ask more questions, but this time to the attention paper. What is multi-head attention? So here is the output. Multi-head attention is a type of attention mechanism, and the output kind of follows.
So if you have—so imagine a use case wherein you have 10,000 documents, you just stuff it in and you kind of pick a good, scalable vector database, which is where I've kind of chosen Cassandra because of the scale that you can kind of achieve. So then once you have all the PDFs loaded into a vector database, you can ask questions, n number of questions to all the PDFs or documents that you have. I can also ask what is positional encoding? And let's wait for the answer. So it says positional encoding is a technique used in Transformers and yeah, it gives me, it's giving me the exact results. It's also giving me sometimes a quotation in terms of where this exists in the PDF. Overall, this is an amazing solution where you're kind of using the latest and greatest LLMs.
So this is something that I wanted to demonstrate in today's video. I wanted to show you the power of how you can use LlamaIndex, open source LLMs using Gradient, Apache Cassandra using DataStax AstraDB and Python and create an end-to-end amazing solution. The initial demo that I showed you is something that I've created using Gradio. The code will be available in the description section of the video so you can check that code out as well, but I've kind of packaged the entire Python code that I've shown you into a Gradio based application.
I hope you enjoyed this video and if you do like the content that I create on my channel, it would be super motivating if you can press the Subscribe button and also press the Bell icon to be notified for amazing videos on data science and machine learning. Thank you so much for watching the video.
