Made withLlamaIndexTutorial

Llama Index 101 with Vector DBs and GPT 3.5

J

James Briggs

Creator on the LlamaIndex channel

Watch on YouTube ↗

41K viewsJun 7, 202318:40 runtime

Learn how to use Llama Index in a production-style environment with the Pinecone vector database. This introductory tutorial guides you through building a retrieval augmented generation (RAG) pipeline to enhance large language models like GPT-3.5 with external knowledge…

Chapters

#llamaindex#tutorial#aillm

What is Llama Index?

Llama Index (formerly GPT Index) is a data framework for building retrieval augmented generation (RAG) pipelines. It connects your custom data sources to large language models (LLMs) like GPT-3.5. This process allows LLMs to access external, private, or real-time information, which helps in reducing hallucinations and enabling source citations.

Core Llama Index Objects

Document: The basic data container in Llama Index. It's a wrapper around a piece of text (e.g., a paragraph from a PDF) and can hold associated metadata.
Node: A Document object that also contains information about its relationship to other nodes. For example, a node can have pointers to the next and previous nodes, which is useful for maintaining the original structure of a long document.

Building an Indexing Pipeline with Pinecone

This video demonstrates an end-to-end pipeline for loading data into a Pinecone vector database using Llama Index.

Data Preparation: The example uses the SQuAD dataset. The raw data is cleaned by selecting relevant columns, and duplicates are dropped to create a clean set of unique text contexts.
Object Creation: The clean data is converted into a list of Llama Index Document objects, each with a unique ID and metadata. These are then converted into Node objects for indexing.
Pinecone Setup:
- Initialize a connection to Pinecone using your API key and environment.
- Create a new index if one doesn't exist, ensuring the dimension matches your embedding model (e.g., 1536 for text-embedding-ada-002) and the metric is set (e.g., cosine).
Llama Index Contexts:
- StorageContext: This tells Llama Index where to store the data. It's configured to use a PineconeVectorStore which points to your Pinecone index.
- ServiceContext: This defines the processing pipeline, primarily for embeddings. You specify the embedding model (e.g., OpenAI's text-embedding-ada-002) and other settings.
Indexing: The GPTVectorStoreIndex.from_documents method orchestrates the entire process. It takes the nodes, uses the ServiceContext to generate embeddings, and uses the StorageContext to upsert the embedded data into Pinecone.

Tip
To significantly speed up the embedding and upserting process, increase the embed_batch_size in your ServiceContext. A value of 100 is recommended to reduce the number of API calls and network latency.

Querying Your Data

Once the index is built, you can create a query engine using the index.as_query_engine() method.
This engine provides a simple .query() interface to ask questions. Llama Index handles retrieving relevant context from Pinecone, constructing a prompt for the LLM, and returning a synthesized answer.

Note
The video notes that Llama Index's built-in data uploader can be slow for large datasets. You can monitor progress by checking the vector count in the Pinecone UI. For bulk ingestion, using the Pinecone client library directly might be faster.

Key Takeaways

Llama Index provides powerful abstractions for building complex RAG applications.
Combining Llama Index with a managed vector database like Pinecone creates a production-ready stack.
Performance tuning, like adjusting batch sizes, is crucial for efficiency.

Important
Always remember to delete your Pinecone index after you are finished to avoid incurring costs and to free up resources, especially on the free tier.

Introduction to Llama Index for RAG

Today we're going to take a look at how we can use Llama Index in production with Pinecone. Now this is an introduction to the Llama Index library that was previously known as GPT Index. We're not going to go into any details on the more advanced features of the library, we're just going to see how to actually use it and get started with it and do that in a way that would be more production friendly with a vector database like Pinecone.

Now for those of you that don't know, Llama Index is a library that helps us build a better retrieval augmentation pipeline for our LLMs. So we would use retrieval augmentation when we want to give our LLM source knowledge, so knowledge from the outside world or maybe some internal database or something along those lines. And that will help us: one, reference that for the knowledge, so we can add in citations and things like that there, and it will also help us reduce the likelihood of hallucinations.

So Llama Index is a library that was supported in doing that. Now Llama Index can do a lot of things, not all of those are going to cover in this video, but the main features of the library include the data loaders that allow us to very easily extract data from APIs, PDFs, SQL databases, CSVs—all of the most common types of data sources. It also gives us some more advanced ways of structuring our data. So we can add in connections between different data sources, which is kind of useful. So imagine you have a load of chunks of text from PDFs. What you can do is add in connections between those chunks, so the first chunk in your database would be connected to the next chunk with a little connector that says this is actually the next chunk and this is the previous chunk.

And they also support things like post-retrieval re-ranking as well. So there's plenty to talk about, but first let's get started with a simple introduction to the library.

Environment Setup and Data Preparation

So we're going to walk through this notebook here. There will be a link to this notebook at the top of the video right now. So the first thing we need to do is install the prerequisite libraries. So go ahead and run that.

Now for the runtime here, we don't need to be using GPU, so you can just check if you are doing that or not. It costs money to use GPU on Colab, so you can just set hardware accelerator to None to save that money.

Okay, and once you're doing that, what we're going to do is just download a data set. So I'm going to use the SQuAD, which is the Stanford Question Answering dataset. Okay, so there's a few things I'm doing here. First, I'm just getting the relevant columns I need, so the ID, the context (which is like a chunk of text), and the title, so basically the page title where that context is coming from. And then what I'm doing is dropping duplicates. So in the SQuAD dataset, you will basically have like 20 contexts and 20 different answers, but those 20 contexts are all identical for those 20 different questions. So you end up with a lot of duplicate context in there, but because we are just using the context, we actually need to remove all that duplication. So that's what I'm doing here.

And then we get this. Okay, so we have our ID, so it's like the document or context ID, the context itself, and then we have where that is coming from. Okay, so we have the first few there are all from the University of Notre Dame media page, and in total we have just almost 19,000 records in there.

Creating Llama Index Document Objects

So Llama Index uses these Document objects, which you can think of as being basically the context, or it revolves around the context of your data, right? So this chunk of text, and it will obviously include other bits of information around that context. So for us, it's going to include the document ID, right? So every document is going to need an ID.

And this is optional, so we can also add extra info, which we can think of as metadata for our context. Now for us, we just have title, but obviously we could add more. This is a dictionary, so we could add, I don't know, something else here, right? And we could just, you know, put something, but of course we don't actually need that, so we will remove that. But yeah, you can put as many fields as you like in there. Yeah, let's run that and take a look at one of those documents and see what it looks like. So you can think of this as a core object for for Llama Index.

All right, so we have this document. We have the text, and then if we go through here, we have the document ID and that extra info. Embedding, we don't have an embedding for it yet, so we're going to create that later. But the embedding is also very important because that's what will allow us to search through that data set later on.

From Documents to Nodes

Okay, so now what we need to do is actually create those embeddings. So to create those embeddings we're going to be using OpenAI. So for that you will need to get an OpenAI API key from platform.openai.com, and then you just put that in here. I have already done it so I will move on to this.

So one step further from our document is what we would call a Node. So a Node, the way that I would think of this is it's your Document object with extra information about that document in relation to other documents within what will be your your database. So let's say you have the chunks of paragraphs or chunks of text from a PDF, a Node will contain the information that chunk one is followed by chunk two, and then in two, it will say chunk one was the preceding chunk before this. So it has that relational information between the chunks, whereas a Document will not have that. So we would need to add that in there. We're not going to do that here, we'll talk about that in the future, but we still need to use the Nodes here. So we're going to just run this.

So our Nodes in this case are basically going to be the same as our documents in terms of the information that they carry. But Node is the object type that we will build our vector database from. So let's run this. I should say here we've set our OpenAI API key, we don't actually need to use it yet, I should have really done that later, but it's there now. So we have that ready for when we do want to use it. Okay, so we've just created all the nodes from the documents. Let's check, take a look at those nodes. Okay, obviously we have the same number of nodes as we do documents.

Setting Up the Pinecone Vector Database

Now we are going to be using Pinecone, which is a managed vector database, as the database for our Llama Index data. Okay, so to use that we need to get our API key and environment, which we do from app.pinecone.io. And within there, once you are in at app.pinecone.io, you should be able to see API Keys over on the left. You'll see something that looks like this. And you just want to copy your API key and also remember your environment here. So I've got us-west1-gcp. So your API key you put in here, and here I'm going to put us-west1-gcp. Okay.

And after running that, let me walk you through this. What is going on here? So we initialize our connection to Pinecone. We create our Pinecone index, so I'm just going to call it llama-index-intro. You can call this whatever you want. The things that we do need to do is one, create our index if it does not already exist, which if you're running this for the first time, it won't. And to create that index, you need to make sure dimensionality is the same as the text-embedding-ada-002 model, which is the embedding model we're using. And that dimensionality is 1536. And we also need to make sure we're using the right metric. We can actually use any metric here, so you can use Euclidean, dot product, or cosine, but I think cosine is the fastest in terms of the similarity calculation when you're using text-embedding-ada-002. Although in reality, the difference between them is practically nil, so you can use any, but I recommend cosine. Now, after that we will just connect to the index. Okay, so here we're connecting to Pinecone, creating an index, and then connecting to that index.

Building the Indexing Pipeline

Okay, once that is done, we can move on to connecting. So we've just created our index, connected to it. Now what we want to do is connect to it through the vector store abstraction in Llama Index. So to do that, that's pretty simple, PineconeVectorStore, and then we just pass in our index. That's it. That's pretty easy. So this will just allow us to use the other Llama Index components with our Pinecone vector store.

Cool, so I think that is all good. And then we have a few more things going on here. So let's talk through all of this. Let me make it a little more readable. So yeah, there's a few things going on. We have, basically what we're wanting to do here is create our index, which is this GPTVectorStoreIndex. So we're basically going to take all of our documents and we're going to take the ServiceContext, which is like your embedding pipeline, and we're also going to take the StorageContext, which is the vector store itself. And this will essentially act as a pipeline around that. So it's going to take all of our documents, it's going to feed them through our embedding pipeline, so this ServiceContext, embed all of them, and put them all into our vector store. So, I mean, it's not in reality, it's pretty straightforward. Let me just explain that from the perspective of where we're actually initializing these.

StorageContext.from_defaults, it's really simple, we're just using our vector store. There are other parameters in here, but we don't need to use any of those because we're just using our vector store with the default settings. With the ServiceContext, like I said, this is the embedding pipeline. Again, we don't really need to specify much here. We just need to specify, okay, we're using OpenAI embeddings. This is going to automatically pull in our API key which we set earlier on, up here. Okay. So it's going to automatically pull in the API key. We do need to set the model, so text-embedding-ada-002 at the time of me going through this is the recommended model from OpenAI. And we have our embedding batch size. So this is one important thing that you should set. Basically, it will embed things in batches of 100. I think by default the value for this is much smaller, 32 or 16 or something like that. So that basically means if ever it's going to take 16 chunks of text, it's going to send them to OpenAI, get the embeddings, and then it's going to pass on to the StorageContext and upsert those to Pinecone. But what we've done here is set the embedding batch size to 100. So it can take 100, send it to OpenAI, then sends to Pinecone. That means that you need to make less requests. It was, it's like six times less requests if you set this to 100, which means in essence, you're going to be roughly six times faster, because the majority of the wait time in these API requests is actually the network latency. So it's making the request, receiving the requests. So by increasing that batch size, it means things are going to be faster, which is I think what we all want.

So yeah, we set that and then with that we initialize our ServiceContext, right? So embedding pipeline, or maybe we can even think of it as a pre-processing pipeline for our data. And then we just set everything up together, right? So our, that's our full indexing pipeline. We can initialize that.

Indexing Data and Monitoring Progress

Now this can take a long time, and unfortunately Llama Index doesn't have like a progress bar or anything built into it, but we can actually check the progress of our index creation. If we go down to... so our llama-index-intro here, we can go to index info and then we'll see the total number of vectors that are in there. Okay, and we can also, you can also see the rate of them being updated as well, and you can then refresh and you can see where we are. Okay, so we're 4.3 thousand and we need to upsert how many? Quite a few actually. So it's going to take a little while. What I might do is just stop that for now and we can just jump ahead and begin asking questions, so I'm not waiting too long for that. But yeah, that's just one of the unfortunate things with Llama Index, but we can kind of get around that by just taking a look in the in the actual Pinecone dashboard at how many vectors we actually currently have in there.

Okay, so yeah, let's stop that. Right now it is very slow to do this with with Llama Index. If you're just wanting to get your vectors and documents in there, I would just use Pinecone directly, it's so much faster. I mean, for more 18,000, 16,000, whatever that number is, you're going to be waiting, I don't know, not long, like maybe a couple of minutes at most, because you need to embed things with OpenAI and then send things to Pinecone, yeah, a few minutes if you set that code up properly. But anyway, that does mean that we wouldn't benefit from the other things that Llama Index offers. So in some cases, it might just be a case of being patient. But that Llama Index, the embedding process will be optimized in the near future, so that hopefully that will not take quite as long to actually upsert everything.

Querying the Index

So from here, let's pretend we've upserted everything and now what we want to do is build our query engine. So the query engine is basically, it's just the index and we have this method called .as_query_engine(). It basically just reformats that into something that we can begin querying. Okay, so we have our query engine, then we do query_engine.query(). And our question is going to be: 'In what year was the College of Engineering established at the University of Notre Dame?'

We saw that the first few items in there were talking about the University of Notre Dame, so we would expect that that will work. Why? Okay, so it looks like that hasn't actually initialized the index properly because I kind of stopped it midway through. So what I'll do is I'm just going to take like 100 docs quickly. Okay, so it's a bit quicker. Let's check. Okay, so we still have all those other documents in there. So now let's try that. Okay, and we get, 'The College of Engineering was established in 1920.'

I'm sure it's one of the first items, it's probably where I got the question from. Yes, so like question four here I think. If we take a look at that, so data.iloc[4], yeah, and we can have a look at the context. Okay, so it's pulling this information: 'established in 1920.' Okay, cool. So yeah, that's how we would set up with Llama Index with a vector database like Pinecone.

Cleanup and Conclusion

Once we're done with that, or we don't want to do is, if you're finished and you maybe want to ask some more questions, so obviously go ahead and do that. But once you are done and you're not going to use the index again, we delete the index, just so that we're not wasting resources there. And we can actually use, at least if you're on a free tier, you can use that index for something else after that.

So that's it for this video. I just wanted to very quickly introduce Llama Index and how we would use it. Of course, like I said at the start, there is a lot more to Llama Index than what I'm showing here, but this is very simply an introduction to the library. But anyway, I hope this has all been useful and interesting. So thank you very much for watching and I will see you again in the next one.