Made withLlamaIndexTutorial

Multi-Modal RAG: Chat with Text and Images in Documents

P

Prompt Engineering

Creator on the LlamaIndex channel

Watch on YouTube ↗

20K viewsJul 12, 202415:45 runtime

Learn to build a complete end-to-end multi-modal RAG system that can chat with both text and images in your documents. This tutorial guides you through the process using GPT-4o and LlamaIndex, demonstrating how to handle mixed data types. You'll see how to set up separate vector…

Chapters

#llamaindex#tutorial#aillm

Core Architecture of the Multi-Modal RAG System

The system integrates both text and image data to provide context-rich answers. The high-level workflow is as follows:

Data Ingestion: Collect documents containing both text and images (e.g., Wikipedia pages).
Indexing: Process and store text and images in separate vector stores.
Retrieval: Given a user query, retrieve relevant text chunks and images from their respective stores.
Augmentation & Generation: Combine the retrieved text and images into a single context and feed it to a multi-modal LLM to generate a final response.

Key Components:

LLM: GPT-4o
Framework: LlamaIndex
Image Embedding Model: CLIP
Vector Store: Qdrant

Indexing Process

The core of the system is a multi-modal index managed by LlamaIndex, which uses two separate collections within a Qdrant vector store.

Text Indexing: Text data is chunked and embedded using a standard text embedding model.
Image Indexing: Each image is processed by the CLIP model to generate a vector embedding.

Note
Qdrant and ChromaDB are mentioned as open-source vector stores that support multi-modal data, making them suitable choices for this architecture.

LlamaIndex's MultiModalVectorStoreIndex handles the logic of creating and querying these dual stores.

Retrieval and Generation Pipeline

The query process leverages the dual-index structure to fetch comprehensive context.

Multi-Modal Retrieval: The retriever queries both the text and image vector stores simultaneously. It's configured to return a specified number of top results from each (e.g., top 3 text chunks and top 3 images).
Context Augmentation: The retrieved text chunks and raw images are packaged together.
Response Generation: The user query and the augmented context (containing both data types) are passed to GPT-4o.

Important
LlamaIndex's multi-modal wrapper for GPT-4o is essential here. It abstracts the complexity of formatting and sending both text and image data to the OpenAI API.

Alternative Strategy: Generating Image Descriptions

An alternative method for incorporating image information is to use a multi-modal model like GPT-4o to generate descriptive text for each image.

Tip
These generated descriptions can then be indexed as regular text chunks. This approach can improve retrieval by adding rich, semantic text that describes visual content, which can be searched alongside the document's original text.

Future Enhancements: Agentic RAG

The video suggests Agentic RAG as a powerful next step for improving system robustness.

An agent can perform multi-step retrieval. If the initial search results are insufficient, the agent can rephrase the query and try again.
This allows the system to iteratively refine the context, assess the quality of retrieved chunks, and ultimately generate more accurate and reliable answers.

We're going to continue our discussion on how to build multi-model RAG systems. In the previous video, we looked at different architectures that are possible for building a multi-model RAG application which combine both images as well as text data. In this video, I'm going to show you how to build an end-to-end system using GPT-4o and Llama Index. This is going to be an extension of our previous video.

Now here is how the architecture is going to look like for this specific case. So first, we're going to collect data. Again, this is going to be a combination of both text and images. For text, we're going to chunk those, create a separate text vector store. We will take each image separately, run them through an embedding model, which is going to be a CLIP model again in this case, and create an image vector store. During user query time, we're going to process the query and compute embeddings for it. And then we'll do retrieval based on the text vector store as well as on the image vector store. We will combine them together to augment the context for our LLM, which is GBT-4o in this case, and that will generate responses which are going to be returned to the user.

So here's a quick overview of how this video is going to look like.

Video Outline and Core RAG Concepts

So first, we'll simply talk about multi-model RAG system. Then we'll talk about the setup, installation, how to prepare the environment. In this case, I'm using Google Colab notebook, but you can run this on your local machine as well. Then we'll quickly talk about data collection and preparation, and then how to create multi-model indexes based on both images as well as text. Then we will implement a multi-model retrieval pipeline, which is going to be retrieving both images as well as text chunks based on the user query. And we're going to wrap everything around in a single pipeline that is going to use GPT-4o to generate responses.

So in the multi-model RAG system, you combine both text and image data to enhance the capabilities of the large language models. So there are four different steps, very similar to the traditional RAG systems. So first one is indexing, where we combine both image and text data and store them in separate vector stores. I'll also show you how you can use the multi-model model like GPD-4o to generate description of images that can potentially be combined with the text chunks to augment your vector store. Then you do retrieval. So basically, the user provides a query and you retrieve both the text and images that are relevant to your query. And after that, we do augmentation. So the retrieved information is used to augment the input to the LLM, and the LLM will generate a response based on the augmented input plus the user query.

Setup and Environment Preparation

Okay, so let's talk about the setup. We will be again installing the CLIP model. This model is going to be used to generate the image embeddings in our case. We will use OpenAI for our LLM. Qdrant is going to be the vector store because we're looking at a multi-modal vector store, and Qdrant is one of the vector stores that supports both image as well as text data. If you're looking for another alternative, ChromaDB is another open source vector store that supports both, so that can be an option. And we will also install some other packages that are going to be just auxiliary packages.

Okay, so first we need to import the packages that we will need. We need to set up our OpenAI API key. I am again using Google Colab notebook secrets, and actually the person who implemented this needs an award because I think this makes life so easy now. After that, we are creating a couple of different folders. So one is input_images. This is a folder that is going to contain some of the input images that I'm going to provide as an example. We're going to see like how you can use GBD-4o to generate descriptions. Then another folder that we are creating is called mixed_wiki. This will contain both images as well as text that we are going to be reading from a Wikipedia page. So first, we check if those folders do not exist, this script is going to generate or create those folders for you.

Generating Image Descriptions with GPT-4o

Now to show you how you can use GBD-4o to generate description of images, we are going to download a number of different images. So these are images related to different Tesla vehicles. Actually, these are I think all of them are related to Model Y. So for example, there are different configurations of Model Y, and these images have the specs for different configurations. So this one is performance, I think the first one was long range, and you can see there are information related to the weight class, what is the speed, and I think also the pricing is here on this specific image. Now this small script will just display all the images contained in a folder. So these are the images that we looked at.

Now the first example that I'm going to show you is how to use GPT-4o to generate description for each of these images. And you can use these descriptions as text chunks and put them in your vector store. We're not going to do that in this specific video, but in a subsequent video, I'm going to show you how you can perform that process. So here, we're providing the input_images folder which contains all the images that we downloaded. And then we're using the OpenAI multi-model function from Llama Index. So they have written a function that simply wraps GBD4, or the OpenAI API for multi-model applications. So the first prompt is simply, "generate detailed text description for each image." So basically, it takes one image at a time by reading those images from the directory and feed it into the LLM to generate responses for those. So here is a detailed response to image number one. I think this is one related to the specs. So basically seems like it did OCR on the image and generated a text response. The second one is structural diagram of the vehicle. This was I think this specific diagram. So based on the diagram, it generated a description of what it sees in the image, right? So normally, if you're using, if you're running these images through a multi-model model like GP4o, you can use these text description as text chunks to augment your existing vector store. But for the purpose of this specific video, we're going to be sticking to using CLIP to generate embeddings that's going to be used for retrieval purposes.

Data Collection from Wikipedia

Then we have this helpful function that is going to download images from Wikipedia pages. So in this case, I am providing links to different articles in on Wikipedia. So for example this Tesla Model XY, then Kia, Rivian, so these are different electric vehicle. We want to get the information regarding all of those. We run that script and we will download images as well as text from those Wikipedia articles.

Now in my case, I was running into, there's too many requests. So you could potentially add a small pause or that will probably help you not getting into this specific error, but I was able to download some of the images. So we can look at some of them. For example, this seems to be, I think Model S, and it downloaded data for Rivian, Model S, X, and we're going to also download a 10K form for Tesla. So here a truck that Rivian is creating for Amazon. So there are both Rivian images as well as Tesla images. We have the images.

We're going to create two different vector stores and then combine them together using a multi-modal vector store. So the first one is going to be for text and the second one is going to be for image. We are going to be using Qdrant as I said. So we create the Qdrant client. Then if you're trying to create a multi-modal vector store using Qdrant, you just need to provide the names of two collections. So we create one collection for text, the second collection is for images. So think about these are two different tables in your database. One belongs to the text chunks and the other belongs to your images.

Since we're using Llama index, so we will need to create a storage context. If you're not familiar with this idea, I have a couple of videos on Llama index which really explains the concepts, so I'm going to put links to those. Storage context basically contains all the information regarding where to store your vector store, what type of embeddings to use, what are going to be the chunk size, and so on and so forth. So we load the data from this mixed_wiki directory that contains both text as well as the image data. And then we create a multi-modal vector store based on all the documents that are contained. The storage context is going to be the one that we created using both the image as well as the text collection. So this basically combines embeddings of both the text as well as your images.

Implementing the Retrieval Pipeline

Next, we will create a RAG pipeline. Now, in the first part, we're going to just look at what kind of retrieved context we are seeing, and then in the next part, we're going to feed it into the LLM to generate our final responses. So we need to set up some hyperparameters for index for retrieval. So we take that index, create a retrieval on top of it, and we say that for every query, it's supposed to return the top three text chunks and the top three images that it can find which are similar to the query that the user is providing.

So we write a helpful function which basically takes that retriever, gets the user query, and we're just inputting only the top 50 tokens. You don't really need to put this, but this was just to show you like for extremely long user queries, you can put how many tokens you want to process. This will just help you save some money. All right, so we get the retrieved results. Now the retrieved results are going to have two different parts. One is going to be text chunks and the second one is going to have images. So we take or separate both of them. So here we are looking at the images and putting them into a retrieved_images list, and the second part is the text that we are going to be displaying, right? And if there are images, then we just show those images, right?

So here is a simple prompt: "what is the best electric sedan?" And you can see that these are three chunks that are retrieved by the text embeddings. So the first one is, "The Tesla Model S is a battery electric executive car with a liftback body style built by Tesla." The second one talks about Model X, which is not a sedan, but if you look at the images, it only retrieved images of Model S, which is pretty good because we are specifically talking about a sedan.

Building the Full RAG Query Engine with GPT-4o

So in the last step, we can just take all this context, put it, put it together and feed it into GBD-4 to generate a final response. So here is how it looks like. We will be using a prompt template, and in that prompt template, we say that, okay, context information is below. So this is the context that the model is going to get. And then, given the context information and not prior knowledge, answer the query. So we ask it to answer the specific query that the user is providing.

Now, in this case, we also want to run both the images through this whole process. So we're going to create a prompt template, then we are going to run that prompt template on the index that we created. In this case, we're going to be using GPT-4o as our multi-modal model. So again, keep in mind that this is not the normal GPT-4o, but the multi-modal version that Llama Index implements. So it's going to receive both the images as well as the text to generate the final response. So this pipeline now receives the user query and the LLM along with the context that is going to be retrieved by our index. When we run our query on the query engine that we just created, we're going to get a response. And then we also look at what images were retrieved as part of the context, so we're going to display those as well. So here I'm asking it to compare the design features of Tesla Model S and Rivian R1.

Analyzing Results and Future Directions

So the responses that you are going to get are going to be based on the text as well as the images that are retrieved. So here it talks about Tesla Model S: the body style, powertrain, and design evolution, range, interiors, additional information. And it looks at the exact same features for Rivian R1 as well. And I think at the end, it gave us a small comparison or summary of whatever it has found. So it says that Model S is a sedan which is focused on performance, range and luxury, whereas the R1 or R1S are designed as a pickup truck and SUV respectively, with a focus on utility, off-road capabilities and utility.

Now we can also look at the sources, so basically the text chunks that were retrieved as well as the images. In this case, it only retrieved two images. One is that of Rivian R1 and the second one is Model S. Now for this specific project, we combine both images as well as text chunks and created an end-to-end RAG system or RAG pipeline. Now you can further improve this by using an agentic RAG. That's going to be a topic of our subsequent video. So we're going to look at how to create agents that can perform RAG for you. That's extremely helpful in situations in which you cannot find the answer in a single shot. So for example, in the first case, if it's not able to retrieve the proper chunks, then the agent is going to do a second pass of retrieval by changing the prompt. So that will change the retrieved context and it can also look at how good the responses or the chunks are before generating final responses. So I'm going to be creating some videos around how to do agentic RAG. I think that's going to be a topic which you will find extremely helpful.

Now there are a number of different approaches that we can take when it comes to multi-modal RAG, and I'm going to be building a lot more content on on this. I still think RAG is one of the most practical applications of LLMs and that is being used in industry more widely than compared to anything else. For example, the concept of agents is great, but it's not there yet. So if you are interested in RAG, agents, or anything related to LLMs, make sure to subscribe to the channel. I hope you found this video useful. Thanks for watching and as always, see you in the next one.