Introduction to the Multimodal RAG Chatbot
Hello everyone. In this video, we'll be building a multimodal AI agent with RAG capabilities, leveraging NVIDIA's GPU-accelerated models and open-source technologies. Let's start with a quick demo of our chatbot in action.
As you can see, our chatbot can quickly process and respond to the question about the data provided. So, in this tutorial, we'll cover integrating a pipeline that accepts multimodal kinds of data that includes text and vision. As part of the process, we'll cover how to set up a vector database, generate embeddings, and how to use GPU-accelerated models.
Now, let's break down how we build this.
The NVIDIA AI Ecosystem
Before we dive into the specifics of our chatbot, it's important to understand the ecosystem we are working with. NVIDIA provides a full-stack ecosystem for large language models that integrates seamlessly with popular open-source tools. This ecosystem offers flexibility and performance, allowing developers to switch between different components as needed.
NVIDIA not only provides proprietary tools but also contributes significantly to the open-source community. Tools like NeMo for building and training large language models, and Triton for serving, are prime examples of NVIDIA's contribution to open source. These tools integrate well with the broader AI ecosystem, providing developers with powerful GPU-accelerated options for building LLM applications.
NVIDIA NeMo goes beyond just a training framework; it includes open-source components that enhance the capabilities for developers. Two notable open-source components are NeMo Data Curator, that simplifies the complex process of data curation. It helps extract, deduplicate, and filter information from large amounts of unstructured data, ensuring high-quality, relevant data sets for training.
The second component is NeMo Guardrails. This component implements safety measures and controls for model output. It allows for developers to add a guard that prevents inappropriate responses, enhancing the reliability and safety of AI applications. You can learn more about this at github.com/NVIDIA/NeMo.
Understanding RAG and Multimodal Pipelines
Let's quickly understand what RAG, or Retrieval Augmented Generation, is. So, RAG enhances large language models by giving them access to external up-to-date knowledge. This process allows our chatbot to provide responses that are both general and specific to our patent data, combining retrieval and generation for more accurate and contextual answers.
Let's explore how we are extending RAG to handle various types of data, or what we call a multimodal RAG pipeline. Now, there are various ways to handle multimodal data, but in this case, we are grounding all the modality into a single one, which is text. For this specific application, we have introduced a specialized Vision Language Model to process visual data.
So, in this case, one VLM is handling general images, while the other Vision Language Model is specifically tuned for charts and plots. Once the conversion of information is done, the rest of the pipeline remains the same as the traditional RAG pipeline.
The Tech Stack: NIM, Milvus, and LlamaIndex
For our implementation of this application, we are using cutting-edge tools from NVIDIA and the open-source community. So, let's break down the components of this specific pipeline. To build our knowledge base, we first have to convert all the documents into text form, and to do that, we'll use Vision Language Models, as shared before. For general image understanding, we'll be using Neva 22B model, which is NVIDIA's fine-tuned variant of LAVA.
For charts and plots, we are employing Deplot by Google, which is specialized in understanding graphical data, including charts and plots. For this specific example, we'll have some mix of documents, which is PDF and images that contain some of the blogs and slides from NVIDIA. So, this data fuels our chatbot's knowledge base.
For vector storage and similarity search, we'll use Milvus, a GPU-accelerated vector database that can handle large-scale similarity searches with impressive speed. So, using GPU indexing and querying requires numerous queries, vectors, or exceptionally high request pressure for maximum throughput and reduced latency compared to CPU indexing.
For embedding, we'll use NV-Embed, which leverages NVIDIA's GPU acceleration to quickly transform text into high-dimensional vectors. The GPU acceleration in both NV-Embed and Milvus significantly enhances the performance of our chatbot, allowing for faster indexing and querying of our knowledge base. For the actual question answering, we'll use a large language model, specifically Llama 3 from Meta, the 70B instruct variant, which will be accessed through NVIDIA's NIM API.
So, this gives us the power of state-of-the-art language models, which is GPU-optimized access through an API. Finally, we'll use LlamaIndex to orchestrate the entire process, from query processing to retrieving relevant information and generating responses. We will also use Streamlit to create a user-friendly interface for our chatbot, where the user can directly upload or just provide the directory path of their documents, and then simply query or ask a question based on those documents.
Project Code and Directory Structure
Now we have learned what the pipeline is about. Now let's dive into the code. Before we dive into the individual script, let's take a quick look at our project structure. Our project consists of three main Python scripts. The first one is app.py, which is the main application file that sets up the Streamlit interface and manages the overall flow of our multimodal RAG system.
The second is document_processors.py, which is responsible for processing various document types like PDFs, PowerPoint presentations, and images. Then comes the utils.py, which contains utility functions for image processing, API interaction, and text handling.
In addition to these Python files, our project also includes several directories, such as Vector Store, which stores all the vector database and related files. In Vector Store, we have image references and PPT references, which store image-extracted or converted PowerPoint slides and its related files. This structure allows us to efficiently manage different types of data and maintain a clean separation of concerns in our code.
Generating an NVIDIA NIM API Key
Before we dive deep into our RAG Q&A chat application code, let's generate an API key for NIMS by going on to build.nvidia.com, and we'll select Llama 3 as our LLM model for the application. Here you get various options to try out the model or just boilerplate code to get this model on NIM optimized NIM onto your application, but here we are trying to generate an API key, so just click generate, and there you have it.
Just copy this API key, and you can use it wherever you are setting the environment variable. But in my case, I'll set this on my terminal, assign that API key to a variable named NVIDIA_API_KEY as an environment variable, and you can do that with this command and press enter.
Code Deep Dive: utils.py
Let's start with utils.py, which contains various utility functions for our multimodal project. Here we are importing necessary libraries and setting up our NVIDIA API key. This is an alternative way from what we did in the terminal, but you can choose either one of those, whatever is better for you.
And then now we load an image processing function, which can handle image conversion to base 64 format, which is necessary for sending images to NVIDIA NIM APIs. And then we use an image analysis function, which helps us identify and process the graphs or charts in our images. We use NVIDIA's NIM APIs, where the model are hosted, to generate a description of these visual elements.
And now we are using these functions to interact with NIM API to describe images and process graphs, which are crucial for our multimodal understanding capabilities. Next, we'll create some text processing functions to process text from PDF, allowing us to extract context around images and tables and group text blocks efficiently.
And now we'll do some file handling, which handles saving uploaded files, which is important for processing user-provided documents. So that covers the main utilities in our utils.py. These functions form the backbone of our image and text processing capabilities. Now let's move on to the document_processor.py.
Code Deep Dive: document_processor.py
Now let's go through the document_processor.py file, which contains functions for processing various types of documents. So, first, we start by importing necessary utilities, and notice that we also importing functions that we just defined in the utils.py.
Now we'll create a function that is crucial for extracting content from the PDF files, including text, images, and tables. Now we write a function for handling the extraction of tables and images from the PDF files, including processing them and storing them.
Now let's work on the PowerPoint processing, and here we are writing four different functions that work together to process PowerPoint files, converting them into PDFs, and then to images, while also extracting text and notes. And now write the main function, which handles the loading and processing of various file types, whether they are individual files or entire directories.
So that covers the main component of our document_processor.py file. Now let's move on to the main script, which is app.py, which includes all the GUI and function calling for all these scripts that we have just written.
Code Deep Dive: The Main app.py
Now let's move on to app.py, which is the main application file for our project. We start by importing necessary libraries and set up our Streamlit page. Notice how we are using Llama Index components and our custom modules. Then we set up our embedding models and language models using NVIDIA's NIM integration with Llama Index.
And then we create the index, which uses the vector database using Milvus and stores all our document embeddings. Now this is the main application, where we set up our Streamlit interface. It provides options to upload or directory processing and sets up the chat interface.
And now this section handles the chat interface, processes user queries, and displays responses. So that covers the main component of our app.py files and all the other Python scripts. Now let's try running this code on Terminal.
Setting Up the Python Environment
Before we run the code, let's create a conda environment and install all the dependencies that we need for this project. First, we'll create a conda environment using the conda create command. I'll name this environment as GPU_Rag, and I'll have Python 3.10, so it will create the environment.
Now let's activate the environment that we just created by using conda activate and the environment name. Now we are into the environment. Let's install all the required dependencies by using pip install method and passing in the requirements.txt file, which includes all the dependencies.
Running the Streamlit Application
Now dependencies are installed. Let's try running this app using Streamlit. To do that, you mainly use Streamlit run and app.py, so it should generate a port number where you can use to look at the web app. So here it has given us the local URL, Network URL, and external URL as well. So, it's 8501. So let's try to go into the browser and see what's there.
So this is how the UI looks like, where you have an option to upload files or enter a directory path, where it will process all the documents present in that directory. So in this case, I'll upload a mix of documents, which is a PDF of a blog from NVIDIA developer, which includes images as well as graphs doing various performance comparisons or the speed up, and then I'll also add an image or a slide about TensorRT. So let's try importing those files.
So once the documents are uploaded to the server, I can process these files, where it is generating embeddings and creating a vector database, where you can query or ask questions about these documents that you just uploaded. So it may take some time or a while, based on the size of the documents.
Live Demo and Q&A with the Chatbot
Now that the index is created, let's try asking a question based on the document we just uploaded. So the first question I'll ask is like, what is the variant about Llama that is the fastest? So based on the chart in this blog, it answered correctly, which is 7B Q4.
Now let's ask, how does CUDA graph lead to substantial performance? So again, it uses the context. And then let's wrap it up by asking, who's the author, which is Alan G, based on the doc we just uploaded. So there you have it. Users can now upload documents of various input types, including text and vision, ask questions, and receive answers based on the content of those documents in real time.
Conclusion and Wrap-Up
So in this tutorial, we have covered how to build a multimodal AI-based RAG application for enhanced understanding using GPU-accelerated components from NVIDIA and open-source tools. Feel free to experiment with the provided scripts and explore further. For more information and resources, check the links in the description below.
