Made withLlamaIndexClip

META LLAMA 3 8B INSTRUCT LLM – How to Create Medical Chatbot with LlamaIndex FastEmbed Colab Demo

A

AI WITH Rithesh

Creator on the LlamaIndex channel

Watch on YouTube ↗

12K viewsApr 20, 202413:53 runtime

Learn to build a sophisticated medical chatbot using Meta's Llama 3 8B Instruct model. This guide demonstrates how to create a Retrieval-Augmented Generation (RAG) application that answers questions about fibromyalgia based on a PDF document. Discover the complete workflow, from…

Chapters

#llamaindex#clip#aillm

Project Overview

This video provides a step-by-step guide to building a Retrieval-Augmented Generation (RAG) chatbot using Meta's Llama 3 8B Instruct model. The chatbot is designed to answer questions about a specific medical condition, fibromyalgia, by sourcing information from a provided PDF document.

Core Technologies

Large Language Model (LLM): meta-llama/Meta-Llama-3-8B-Instruct
RAG Framework: LlamaIndex
Embedding Model: FastEmbed with the BAAI/bge-small-en-v1.5 model.
User Interface: Gradio
Vector Store: An in-memory VectorStoreIndex from LlamaIndex is used in the demo.
Execution Environment: Google Colab with a paid L4 GPU instance.

Important
The Llama 3 8B model is resource-intensive. The demo shows it consuming nearly 16 GB of GPU RAM, making a capable GPU like the NVIDIA L4 a necessity.

Implementation Steps

Get Model Access: You must first request access to the Llama 3 model on its official Hugging Face page. Access is typically granted quickly.
Generate Access Token: Create a Hugging Face access token to authenticate and download the model in your environment.
Environment Setup: In a Google Colab notebook, install the necessary Python libraries, including transformers, llama-index, fastembed, PyMuPDF (for PDFs), and gradio.
Load Data: Use SimpleDirectoryReader from LlamaIndex to load the content from the source PDF file.
Initialize Models:
- Set up the FastEmbed model for creating vector embeddings.
- Instantiate the Llama 3 8B Instruct model using HuggingFaceLLM from LlamaIndex, providing the model name, tokenizer, and other parameters like context_window and max_new_tokens.

Tip
It is crucial to correctly configure the stopping IDs for the Llama 3 tokenizer to ensure the model terminates its generation properly. This involves fetching the eos_token_id and a specific eot_id.

Create Vector Index: Build a VectorStoreIndex from the loaded documents. LlamaIndex uses the configured embedding model (FastEmbed) to convert text chunks into vectors and store them in an index.
Build Query Engine: Create a query engine from the index (index.as_query_engine()). This engine handles the entire RAG process: embedding the user query, finding relevant context, prompting the LLM, and returning the final answer.
Deploy UI: Wrap the query engine's logic in a predict function and use Gradio to create a simple chat interface for interacting with the chatbot.

Key Concepts & Takeaways

RAG Architecture: The chatbot leverages the RAG pattern to ground the LLM's responses in factual data from the provided document. This involves retrieving relevant text chunks (context) based on a user's query and feeding them to the LLM along with the original question.
Prototyping, Not Production: The demonstrated application is a proof-of-concept. It is not a medical-grade chatbot and should be treated as an educational example of how to use Llama 3 for a specific task.
Flexibility of LlamaIndex: LlamaIndex provides a modular framework where components like the LLM, embedding model, and vector store can be easily swapped, allowing for experimentation with different technologies.

Introduction & Chatbot Demo

Meta AI has released Llama 3, their newest large language models, and these models are available for download as open weights models. So I thought I will build a chatbot, a retrieval augmented generation based chatbot using the Llama 3 8 billion instruct model.

First, let's look at a quick demo. So this is a chatbot which is built to answer questions on a particular condition known as fibromyalgia because I'm using this document, which is a guideline or which is a booklet which talks about this particular disease called fibromyalgia. So I'm going to do retrieval augmented generation chatbot over this particular PDF file, okay, using Llama 3 as my, Llama 3 8 billion instruct as my LLM and a couple of other libraries. Okay.

So I asked a question, 'What is fibromyalgia?' over here and this was the response. There is a typo over here in this particular disease condition, whatever I put over here, but then it has given a proper response from, you know, the file, right? 'Fibromyalgia is a long-term condition, you know, tenderness all over the body.' It gives this.

Chatbot Performance & Symptom Query

Now if I were to ask it, like, what are the symptoms? Okay, so it's actually generating over here, all right. So if you look at it, for this particular query, it is taking somewhere I think a little over 20 seconds or more, but the generation is happening. Yeah, somewhere it took close to 20 seconds, right?

So it says, 'According to the provided information, the main symptoms are this: pain, feeling as tiredness, fatigue, stress, worry, or low mode.' Okay, these are the this thing. Additional, some people may experience symptoms such as, right, 'pain getting worse' and see if you over here, 'difficulty'—and it has not continued over here, probably it's because of the generation window which I have put. Right, but if you look at this thing, it is actually pulled up from over here, okay, and you know, it has taken a couple of these things and then it's actually created the symptom list, right? 'What are the symptoms?' It has created from over here. Okay.

Diagnosing Fibromyalgia & Demo Summary

Now if I were to ask, 'How do you diagnose fibromyalgia?' Again, I think somewhere close to 20 seconds. So this demo is actually running on a GPU instance, Colab GPU. Okay, somewhere close to 20 seconds.

It says, 'According to the information, it is difficult to diagnose, vary from person to person,' right? 'There aren't any specific blood tests, X-ray scans, or this thing to do it, but you might have some blood tests to check for other conditions. Your doctor may also suggest a referral to a rheumatologist, physiotherapist,' so on. So it's again pulled that information from here, right? It's kind of made a summary out of it, right? So your doctor might suggest a referral to this part and then, you know, this part, right? So it's actually pulling information from this file and the large language model is able to generate responses. Okay.

Accessing Llama 3 on Hugging Face

Now let's go to the technical details. Okay, so first and foremost, this particular model is released on Hugging Face, so you have to actually go and apply over here for access. It asks you some details like your name, organization, and other details. So once you apply over here, you'll get access to the model, and the access is quite fast. I got access in somewhere around 5 minutes or something. I applied and immediately got access to the model. So this is the model which I'm using: Meta Llama 3 8 billion instruct. So this is an instruction fine-tuned model. Okay. This model can generate text and code only. It has a context length of 8K, right? And the other details are present over here about this particular model. So I'm making use of this particular instruction fine-tuned Llama 3 8 billion parameter model. So I got access over here.

The next thing is that you need to create an access token over here. Okay. Uh, let me open over here, right. Uh, if you go to Hugging Face, you need to create an access token over here, all right, and you need to copy this access token in your Colab notebook. Okay.

Setting Up the Colab Environment

Now let's go to the Colab notebook. All right. Um, so what I'm doing over here is that I'm making use of a L4 instance. So this is a paid instance, this is not a free instance. And if you look at over here, let me look at the resources. This particular model has taken close to 16 GB RAM. Okay. So this particular L4 instance is a good instance, it is cheaper. Okay, the L4 GPU instance, and it has 22.5 GB GPU RAM. Okay, this model took somewhere close to 16 GB RAM. Right.

Now let's go to our code. So first, you need to upgrade Transformers. Okay, that's what I've done over here. Install Transformers. Then I'll be using LlamaIndex as my library for this RAG framework. Okay, so I need to install LlamaIndex. I'll be using a PDF file, I'll be reading a PDF file, so PyMuPDF. Gradio for my UI, and a couple of other libraries over here. That is what I do over here. Okay.

I create a folder called 'data' over here and I upload my PDF file, right? And that is what I do. I have to upload the PDF file over here, so that is also done. Then what I do over here is that I need to install LlamaIndex. Okay. So here, I've installed this particular version of LlamaIndex. For me, it works with this version, maybe the newer version also can work, but I've installed this version. Okay. Then what I do is that I install llama-index-llm-huggingface. This needs to be installed to make use of Hugging Face LLMs with LlamaIndex. Okay, so that is also done.

Understanding the RAG Architecture

All right, now after doing that, I need to install FastEmbed embeddings. Okay, so that is what I'm doing over here, all right? And then I install FastEmbed also because I'll be making use of FastEmbed for embeddings, right?

And then what I do over here is that I create a SimpleDirectoryReader and I load the data, basically this PDF file. Okay. So once this PDF file is downloaded, I also load my Fast embedding model, which is this is the embedding model, okay, BG-small. So I'm loading that particular model.

Okay, so why is this embedding model required? So if you are familiar with retrieval augmented generation or RAG framework, okay, what happens in RAG framework is that you have a set of documents, okay, right? Let me open an image, one of these images. Any one of these images should be fine. Okay, maybe I'll open this one. All right.

So in a RAG architecture, what happens is that you have a document which is already present, right? You have a set of original documents. These documents are converted into vectors using an embedding model and they are stored in a vector database. Let me zoom this image and... yeah. So what happens in a RAG architecture is that you have a set of documents, your referral documents from which you want to query information. Now these documents are converted into vectors using an embedding model. Now when a query comes from a user, okay, what this RAG framework will do is that it will actually send the query, convert the query into embeddings using the same embedding model, and it will do a vector database search, right, for finding relevant context. And that contextual data is used by the framework along with the query, and it is sent to the LLM. The LLM then does, you know, it generates a response based on the context, right? And that is given back to the user. So what the framework does is that it takes this relevant context, it appends it to the original query and the prompt and sends it to the LLM, and from the LLM you get the response back. It post-processes, the framework post-processes the response and gives it back to the user.

So the framework which I'm using is LlamaIndex. The embedding model I'm using going to use over here is FastEmbed. The vector database is ChromaDB. The LLM is Llama 3 8 billion instruct model.

Implementing the RAG Pipeline with LlamaIndex

Now let's go back over here, right? Then what I do over here is I have to create a prompt template. So prompt template system prompt is 'You are a question and answer assistant. Your goal is to answer questions as accurately as possible based on the instructions and context provided.' Okay. So here I create a query wrapper prompt, and this is not required because I have created a Hugging Face token over here and I copy the access token over here, right? So I need not do, what do you call, Hugging Face notebook login.

Then what I do over here is that for Llama 3, you need to get the stopping IDs for the token model. So how you do it is like this. You get the tokenizer, and from the tokenizer, you get the stopping IDs, basically end_of_text token ID and convert_tokens_to_ids. There is an eot_id. So that is what is stopping IDs.

Here is where I'm instantiating my Llama 3 8 billion instruct model. So this is how I instantiate it over here. So HuggingFaceLLM, if you see, tokenizer name and model name is 'meta-llama/Meta-Llama-3-8B-Instruct'. So here is how I'm initializing it, okay? And the context window is 8k and max new tokens is 256. I can increase this to generate more output. Okay. So this is how system prompt and query prompt and all those I set up over here, and Settings.llm is equal to LLM. So I have instantiated the LLM over here. Okay.

Then what I do next is that I create a vector store from my documents. So internally what it will do is that, here I would have set the LLM embedding to, you know, embedding model I'm setting over here, okay? So using this embedding model, it will actually convert the documents into vectors and it will store it in the vector database. Okay, so that is what is happening over here. So here I'm actually not using ChromaDB, I'm just using the internal VectorStoreIndex. Okay, it is an in-memory vector store which is present over here.

Creating the Gradio UI and Final Thoughts

Right, then what I do over here is that I create a query engine and I create a predict method over here which takes an input, and based on the input it gives it to the query engine, right, and a response is generated.

So internally what happens is that when an input query is given, it tries to find out the relevant context from the vector database, it retrieves the context, attaches the prompt, and then gives it to the LLM. The LLM is able to generate a response and that response is actually returned by the query engine after processing. And so this predict method I'm creating for creating my Gradio UI. And here I'm sending my predict method with the input and history.

So what happens over here is that when a user sends a query over here, that is sent by the Gradio chat interface to this predict method, which is actually then sent to the query engine. Okay, what this query engine does is that internally, it will take the query, find the similar vectors in the vector database, retrieve that textual context or content, and then send that along with the user query and prompt to the LLM, get the response back from the LLM, post-process it. That is what is returned over here and that is returned back to the UI. So that is what you have seen in this particular demo.

So in this way, you can actually create a RAG-based application using Llama 3 instruct 8GB model and using various frameworks like LlamaIndex, FastEmbed for embedding, right? So here you can play around with the different vector databases and other things which LlamaIndex supports. So this is just a prototype. You know, this is not a medical grade chatbot or anything. This is just a prototype of how you can use Llama 3 HGB instruct model for RAG, for retrieval augmented generation chatbot.

I hope this video is useful to you. I'll share all the links in the description of the video. If you like the video, please like, share, subscribe to the channel. See you in another video.