Made withLangChainClip

LangChain RAG: Optimizing AI Models for Accurate Responses

I

IBM Technology

Creator on the LangChain channel

Watch on YouTube ↗

42K viewsFeb 13, 20258:11 runtime

Learn how to overcome the limitations of large language models (LLMs) using Retrieval-Augmented Generation (RAG) with LangChain in Python. This tutorial demonstrates how to provide an LLM with up-to-date information by building a knowledge base from web content. Follow a…

Chapters

#langchain#clip#aillm

The Problem: Outdated LLM Knowledge

Large Language Models (LLMs) are trained on vast datasets but have a knowledge cut-off date. This means they are unaware of recent events, products, or information published after their training was completed. As demonstrated in the video, an IBM Granite model with a 2021 knowledge cut-off could not answer a question about a 2024 partnership announcement.

The Solution: Retrieval-Augmented Generation (RAG)

RAG enhances an LLM by providing it with up-to-date, external information at query time. Instead of relying solely on its internal knowledge, the model can access a custom knowledge base to find relevant context for answering a user's question.

Note
The RAG workflow is: User Query → Retriever → Knowledge Base → Retrieved Context → LLM → Formatted Response.

RAG Implementation Steps in LangChain

1. Prepare the Environment

Obtain an IBM watsonx API key and Project ID.
Store these credentials securely in a local .env file.
Install the required Python libraries, including langchain and ibm-watson-machine-learning.

2. Build the Knowledge Base (Vector Store)

Load Data: Use a LangChain WebBaseLoader to ingest content directly from a list of URLs.
Clean Text: Remove unnecessary whitespace and newline characters from the loaded documents.
Chunk Documents: Split large documents into smaller, manageable chunks using a RecursiveCharacterTextSplitter.
Tip
A chunk size of 512 characters was used in this example. This size is small enough for manageable processing but large enough to retain semantic meaning.
Create Embeddings: Use an embedding model, such as IBM's Slate model, to convert the text chunks into numerical vectors (embeddings).
Store Vectors: Load the embedded chunks into a vector database. The example uses Chroma for a local vector store.

3. Configure RAG Components

Retriever: The vector store is configured to act as a retriever (vector_store.as_retriever()), enabling it to search for and fetch relevant chunks based on a query.
LLM: The generative model (IBM Granite) is instantiated using the watsonx API.
Prompt Template: A prompt is engineered to instruct the LLM on how to use the provided context (from the retriever) and the question to formulate its answer.

4. Create and Use the RAG Chain

All components—the retriever, prompt, and LLM—are combined into a single, executable RAG chain.
Invoking this chain with a question automates the entire process: searching for context, augmenting the prompt, and generating a final, informed response.

Key Technologies Used

Framework: LangChain
Language: Python in a Jupyter Notebook
AI Platform: IBM watsonx
Generative LLM: IBM Granite
Embedding Model: IBM Slate
Vector Database: Chroma

The Problem with Outdated LLMs

Hi, my name is Erica and I'm going to show you how to use LangChain for a simple RAG example in Python.

Large language models, LLMs, can be great for answering lots of questions, but sometimes the models don't have the most up-to-date information and can't answer some questions about recent events. For example, I was reading this recent announcement about the UFC and IBM partnership on IBM.com and wanted to ask an LLM about it.

But when I asked the IBM Granite model to tell me about the UFC announcement from November 14th, 2024, it didn't know what I was talking about and mentioned it was trained on a limited data set up to only 2021. How do I give this LLM the most up-to-date information so it can answer my question?

The answer is RAG, Retrieval-Augmented Generation.

How Retrieval-Augmented Generation (RAG) Works

Let me show you how it works. Typically, we have our user asking the question to the LLM, which generates a response. But as you just saw, the LLM didn't have the right information—the context—to answer my question. So we need to add something in the middle between the question and the LLM.

First, we'll add a knowledge base to include the content we want the LLM to read. In this case, it'll be the most up-to-date content from IBM.com pages about some IBM products and announcements.

Second, we'll set up a retriever to fetch the content from the knowledge base.

Third, we'll set up the LLM to be fed the content.

Fourth, we'll establish a prompt with instructions to be able to ask the LLM questions. The top search results from search and retrieval will also be gathered here.

Once we've completed these four steps, we can start asking our questions about the content in our knowledge base. Our query is searched for in our knowledge base vector store. The top results are returned as context for the LLM. And finally, the LLM generates a response.

Setting Up the Python Environment

I'll walk through all these steps again in the Jupyter Notebook, linked in the description to this video.

Before we can begin, we need to fetch an API key and project ID for our notebook. You can get these credentials by following the steps in the video linked in the description below. We also have a few libraries to use for this tutorial. If you don't have these packages installed yet, you can solve this with a quick pip install, and here we can import the packages.

Next, save your watsonx ID and watsonx API key in a separate .env file. Make sure it's in the same directory as this notebook. I have my credentials saved already, so I'll import those over from my .env file and save them in a dictionary called credentials.

Okay, now we can get started with the workflow.

Step 1: Building the Knowledge Base

First, we'll gather the information from some IBM.com URLs to create a knowledge base as a vector store.

Let's establish a URLs dictionary. It's a Python dictionary that helps us map the 25 URLs from which we will be getting the content. You can see at the top here, I have the article about the UFC and IBM partnership I asked about before. Let's also set up a name for our collection, AskIBM2024. Next, let's load our documents using the LangChain web-based loader for the list of URLs we have. Loaders load in data from a source and return a list of documents. We'll print the page content of a sample document at the end to see how it's been loaded. It can take a little while for it to finish loading, and here's a sample document.

Based on the sample document, it looks like there's a lot of whitespace and newline characters that we can get rid of. Let's clean that up with this code. Let's see how our sample document looks now after we've cleaned it up. Great, we've removed the whitespace successfully.

Before we vectorize our content, we need to split it up into smaller, more manageable pieces known as chunks. LangChain's recursive character text splitter takes a large text and splits it based on a specified chunk size, meaning the number of characters. In our case, we're going to go with a chunk size of 512.

Next, we need to instantiate an embedding model to vectorize our content. In our case, we'll use IBM's Slate model. And to finish off this step, let's load our content into a local instance of the vector database using Chroma. We'll call it vector_store. The documents in the vector store will be made up of the docs we just chunked and they'll be embedded using the IBM Slate model.

Step 2 & 3: Configuring the Retriever and LLM

For step two, we'll set up our vector store as a retriever. The retrieved information from the vector store—the content from the URLs—serves as additional context that the LLM will use to generate a response later in step four. Code-wise, all we need to do is set up our vector store as a retriever.

For step three, we'll set up our generative LLM. The generative model will use the retrieved information from step two to produce a relevant response to our questions. First, we'll establish which LLM we're going to use to generate the response. For this tutorial, we'll use an IBM Granite model. Next, we'll set up the model parameters. The model parameters available, and what they mean, can be found in the description of this video.

And finally, in this step, we instantiate the LLM using watsonx.

Step 4: Constructing the RAG Prompt & Chain

In step four we'll set up our prompt, which will combine our instructions, the search results from step two, and our question to provide context to the LLM we just instantiated in step three.

First, let's set up instructions for the LLM. We'll call it template because we'll also set up our prompt using a prompt template and our instructions. Let's also set up a helper function to format our docs to differentiate between individual page content.

Finally, as part of this step, we can set up a RAG chain with our search results from our retriever, our prompt, our helper function, and our LLM.

Step 5 & 6: Testing the RAG System

Finally, in step five and six, we can ask our questions about our knowledge base. The generative model will process the augmented context along with the user's question to produce a response.

First, let's ask our initial question: Tell me about the UFC announcement from November 14th, 2024.

On November 14th, 2024, IBM and UFC announced a groundbreaking partnership... and it looks like the model was able to answer our question this time, since it received the context from the UFC article we fed it.

Next, let's ask about watsonx.data. What is watsonx.data?

watsonx.data is a service offered by IBM that enables users to connect to various data sources and manage metadata for creating data products.

Looks good.

And finally, let's ask about watsonx.ai. What does watsonx.ai do?

watsonx.ai is a comprehensive AI platform that enables users to build, deploy, and manage AI applications.

It was also able to respond to our watsonx.ai question. Feel free to experiment with even more questions about the IBM offerings and technologies discussed in the 25 articles you loaded into the knowledge base.