Introduction to Hybrid RAG & Text-to-SQL
Hey everyone, Jerry from Llama Index here. And today's topic will be on combining RAG and text-to-SQL in a single query interface. And so a common use case that a lot of people have is you not only have just unstructured data, but you also have some structured data. And you want to basically create some sort of QA system that can pull information whether from unstructured data or structured data dynamically to basically solve any sort of user task. And this goes towards this goal of building a more general knowledge assistant where you can plug in different data sources into the same interface.
So this is a pretty straightforward tutorial, but basically this notebook shows you how to create a custom agent that can query either your unstructured data through a Llama Cloud index for RAG-based retrieval or a separate SQL query engine as a tool. And so in this example, we'll use PDFs of Wikipedia pages of US cities. It's a pretty simple toy example along with just a SQL database of their populations and states. And again, you know, this obviously in an enterprise setting, there's going to be more complex cases and oftentimes the SQL database itself can be quite complicated, but this hopefully gives you a sense of how you can basically create a general agent architecture that can query both sides.
So at its core, it's basically just an agent loop with a router. So given some sort of user input, we are going to to feed this into some router. And actually this router is implicit in the agent reasoning loop. The agent, reasoning through tool use, will decide whether to query a collection of structured data or to query a retrieval tool that's based on a vector database of unstructured documents. If it decides to query the structured database, then we're going to have to run text-to-SQL to basically convert a natural language to a SQL query to then query the SQL database and get back the relevant rows. If it decides to query the RAG tool to basically query the vector database, then it'll return relevant unstructured context from the database, pull that in, and depending on what tools you call, you feed in all the context into the LLM prompt window and you generate a response.
Security Considerations for Text-to-SQL
Just a quick disclaimer is that obviously, for running stuff like text-to-SQL and running anything that requires some degree of code execution, you have to be aware that there's some degree of like security risks. You basically need to make sure that you're not, for instance, like doing things that will cause you to delete data or drop a database or, you know, worse, kind of execute arbitrary types of code on your device. So you basically just need to make sure that you're executing this in a secure environment. So let's get started.
Ingesting Unstructured Data with LlamaCloud
The other piece here is that, you know, we also have integration with Arize Phoenix, and we're going to be logging our traces and showing you how they show up in LlamaTrace. LlamaTrace is basically the joint name that is basically just hosted Arize Phoenix that has a really nice integration with LlamaIndex.
We'll start by basically loading some documents into LlamaCloud. If you haven't done this exercise, all you have to do is basically download the following Wikipedia pages, print them into as PDFs, and then basically upload those PDFs to LlamaCloud. So here's an example of the Wikipedia page for New York. All you have to do is, you know, save it to PDF and then create an index in LlamaCloud right here. For those of you who don't have access to LlamaCloud yet, you can certainly request access by adding yourself on the wait list. And of course, there's also a purely open source way to do this too.
But to basically, all you have to do is click files, upload all your files, select your data sync. You can configure whatever vector database you want, and then configure the chunking options if you want or not. And basically you get back your index of cities. So going back right here, we're going to go into our already created index of cities, and you can see we uploaded all these files. And then it goes through our ingestion... ingestion which includes parsing, chunking, and indexing. And you can see the intermediate outputs throughout all the stages. So here you can see the parsed outputs, which parses out this doc into Markdown. And here you see the chunks where by default we basically do page-level chunking on the PDF. So you can actually validate the capabilities of this by, for instance, going into the chat playground and, you know, for instance, if you ask a question over the Wikipedia pages like 'museums in New York City,' you ask this question and then you'll be able to get back a chat response. And you can see the sources too. Okay, let's go back.
Creating the Structured SQL Database
And then um the other thing that we're going to create besides the database of unstructured data is the SQL database. And the SQL database in this example basically is just a toy database that contains the the cities along with their populations. And so it's a pretty simple example of structured data. And by storing this in a structured manner, you ideally are able to answer questions over a large quantity of structured data through stuff like SQL that would be hard to do if you were to directly query a vector database.
For instance, if you were to look at like the average population across all the cities or the max population, that type of stuff, this would be a lot easier in a SQL database. You can see we use a SQLite, and a SQLAlchemy wrapper on top of that, and we basically just inject a bunch of the cities along with the population information into into this database.
Next, we are going to define that wrapper or the integration with the LlamaCloud index, and we are going to wrap both the LlamaCloud retriever as well as the SQL database as tools. And these tools will then be given to an agent to call, and this agent will decide which tool to use to basically answer the question.
Building the Agentic Workflow in LlamaIndex
Now that we've defined these tools, we are going to create an agent around the two query engines. Because we already defined them as tools, we basically just need to write some sort of agent reasoning loop that can call these tools and then continue iterating and adding the information to chat history until the LLM decides that, you know, it's done and it's fully answered the question.
And so all the steps that are listed here are basically the steps you need to implement what we call like a function calling agent. If you're familiar with like a ReAct agent or a function calling agent, it's basically a sequential process where given some input task, you call the LLM with the set of tools, you rely on the function calling capabilities of the LLM to decide what tools to call along with which inputs. By inferring the inputs, you implicitly do Chain of Thought by being able to break this complex question down into the subquestions or the tool calls that have more simple inputs. And then the LLM will be able to repeatedly and sequentially call the tools until it's able to fully answer the question.
So here we are creating a workflow that basically acts as an agent around these two tools, and it consists of the following steps. You know, you're able to kind of continuously append the message to the chat history, call these tools, gather the results from the tool calls, and then feed it back in until there are no more tool calls to be had. And this is defined via a LlamaIndex workflow. We've covered this in some of the other videos too, but I'll basically just jump straight to it. And you can see that the input takes in a set of tools. And there's basically just like four main steps. One is you know, given whatever the input is, we are going to add the first user input to the chat history. And then we are going to go into this loop. Given the input event, we will call llm.achat_with_tools, or its async version of making an LLM call with tool use. So we pass in the tools, we pass in the current chat history, and then we look at what the chat response is. Specifically, we try to get the tool calls, see what tools the LLM has decided to call given the inputs and the tools that we've given it. If these tool calls are non-zero, we are going to actually call these tools through the downstream events here. If there are no more tool calls to be done, then we're done with the agent reasoning loop and then we stop. If there are tool calls to be called, we call dispatch_calls, which will look at all the tool calls to be done, and then we emit a tool call event for every single tool call, and this will actually execute the tool and then get back the output of the tool and append it to the chat history. call_tool is the thing that's responsible for executing the tool. So here await tool.acall and then we get back a chat message. After calling all these tools, we basically collect all the events of the tool call event results you see here. You know, we pass this, which passes it to gather. And then once all the tools are finished executing, we append the tool call inputs and outputs to the chat history, and then we basically restart the process. And basically this reasoning loop through the LLM function calling capabilities will keep iterating through this process until it's done. If it's decided that it's answered all the questions in the chat history, then we'll no longer call these tools. So this is basically an example of... it's like a fancier version of this router that's described right here, right? Because a router, the most simple version is a single-shot prompt where you just take in some input and you decide what choice to actually take given that input. Here, we actually wrap this instead of a single-shot prompt into almost a while loop of sorts. We'll be able to first decide, you know, what tools to call, and then before actually finishing, we're going to go back into this loop to decide if there are other tools we need to call, and only when there are no more tools to call do we decide to return a response.
So let's decide to plug this in to this workflow. We are going to look at what this workflow is overall. And you can see, you know, there's some start event, prepare_chat, and it goes into this core chat loop. If there are tools to be called, it'll call every tool, call the tools and then gather, and then finally stop.
Querying the Hybrid System
And finally, we are going to execute some sample queries and just see, you know, how the overall agent, where the agent reasoning loop takes us. So let's first ask, 'What city has the highest population?' You know, recall this is information that's basically found in the SQL database. And so we are going to go into this right here and see that basically it's calling the input event, made a tool call, calling gather tools. And we, in verbose mode, you can see it's calling the function sql_tool with message 'SELECT city FROM city_stats ORDER BY population'. So it figures... this means that when you pass both tools into the agent, it decided to call the SQL database to actually answer this question, probably because they knew the SQL database was oriented around population stats.
And finally it gives you back the final answer. Let's try another example and maybe let's just do like, 'List all the places to visit in Miami,' right? And you can see here, you know, it's running through the same process, gathering tools. And instead of calling the SQL tool, this time it's calling 'places to visit in Miami'. And so this is actually querying the, you know, retriever or vector database that's implicitly hosted by LlamaCloud. And then here is the final response right here. If you wanted to, you could actually try doing something that's comparing between two cities. And so for instance, like let's say 'List all the places to visit in Miami and Chicago'. So let's try this.
You see, it's actually doing parallel tool calling in this case. When you ask the LLM to make a tool call decision, it's outputting both 'places to visit in Miami' and 'places to visit in Chicago'. then it's actually aggregating two tool calls. There's two tool call event results, and then finally giving you back a chat response that combines both.
Tracing the Agent with Arize Phoenix
Last but not least, you can basically see all these traces logged in Phoenix. And so if you actually go into this trace right here, which lists 'all the places to visit in Miami and Chicago', you can see all the steps of the workflow. So there's prepare_chat, there's the chat call. You can see here is the achat call, right? And this actually stores basically the entire history of conversation. 'Which city has the highest population?', 'list all the places to visit', so on and so forth. And then for this call specifically, you know, we are going to call the different tools. Here, basically places to visit in Miami, and here places to visit in Chicago. If you go all the way down into the weeds, you can actually see the raw LLM calls, you know, where it's actually answering this question. And you can see it's doing RAG over this data, and then trying to answer that question. It gathers all the results and then finally does a final call right here where it's basically feeding in this entire chat history and giving you back a response.
Conclusion & Next Steps
That's it. And then, you know, there will probably be other advanced RAG topics, and besides that, we're also... we'll also release some videos on report generation. So thanks and see you next time. And then if you want access to LlamaCloud, please check the links right after this. You know, we're adding a lot of people to the wait list and hopefully we'll make this public soon. So thanks and see you next time.
