The Limitations of Basic RAG
Hey everyone, Jerry from LlamaIndex here. And today I'll be giving you an introduction to report generation. So if we think about a basic RAG pipeline, the output application is typically just a simple chatbot. But the issue with a simple chatbot is that it's typically not very useful for your end users. So a simple RAG pipeline can typically only handle very simple questions and output very simple responses. And so when you translate that into an end-user experience, this means that the user is basically treating this tool as a simple search and retrieval tool.
And it also means that they're basically the knowledge worker who's using this tool is responsible for translating the output of that tool into the knowledge work that they're responsible for producing. So this means the agent that's powering this RAG pipeline is not directly producing the output that the knowledge worker is responsible for and at best is just like a simple lookup tool, almost like an internal Google Search or something like that for the existing user.
And so what this means is that in this like kind of naive or basic RAG setting, we're still requiring humans to do most of the knowledge work and therefore not really saving them that much time. We think there's basically limited time savings or limited decision-making enhancement in giving the end user like a basic search and retrieval tool. And that the potential for GenAI or agentic applications in general is that you should be able to actually do things that actually automate entire chunks of what the knowledge worker is able to produce, therefore actually letting them be, you know, not just like 1 or 2% more efficient, but like 10, 50, even like 2 to 300% more efficient.
And so the question we ask ourselves is how do we automate more decision-making and actually create knowledge-powered agents that can, you know, produce reliable, consistent outputs for the end users and therefore make them basically supercharged? They can do a lot more things in a shorter amount of time.
Introduction to Report Generation
The answer to this is of course something around agents. And so there's a lot of hype around building agentic applications these days, especially with state-of-the-art models getting better and better at multi-step reasoning. We think that, you know, knowledge assistant agents should have the capability to not only generate just simple chatbot responses, but also produce entire units of knowledge work, for instance, producing like an entire PDF or slide deck for you or taking actions.
And so this means that instead of the human having to go into, for instance, like a like a document editor like Google Docs or Microsoft Word or having to click a few buttons in a UI, the agent should increasingly be able to do this for the user and therefore allow them to automate a lot of rote or routine tasks. And so we think that this overall pattern of going more towards multi-step reasoning, action-taking, and output generation can potentially lead to much greater ROI in terms of time savings and capability improvements.
Some of the top use cases that we see include, you know, report generation, data analysis, and action taking. And for the purposes of this video, we'll talk about report generation. Just as a very broad concept, we think that report generation is a very natural next step for any sort of agentic RAG application or adding an agentic layer on top of an existing RAG pipeline. And it turns out it's also one of the top Enterprise agent use cases that we see among our users and customers within our community.
Common Patterns and Use Cases
So anywhere from banking, finance, to consulting, to legal, to tech, to manufacturing and other industries, we actually see a lot of common patterns around, you know, often times a company will start with building a very basic or simple RAG pipeline, but then they'll generalize beyond that. Once they get some of the basics down, you know, they're able to index a bank of unstructured data and start generating simple Q&A over that data. The next step is typically instead of just, you know, giving back or giving the user a simple search tool, actually starting to synthesize insights in a more end-to-end fashion and basically producing entire outputs for the knowledge worker.
As we mentioned, report generation is a pretty broad category, and the overall idea is that you can also call this artifact generation, generating like an artifact or a unit of output for the end user. And so some sub-patterns that we see include generating, for instance, like an entire research report or presentation. So for instance, generating like a survey paper over existing documents or generating an entire slide deck where you know, maybe instead of, in addition to actually figuring out the content, you also have to figure out the structure and the overall layout. Other use cases include filling out an example form or questionnaire. This is typically pretty common in many kind of routine tasks where you have some existing bank of context and you want to use that to actually fill out, you know, it could be anything from a tax form to like a due diligence questionnaire. There's some sort of input template that defines what are the specific sections that the worker needs to fill out, and you basically need to leverage both your prior knowledge as well as existing company-specific contexts to fill that out.
This also includes more tabular settings, like being able to fill out an entire Excel sheet and potentially doing that end-to-end from an existing bank of context. So instead of, you know, an analyst directly opening up Microsoft Excel and filling in every single value based on, say, reading only 10K reports, an agent can do this automatically end to end.
Industry Examples: ChatGPT, Claude, and Cursor
We also see this in emerging examples and existing tools that everybody uses. So, you know, ChatGPT recently released this this feature called Canvas. So OpenAI has this Canvas feature where for everyone using ChatGPT, you know, you typically use it in like a simple chat, user chat, get back response format. But the Canvas tool is basically a layer on top of it where instead of just generating, again, like a chatbot response, it gives you an entire output or report that you can continue to interact with, edit, and refine. And you can basically go into specific sections, for instance, here, and actually give feedback on very specific sections, and you basically collaborate with the AI to generate this entire unit of output. And this is pretty much an example of report generation.
It's very clear that, I mean, I'm sure OpenAI and Anthropic have also worked with a lot of enterprise customers that have these exact use cases. Claude has had this as well. You know, I'm a heavy user of Claude, and if you just look at, for instance, some of the recent updates, you're able to—it basically is very eager, especially with the new Sonnet models, in generating entire outputs for you, whether it's writing like a markdown report or writing code. And you can continue to select and work with it to edit and refine different sections.
We focus on now these next sections a little bit less, but this also exists in code. For instance, like if you if you're a Cursor user, you know, you can basically generate entire units of code and actually have the right UX to basically directly refactor your existing code directly in your editor. And there's basically some back and forth interactions between what the AI generates and how it plugs into your existing workflow. So we'll talk about some of this in future videos too.
Component 1: Structured Outputs
But, you know, there's some core components of a report generation architecture, and today we'll go over some just very basic building blocks of what a report generation architecture looks like. And then in subsequent videos, we'll go over some tutorials and how do you actually leverage some of these components to build both like basic to slightly more advanced report generation workflows.
The first concept is structured outputs. And structured outputs is basically a foundational concept for anyone trying to build or develop with LLMs and build applications. It's basically the idea that instead of prompting the LLM to output something that's an unformatted output, prompt it to output something that's more structured in nature. And so something that follows a certain predefined schema. And this is important because if you think about most versions of reports or artifacts, they basically require you to adhere to some structure, whether it is something that's super explicit like filling in, you know, every single cell in an Excel sheet or, you know, being able to generate even like a markdown report. There will be like a heading, different sections, there might be like interleaving text, tables, and images.
And so if you look at this example right here, this is just a very simple schema that defines what a sample report output could look like. There's a top-level report output class, and underneath it there's text blocks and also image blocks. And basically a report output can contain interleaving text and image blocks. And this is just one of the examples we'll go over in one of the subsequent tutorials. And the result is that you're able to generate this entire output with text sections, and then there's images in the middle, and then more text after that. And there's other versions of this, of course, but fundamentally, you know, you want to be able to prompt the LLM, probably through like function calling capabilities or through regular text prompts, to basically output something that's in a structured format.
Component 2: Knowledge Base
Number two is having a knowledge base. And so, you know, most instances of enterprise-ready report generation require you to have an awareness of private context. And so to to basically be able to access this context, you basically need to create some sort of search or Google-like interface into the knowledge base to allow any sort of agent to look up relevant information from this knowledge base.
And so there's different ways, you know, LlamaIndex of course has a ton of tools across both Llama Cloud and the open source to help you index a bank of multimodal data and retrieve from it. But basically, we help—we'll show you in subsequent videos how you load in a bunch of unstructured documents and define a different set of like retrieval endpoints on top of it so that when you're building like a report generation agent, one of the steps is typically research. And this research step can call the retrieval endpoints to pull and look up the correct context to look up the information that you then feed into the rest of the agent loop.
Component 3: Multi-Agent Workflows
The third is building or defining some sort of multi-agent workflow. And so a multi-agent workflow is something that you can build using our core abstraction within the open source framework. It's called LlamaIndex Workflows. And this allows you or gives you a very flexible agent primitive to orchestrate, you know, any sort of agentic steps in an event-driven manner. And this will allow you to build something like this type of flow where you're able to break this overall report generation task into independent agent steps, for instance, having like a researcher plus a writer, and then having every step solve that task before moving on to the next one.
Component 4: Template Parsing
And then another piece here is template parsing. So if you think about what the inputs to a report generation task are, you know, it could be through natural language. For instance, if you go into ChatGPT, you can just say something like, "Oh, please generate this short story for me," or, "Generate like a research report," just in a very simple manner.
But besides just parsing an explicit natural language user task, oftentimes, and this is something that we've also discovered through conversations with customers, is that they you want to input almost like an existing document template. And this template basically is the input that forms that defines the overall structure of, you know, what is the report that you want to generate. For instance, if you input like a questionnaire, then the output of course needs to be a set of answers that that actually answer every question in the questionnaire. If it's like a template or, you know, just class instructions, for instance, you need to generate a report that actually adheres to the guidelines that this template that this template defines. So or it could be an Excel sheet, like a blank Excel sheet, and of course, you need to fill in the cells of this Excel sheet. So it's a pretty important step, and we'll start baking this into some of the later tutorials.
Component 5: Human-in-the-Loop
There's many other components too. And you know, this again is just the core set of building blocks that we believe are important to actually help users to build, you know, some sort of report generation agents. In the end, something that is production-ready will will likely require very custom components that are likely more advanced than some of the core things that we show you here.
But the last step that we'll talk about is just human in the loop. And this is something that increasingly as workflows are more agentic and automate more and more steps, then there's a lot of value in letting the human actually inspect what's going on and give feedback in both an implicit and explicit manner at every step of this multi-agent workflow.
So implicit feedback would be, for instance, giving back some natural language unstructured input back to the agent, and the agent then can have the freedom to actually translate that feedback into certain steps that it will take, you know, to to basically try to take the human feedback into account. Explicit feedback is direct. The human actually injects, you know, direct control over either the agent decision-making or over the generated output. So for instance, it can say, you know, please throw away this information and actually have a toggle, and then it's not an LLM figuring out, you know, what to do. It's actually just directly throwing away this this information in in the control flow of the program. It could also mean the user directly injects an edit into the output report. And this doesn't mean that it's up for interpretation; the human literally directly edits the report, and then the AI just has to take that into account.
And so this requires some sort of back and forth interaction that you can model through something like an orchestration framework like Llama Workflows. And we'll show you some initial examples of this towards some of our later videos.
The LlamaIndex Stack for Report Generation
So in some of the subsequent videos, the stack will typically consist of LlamaIndex Workflows. So as I mentioned, it's a core agent orchestration framework that's very flexible. It'll leverage some of the pre-builts that we have in the framework, but fundamentally basically lets you define any sort of event-driven async workflows over your data and gives you the ability to step through it, debug it, and inspect what's going on.
There's also, uh, some of the videos will take advantage of Llama Cloud, which is which is our enterprise end-to-end advanced RAG platform. It will let you easily connect a set of data sources and give you a set of persistent indexes that you can query. And then some of the some of the use cases will also use Llama Parse. And so Llama Parse is our state-of-the-art advanced document parser, really really good at text, tables, and images. It's a part of Llama Cloud, but it's actually available for everybody to sign up and start using today. You get a thousand free credits per day, and then you also can upgrade to the Premium plan if you want more. And so the set of tutorials will use these three components.
Resources and Conclusion
And here is just a general list of resources to to get started and also start building our workflows. If you don't want to wait for some of our future videos to come up, we basically just linked a core set of Jupyter notebooks that you can directly start using. And some of these, you know, again, are based on Llama Parse, some of these are based on Llama Cloud, but they basically showcase different examples of report generation.
So that's it. Thank you all. And so if you want to sign up for Llama Cloud, just scan this QR code right here. Again, you get immediate access to Llama Parse, and then the overall Llama Cloud platform will be is is currently in a waitlist state. And then if you're an enterprise looking to build enterprise-grade RAG and agents, please get in touch on the QR code at the left. So thanks and see you see you all next time.
