Introduction to the Talk and LlamaIndex
Uh hi everybody. Uh you probably met me this morning uh when I greeted you all to the conference. My name is Lori. I'm VP of developer relations at Llama Index. Uh today I'm going to be talking about Llama Index and what it is uh very briefly because I've only got 15 minutes. uh and then we're going to talk about agents and how they are built plus a very very brief refresher on RAG uh and why it's necessary for agents. Then we're going to look at some highlevel uh design patterns that improve the performance of your agent and we won't have time to dive into how to build an an agent as well as a multi-agent system because we've only got 15 minutes so let's go fast. So what is
Overview of the LlamaIndex Ecosystem
Llama Index? We are a bunch of things. Start with the most obvious. We are a framework in Python and Typescript for uh building generative AI applications. We are particularly good at building agents. We also have a service called LlamaParse. Llama Parse uh will parse complicated document formats for you. So PDFs, Word, PowerPoints, those sorts of things. These are crucial to uh building an effective agent uh is being able to parse your unstructured data in an effective way. and LlamaParse will uh demonstrably improve the quality of the agents you build by making the data easier for an LLM to understand than if you just try and uh feed these things in through an open source parser or something like that. Uh we also have an enterprise service called LlamaParse, sorry LlamaCloud. Uh if what you want to do is stuff documents into one end and get a retrieval endpoint out of the other, uh then that is the service for you. Uh unlike the rest of Llama Index it costs money. It is available as a SaaS at cloud.llamaindex.ai or you can get it deployed uh onto your own private cloud. Uh we also have a website called Llama Hub uh which is a huge registry of open-source software uh that plugs into the framework that integrates with everything. So if you need to get your data out of Notion or out of Slack or out of any database in the world, that is where you find the adapters. If you want to store your data in any vector database that exists, the adapters exist for that. And we also integrate with every LLM that exists.
Why Use LlamaIndex and What to Build?
So, uh, 400 different models over 80 different LLM providers, uh, including local ones like Llama 3. Um, oh, and it also has pre-built agent tools. So, if you are building an agent, you can just plug in an existing agent tool without having to build one yourself.
Uh, why should you use Llama Index? Because we will help you go faster. That is the base promise of a framework. Generally, uh, you have actual business and technology problems to solve and you have limited time. A framework is going to help you uh get past those with uh by skipping the boilerplate, getting best practices for free uh and getting to production faster in general.
Uh what can you build in Llama Index? Well, anything obviously, but there's two things that we are particularly good at. One is retrieval augmented generation. Uh and the other one is agents. Uh both of which you are uh probably familiar with already from uh just being at this conference because we won't shut up about agents at this conference.
Understanding Agents and Their Applications
Uh it's worth covering why you would want to build those things though. Uh so what is an agent? An agent is uh a dramatically overused term in the industry. Just about everything is an agent in 2025. Uh but what I mean when I say an agent is it is a bit of semi-autonomous software that can use tools to achieve a goal without you having to explicitly specify what steps it's going to take to achieve that goal. Uh and the tools can do anything which is great. They can take retrieve information or they can take action. Uh agents are a dramatic departure from traditional programming. Uh LLMs are given decision-making power to decide what tools to use and stuff like that. That makes them extremely flexible and powerful. So the time to build an agent is when that flexibility, that uh ability to deal with the unexpected or the unknown uh is going to come in useful. And when that's really useful is when you have a a bunch of unstructured data. Uh LLMs are extremely good at handling messy inputs and the world is full of messy inputs. So there's a whole lot of applications for LLMs. Um in general, I regard a good agent use case as any situation where an LLM is required to turn a large body of text into a smaller body of text. I think that is a key principle of agent design and LLM use in general is that they are not good at taking a small prompt and turning it into a big body of text. They are very good at summarizing stuff down.
So, interpreting a contract, processing an invoice, uh applying regulations, summarizing documents, uh a thing where you need to turn text into less text. So, a calendar event, a decision or report, an answer to a question. Those are good applications for LLMs and good applications for agents. The most obvious application of LLM is a chat interface where you give it questions and it answers. I encourage you to think beyond the chatbot. Uh chat bots are a very 2023 of way way of using an LLM. uh we believe a much greater addressable surface is if you integrate them into existing software. So you use the LLM's capability to handle messy inputs to handle unstructured data and turn it into structured data that it can then feed make decisions about and feed into your regular software. That is a really productive and powerful set of use cases. Uh and we think it's much bigger than the market for chat bots.
The Necessity of Retrieval-Augmented Generation (RAG)
Um, I've talked a lot about unstructured data. The reason I talk about it is because that is where LLMs become really useful. Um, unless you are building something extremely generic, uh, you are not going to be able to get anything useful out of an LLM by just asking it questions, you're going to have to give it contextual data relevant to your company, to your domain, to whatever problem set it is that you are working on. Uh, so you have to feed the LLM your data. And the problem is that you have tons of data. And this is the use case for RAG. Uh you take all of your data, you embed it, which is turning it into vectors that you can search for in a vector search. Uh and the you can then take your query, take your questions about your data uh and turn embed them in this into the same vector database and they will end up mathematically nearby in vector space to the context that you fed in that is relevant. So instead of having to take all of your data and feed it to the LLM every single time, which would be tremendously slow and tremendously expensive, you can just feed in the most relevant context out of your data corpus. Uh and uh answer questions about that. Um this is why RAG will never die. People keep talking about like larger and larger contexts and you know more and more powerful models. Uh it's always going to be cheaper and faster to send less data that the LLM has to think about less. it's always going to be uh you're always going to get more answers better answers if the context that you have given your LLM is uh more specific.
The Symbiotic Relationship Between Agents and RAG
uh so agents can use RAG as one of their tools but also uh um agents so agents need RAG uh but RAG also needs agents. RAG by itself, naive top-k RAG where you just you know throw in a query and retrieve the uh retrieve the most relevant context and feed that to the LLM, that's not going to work very well for a variety of situations. But what we've found through lots of production use cases is that layering an agent on top of your RAG uh will produce significantly higher quality results. Um and they're capable of doing things that RAG just can't do. RAG is a simple question and answer uh robot whereas an agent can do stuff like introspection like could this complicated question be answered more easily if it were a series of simpler questions. uh do I need to try extracting that data again because the data that I got out is nonsense. Did I just give a sensible answer to your question or should I try again? That is something that an agent can do. It can look at its own responses and improve itself which RAG by itself obviously doesn't do. Uh so agents improve the performance of RAG both in terms of speed uh and crucially in accuracy.
Design Pattern 1: Chaining
Um, in December of last year, Anthropic did an excellent post about how to build agents, uh, in which they codified, uh, some design patterns about how to build an effective agent that we immediately recognized, uh, from our own work, uh, building agents. So, I'm going to go through them very quickly, and given the amount of time I have, that's probably all I'm going to be able to cover. Um, they are chaining, routing, parallelization, orchestrator workers, and evaluator optimizers.
Um the first and most obvious is the chain. Uh you can use an LLM to do some work and you pass the output of that to another LLM and you pass the output of that to another LLM. Uh it is trivial to build uh especially in Llama Index. Uh this is what uh a chain looks like in Llama Index. We use uh an abstraction called workflows. Uh where you define regular Python functions that do whatever you need them to do. uh and you use event annotations uh to define how you sorry you use type annotations to define how events pass from step to step within a workflow. Uh it is a very simple and flexible pattern uh that our users like a lot.
Design Pattern 2: Routing
Um so llama index workflows have a built-in visualizer the output of which you can see here. Uh this is obviously a chain. Um there is much much more to LLM applications than a chain though despite uh what the names of some other frameworks might indicate.
Um the next pattern it called out is routing. Uh in this one you create several LLM based tools uh to solve a problem uh in different ways or to solve different types of problems and you give the LLM decision-making power to say which of these tools should I call which of these different LLM paths should I follow? Uh again, not that complicated a a concept and simple to build uh in Llama Index. I'm going to uh spare you the code. This time you can do it using uh branches. You can just decide that you're going to split off into your own chain uh and do a series of uh do another series of work uh based on the original decision made by the LLM.
Design Pattern 3: Parallelization (Sectioning & Voting)
Um the next pattern is parallelization, which is where things begin to get interesting. Uh Anthropic defines this as running several LLMs in parallel and then aggregating their results. Uh and they define parallelization as having two flavors. Uh the first is sectioning. Um this is where you take the same input and you act on it in completely different ways. The uh sort of canonical use case of this is guard rails. So the user has a query or a piece of input that they want processed uh and you use one of your tracks to actually process the data or to answer the query and you use the second one of your tracks to query is this an illegal request? Is this against my rules? Is this those two qu those two questions are related but they can be answered in parallel? And you can use your guardrails to cut off the answer uh from your processing if it turns out to be uh you know illegal or otherwise undesirable.
Um the other par flavor flavor of parallelization is voting. This is where you take exactly the same query and you give it to uh three different uh three different tracks. The tracks can be literally exactly the same LLM because they are non-deterministic. So it might not give the same answer every time or you could give it to multiple different LLMs which have different capabilities and different specialties. And then you take the answers and you allow them and you see if uh the tracks came to the same answer. You can take a majority vote. you can take a unanimous vote. And what this does is it allows you to uh limit the amount of uh uh hallucination that is happening. Uh it's a great way of reducing hallucination because if you know three different LLMs come to the same conclusion, then it's probably not the LLM just making stuff up because LLM's hallucinate, but they hallucinate in different ways. So they seldom hallucinate to the same answer.
Uh whichever flavor you use, it's implemented the same way uh using concurrency. Um we uh Llama Index workflows allow you to emit multiple events simultaneously uh and then collect those events uh at the other end. So you can do you can do work concurrently.
Design Pattern 4: Orchestrator-Workers
and uh yeah and it I think the visualization here is particularly pretty. Um the next pattern is orchestrator workers. uh you can use an LLM to look at a complex task like a multi-part question uh and split it into several simpler questions and ask each of those questions in parallel. So this is how deep research works basically. Uh it takes a very complicated question. It says I'm going to I'm going to look at all of the possible questions that could come from this deep question. I'm going to answer them all at the same time and then I'm going to aggregate all of the answers that I've got and turn them into one single coherent answer. This is a very powerful pattern that is uh doing a lot of good in the world right now. Uh this is also implemented using parallelization.
Design Pattern 5: Evaluator-Optimizer (Self-Reflection)
Um and the final pattern that Anthropic called out is the evaluator optimizer which is also called self-reflection. Uh in this pattern you use the LLM to decide whether or not the LLM has done a good job. So uh you take your output, you feed it to an LLM and you say here was the original question or here was the original input and the goal that I had. Have you actually reached the goal that I had? Uh and if not, you can get the LLM to generate feedback and send it back uh to the original first step and say okay you almost got the answer but you hallucinated something or uh you missed a part of the question or you know something like that.
Implementing Patterns & Multi-Agent Systems in LlamaIndex
Um this is again easy to do in Llama Index. In workflows, you just create a loop uh and you can send yourself back to the step one.
Um and the real power here is obviously combining all of these patterns. You can create arbitrarily complex workflows uh to handle any combination of circumstances.
Um right near the beginning, uh when I was defining an agent, I have 60 seconds left, so I'm just going to go through the syntax super quick. Uh I said it was that agents are defined by their ability to use tools. In Llama Index, this is what a tool definition looks like. It is just a Python function that you have wrapped uh in a step wrapper.
Um, and the way that you use your tool function is you just give it to an agent and the agent will figure out that it is a tool function and start using it. Uh, this allows you to create workflows that are multi-agent systems. I do not have time to uh explain how multi-agent systems work for uh this but this is how you create a multi-agent system in in llama index. Uh you create uh a function agent which gets takes a system prompt. It takes an LLM uh and it takes uh a set of tools and you can feed uh an array of agents into a multi-agent system which then just sort of figures it out by itself passing control back from one agent to another. Uh this is technically one line of code and we're pretty proud of it. Uh and that is about it. Um if you want a full agent workflow and workflows tutorial, this was the simplest possible one. The it is available uh at that this notebook will teach you how to build a deep research of your own. Uh and with that, I am pretty much out of time. If I if you have any questions, I will be outside in the hallway. Thank you very much.
