Introduction & Live Demo
Hey, this is Lance with LangChain. We've seen that research and summarization is one of the most popular use cases for agents. So I want to show you an agent that I've built that performs automated research and summarization for me using local LLMs. So all the LLM calls necessary to run this assistant are free to use.
Now I want to show you how it works first, and then I want to actually walk through the code and build it from scratch. So right now I'm in LangGraph Studio. This is our environment for testing and debugging and working with agents. If you follow the quick start, you'll spin this up in like three steps, it's very simple. Go to the configuration, import any local LM that you've already pulled using Ollama. So in this case, I'm going to try Qwen 2 14B here. You can set the number of research iterations that you want the agent to take. Now I'll talk about this a lot in detail later, but for start, let's just set two iterations. I can add a research topic. Let's say I want some quick notes about how Qwen 2.5 was trained. I just hit submit.
So it's going to run whatever local LM I specify to generate a search query based upon the topic. We'll run web research, and then Qwen in my case is going to summarize the output of the research. It'll then reflect on that summary and find any gaps, ask a follow-up question, do more research on my follow-up, add the follow-up notes to my summary, and it will reflect. And this will continue as many times as I configured I want it to repeat.
So this is performing the final summary in my case, and now it finished. I can scroll down. Looking at the final summary, you can see a kind of nicely formatted Markdown document with a summary, a bunch of nice notes that I obtained from web research, a nicely formatted list, and a set of sources I can go dig into. So this ran in about a minute. It was all free because I ran it with Ollama locally, and this is a really nice tool I use to automate web research and summarization really quickly and free.
Motivation and Iterative Approach
Now I want to spend about a minute talking about the motivation for this, now that we've seen it in action, before I walk through all the code from scratch. So first, we put out a survey about a month ago where we asked around 1300 professionals in the industry top use cases for agents, and research summarization came out as number one. So it's a huge use case, people are very interested in this.
And the approach here is inspired loosely by some interesting work that I've seen. This paper in particular talking about iter-RAG is cool. The main idea is this: iteratively retrieve documents, generate answers, and continue to generate new research focuses based upon what it discovers from prior loops. Now, why do I do this here with local models particularly? I found that this is actually a really nice approach for local models because given local resources on my machine, I can't run, for example, five or six calls to a model simultaneously. So I can't do a research approach where I like take a topic and fan it out into 10 subtopics and run all those in parallel; that won't work given my system limitations. So instead, I like this iterative approach where I just very simply take the topic, generate a question, do web research based on the question, get something back, reflect on it, do it again, run that for some number of cycles, iteratively update the summary. I found that approach is really simple and works really nicely.
Now, let's talk about local models very briefly. Now for choosing models to run, I like to poke around Local Llama, I like to poke around Twitter, of course, and I like this Hugging Face local LLM leaderboard. You can set the number of parameters here and you can look at models rated by their performance on a bunch of different challenges. I've been working mostly with Qwen 2.5 and Llama 3.2.
Building From Scratch: Agent State Definition
Now as noted in the readme, all you need to do is just run ollama pull whatever model you want and it'll pull it down locally. Now, so now I'm in a fresh notebook and let's build this from scratch. I'll show you how easy it is to put this together. So first, specify whatever Ollama model you want to work with. In this case, I'll use Qwen 2 5 14B. I've already pulled it.
Now second, we're going to use LangGraph to build this agent, and so I want to show you how to define your state for this particular use case. So LangGraph uses the notion of state to basically preserve anything you want to live over the course of the lifetime of your agent. In this case, there's a few things we're going to want. One is we want the user to provide a research topic, we saw that previously. So that's one state key that's going to be a string. Now that is the only thing we want to expose to the user, and so I'm going to create a particular state called my input_state for that key, and you'll see why this is important later. The only thing we want output is a summary, again, that's just going to be a string. We only want the research to be exposed to the user on the input and the summary on the output, but my assistant itself needs a bunch of other things. So we're going to track a web search query, a collected set of web search results which we'll basically append to over the course of our agent, a set of sources gathered—again, we're going to append to over the course of our agent. Only difference here is one is going to contain all the content for web search, the other is going to only contain the URLs.
Defining the Agent's Graph and Nodes
So now let's build and view the overall graph that our assistant's going to use. So it's going to have a few different nodes. First, it's going to start with that input from the user. It's going to generate a search query, it'll run web search, summarize results of web search, reflect on the summary, repeat this process for some number of cycles and then finish. Pretty simple. Let's walk through each node in detail.
So first, we're going to run generate_query. For this, we're really simply just going to use a query writer prompt that's up here, query_writer_instructions. So here all I'm doing is I'm going to ask an LLM, in this case my local LLM, generate a targeted web search query. The query will gather information for web search and the rationale for that query to really force the model to think about the generation of the query itself.
Now for this, I'm going to use the LLM that I configured using Ollama to use JSON mode. I invoke it very simply with my formatted system prompt which passes in the topic from the user, and I instruct it, generate a search query. I get the query back out and I extract the query key from the structured output.
Implementing Web Search with Tavily
Now I'm going to perform web research. Now in this particular case, I'm using Tavily search to do my web search. Now again, I mentioned I like Tavily, it's free to use up to 1000 requests and it provides nice results with already scraped sources, so it saves a lot of work. This particular node could be replaced with whatever search you want to use. The key point is it does a web search, returns the results from search to this sources_gathered key in my state. That's all that happens. Now what's nice is this key in my state is going to accumulate a list of sources as the agent runs, and so all my search results are going to be saved to this key and I can go inspect all of them if I want.
You'll also notice I run this deduplicate_and_format_sources. This is just a utility function. You can see it in the utils folder of the repo, and it just does deduplication of the sources and cleans them up a little bit, turns them into a string. Now what's nice is you could just modify this deduplicate_and_format_sources if you want to change the search engine. So all you need to do is just modify this function to return a string that formats the search results from the raw output from whatever search engine you use. So this is like a nice adapter that goes from your raw search results to just a string that you're just going to go ahead and save to state. That's really it.
Now you will notice a few small things here. One is I save sources_gathered, so this is going to use that format_sources function to actually just get the URLs. The raw_web_research_results will return basically the entire formatted output and we'll also go ahead and increment the number of research loops we've done once after research is finished.
The Summarization and Reflection Nodes
So now we've run web research, we've saved all of our sources to state, let's go to summarize sources. Now here, I get the latest results from web search. Remember, these are all saved to a list, so I get the most recent addition to the list. And if there's an existing summary that I've already generated, I'm going to modify my prompt a little bit to basically say, 'Hey, if there's an existing summary, extend it and add all the new stuff I've learned.' Alternatively, if I don't have an existing summary, if I'm just starting this research process, I'm just going to go ahead and generate a summary for these search results. Super simple. Then I run my local LLM with my formatted prompts and I return that summary as a string, save it to state.
So now I have a summary. Let's go to reflect on summary. So here we go. Now I just tell the LLM, 'Identify knowledge gaps and generate a follow-up search query based upon our existing knowledge,' and I provide some additional instructions in this system prompt, reflection_instructions, which you can see right here that tells the LLM to reflect and to identify knowledge gaps and produce a follow-up question. So I've done reflection, I've generated a follow-up question.
Conditional Routing, Finalization, and Testing
Now all we do is we use this route_research conditional edge to decide whether or not to, you can see right here, go back to web research or finalize. This just looks at the number of research loops we've already done relative to our configured max loops performed. That's really it, very simple conditional edge which tells us where to go next. So depending on how many loops we've already done, we'll either go back, try again, or we'll just finish.
When we finalize, all we do is very simply, we take our running summary and we take the sources that we gathered here, that's that all sources saved in the sources_gathered key in our state, and we just format it by appending summary to the front and sources to the end. Cool.
Now we've seen how to lay this all out, let's just go ahead and test it. Now this is doing exactly what I already showed previously using Studio. Now I'll just show you how to run this in a notebook. So you've compiled your graph, that's done right here, we visualized it right here, and all I need to run is graph.invoke and I pass in an input, in this case I'll also question who developed Qwen 2.5? So that runs. We get this pretty nice summary which you can look at here, pretty detailed, and a bunch of nice sources. So this again all runs locally and it's free for me to use with, in this case, Qwen 14B and using Tavily API.
Setup and Local Deployment in LangGraph Studio
Now I just want to briefly walk through the readme so you see exactly how to set this up. So first, you want to get a model. For that, you can use Ollama, just simply run ollama pull in a terminal, whatever model name you want. Think careful about the resources available to you. For example, ollama pull 3.2 will grab a 3 billion parameter model which is actually pretty nice for general use. I can push it with a 14 billion parameter model on my Mac with 32 gigs, but even that's pushing it. I like to stay 3 to 7 billion parameters is kind of a nice regime that I like to operate in.
Now I also use Tavily for web search, I mentioned that. You just go to tavily.com, you can sign up, you get an API key, and again, you get 1,000 requests for free. It's a really nice API for accessing the web basically. I kind of like this and it also scrapes for you. Um, so that's all you need. Set your API key and then just run these commands, clone the repo, go into the directory. So basically I'm going to go here, I've already cloned and all I'm going to run is this command very simply to spin up a LangGraph server locally. So let's build an environment and there you go. It's going to spin up this project and this will automatically open your project in a browser. There we go. So you can see this is LangGraph Studio, and this is what I showed at the beginning of the video. This is a very nice way to interact with our system, it's a nice UI, you can see it running like we saw earlier. You just pass in the user topic here, you can configure whatever model you want to work with here as well as the max research loops. So again, I can kind of set this to say two in this case. I'll run whatever the default model is set to, in my case in this repo it's set to uh, Llama 3.2. And again, I can pass the research topic and we can watch the research progress just like we saw before, which is pretty cool. It streams all really nicely here, you can see it going through iterations, updating the summary as it goes.
Debugging with LangSmith and Conclusion
And again, here we just get a nicely formatted summary of our research as well as sources. I'll show you something else that's pretty neat. You can just click on this thing, 'Open in LangSmith', open this up, and this actually shows us a trace of everything that happened under the hood with our assistant. We can open these up, we can see the initial query generation, we can see I used chat-llama Llama 3.2, and we can really look at all the individual outputs of each step. Pretty nice. So sometimes I like to kind of audit the details of the assistant and what it did. We can look at the web research steps, we can look at the summarization step using a Llama, uh, Chat Llama in this case, Llama 3.2. So we can look at that initial summary, we can look at the reflection, we can see what Ollama determined was necessary for follow-up questions. We can look at the output here of reflection, we can see it returns a structured object just as we requested using Ollama JSON mode with the apparent knowledge gap and a follow-up question. Web research kicks off again with the follow-up question and so forth. So we can really nicely audit our local summarization and research assistant using this LangSmith trace.
So hopefully that gives you a nice overview of this very convenient, lightweight, very simple research and summarization assistant that I built. I use it all the time, runs locally. What I like about it is I can configure it for any local model I want. As new models come out, I very simply update my configuration to test them out like I just showed. I can drill in, look at the traces if I want, and I often will just copy the summaries generated, paste them to, for example, if I want to store them in Obsidian or some other place where I'm storing markdown files. I just do that. So it's a really nice way to automate web research and really collate sources that I can then dig into a little bit later if I want to. So again, it's a very nice tool. Feel free to leave any comments below, and this will all be open source, so feel free to play with it and enjoy it. Thank you.
