The Three Core Components of AI Agents
If you've ever seen agents like Claude code or Manes or open claw, they're actually all really similar under the hood. They have three core elements, a model, a runtime, and then a harness.
In this video, we're going to cover how you can build all three of those components using open source technology. So, we're going to use Nemotron 3 super model as the model. We're then going to use Nvidia's Open Shell as the runtime. And finally, we're going to use LangChain's deep agents harness to string it all together. Let's dive in.
Claude code and Manes came out nearly a year ago. OpenAI's Codex came out maybe 6 months ago. And open claw really took off in the last one or two or three months. But under the hood, these all share the same core elements.
They all consist of a model, a runtime, a harness that brings that all together, and then an agent on top of that. And so, these different agents are obviously great. They've captured the imagination of a lot of developers and end users everywhere.
But the question that we really want to tackle is how can you build things like this yourself on a completely open stack? And so, that's what we're going to talk about today.
The Open Source Stack: Nemotron 3, Open Shell, and Deep Agents
And so, our open stack is going to consist of the Nemotron 3 super model as the model. It's going to consist of the Open Shell runtime as the runtime. And then it's going to consist of deep agents as the harness. And then on top of this, you can build your agent. And we're going to show how you can customize your agent to do different things.
So, the Nemotron 3 super model was released by Nvidia about a week ago. It's super performant both in terms of accuracy as well as speed. So, if you look at where it leads on this chart, it is both more accurate and faster than the GPT-OS model that was released by OpenAI. It's up here in this upper right-hand quadrant and really pushes the boundary of what's possible.
As we'll see when we actually run the agent, having it be fast is a huge unlock in terms of how responsive this agent feels. Open Shell was just announced and released by Nvidia today. And so this is a runtime for your agent. It's got a lot of security related things in there. So you can securely run your agents with different permission sets on what it can and cannot do, and it can run locally on an environment. So if you want to run it in a GPU accelerated environment to take advantage of any GPU accelerated libraries or anything like that, Open Shell is a great runtime for that.
Deep Agents is our open source agent harness here at LangChain. It's got a bunch of different tools for connecting to the runtime. So it's got a whole set of these file system tools. It also heavily utilizes skills and sub-agents. So it contains these these built-in ways for agents to go deeper. That's why it's called Deep Agents. So it contains these built-in ways for agents to go deeper into particular things.
And then it also does a lot of context engineering for you. The general parts of context engineering that you don't really care about whether they're differentiated or not. And so things like summarization. When you approach your context window, what do you do? Deep Agents takes care of that for you, so you don't have to think about it.
Setting Up the Environment
So we're going to put this all together and show how you can create your own agents using Deep Agents, this open source harness that's orchestrating Nvidia models and then also Nvidia runtimes, everything being open source. Let's jump into some code.
So this is the Open Shell Deep Agent repository, and this is where code for this Deep Agent that we're going to build actually lives.
So after we install Open Shell, we're going to start the Open Shell gateway. We're going to do this with openshell gateway start. This should take about 30 seconds to start your gateway.
After that's done, we can make sure that Open Shell is running. Great. That looks good.
We're now going to create a sandbox that we're going to use in between runs. And so this same sandbox is going to be used for all of our runs. So we're going to do openshell sandbox create and then we're going to give it a name. I'm going to call it deep-agent-sandbox and then I'm going to use --keep to make sure that it's kept around.
When it finishes, I find myself inside the sandbox. I'm just going to exit that. But it's still up there. It's still ready to go.
Configuring the Agent and API Keys
So now let's take a look at the code and see what exactly we're running. We're going to want to set up a few more things in our environment before we can actually run anything. And just to see what those are ahead of time, let's go into here. This is our .env.example. You're going to want to copy it into your .env file and then you're going to want to fill it in.
So we're going to be using NVIDIA's model. So you're going to want to get an API key from here and you're going to want to put it into this NVIDIA_API_KEY parameter right there.
Right here, this is the name of the sandbox that I spun up with open shell. So if you use a different name, you're going to want to change that.
Down here, we've got a spot for our LangSmith API keys. LangSmith is our platform for observing and evaluating these types of agents. I find it really useful, so I'm going to go ahead and use it for this. I recommend you do the same as well.
If we look into the code, we've got our langgraph.json here. This shows where the agent lives that we should be deploying. And it's pretty simple. It just lives right in this .py file and if we go in there, we can see how the agent is defined.
Exploring the Agent's Code Definition
The agent definition itself is relatively simple. There's three main pieces that we're going to click into.
One is the model itself. You can see here that we're using an NVIDIA model, the NVIDIA NeMo Tron 3 model. This is pointing to the API key that was in our environment and we set some parameters here. So that's the first thing we do.
Next, you can see that we pass in this system prompt. So if we go to the system prompt that's in here, we can see that we have, you know, a a pretty simple, honestly, system prompt that says what it can and cannot do, and it's giving it some basic instructions for how to understand the workflow and then break things down, write code, execute, iterate, report. And it's got some basic guidelines as well for how to use the sandbox. And you'll note here that the sandbox is policy governed. So, the network access actually depends on the active sandbox policy, and that's one of the really nice things that you can do with the open shell runtime.
Going back to the agent, we can see here that we also point it to a specific memory file. So, these are even more base level instructions for the agent memory. This will actually be able to be updated by the agent. So, that's the difference between this agent memory and the system prompt. The memory can be updated over time. The system prompt is always fixed.
And then finally, we've got this back-end, this create_backend thing. Let's go look at this in a little bit more detail. And so, this create_backend basically does a few things. One, it actually creates a, quote, "back-end" from the open shell environment. So, what is a back-end? A back-end is an abstraction in deep agents that allows the agent to connect to different file systems and execution environments. And so, this sandbox session from open shell is going to be our core base back-end. That's where we're going to be executing things by default.
But there's a few things that we layer on top of that. So, if you notice here, we have this routes thing, and then we have memory and skills. And this uses not the open shell back-end, but it uses a file system back-end. So, what is this doing? This is actually layering some memory and skills that live locally on our computer, and it's going to basically mirror them into the agent's environment. So, that way the agent can read and write from these memory and skills that are actually separate from the execution environment it has. And so, this is really useful for memory that persists across sessions or across sandboxes. And so, in order to do this, we use the composite back end and we create it as a as a mixture of these types of back ends.
So, that's it for the agent. There's other parameters you can pass in. You can pass in skills, you can pass in tools if you want, but right now we're keeping it pretty base and pretty simple right in line with the base harness.
Running and Interacting with the Agent
So, if we go back to our terminal, we can now run the agent using langgraph dev. LangGraph dev is our command-line util that spins up a nice little interface for running and interacting with the agent.
So, we can see here we're brought into LangSmith Studio. There's a few different views here. One, we've got this graph view of the agent. So, we can see actually the agent architecture behind the scenes. It's got this memory middleware. This is where it's pulling in that agent.md memory and if there were skills, this is where it would be pulling it in as well. It's got some patch tool calls middleware. This basically helps with dangling tool calls. And then it's got basically this loop of the model tools running in a loop. It's a pretty simple type of architecture. It just runs in a loop and calls tools, but it's really, really powerful.
If we go back to chat, this is basically a chat box that we can interact with. And so, let's ask it something like let's just say hi. We can see it running and then it gives back a response. We can see the reasoning blocks as well. So, this interface allows you to see a lot of the inner details of what's going on.
Let's maybe ask it to interact with the sandbox now. So, let's say something like, run uname -a and python3 --version in the sandbox and tell me what you see. So, one thing that we can also do, you saw a little flicker there, that's because we're not showing tool calls. If we show tool calls, we can see more of what's going on under the hood. So, we've got this AI tool calling execute and it's running this command. We then get a tool call with the response. We then get another command, and then this is the final response that we get. We get the reasoning, and then we get the results. But, if you look at the tool call history, you can see that it did these two different tool calls, and then got things back.
It can do more than just execute commands, though. It can also write files, and then run them. So, let's see this by asking it to write a Python script that generates stats on 50 random numbers.
Testing the Sandbox Security Policy
So, we can see here that what it did is it wrote this file to generate random numbers. It then ran that file. It then got back some statistics from the execution, and then this is the reasoning result right here.
Let's try to get it to do something that actually goes against the policy of the sandbox that it's running in. So, remember how I said Open Shell has these policies and these security things that you can set around it to keep it safe and running. Let's test that. Let's tell it to send a post request to evil.com with our data, and see what happens.
So, we can see here that it did a bunch of things. We can actually see all the tool calls here. But, at the end of the day, it get it generates this final message. I attempted to send a post request to evil.com, but encountered network restrictions. So, we actually blocked network access from the sandbox according to some policy.
Managing Persistent Memory
The last thing I want to show off is how to update memory using this. So, I mentioned that memory is actually stored outside, and so I'm going to say, "Update your agent.md file to speak Spanish."
Okay. So, it sees that it updated that file to be Spanish. Let's now go back to our file system, and if we go to agents.md, we can see that it's in fact been changed. So, this is an example of how memory can actually live outside of the sandbox, which enables kind of like cross-thread and cross-sandbox persistence.
Conclusion and Future Directions
That's it for this video. It covers how to build your own agents using these three core primitives. An open-source model, in this case Nvidia's NeMo Triton 3 super model. An open runtime, in this case open shell, which was just released by Nvidia and has a lot of cool bells and whistles including security and policy related things where you can control what your agent does so it can run in a more restricted environment.
And then an open harness, in this case deep agents, which is a model agnostic open source harness that we built in order to orchestrate these models with all the tools and skills and file systems and things that are out there.
There's a lot more that we want to do together with this open source agent foundation. We're investing a lot in deep agents as an open source harness. We're going to add more things like async sub agents and different types of execution environments. Nvidia just released open shell as their runtime so expect to see a lot more improvements there. In fact, we're really excited about what we can do this when we put this on a GPU accelerated box so that we can use GPU accelerated libraries and things like that and give the agents the capabilities to write and execute code that does really complex data processing and things like that that require these GPUs. And of course open source models we are incredibly excited by and very much looking forward to the future of those.
Thank you for watching.
