Why owning your AI stack matters
People are debating whether AI or AGI is going to be superior to human intelligence, but it really doesn't matter. Let me tell you why. Even if the human brain remained superior to AI, we can't deploy our own brain to do a dozen tasks at once 24 hours a day, 365 days a year. So, to be productive and competitive, we have to lean on AI. That's the world we are heading towards.
Now, the question is, are we going to rent it and turn our life into a real Black Mirror episode, or are we going to own the intelligence, set up our own AI stack, and take back the control? And taking back control isn't just running a model on your own machine. You already know how to do that. It's everything around it, the tooling. That's where the real power actually is, and it's the part very few talk about. So, today we're going through the foundational layers every AI stack is built on, and exactly how to set each one up on your own local system. And at the end, I'll throw in some bonus tips that turn this from a weekend project into an enterprise-grade AI setup you can actually rely on without much extra effort. Let's build it.
Okay, so your AI stack isn't one flat monolithic block. It's more like a tree, and every tree starts with a seed, which we're going to grow into a giant tree, your whole AI stack. The seed here is the engine, llama.cpp. It's the one that actually drives the models, runs the LLM network, and produces the generated tokens for you. And the roots of the tree are all the hardware that the seed, llama.cpp, pulls its resources from.
Now, once the seed is planted, you need a way for the tree to grow and pass its nutrients out to all the branches it develops. For that, you need a model server and a router. Llama.cpp comes with a built-in server called Llama Server, and it has a built-in router you can use when you start Llama Server in model preset mode. I've already done a video on how to set all this up, how to tune Llama Server and llama.cpp to your hardware. You can check that out in the top right corner.
Alternative model routers and OpenAI-compatible endpoints
Now, some people have reported issues with Llama Server as the router. That can happen. The router itself is still early, still experimental. So, you've got an alternative, Llama Swap. It works off of a single config file, sits on top of llama.cpp and serves the model and acts as the router for you. And here's the key part about this model serving. It exposes the model and the server as an OpenAI compatible REST endpoint. That's what lets us plug into a whole range of different tools and harnesses later on. OpenAI compatible endpoints have been the standard for a long time now. The majority of open-source tools accept it to connect to any LLM inference endpoint.
Chat UI: AnythingLLM and Open Web UI
Every AI stack needs the basic chat UI. Somewhere you can ask the LLM a question and get a response back. For that, there are a couple of great options. The one I recommend is AnythingLLM. The Open Web UI is another solid choice if you'd rather.
AnythingLLM is a free open-source tool that lets you do a lot with an LLM. And it's built by a developer called Tim Karambat. He's got a YouTube channel with some great stuff on it, too, worth checking out. Now, AnythingLLM also comes as a desktop app, but I like to run things in headless server mode. So, I'll use Docker. Here's the GitHub page. I'll go straight to the Docker section and grab the Docker Compose content.
I'll drop that into a new Arcane project, paste it in, let Docker manage the volume, and strip out the embedding and model selection settings. We'll set those from the UI. Since to talk to the Llama Server, the easiest thing is to set the network to host mode. That way it can reach the other services on the same machine. Once it's running, you take the server's IP and that port, open it in your browser, and there's AnythingLLM.
Now, inside AnythingLLM, go to Settings, then Providers, then LLM. That lets you pick the LLM provider.
Scroll down and choose Generic OpenAI. For the base URL, put the IP of your machine and the port your llama.cpp server is running on. In my case, 8080. It'll automatically pull the models being served through your Llama Server, and you'll see them in the drop-down. Set the model's context window limit and the max token limit, and we're done. Say hi, and it responds. Now, you can ask it whatever you want. That's the basic chat layer of our AI stack sorted. And with that, we're one step closer to taking back control of our life.
RAG: private retrieval with LanceDB and embeddings
Now, you've been around these models enough to know we can't always trust what an LLM tells us.
We want it to answer truthfully, grounded in our own data. And we want that data to stay on our own devices, not get hoovered up and trained on by the AI labs.
For that, you need a vector database and a RAG system. RAG, retrieval augmented generation, is where you upload your own PDFs and documents, and the system chunks them, indexes them, and stores them in a vector database. That's what lets you query your documents by meaning.
This is crucial if you're a researcher digging through hundreds of past papers trying to find exactly what you need. You can set this up right in AnythingLLM or in Open Web UI if that's what you're running.
Here's how it works in AnythingLLM. First, let's look at the vector database options. Go to Settings, then Vector Database. By default, it uses a local one called LanceDB.
You can swap in a different one if you want, but AnythingLLM does a really good job with LanceDB, so I'll leave it as is.
Then there's the embedder. It reads the document content and turns each chunk into a semantic embedding, which is what makes it searchable by meaning.
You can tweak it, but I'll keep the defaults. Now click the document upload button on the workspace header. That opens the upload window, and I'll drop in all the research papers I've got on my machine. It processes them and embeds every one, so the LM can query them.
Once they are all processed, edit the workspace settings and switch the chat mode from agent to chat. Then you just ask it anything about those documents, and it'll answer straight from them. For example, I'll ask, "What is a Swin Transformer?" And you can see it's read the document and answered straight from what's written in there.
And it shows me which document it pulled that from, so I can go verify it. And that's your own RAG system. Your data, your AI, with no third-party snooping around in it.
Local coding agents: Pi and plugins
Now that you got the RAG system, which covers most of the academic side, we need something that lets us build things. We need a coding agent. And even though I personally use Claude Code for most of my work, I want something local for this too. I don't want to wake up tomorrow morning and find out all the coding models have been banned because they are too dangerous to use. So for coding, I go with the Pi coding agent.
It's really lightweight and perfect for local LMs, but you can also go for OpenCode, which is another great option. To set up Pi, head to the pi.dev website. It'll give you a bunch of different ways to install it. I'm going with the npm option since it's OS agnostic. I select npm, grab the install command, run it, and Pi is ready to go. Now, Pi is a very modular coding agent. It comes with a bare-bones setup that's already good enough, but you can extend it exactly the way you want.
For us, we want it running on llama.cpp, so we'll set up the llama.cpp Pi plugin alongside it. Once the llama.cpp plugin is installed, we configure the llama.cpp URL in Pi's settings. I'm just using the global settings file. You open it up and add the llama.cpp URL parameter with the same address as before. Your server IP, port 8080, the Llama Server port, and then /v1, our OpenAI compatible endpoint. Once the plugin's set up, you can run the command /models, and that'll let you pick any of the models you've got configured in the Llama Server.
And here's an example where I asked the agent to analyze my older codebases, and it did. It found everything, the whole architecture, the caching layers, the cache-aside strategy, the real-time setup, the websockets, all of it. It mapped out everything in that codebase perfectly. I also tried it on a 6-year-old Angular project of mine that wasn't building at all, and it found the exact build errors and fixed the configuration so the project starts up again, completely on its own.
Automation: making your stack act autonomously with n8n
Okay, up to this point, everything we've built is something you sit down and use. You open the chat, you ask the question, you run the agent, you're driving. But the real shift, the thing that changes how this whole thing feels, is when your stack starts doing work on its own, while you're asleep or while you're busy with something else entirely, and that's automation. And the best tool for this, I believe, is n8n.
What it lets you do is connect to all sorts of applications, your email, your news, any website or app you're interested in and then based on different conditions run automations on top of them. Let's set up n8n first and then we'll see exactly what we can do with it. We'll use the same method we used for AnythingLLM, a Docker Compose file. I'll use my Arcane server, drop the Docker Compose into the project section, and start the service. It automatically downloads the Docker image and starts the server, and you can open it on whatever port you set in your Compose file.
Now, once n8n is running, you set up the credentials for OpenAI. We're not actually going to use OpenAI, but our server is OpenAI compatible. So, in the credential section, you set up an OpenAI credential, and under the hood, you just change the base URL to our server. That's it. You can put any value in for the key, it doesn't matter. It'll act as an OpenAI endpoint, but instead of reaching out to OpenAI's servers, it reaches out to our local machine.
Now that the credential's set up, we can start building our workflow. We'll start with a trigger that listens for all our emails every hour. Once that trigger fires, we want an agent to look at the content of the email and decide whether it's important to us or not. We'll pass the subject and the body of the email, which we get from the trigger, into the user message. And in the system message, we specify the instructions we want it to follow.
The agent needs a chat model, and that's where we use the OpenAI node, the one that's actually connecting to our local system. We pick the OpenAI chat model, select the credential we set up earlier, and that's it. One thing, we'll uncheck the Responses API. That's the newer OpenAI method, the older one uses the standard OpenAI and if you uncheck it, it falls back to the default OpenAI compatible method.
Now, we need to give the agent a way to actually do something. That's called a tool. So, we add a tool, and the one we want is to add a label to the Gmail message. It's a Gmail operation, so we pick the Gmail action and select Add a Label. We point the message ID at the dynamic one coming from the trigger, and we pick which label to tag it with. You can also write a custom description, so it's obvious to the AI what this tool does. And that's it, we've got an automation. It runs every hour, and any email it finds important, it'll mark for you automatically after analyzing it. And not one of those emails, some of them pretty personal, ever touched a cloud AI.
This isn't a pile of separate apps anymore. The engine, the model, the trigger, the action, all wired into one system that acts on its own. That's the moment a bunch of tools becomes a stack. And from here, you can take it as far as you want. Build a whole little army of these, agents running around the clock, each one handling a slice of the boring stuff, and quietly making your life easier.
Operational tips: turning a project into an always-on AI lab
Before we wrap up, a few bonus tips. These are from my own setup for when you want to take this from a weekend project to something you actually rely on. First, give it a dedicated machine, a rig that does nothing but run your AI and stays on. Once things like that email tagger are running around the clock, you won't want this on the laptop you carry around with you.
Second, and this one saves you a real headache, go into the BIOS and set it to power on automatically after a power loss. So, if the power blips while you're out, the machine just boots itself back up, and everything comes back online on its own. You're not driving home to press a button.
Third, put a container manager on it. Something like Portainer or the one I use, Arcane. It gives you a clean web dashboard to see and manage all your containers instead of living in the terminal every time something needs a restart. And finally, install Tailscale. It puts your machine on a private network that follows you. So, you can reach your whole stack from your phone or your laptop from anywhere like it's sitting right next to you.
Do those four things and you've got a real always-on AI home lab. So, that's the whole stack, one engine, and every tool you would actually reach for, chat, your own knowledge, a coding agent, automation, all of them branching off that one local endpoint, all of it running on hardware you own. We started this by talking about cutting the strings. This is what that actually looks like. You're not renting your intelligence anymore. You're not one price change or one policy decision away from losing it. It's yours. It works alongside you. It does what you need and nobody can take it away. That's what taking back control actually looks like. I'd love to know how you're planning to build yours. Drop a comment and tell me what you'd wire up first. And if there's a piece of this you want me to go deeper on, tell me that, too. I'll see you in the next one.
