Introduction: A New Contender in Local LLMs
Everyone runs local LLMs differently. Llama CPP, Olama, LM Studio, those are just some of the more popular ones that run on really simple hardware and pretty much anything. But Llama CPP just dropped a new web UI that might change the equation for some of you. I'll show you how to build it from source, push it to the edge, and I'll show you a behavior that might make you rethink your stack.
Here's a discussion that you might want to take note of. Now, this is probably going to be merged into the main documentation, but it's too early right now, so it's not yet. But this was created by Georgie Ganov himself. He tells you the benefits of using this thing and how to get set up with it. I'm on a Mac, by the way, in case you're wondering. It's the M4 Mac Mini with 16 gigs, so we're going to run small models, but the process for running larger models on bigger hardware is pretty much the same. One way you can install it is the simple way, just using Homebrew by doing brew install llama CPP, and then you have it. Pretty easy.
Building Llama.cpp from Source on macOS
But, uh, this is a developer focused channel, so we're going to do it the hard way. But, it's also going to be important for you to know this way if you're going to be building it on other systems that don't have a ready build file, which is actually quite a bit of systems. So, I'm going to go to the build step. This is building Llama CPP locally. The first thing we do is we need to clone this repository. So, let's say you're on the home of the repo, you could just go to code and then copy the URL and clone it. But, this provides you a nice set of steps. So, I'm just going to follow that. I've created the code directory here already and I'm going to do get clone that URL. Then I'm going to cd into that directory.
Now we just need to build it. It says CPU build. Do we want to do that though? We don't want our models to run on the CPU, right? So if you're doing this on Nvidia or AMD, you're not going to be doing this steps. You're going to be doing other steps. But on a Mac, the metal build is enabled by default. You don't have to do anything extra. You just build it the way it is right here. cmake -B build. This will set it up. Boom. No extra flags needed cuz we're on Apple Silicon, which is the machine that I think he uses. So, it's supported right out of the box. And then I'm going to use this command to actually do the release build. But I'm going to add one more flag to it. And that's the -J flag. Let's do eight cuz that's the example he has here. He says for faster compilation, add the -J argument to run multiple jobs in parallel. And believe me, you'll notice it's a lot faster doing it that way. There we go. It's building. Done. That was pretty fast.
Starting the Server and Preparing to Choose a Model
Now that it's built, your built binaries are going to be in a different directory. So, you're going to want to go to build/bin. And if we take a look at what's going on here, you have all these tools that you can run. Llama CLI, Llama Bench.
So, now we're on to step two. Start the Llama Server tool. Here, if we start this Llama Server tool, we're pointing it to the HF, the name of the organization, and then the model name itself. And sometimes they actually have the quantization appended to it too. Right now it's GPTOSS20B GGUF which is a larger model that would fit on the 16 GB machine. So I'm going to go to hugging face and pick a different model. This is kind of like GitHub for machine learning.
So these days I'm constantly flipping between models. GPT4 row for notes and email.
Claude for code refactors. Flux for image generation. Cling for video. Four tabs, four bills and counting. Enter Chat LLM teams. There's one dashboard that houses every top LLM and route LLM picks the right one for you for a given task. 04 mini high for fast answers. Claw SA 3.7 for coding. Gemini 2.5 Pro for big context and even adds GPT 4.1 before Chad GPT has it. Chat with PDFs and PowerPoints. Then generate decks and docs and do deep research all in the same chat. Need human sounding copy? The humanized toggle rewrites text to beat AI detectors.
Spin up agents and run code with AI engineer. I built my first bot in just minutes. Track artifacts, create GitHub pull requests, and debug from the same interface. Need visuals? No problem. Use Flux or Ideoggram and Recraft for images. Cling, Luma, and Runway for video, all builtin. And the kicker is just $10 a month, less than one premium model. Head over to chatlm.abacus.ai or click the link in the description and level up with Chat LLM Teams.
Navigating Hugging Face for GGUF Models
I'm going to just take a look inside GGML or see what they have available. I'm going to just search for it right here. Hey, look at that. There they are. Organizations, GGML AI. Let's go in there. Here's the organization page. If you go down here, you'll see models. Click on that and you can filter by name or just search for something that's going to be relatively small that's going to fit inside of this machine. I'm going to search for Quen 34 billion. There we go. Uh, so we have the thinking version with Q8 and we have another one that's Q8. Q8 means quantized to 8 bits. And GGUF means the format that Llama CPP supports. There's different formats out there. Typically models don't come in GGUF formats. They come in safe tensors format for example. So if you take a look at Miniax and you take a look at the files and versions, you'll see all these safe tensors right here. This is the typical and standard format that models come in. But when you want to run them with llama CPP, they have to go through an extra step of being converted. You don't have to do that, but it's usually done for you. Unless there's a model that you want to use that is only available in safe tensors, then you have to convert it to GGUF.
But here, since this GGML.AI is directly related to Llama CPP, pretty much all their models are going to be in GGUF format already. And it says it in the name. Now, the name, you can't always go with a name because it's not always accurate and it's up to the creator to name it properly using standards that don't actually exist.
Launching the Llama.cpp Server and Web UI
So, if we take a look at files and versions, we have three different GGUF ones here. F16, that's 8 GB. That's the original floatingpoint 16 model, but converted to GGUF. Then we have Quen 34B Q8 GGUF. It's half the size because this one is quantized to 8 bit. And finally, we have the small one, 2.5 GB. And that's the one I'm going to use. Quen 34 billion Q4KM. That's quantized to four bits. So, it's even smaller. It's half the size of the previous one. So, a quarter of the size of the full 16 floating point one.
Now, it says here in hugging face to use this model. You can run it using any of these tools. We're using llama CPP. So, if we take a look at that, you'll see this is the command to run it. You can also have a little selection here. I I don't like this interface cuz it's not easy to copy and paste. Now, we can get around that easily. Um, select which one you want. We want the Q4KM. I'm going to copy that command. Go back to our term and I'm going to paste that in. It copied that comment, which that's silly if you ask me.
I'm going to do ./lamaserver to be explicit about where we're running it from. In fact, I did not install this globally. I installed it here isolated into this bin folder. So I kind of have to use slash instead of just running llama server. If you just run llama server, it'll expect that tool to be available globally. And that's if you install llama cpp globally. I prefer not doing that cuz I want to always have control over my versions so I can have multiple versions in different folders. So I do it this way.
If we go back to the documentation, llama server hf ggml. Then we have a couple of flags here. C 0 means context of maximum, but you can set your own context here. Let's add those flags to the end of this. And notice that our model identifier does have the organization, the name of the model, and then also colon quantization in this case. Paste that in. And just to make it quicker and take up less memory, I'm going to set the context to 4,000. Boom. It says server is listening on HTTP localhost port 8033. Let's copy that and open that up right here. Boom. There it is. The new web UI for Llama CPP.
A Tour of the New Llama.cpp Interface
It tells you what model. It tells you what context. This UI is really nice. It's simple, but it's good because it offers you a lot of flexibility. As you can see, I have some of my old chats here. Write a story was a common one. Write a story. Boom. And it's reasoning. You can expand the reasoning if you want to, but look at this. It says the context and how much is filling up as it's working. That is really useful information. There's the output, how many tokens it's outputting. And now it starts typing. This is now past the thinking stage. This is now the generation stage. They're all generation, but some of it is divided into thinking and not thinking. They're all still going up against your total context. Whether it's reasoning part, the thinking part, or just the generation part, the actual story itself.
There's also a bunch of controls here in the settings. things like temperature, temperature range, show thought and process, so you can expand it automatically. You can import all your conversations or export all your conversations. And then there's the developer section, too. Custom JSON to be sent to the API. Pretty good stuff here. And it shows you the statistics at the end. 29 tokens per second, total tokens created, how long it took.
Concurrency Test: Ollama's Sequential Processing
Now, what about Olama? A tool that really, really easy to install. By the way, if you don't know how to install Olama, I made a whole video about it, deep diving into everything. I'll link to it down below. But if you ask me, they are probably changing direction in my opinion. I don't know if that's what's happening or not, but it looks like a lot of their models are now available in cloud. I think they're heading towards the cloud side, and they're probably going to be offering more and more models that are cloud only. And if it's cloud and local, they're going to offer more options for your cloud offerings. That's just my guess. I don't know if that's what they're doing or not.
If we just compare the UI that they've just deployed, you can't really do much here. Yeah, there's settings if you go here, but model location, expose llama to the network, context length, that's pretty much all you get. You don't even get statistics like tokens per second in Olama uh the UI part. If you want to get tokens per second, you can run Olama through the terminal. And you're probably already aware about how to do that. llama list or llama run same model that we're going to be doing with web UI for llama CPP and we say verbose at the end of that so we can get some statistics about it. let's do a high here and there we go eval rate 36 tokens per second.
Besides all the different functionalities you get with a new web UI with Llama CPP you also have the ability to run things in parallel which is useful for things like agents or if you're doing something programmatically against the API, you can run things in parallel, which is handy. For chat, it doesn't really matter unless you want to have two conversations at the same time. This is more for programmatic cases. When you have a lama running like this, it only handles one message at a time, no matter where it's coming from. Write a story in the UI here and then say, write a story here. Let's go here and let's go here. And it's doing it on the right side in the terminal. So that's tying up that server. The server is only answering one instance right now. There is no concurrency here. Now it's done over here and it's switched to doing it over here. So it's kind of like waiting around. Imagine if you have a process that has to wait around like that. Or if you got more than one user using the system, they're going to have to wait. But what if you have more than one Olama instance running in the terminal? For example, let's do write a story here and let's do write a story here. Boom. And boom. Same thing. You've got one of them is writing and busy and the other one's just sitting around waiting. Come on, get up. Do something. I'm not really angry. I'm just It's just the implementation. This is not what it's designed for. All right, this one's done. Now that one's doing it. We're around 34 tokens per second.
Concurrency Test: Llama.cpp's Parallel Power
Let's quit that and let's go back to Llama CPP. I'm going to start up another one, another instance of the site. Let's have a new chat here and a new chat there. And I'm going to say write a story about Alex. And in this one, I'm going to say write a story about Bob. Boom. And boom. And look at that. They're both actually reasoning. You can see the context counter going up on both of them at the same time. The server is responding to multiple requests.
In fact, I'm going to open two more. Write a story about Tracy and write a story about Alice. Boom. And boom. All four of these are now generating. Looks like this one is about Alex. This one's about Bob. These two are thinking. Let's take a look at our activity monitor here. You can see that we're using 13 gigabytes out of 16. And if we take a look at GPU history, this is in fact happening on the GPU. So, this is kind of proof that Llama CPP has taken advantage of Apple Silicon here instead of running on the CPU only. It would be a lot slower if it was. Here's the story about Tracy and here's a story about Alice. This one is done and so is this one. But notice the tokens per second. Here we are at 16 and here we're at 17. So, our tokens per second took a dive here because we're now splitting up the processing that GPU between these processes. But at least we can now work with multiple processes.
This one 13.4 tokens per second and this one 13.07. But we do get a little bit more speed. Right now, this is parallelized where you can actually run multiple conversations. You can do multiple requests, but it's using the same exact process. If I run this one more time on a separate terminal window, but change the port to 8034, now I'm actually going to have two separate instances of this running. So there's a lot more flexibility when you're running Llama CPP this way. So now I can run it on a separate port. There we go. Write a story. Now remember how O Lama ran a little bit faster than Llama CPP?
Well, if I'm running things in parallel, boom. And boom. There we go. They're going at the same time. This is just llama CVP as one process. You can see that we got 24.3 tokens per second here and 25.92 over there. That's almost 50 tokens per second in aggregate. So when you're using it this way with agents or programmatically, you're going to get better output.
Conclusion and a Look at VLLM
So, Lava CPP works across multiple different systems as I've mentioned before, but there is one more tool that's very popular that's actually going to run on things like the DJX Spark, Nvidia clusters, and AMD clusters, but not on Mac and not on Windows. And that's called VLM. And I made a video about that right over here. Watch that. You get some pretty impressive numbers. Thanks for watching, and I'll see you next time.
