What is Quantization and Why Does It Matter?
This tiny laptop is running a 70 billion parameter AI model right now. But wait, shouldn't this model be too big to fit? The math just doesn't add up unless you know about quantization.
Today I'm going to show you the clever trick that makes this possible and why those mysterious Q2, Q4, and Q8 tags in Ollama are about to become your best friends. In the next 10 minutes or so, you'll learn how to run massive AI models on sort of basic hardware, what Q2, Q4, and Q8 actually mean—and it's simpler than you think—and which quantization is right for your projects. Plus, I'll show you a brand new trick with context quantization that could save you gigabytes of RAM.
Now, quick pause. If you're serious about running AI models locally, hit subscribe and the bell. Every week I share new tricks like this that can save you hundreds in hardware costs. And next week, well, let's just say you'll want to be here for that.
The High Cost of 32-Bit Precision
Here's what nobody tells you about AI models: they're just giant collections of huge numbers, billions of them. Each one needs to be stored with incredible precision. Think of it like this: normally these numbers are stored with 32-bit precision. It's like having a ruler that can measure down to microscopic levels. Sounds great, right?
But here's the problem: all that precision comes at a cost, a pretty massive one. Let me show you exactly what happens to your RAM. Let's do some quick math. Take a 7 billion parameter model, multiply each of those parameters by four bytes, and you need 28 GB just to store it. That's more RAM than a lot of gaming PCs have. You're looking at $2,000, $3,000 for a GPU just to run one model. And this model won't even fit in one of those. But here's where quantization changes everything.
Explaining Quantization with Analogies
Think of quantization like choosing different rulers. Maybe the full model is using millimeters. Q8 is measuring in centimeters—less precise but still pretty accurate. Q4, now we've got a marker every 5 cm or so, but look how much extra space we have. And Q2, well, this is the extreme option, like measuring with a random stick from your yard, but sometimes that's all you need.
Here's my favorite way to explain this: imagine you're running a mailroom, but instead of postcards and packages, you're storing numbers. In the original, unquantized model, every number gets its own custom-sized mailbox. It's perfect organization, but imagine how much space that takes up. That's exactly what happens with full 32-bit precision. It's like having an infinite mailroom where every tiny value gets its own special spot.
Here's what regular Q4 quantization looks like in action. Imagine having just 16 mailboxes for all your numbers. Every number has to pick the closest available slot. It works, but it's not very smart. What happens when you have two big bunches of numbers: one bunch of really tiny numbers and the other bunch of really huge ones?
A Smarter Method: K-Quantization
But wait, it gets better. See those models tagged with K in Ollama? They use something called k-quant, and they are brilliant. Instead of one rigid system, k-quant creates multiple specialized mailrooms. Small numbers get their own precise area, and big numbers get their space. It's like having a smart assistant organizing everything perfectly.
The different sizes you see—KS, KM, and KL for small, medium, and large—are different levels of detail in the notes that are in the mailroom. The small version keeps brief notes, medium keeps more detailed records, and large maintains very precise information about how everything was arranged.
This smart system means areas with mostly small numbers get more precise, small slots, while areas with bigger numbers get appropriately sized spaces. It's like having a mailroom that adapts to what you're storing rather than forcing everything into the same size box.
Beyond the Model: Context Quantization
But here's what really matters: how this affects your actual usage. Look at memory. Watch what happens when we load a Q4 model. Now check out speed. Notice the startup time and performance. Let's run the same prompt on different versions.
Now, here's where it gets really interesting. Ollama just dropped a game-changing feature: context quantization. See, we've been talking about making the models smaller, but there's another memory hog we haven't discussed yet: your conversation history. A year ago, this wasn't really a big deal. Models could only remember 2,000 to 8,000 tokens of conversation. But now, well, we're talking about models that can remember entire books' worth of conversation. That's 128k, or 128,000 or so tokens or more.
Here's the problem: all that conversation history eats up RAM. But context quantization fixes this. Let me show you how to enable this.
Demo Setup: Enabling Flash Attention and Maxing Out Context
It's super simple. First, we'll turn on flash attention by setting OLLAMA_FLASH_ATTENTION is true, and then set the cache type with OLLAMA_KV_CACHE_TYPE F16. What happens to the memory usage when we enable these?
So let's get practical. I'm going to show you exactly how much memory you can save with these tricks. For this demo, I'm using Qwen 2.5, a 7 billion parameter model. Out of the box, it uses Q4_KM quantization and takes up 4.7 GB on disk.
First, let's max out the context size. By default, Ollama gives us 2k tokens, or 2,048 tokens, but Qwen can handle 32k or 32,768. Here's an easy way to set that up. First, go to ollama.com and find the model. Then run the command shown on the right. Now type /set parameter num_ctx 32768 and hit enter. And then type save qwen2.5-max or whatever you want to call it and hit enter. Finally, quit Ollama. Now if you run ollama ls, you should see that the model has been added to your list of models. Perfect, we have our maxed-out model ready to go.
Demo: Quantifying the Memory Savings
Now I'll run the model and watch the output from asi-top, which is a similar tool to nvtop on Linux but for Mac OS. Before I hit enter on the command, which is this, I see the memory usage hanging out around 25 gigs. The Mac uses unified memory, so that's the total amount of memory used right now by the system and the GPU. Now I hit enter, and I see that memory usage climbs to around 40.9 GB. So this model with a 32k context uses about 15 GB.
Now, I'll quit ollama serve and run this command, which runs ollama serve again, but this time with the flash attention option enabled. I'm doing this at the command line to easily switch back and forth between settings rather than setting up the environment variables the correct way. I'll run the model with the same question, and this time I see the memory top out at 33.7 GB. That's a massive drop in memory usage from 40.9 GB. That's a savings of about 7 gigs.
Let's do this one more time, setting the KV cache quantization to Q8. Now it only climbs to 30.6 GB. That means it's taking up closer to 5 gigs of memory with the maxed-out context size. Let's try this. Okay, one more time, going back to the original model with the default context size of 2k without using flash attention at all. Now it goes from 28.5 to 30.3, so with a normal context size, it needs less than 2 GB. The maxed-out context needs 15 gigs, and using context quantization for this model set to Q8 saves us 10 gigs, bringing it down to only 5 gigs.
Caveats: When Optimization Doesn't Work
But it's important to note that not every model is going to benefit in the same way. I saw one model take less memory with flash attention on, but take more memory with KV cache quantization set to Q8, more than without using flash attention at all. That was a brand new model by IBM. So it's not always guaranteed to use flash attention and context quantization.
Choosing the Right Quantization Level
Hopefully now you have a better understanding of how quantization works and how it affects the performance and efficiency of the model and its context. Well, the next question is probably going to be how to pick the right model for your specific needs. In general, it's usually best to start out with one of the Q4 models. This could be the Q4 or even Q4_KM, which is what Ollama seems to be defaulting to these days. If you notice issues with the generation quality, you can move up to Q8 or even FP16. If it seems to work well, you can try dropping further down to Q2. It's amazing how often a Q2 model will work just as well for most tasks for most people, and the memory usage is so much lower. Combine that with a quantized context, and you're getting a lot of power on a machine with even the most limited memory.
Your Game Plan for AI Optimization
Now let's wrap this up with what really matters. Whether you're running AI on a Raspberry Pi or a beast of a workstation, here's your game plan.
Start simple. Begin with a Q4_KM model and then enable flash attention and test your specific use case. Then optimize it a bit. If that's running smooth, try Q2. But if it's having issues, move to Q8. Need more context? Try playing with Q8 KV cache quantization. The best part? These tricks can turn a large model into something that runs on a mostly normal laptop. That's not just theory, you saw it happen right here.
Quick action steps: download a Ollama Q4 model, enable flash attention, and test with your use case. Experiment with lower quantization. You can join our Discord or the Ollama Discord for more optimization tips. And don't forget to like and subscribe for next week's deep dive into something else.
Remember, the perfect setup isn't about using the highest settings, it's about finding what works for your specific needs. Thanks so much for watching, and I look forward to seeing you in the next video. Goodbye.
Bloopers & Outtakes
Are, uh, something for a video? Are you wasting...? A little bit.
Jase? Jase...
One, two, three, four, five.
Dad! Dad, this is emergency tape!
I know, it's being used for emergency.
No, it is not! Dad, you are not hurt and you are wasting tape that we might need for something medical!
My videos are emergencies!
No, they're not!
Okay, this is my last one.
