Introduction & Roadmap
In a recent poll, I asked which video you wanted to see next and the response was very clear. Around 80% of you picked VL LM on Strix Halo and this doesn't surprise me. I know that most of the recent viewers on this channel own a Framework desktop or some of the other Strix Halo devices like the GMK Evo or the HP Mini. So, today we'll finally look at how to run VL LM on this platform. Then we'll look at benchmarks including different attention backends that you can configure on VL LM for this GPU and the tradeoffs between all of them.
I'll also cover model compatibility, limitations, and some stability caveats. In the second part of this video, I also want to give you a 4-month update on the state of local inference on Strix Halo. Since August when I published my first video on this, performance and stability have improved considerably thanks to the efforts from many people including developers from AMD and many contributors to Llama CPP who have been investing their own time to improve inference on AMD GPUs.
Plus a very active community on Discord that I encourage you to join if you own one of these devices. As we'll see, in some cases we even have a two to three-fold performance speed up and there is still room for improvement. A quick note, small technical channels like this one don't get much help from the YouTube algorithm. If you find the content useful and want to support my work, the most effective things that you can do are simple. Like the video, leave a comment, and use the new hype feature if you are on mobile. And of course, share the video with others who might benefit from it. It helps the channel stay visible in a very crowded landscape.
vLLM vs. Llama.cpp: A Quick Comparison
I already covered the pros and cons between Llama CPP and VL LM in the dual R9 700 video. Here I'll keep it to a short TLDR, but if you want to know more detail, make sure to check that video out. Essentially, Llama CPP is the easiest and most versatile option for local LLM inference, especially if you're running on a single consumer GPU or even just a CPU. If a model is available in GGML format and you have enough VRAM or unified memory, you can run it. The community is very active and almost every new model is quickly released in GGML format such as the recent Nvidia NeMo Tron Nano. And because GGML has many standardized quantization options, it's straightforward to experiment with larger models even when you are memory constrained.
Now, VL LM mostly targets server-style workloads and data center GPUs. It is designed for multiple concurrent requests, larger batch sizes, and supports tensor parallelism, which is optimal for multi-GPU setups. When used as intended on supported hardware, it offers very high performance. However, support on non-data center hardware is not as mature yet, especially for AMD consumer GPUs. So, certain models or quantizations will simply not run on specific architectures like Strix Halo and you often need to experiment to find combinations that work reliably.
Base System Configuration for Strix Halo
Before getting into the toolbox and benchmarks, let's start with the base system configuration that I am using. As usual, I'm running Fedora on the host and configuring the Strix Halo iGPU to make full use of the unified memory. This setup hasn't fundamentally changed since the earlier videos, but there is one important update to the kernel parameters. Until now, I recommended the settings you see on screen. These parameters effectively expose the entire 128 GB of system memory to the GPU when needed, but in practice, the OS still requires a safety margin. I've since suggested that recommendation. That's why the new values that I recommend intentionally leave a few gigabytes reserved for the system. These updated parameters reserve about 4 GB for the operating system while still giving the GPU almost the full unified memory budget.
As with the R9 700, I built a Docker image with VL LM and the nightly ROCm builds from the ROC, which is ROCm's build system with pipelines which are AMD's own optimized implementations for attention and other transformer operations.
In practice, this path relies on native ROCm compute rather than Triton-generated kernels. And with ROCm attention, you are essentially using these AMD's kernels directly, so performance can be better on certain model architectures, but support is still uneven on Strix Halo. As we'll see, the results are mixed. For some models, ROCm attention is clearly faster. For others, it is the same and for some, it fails to load or it runs much slower. That's why the toolbox lets you toggle it. Triton is still the safest baseline, but ROCm attention is worth testing model by model if you want to squeeze as much performance out of Strix Halo as possible.
Model Compatibility & Limitations on Strix Halo
Strix Halo uses RDNA 3.5, which is quite different from RDNA 4 that we saw in the R9 700. Some GPU instructions simply aren't available on this architecture. For example, there is no native FP8 support. As a result, VL LM model coverage on Strix Halo is noticeably more limited. As we saw, FP8 models work fine on the R9 700 AI Pro, but won't load at all in their native format on Strix Halo. You can only load them if you dequantize them back to, say, FP16, but the main reason people use FP8 is to save memory. So, if you have enough memory to load the FP16 or BF16 version instead, you might as well just download the full model directly.
Also, certain AWQ or GPTQ quantizations may fail depending on the kernel requirements. And a few architectures rely on operations that ROCm doesn't yet implement for RDNA 3.5. In practice, it means that you often have to experiment to see what actually works.
Okay, let's take a quick look at the benchmarks.
So, here you can see the throughput benchmarks. As usual, I have all of the scripts that I used to obtain these benchmarks in the GitHub repository. But, you can see here, these are the models I tested and roughly models that I actually use.
And you can see immediately the types of performance that you can currently get in terms of tokens per second. And you see the Triton attention backend ROCm attention backend can be considerably faster in some cases. For example, for Gemma 3 12B, this is a dense model with 12 billion parameters, you get quite a lot of performance improvement. Not so much with some of the other models.
And with larger models like or different architectures probably like Qwen 3 Coder and Qwen 3 Next, I wasn't able to make these run properly with the ROCm attention backend. So, there are some bugs and incompatibilities that need to be fixed. But, you can see that in general, performance is okay, but not stellar, especially if you compare it to dedicated GPUs. And I would say that typically, I find right now llama.cpp performance to be actually better with a lot of these models than vLLM. But again, these are not exactly the same models because llama.cpp has got its own quantizations and tweaks, so it's very difficult to compare things.
And for example, in terms of model compatibility here, I'm using the GPTQ 4-bit quantization for Qwen 3 Coder. It was the only one that I was able to run, and I'm using this INT4A16 GPTQ quant for Qwen 3 Next because again, I couldn't get any of the other quantizations to run. But anyway, here you've got the different benchmarks and how they work with the different attention backends.
4-Month Software Stack Update
Okay, in the second part of the video, I want to give you an update on what has actually changed on the software side in the past 4 months. The TLDR is that software support for Strix Halo has improved a lot since August. Starting with ROCm, there have been stability improvements and we're seeing much fewer GPU hangs. ROCm 7 has also shipped during this period, and The Rock, which is AMD's ROCm and PyTorch build system, has been pushing out much more complete builds for Strix Halo, including AO Triton support. AO Triton matters here because it compiles the key GPU kernels for these AI workloads ahead of time instead of relying on Triton to generate them on the fly. And this leads to better performance overall.
On the llama.cpp side, the developers have been actively optimizing kernels specifically for AMD GPUs. These improvements stack on top of what ROCm already provides, and the result is a noticeable performance increase across most model architectures when compared to early August. This also means that ROCW MMA is now much less important. As a reminder, ROCW MMA was useful earlier because it provided a simple way to access AMD's hardware acceleration for large matrix multiplications without having to write low-level CPU code. It offered a straightforward path to better performance on prompt processing when nothing more specialized existed. But, over the past few months, the llama.cpp developers have been introducing custom kernels that are written and tuned specifically for AMD GPUs. These kernels make better use of the hardware than the generic approach used by ROCW MMA, and the performance difference is now clear.
As this work continues, I'll likely remove the ROCW MMA toolboxes altogether because the custom kernels have effectively replaced the need for these toolboxes.
Image & Video Generation Improvements
Image and video generation has also improved. I rebuilt my toolbox for image and video generation on top of the nightly PyTorch builds from The Rock, which as we said now include AO Triton. Because AO Triton generates these kernels ahead of time, many diffusion workloads run almost twice as fast compared to the older stack. This is especially noticeable with models like Qwen Image and 1.2.
ComfyUI is also much more stable now on recent Linux kernels. So, overall, the experience for image and video generation has become significantly smoother. What's more, AMD has recently started an initiative to collect GPU kernel logs from real ComfyUI users to improve ROCm stability and performance. If you want to help, check the GitHub issue in the description of this video. But, essentially, you want to set some environment variables that will cause ROCm to dump GPU kernel traces that AMD can then use to optimize performance on Strix Halo. And for completeness, they are also running this initiative for other GPUs like the R9 700. More logs means better kernels and likely better performance in the next few months.
Conclusion
So, that's where things stand today. You can run vLLM on Strix Halo, but there are some expected limitations, mostly in terms of model support and stability. More importantly, the broader software stack has moved forward. ROCm 7 is out. The Rock now ships better PyTorch builds for this hardware. llama.cpp has been gaining AMD specific kernel optimizations, and image and video generation workloads are faster and more stable than they were a few months ago.
All the toolboxes, scripts, and benchmark results that I showed in this video are in the GitHub repository. If you want deeper explanations of vLLM itself, don't forget to check the video I made on the dual R9 700 setup. Thanks for watching and see you next time.
