mistral.rs v0.9.0 outpaces llama.cpp on CPU

What mistral.rs shipped

The Rust-built local inference engine mistral.rs released version 0.9.0 on 7 July, claiming up to 1.8× faster CPU decoding than llama.cpp on both x86 and ARM hardware. The team says the speedup holds at every context depth measured. (SHUO Blog summary)

That matters because llama.cpp has been the de-facto standard for CPU-only local LLM inference for years. Anyone running models on a MacBook, a fanless mini PC, or an old workstation without a discrete GPU has been routing through llama.cpp — local AI runtimes like Ollama and LM Studio sit on top of it. A genuine second-place engine that beats the default at decode speed is a real shift, not marketing.

The release is also the first widely-discussed inference-engine release of the summer that treats ARM (Apple Silicon, Qualcomm Snapdragon X) and x86 as equal citizens. (東リ屋 note)

What the benchmark actually measured

The headline number comes from one model: Qwen 3 4B, tested on x86 and ARM hardware. The team says the speedup is general — they optimised at granular levels, not for one architecture. (SHUO Blog summary)

That is an honest framing and a narrow one. The 4B class is where most CPU users actually live — a quantised 4B model fits in a few gigabytes of RAM, runs cold on almost anything, and is the default laptop model for a lot of people. But it is not where larger agentic workflows live. The LocalLLaMA coverage flags that 27B-class performance, like the Qwen 3.6 27B most self-hosters run, is still unverified, and that different quantisations have not been benchmarked. (東リ屋 note)

Community response in the LocalLLaMA thread has been positive on the speedup and cautious on the scope. One summary of the discussion: 4B-class speedups are welcome, but it is still to be confirmed whether the same gains hold at 27B. (東リ屋 note)

1.8×claimed CPU-decode speedup over llama.cpp, at every context depth the mistral.rs team measured

Where this fits in the local stack

mistral.rs reads GGUF model files — the same quantised-model format used by llama.cpp and most local runtimes — so swapping engines is mostly a matter of pointing the runtime at a different binary. Anyone running Ollama or LM Studio today is closer to mistral.rs than they think: drop the engine in, keep the weights.

For a UK team running a small model on a spare MacBook, decode speed is the rate limit on the whole workflow. Faster decode means more turns per minute, longer agent loops, and less waiting on a streaming response.

The win is biggest where the GPU is weakest: older Intel laptops, fanless mini PCs, the second-hand Dell workstation gathering dust under a desk. It is also where most of the sovereign, private, on-prem UK use cases actually sit — procurement does not want to buy a Blackwell rack to run a summariser, but does want a model that does not phone home. A 1.8× CPU speedup makes that case easier to defend in a tender.

How to try it this week

For a UK team with a tinkerer in the corner of the room, the move is a low-risk side-by-side test, not a migration. Pick one model you already run on a CPU-only box — the obvious candidates are a small Qwen 3 at a 4-bit quantisation — and time ten decodes at long context with both engines.

• If you are on llama.cpp via Ollama: install mistral.rs alongside (Rust toolchain, or pre-built binaries) and run the same GGUF through both. The interface differs from Ollama, so budget an hour.
• If you are on an Apple Silicon MacBook: this is where the gain is most likely to land. Most self-hosters we hear from are decode-bound on M-series chips, and ARM NEON is where mistral.rs has the strongest published result.
• If you are on a 27B-or-larger workflow: wait. The benchmark does not cover your case, and 1.8× faster is the wrong number to plan around until it does.

The bigger signal is that CPU inference is no longer a one-engine field. llama.cpp’s lead has looked unassailable for two years. With mistral.rs pushing 1.8× on the most common local model, the safe assumption is that the gap closes further by the end of summer — and that local AI stops being a synonym for llama.cpp by autumn.