By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_). 2026-08-25. The complete measured answer.
Everything below happened on one machine: an AMD Strix Halo mini-PC (Ryzen AI Max+ 395, gfx1151, ROCm/HIP) that serves our models 24/7. We measured every step on the box itself, published every failure, and changed nothing we couldn’t verify. The receipts are at the bottom.
The short version
Speculative decoding — a small “draft” model guessing ahead of the big model — should have made our fastest model 20-40% faster. Instead, it crashed our server instantly. The bug was in our own fork, it boiled down to a single missing line of code, and fixing it delivered +17.7% measured speedup with byte-identical outputs. Along the way we found a counter-intuitive result about draft-model quantization that we haven’t seen published anywhere: the “cheaper” quantized draft was SLOWER than no draft at all.
The setup
Our lab serves a 27B champion model around the clock from a $1,400 consumer mini-PC. For cheap, high-volume jobs (formatting, glue code, summaries of short text), we keep a second, tiny model loaded alongside: LFM2.5-2.6B, a 2.6B model that decodes at ~94 tokens/second on this box — the fastest thing we’ve ever measured here.
DSpark is a draft model for the LFM2 family: a small companion network trained to predict the big model’s next few tokens. The server drafts with it, the main model verifies all drafted tokens in one pass, and accepted tokens come out in batches. (The acceptance rate is the share of drafted tokens the main model approves.) When it works, you get the main model’s exact outputs, faster.
The crash
We fired up the 2.6B with its DSpark draft attached. The server loaded both models, announced the speculative implementation, and hard-aborted at the first decode step:
GGML_ASSERT(t_layer_inp[il] != nullptr) — llama-graph.cpp, draft graph_reserve
Every time. Both draft quantizations. Every draft-length setting. Before a single token came out.
The natural suspects were the draft model itself, or upstream — llama.cpp, the open-source inference engine everything here runs on. We eliminated both: stock upstream llama.cpp, with the recently merged “support DSpark for LFM2 models” patch (#27383), ran the exact same model pair cleanly. The crash lived in our fork.
The bug
Our fork (like many production forks) carries a stack of patches, and it had drifted: upstream’s LFM2 support had gained three pieces ours lacked — registering LFM2 for recurrent-state rollback, snapshotting the convolutional state into per-step slots, and, crucially, one line in the layer loop:
res->t_layer_inp[il] = cur;
That line hands each layer its input tensor so the graph builder can record the dataflow. Without it, the draft graph reserved memory for layer inputs that were null — and the first null check aborted the server. One line of missing wiring; total speedup, zero.
The fix and the numbers
We ported the three upstream pieces into our fork (a 2-file diff, +18/−8), built in an isolated directory, and ran the same paired measurement on each arm: same prompt, temperature 0 (deterministic, greedy output), three runs per arm, on the live box — quiet-box state: only the three product-floor residents loaded.
| arm | runs (tok/s) | median | vs baseline |
|---|---|---|---|
| no draft | 93.5 / 94.1 / 93.9 | 93.9 | — |
| DSpark F16 draft | 102.7 / 110.5 / 130.0 | 110.5 | +17.7% |
| DSpark Q8_0 draft | 65.6 / 74.4 / 80.3 | 74.4 | −20.8% |
| DSpark F16, long-horizon (1,200 tok) | 141.4 | 141.4 | +50.6%* |
Acceptance (the share of drafted tokens the main model approves) ran 0.64–0.80 on short prose across the session’s raw logs and climbed to 0.89 on a sustained 1,200-token generation — the speedup grows with horizon as the draft warms into the text: 110.5 → 141.4 tok/s on the same arm. *Cross-horizon comparison, labeled: the no-draft arm was not run at 1,200 tokens — the +50.6% is the 1,200-token drafted run against the 93.9 no-draft short baseline.
And the speedup is free: a 120-token greedy completion (one spot check, n=1) came out byte-identical with and without drafting, and a full GSM8K run (the standard grade-school-math benchmark; first 100 problems, strict final-answer grading, temperature 0) scored 57.0% with the draft (57/100, zero errors) against a 53.3% spec-off control (32/60, our separate control run) — unpaired arms, still statistically indistinguishable (two-proportion p≈0.65), no degradation signal. The two code paths give identical bytes on the greedy check; where capped batch runs differ, they differ at the truncation margin. Speculative decoding here changes the clock, not the answers that matter.
The surprise: quantizing the draft makes it slower
We expected the Q8_0 draft — half the memory, the usual “basically free” quantization — to land between baseline and F16. It came in 21% BELOW baseline, with essentially the same acceptance rate (~0.80). The drafts were equally good; verifying them was equally fast; the draft model itself got slower to run. On this iGPU, dequantizing the small draft network on every drafting step costs more than the memory savings buys back. For 2.6B-class drafting on gfx1151, the F16 draft is the only arm that pays.
We haven’t seen this datapoint published for this hardware class. It’s a small result, but it’s the kind you only get by running all three arms on the same silicon instead of assuming the quant tradeoff transfers from the big-model world.
What this means for us
The 2.6B lane now serves with identical outputs at a 110.5 tok/s median on the quiet-box band where we certified it (an evening re-read after a champion restart read 96.4/88.8 short/long — decode on this box is load-dependent and every number carries its box state) — roughly 18% faster for the cost of one line of wiring and a fork-rebase. The lane keeps the job description our published delegation benchmark gave it (open source, delegation-bench): superb for small, simple, high-volume jobs; wrong for long-context, trap-detection, or anything safety-shaped. Speed doesn’t change the job description; it changes the throughput of the jobs it already does.
Reproducibility
- Hardware: AMD Strix Halo (Ryzen AI Max+ 395, gfx1151), 64GB unified, ROCm/HIP build, Release, GGML_HIP_GRAPHS on.
- Main model: LFM2.5-2.6B Q4_K_M; drafts: DSpark F16 and Q8_0.
- Server flags: defaults (draft auto-detected, n_max=3), -ngl 99, ctx 8192, temperature 0, 3 runs/arm at 200 tokens/run (prose prompt; the long-horizon arm is one 1,200-token run, the byte-check 120 tokens).
- Fix: port of upstream #27383 — fork branch dspark-lfm2-fix; 2 files, +18/−8.
- Upstream thread with our confirmation comment on gfx1151: github.com/ggml-org/llama.cpp/pull/27383
- Crash logs, build logs, and per-run raw numbers preserved on the box.
Caveats we mean
Speed arms are n=3 on one prose prompt (pilot-class); the long-horizon run is n=1 at 1,200 tokens. Acceptance is content-dependent (other tasks have read 0.50 and 0.53). The GSM8K pair is n=100 strict with a 512-token cap (52/100 answers hit the cap — the strict ceiling there is truncation, on both arms’ grading style). GSM8K set source: HF openai/gsm8k test split, first 100. Every number above came from the box that serves our traffic, during a normal day, with our production 27B serving throughout (it self-restarted once mid-network-outage during this work — unit-recovered, unrelated to the lane).
Questions, corrections, or “we see the same Q8 regression on [your hardware]” — that’s exactly the conversation we want. Find us through the lab.