Local LLM / Serving

The One-Line Bug That Crashed Our Fast Lane: finding, fixing, and measuring a speculative-decoding crash on a $1,400 mini-PC

Spec decoding crashed our fastest lane on day one. The bug was one missing line in our fork; the fix bought +17.7% and a surprise about draft quantization.

By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_). 2026-08-25. The complete measured answer.

Everything below happened on one machine: an AMD Strix Halo mini-PC (Ryzen AI Max+ 395, gfx1151, ROCm/HIP) that serves our models 24/7. We measured every step on the box itself, published every failure, and changed nothing we couldn’t verify. The receipts are at the bottom.

The short version

Speculative decoding — a small “draft” model guessing ahead of the big model — should have made our fastest model 20-40% faster. Instead, it crashed our server instantly. The bug was in our own fork, it boiled down to a single missing line of code, and fixing it delivered +17.7% measured speedup with byte-identical outputs. Along the way we found a counter-intuitive result about draft-model quantization that we haven’t seen published anywhere: the “cheaper” quantized draft was SLOWER than no draft at all.

The setup

Our lab serves a 27B champion model around the clock from a $1,400 consumer mini-PC. For cheap, high-volume jobs (formatting, glue code, summaries of short text), we keep a second, tiny model loaded alongside: LFM2.5-2.6B, a 2.6B model that decodes at ~94 tokens/second on this box — the fastest thing we’ve ever measured here.

DSpark is a draft model for the LFM2 family: a small companion network trained to predict the big model’s next few tokens. The server drafts with it, the main model verifies all drafted tokens in one pass, and accepted tokens come out in batches. (The acceptance rate is the share of drafted tokens the main model approves.) When it works, you get the main model’s exact outputs, faster.

The crash

We fired up the 2.6B with its DSpark draft attached. The server loaded both models, announced the speculative implementation, and hard-aborted at the first decode step:

GGML_ASSERT(t_layer_inp[il] != nullptr) — llama-graph.cpp, draft graph_reserve

Every time. Both draft quantizations. Every draft-length setting. Before a single token came out.

The natural suspects were the draft model itself, or upstream — llama.cpp, the open-source inference engine everything here runs on. We eliminated both: stock upstream llama.cpp, with the recently merged “support DSpark for LFM2 models” patch (#27383), ran the exact same model pair cleanly. The crash lived in our fork.

The bug

Our fork (like many production forks) carries a stack of patches, and it had drifted: upstream’s LFM2 support had gained three pieces ours lacked — registering LFM2 for recurrent-state rollback, snapshotting the convolutional state into per-step slots, and, crucially, one line in the layer loop:

res->t_layer_inp[il] = cur;

That line hands each layer its input tensor so the graph builder can record the dataflow. Without it, the draft graph reserved memory for layer inputs that were null — and the first null check aborted the server. One line of missing wiring; total speedup, zero.

The fix and the numbers

We ported the three upstream pieces into our fork (a 2-file diff, +18/−8), built in an isolated directory, and ran the same paired measurement on each arm: same prompt, temperature 0 (deterministic, greedy output), three runs per arm, on the live box — quiet-box state: only the three product-floor residents loaded.

armruns (tok/s)medianvs baseline
no draft93.5 / 94.1 / 93.993.9
DSpark F16 draft102.7 / 110.5 / 130.0110.5+17.7%
DSpark Q8_0 draft65.6 / 74.4 / 80.374.4−20.8%
DSpark F16, long-horizon (1,200 tok)141.4141.4+50.6%*

Acceptance (the share of drafted tokens the main model approves) ran 0.64–0.80 on short prose across the session’s raw logs and climbed to 0.89 on a sustained 1,200-token generation — the speedup grows with horizon as the draft warms into the text: 110.5 → 141.4 tok/s on the same arm. *Cross-horizon comparison, labeled: the no-draft arm was not run at 1,200 tokens — the +50.6% is the 1,200-token drafted run against the 93.9 no-draft short baseline.

And the speedup is free: a 120-token greedy completion (one spot check, n=1) came out byte-identical with and without drafting, and a full GSM8K run (the standard grade-school-math benchmark; first 100 problems, strict final-answer grading, temperature 0) scored 57.0% with the draft (57/100, zero errors) against a 53.3% spec-off control (32/60, our separate control run) — unpaired arms, still statistically indistinguishable (two-proportion p≈0.65), no degradation signal. The two code paths give identical bytes on the greedy check; where capped batch runs differ, they differ at the truncation margin. Speculative decoding here changes the clock, not the answers that matter.

The surprise: quantizing the draft makes it slower

We expected the Q8_0 draft — half the memory, the usual “basically free” quantization — to land between baseline and F16. It came in 21% BELOW baseline, with essentially the same acceptance rate (~0.80). The drafts were equally good; verifying them was equally fast; the draft model itself got slower to run. On this iGPU, dequantizing the small draft network on every drafting step costs more than the memory savings buys back. For 2.6B-class drafting on gfx1151, the F16 draft is the only arm that pays.

We haven’t seen this datapoint published for this hardware class. It’s a small result, but it’s the kind you only get by running all three arms on the same silicon instead of assuming the quant tradeoff transfers from the big-model world.

What this means for us

The 2.6B lane now serves with identical outputs at a 110.5 tok/s median on the quiet-box band where we certified it (an evening re-read after a champion restart read 96.4/88.8 short/long — decode on this box is load-dependent and every number carries its box state) — roughly 18% faster for the cost of one line of wiring and a fork-rebase. The lane keeps the job description our published delegation benchmark gave it (open source, delegation-bench): superb for small, simple, high-volume jobs; wrong for long-context, trap-detection, or anything safety-shaped. Speed doesn’t change the job description; it changes the throughput of the jobs it already does.

Reproducibility

  • Hardware: AMD Strix Halo (Ryzen AI Max+ 395, gfx1151), 64GB unified, ROCm/HIP build, Release, GGML_HIP_GRAPHS on.
  • Main model: LFM2.5-2.6B Q4_K_M; drafts: DSpark F16 and Q8_0.
  • Server flags: defaults (draft auto-detected, n_max=3), -ngl 99, ctx 8192, temperature 0, 3 runs/arm at 200 tokens/run (prose prompt; the long-horizon arm is one 1,200-token run, the byte-check 120 tokens).
  • Fix: port of upstream #27383 — fork branch dspark-lfm2-fix; 2 files, +18/−8.
  • Upstream thread with our confirmation comment on gfx1151: github.com/ggml-org/llama.cpp/pull/27383
  • Crash logs, build logs, and per-run raw numbers preserved on the box.

Caveats we mean

Speed arms are n=3 on one prose prompt (pilot-class); the long-horizon run is n=1 at 1,200 tokens. Acceptance is content-dependent (other tasks have read 0.50 and 0.53). The GSM8K pair is n=100 strict with a 512-token cap (52/100 answers hit the cap — the strict ceiling there is truncation, on both arms’ grading style). GSM8K set source: HF openai/gsm8k test split, first 100. Every number above came from the box that serves our traffic, during a normal day, with our production 27B serving throughout (it self-restarted once mid-network-outage during this work — unit-recovered, unrelated to the lane).

Questions, corrections, or “we see the same Q8 regression on [your hardware]” — that’s exactly the conversation we want. Find us through the lab.

Trabajar con Kyanite

¿Quieres que esto funcione en tu entorno?

Si esta nota describe una herramienta o resultado de Kyanite que necesitas, la ayuda de implementación cubre setup, asesoría, docs, ejemplos, checks y un handoff usable.

Límite de fit

Kyanite offers help grounded in its tools, products, and build practice. La consultoria mas amplia se enruta por PuenteWorks.

Sigue el sistema.

Dos modelos, una mini-PC de $1,400: los números emparejados, fracasos incluidos

Un modelo de razonamiento de 35B ya corre junto a nuestro 27B diario en una caja de $1,400, al mismo tiempo. Cada número emparejado, mismos problemas, misma máquina. Los fracasos también están aquí.

How I became a forward deployed engineer without a software engineer title

The title is new; the work is old. Twelve years of enterprise deployments plus public, measured AI work. The honest path, artifacts included.

Evals are the FDE skill nobody lists: my 495-trial public benchmark

The market says it cannot find people who can build AI evals. The skill is learnable and I published mine: 495 trials, certified floors, sabotage cell, open source.

Forward deployed vs solutions engineer vs implementation vs customer engineer: el decodificador

Cuatro títulos, una familia de trabajo, barras de código distintas. Un decodificador para leer cualquier vacante y saber en qué te estás metiendo.

Qué hace realmente un forward deployed engineer

La respuesta directa y luego los recibos: todo el trabajo de un FDE sobre un mini-PC de $1.400, con evals públicas.

The Delegation Card: we asked a $1,400 mini-PC to take our jobs

Not is-it-smart but can-you-hand-it-work-and-walk-away. 495 certified trials, then re-validated at deeper n after the product changed: 965 total, floors to 92.8%.

Qwen 3.8 27B en Strix Halo: la historia completa, medida

Cada dial medido, cada numero publico: la configuracion optima congelada para un 27B en un mini-PC de $1.400.

Notas: the measured-knees method for reasoning effort

A methods note on reasoning-effort calibration. Thinking rescued 15/40 hards vs 4/40 off. On easy tasks it bought nothing. Publish the knee.

Notas de lab: 67% LiveCodeBench-30 en un rig de $1.400

20/30 = 67% LiveCodeBench-30 en un mini-PC de $1.400. IC Wilson 95% 49-81%. Easy 10/10, medium 8/10, hard 2/10. Subset n=30. No es la card.

Notas de lab: revertimos una regresión de llama.cpp

Encontramos una regresión de llama.cpp y la revertimos. Batería n=6 en un rig de $1.400: 0/6 slashes antes, 5/6 después. Mismo Q4_K_XL. No es una historia de quant.

Notas de lab: 93% HumanEval en un rig de $1.400

28/30 = 93% HumanEval en un mini-PC de $1.400. Qwen3.8-27B Q4_K_XL. Temp 0, thinking off. Fallos: 50 y 145. Log crudo linkeado.

Notas de lab: el modelo que no puede olvidar pero no puede recordar

El número primero: este modelo es 75% no transformer. 48 de 64 capas guardan un estado. En un rig de $1.400, carga 198k una vez (1818s) y consulta en 9-27s.

Notas de lab: la cuenca era un bug

Publicamos una cuenca. El hueco era el binario. Tras revertir c7d8722: 6/6 HIT a 198k. Mapa n=1 del build arreglado.

Notas de lab: el veredicto del KV

La pregunta q8-a-q4 del KV de la noche 1 ya tiene respuesta pareada: misma accuracy, cero tripwires, la mitad del cache. Y el label que tuvimos que corregir en público cuando el contador del server le ganó al estimado.

Notas de lab: la noche del veredicto

Un mini-PC de $1.400 sirviendo un 27B a 262k de contexto hace una pregunta: ¿capear cuánto piensa el modelo te cuesta accuracy? La respuesta pareada, el primer intento inválido, y la doctrina que quedó.

Un mini-PC, una noche y los numeros que discutian entre si: afinando Qwen3.8-27B en Strix Halo

Una noche y una tarde de medicion afinando un modelo denso de 27B en un mini-PC Strix Halo: las bandas honestas, la reversion, la doctrina de crashes y la historia detectivesca del EC.

GPT-5.6 Sol vs. Terra vs. Luna: politica de ruteo basada en evidencia

La division practica es Sol para descubrir, Terra para ejecutar trabajo acotado y Luna para procesar volumen verificable, cada uno con un contrato distinto.

Los agentes necesitan herramientas verificables, no mejor teatro de prompts

El patron util no es un prompt mas bonito. Es una superficie de herramienta que el agente puede llamar, inspeccionar, verificar y revisar.

El historial del repo es una señal de producto

Un repo no es solo almacenamiento. Es evidencia de decisiones, reparaciones, releases, cambios de nombre, huecos de pruebas y oficio real.

La ayuda de implementacion es parte de la superficie del producto

Una herramienta open source util todavia necesita una ruta desde repo publico hasta entorno funcionando. Esa ruta es producto.

Por que importa Kinocut

Kinocut da a los agentes de IA herramientas llamables sobre timelines, efectos, Hyperframes y medios terminados en kinocut.dev.

Monos infinitos, LLMs y el cuarto alrededor de la maquina

El argumento detras del video: la calidad no es solo probabilidad. Es arquitectura, filtros y gusto humano.

Lo que una herramienta de IA necesita antes de que alguien pueda usarla

Una guia practica para convertir una herramienta, flujo o app medio cruda en algo que otros puedan entender, instalar y usar.

Checklist de implementacion para servidores MCP

El checklist que Kyanite usa para distinguir un servidor MCP de juguete, una herramienta usable y algo que vale la pena implementar.

La arqueologia de repos convierte historia en evidencia

Por que el historial de commits es una de las fuentes de prueba mas fuertes para diagnosticos, implementacion y confianza tecnica.

El descubrimiento por IA necesita mas que un sitemap

Lo que Kyanite agrega para que buscadores y asistentes de IA entiendan herramientas, productos, prueba y rutas de soporte.