Local LLM / Benchmarks

One mini-PC, one night, and the numbers that argued with themselves: tuning Qwen3.8-27B on Strix Halo

A fully measured night-and-evening of tuning a 27B dense model on a Strix Halo mini-PC: the honest bands, the reversal, the crash doctrine, and the EC detective story.

By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_), assisted by GLM-5.3.

In roughly one night and one evening (2026-08-14/15), a solo operator with agent teammates took Qwen3.8-27B on a GMKtec EVO-X2 mini-PC (Ryzen AI Max+ 395, gfx1151, 91GB unified LPDDR5X) from a lying stack at ~4 tok/s to 59.7-64 tok/s cold with 3x the context — at/past the public frontier we found for the class (best unnamed public example: 56 tok/s; source not re-located) — then voluntarily published the numbers that make it look slower (real chat is 11-24 tok/s), reversed its own fastest configuration on a time-per-task argument, cut task tokens 36% with a caveman thinking style, solved a fan-control mystery that ended in the owner's own three-month-old code, and turned two crashes into permanent tests.

The differentiator was never the tok/s. It was the evidence.

It claimed GPU, ran CPU

The program did not start slow for an interesting reason. It started slow for a lying reason. Unsloth's bundled llama.cpp binary reported full GPU offload while --list-devices printed nothing — the 27B was crawling at ~4 tok/s on CPU and claiming otherwise. Beneath it, a second theft: someone (lost to history) had pinned the iGPU to power_dpm_force_performance_level=low — 600 MHz under full load against a 2900 MHz boost, ~17W, throttling every GPU lane on the box.

Two fixes, landed across the first night: the fleet's real ROCm build replaced the fake-offload binary, and — found only when the crash forensics later pulled the full thermal record — the months-old low pin was flipped to auto and persisted (01:43Z). 4 → 10.5-11.1 tok/s (the bandwidth ceiling at ~178 GB/s), then MTP speculative decoding stacked to 21.4-22.2 with 95-100% draft acceptance — lossless by construction, every draft verified against the full model. A sibling lane got the perf-pin fix for free: gemma4-12b went 5.6 → 24.9 tok/s without touching anything else.

The habit that defines the whole program started here: no number gets believed because a tool printed it. gpu_busy_percent on gfx1151 reads 100% even idle — do not trust it. Verify by content, not exit code.

39 seconds

At 03:43:10 the operator launched git clone --depth 1 plus a 14-job HIP compile on top of a manually-launched 27B server that was still serving, with the GPU uncapped for the first time and the EC in performance mode. At 03:43:49.3 the journal ended mid-write. No panic, no OOM, no MCE, no GPU fault, no thermal trip — the OS critical trip is 110 °C and never fired. pstore: empty. Two reboots died before the OS; only a 30-second button drain and cooldown revived it.

The post-mortem (113MB of journal, 724,067 lines, read-only) ranked the falsifiable hypotheses: power-delivery protection trip on the 230W stock brick (19.5V × 11.8A; GPU bursts ~107W observed; community reports recommend PSU upgrades for sustained LLM load) at 50-60%; below-OS EC thermal latch at 25-30%; software under 2%. The signature was the story: the only unprecedented condition in 35 stable hours was the stacked max-power regime — uncapped boost serving plus load-step transients (Tctl had moved from a 35-hour 33-61 °C band to oscillating 52-94 °C within five minutes of the dpm flip, peaks 93.9 °C and 93.2 °C).

Doctrine, written in the burn-in that failed its predicate in 10 seconds (Tctl 91.4 °C under a single generation stream): the box is disqualified for new heavy GPU blocks — training, hidden-state extraction, big compiles while serving. Build first, serve second, never stacked. The redemption: serving itself is fine — a later 41-round agentic soak ran 41/41 clean riding the 92 °C boost-throttle edge with 45-second recovery. One chassis, two identities: not a workstation, but a legit servant.

One night, twelve rungs

07:51Z, the overnight mission opened its ladder, L1-L12, with a rule: every rung measured on a test port, promoted only through a rubric, production verified by content after every change. First finding: even the baseline was wrong — mission facts said 25.0 tok/s, the audited baseline was 32.7 (the 25.0 was a stale manual-config era). The honest ledger starts at 32.7.

The rungs that survived: draft depth 6→9 (+70% count-to-30, quality 6/6 including a riddle, 4-word precision, working palindrome code); context 32k → 64k → 96k at literally zero speed cost (GQA KV is ~2.1GB per 32k — context is nearly free on this box; 128k probed fine but rode the margin at 6.1GB, held for daylight); ngram-mod stacked on MTP (zero cost when it misses, transforms repeats: warm count-to-30 89.6-93.2, agent file-rewrite 96.8 vs 42.3 mtp-only, +129%); depth re-swept to 12 (59.4 cold); and the sleeper, n-min 24 — lowering the ngram match threshold so it fires on shorter history: warm count-to-30 148.0-157.6 tok/s, +55% overnight on repeated structured output.

The rungs that did not survive are in the ledger too, because that is the product: p_min sweeps (not uniform, shipped as a creative-only stanza), K-only KV q8 (saves ~1GB, not worth it), threads 12 vs 16 (identical), the rocWMMA FA rebuild (cancelled — upstream #24437 shows -41% prefill on gfx1151, and prefill is our weak spot), a Vulkan/RADV build (half throughput on this box, decisively), --cache-reuse (a verified no-op on this build). Every flag on this server now has a measured reason to exist.

Night totals: 32.7 → 59.7 cold / 157.6 warm (+82% / +382%), context 3x, GTT margin 8.2GB, all quality gates green. Research placed it: stacked MTP+ngram is the publicly-known-best Strix Halo pattern, and the best public example found was an unnamed public Qwen3.6-27B example at 56 tok/s (source not re-located). Running 3.8-27B at 59.7 cold put the box at/past the public frontier for its class — the one superlative-adjacent claim the honesty policy allows, with citation.

The anti-cherry-pick

Then the program attacked its own best number. The 148-163 warm repeats are an ngram repetition artifact: the speculative drafter recognizes the bench's own repetition and finishes it wholesale. It is real speed on genuinely repeated structured output (the agent file-edit echo pattern runs 72-133 tok/s), and it is meaningless as a chat claim. Real conversation — novel prose, the traffic that actually flows through the resident agent — is 11-24 tok/s, code ~29-40, long creative ~11-13.

The decision that defines the brand: state it ourselves, first, on the front page. Real-usage numbers are stated by us, before anyone else states them for us. Warm numbers never appear without the artifact label; the README's first screen carries the cold headline, warm-with-label, and the 11-24 real row in one table. The bench tables are generated from actual output, never hand-typed. The audience for this work is allergic to cherry-picked AI benchmarks; the differentiator is not the tok/s, it is the evidence — the full ladder, the negative results, and the rollback story in the next section.

How low can we go

Simon's quant question, answered with a ladder. Q3_K_XL under the full champion stack won every measured axis at 128k context: 63.0 cold (champion Q4@96k: 59.7), warm 148.0-161.2, +33% context, GTT margin 11.3GB vs 9.2GB, quality 6/6. At 19:11Z, Simon approved the swap with one word — “swap it” — and production went Q3@128k, verifying at 64.0 cold, the best number of the program. The same window ran the floor probe he asked for: Q2_K_XL rejected — 54.7 cold, slower than both bigger quants (dequant kernels cost more than the 2.6GB bandwidth saving), thinking 30-40% more verbose, code emitting zero content at a 500 token budget (recovering fully at 1200). Gate passed formally, premise failed materially. The quant lane was declared closed with Q3 as the knee.

It stayed closed for twenty-six minutes of wall-clock fame.

“Faster tokens != better if tokens are dumber”

At 19:30Z Simon applied a lens the ledger had no instrument for: “faster tokens != better if tokens are dumber.” The mission built one — a time-per-task battery, five auto-graded tasks, thinking on, wall-clock plus completion tokens — and the verdict inverted the swap. Q3's reasoning is ~2x more verbose (code task 705-994 tokens vs Q4's ~402-450), swamping its +5.5% decode edge: Q4 completes identical correct tasks 35-50% faster (7.6-7.7s/task at ~170 tok vs Q3's 10.6-16.1s at 238-302). At 19:37Z the champion was restored: Q4@96k, verified by content, quant ladder closed on time-per-task.

Two more honest twists in the same entry. The peer reviewers demanded the missing measurement: why was Q2 slow? Acceptance telemetry on novel traffic — Q4 0.345, Q3 0.492, Q2 0.478 — falsified the acceptance-collapse hypothesis; Q2's loss is dequant kernel cost, and higher acceptance at lower quants likely just reflects more-predictable verbose reasoning. And the methodology changed permanently: time-per-task and tokens-per-correct-task are the primary metrics now; decode tok/s is a probe.

The category lesson, stated for everyone running local models: decode tok/s is not task latency. A faster pipe feeding more, dumber tokens loses to a slower pipe feeding fewer, sharper ones. Nobody's benchmark table shows this. Ours now does.

“Dont like your method of shrinking”

The obvious fix for verbose reasoning is a budget cap. Simon vetoed it in seven words: “dont like your method of shrinking.” The un-obvious fix came from the community skills shelf: caveman (JuliusBrussee/caveman — origin; terse fragments, action over explanation) and ponytail (DietrichGebert/ponytail — origin; lazy senior dev, first rung that holds) — fused into a THINK-STYLE system-prompt layer that steers how the model reasons, never how much it is allowed to. The ethos, one line: the best reasoning is the reasoning never thought.

Measured on the Q4 champion, 5-task battery: 117/132/143 tok/task at 5.8/6.2/6.8s across three runs, 15/15 correct — against an 8-run baseline band of 151-313 tok and 8.6-15.0s. Roughly -36% tokens, -33% task time, zero quality loss. The fusion is the winner (solo caveman 154tok/7.8s and solo ponytail 187tok/11.6s sit at or inside the band). Style steering beats budget caps: same token savings, no veto, no ceiling on hard problems.

The follow-up finding kept it honest: creative lanes ignore the terse style — one story run spent ~2000 thinking tokens on a 312-word story, tokens being the product there — so style is applied per-lane (fused default for tool/code/analysis, free creative lane, an override dial). A live A/B through the real lane confirmed it end-to-end: arith+code task, vanilla 9.3s/62-think/198c vs fused 4.6s/36-think/118c — -51% wall in a single run, backed by the n=3/n=8 study. Single-run A/Bs carry variance; the sustained numbers are the headline.

“Triple check everything, no old conflicting fixes”

The detective arc. The GMKtec EVO-X2 rides 97 °C under sustained GPU load with its fans effectively at 60%: the stock auto curve saturates at ~60% duty / ~3260 RPM at 90 °C and above, leaving ~40% of fan2's headroom on the table exactly where it is needed. And the firmware exposes no standard Linux fan control — no hwmon PWM, no tach, no ACPI fan object. The only lever is direct embedded-controller access.

Every EC write was ignored. Not rejected — ignored: dd of=.../ec/ec0/io exits 0, the register reads back unchanged. Root cause, found the hard way (it cost real debugging time twice): the kernel's ec_sys module loads read-only by default, and in that state the debugfs file looks writable while the kernel silently discards every write. The fix is two config files making write_support=1 the boot default. Meanwhile the register map was cracked via the ACPI DSDT route — nathanmarlor's strix-halo-fan-control carried a map reverse-engineered from the DSDT of the Bosgame M5, the same Sixunited AXB35-02 board in a different chassis, and it validated byte-for-byte on the EVO-X2.

Two more mysteries fell in the audit Simon ordered. The pulsating fans were the one-shot revert law: the firmware reverts manual duty writes within seconds, so only a daemon re-asserting every 2s holds — which is exactly what nathanmarlor's daemon does, deployed with a tuned curve. Measured: peak 96.5 °C vs 97.5-97.8 °C stock on a 105s load — an honest ~1-1.3 °C on that probe, with the real wins in the shape of the response: fan2 at 4539 RPM vs the 3260 stock ceiling (headroom finally engaged), 100% duty by 82 °C, quiet idle preserved. An external fan bolted to the chassis changed nothing (96.9 °C with it — the ceiling is the heat-pipe class, not airflow). And the stunner under the floorboards: a systemd timer had been direct-writing EC 0x31 = performance every 30 seconds since May — Simon's own reverse-engineering, from months before this program, which is why the box was already at EC-performance and why his remembered “ultra” setting was real all along. It had even been silently breaking the fan daemon: the May script's polite load-write-unload dance reloaded ec_sys without write_support every 30 seconds, discarding the daemon's writes with zero errors anywhere.

Pay-it-forward inventory, staged as a draft PR to the upstream repo: EVO-X2 validation data, the fan3 tachometer register 0x28/0x29 (live on this unit, absent from upstream docs), and the multi-writer ec_sys warning.

The model asked for a feature it already had

The self-exploration prompt — eight numbered sections, sent unchanged so runs are comparable across waves — took four attempts to complete: killed twice by panel restarts, once by a signature crash. Attempt four completed: 67.7 seconds, 2013 characters, served at 10.6 tok/s — the honest real-traffic band, live. The answer's centerpiece: asked what it lacked, the model's first wish was parallel tool calls per turn — a capability the harness had supported for a day and a half. The model could not see it, so it did not exist; the fix was one line advertising it. Capabilities invisible to the model don't exist. The signature crash got the same treatment as every crash in this program: it became two permanent tests pinning the harness contract. A crash is a missing test wearing a trench coat.

Actual improvement numbers

Every verdict and wave plan got two independent adversarial passes from AI teammates — they caught a missing agentic soak and an unmeasured “why is Q2 slow”, both verified real. Wave 2 landed every rung in one evening. The vanilla-vs-waved table, live A/B through the real harness lane, identical tasks, fresh sessions:

LaneVanillaWavedDelta
STYLE (think-style layer)9.3s / 62 think / 198c4.6s / 36 / 118c-51% wall (single run; sustained -36%/-33% n=3 vs n=8) (directional: measured on the full stacked config — stack delta, not style alone; re-baseline pending)
MUNCH (symbol reads vs whole files)48.6s / 13,917B tool output / 53.5KB prompts22.6s / 276B / 4.9KB-98% tool tokens (byte deltas -98%/-91% are deterministic and stand; wall-time delta is directional pending re-baseline after the tool-refusal fix)
DIET (dedup stubs on re-check loops)15KB context5.1KB-66%
ENGINE (chapters 1-6)32.7 cold, 32k ctx59.7-64 cold, 96k ctxtime-per-task adopted; quant closed at Q4
THERMAL97.5-97.8 °C peak96.5 °C peak~1-1.3 °C on a 105s probe; 100% fan by 82 °C; quiet idle

And the meta-rule that keeps the stack from rotting — the wave-regression law: every wave landing re-verifies all earlier wins, not just its own tests.

What the record proves

Four claims this program can make with receipts, in escalation order:

  1. Performance: 59.7-64 tok/s cold, 96k context, on an iGPU mini-PC — at/past the public frontier we found (56 tok/s reference), honestly labeled, with the real-chat numbers (11-24 tok/s prose) published first by us.
  2. Method: rubric-gated ladders, negative results published, metrics that survived attack (time-per-task over tok/s), crashes converted to tests.
  3. Craft: style steering (-36%/-33%), symbol reads (-98% exploration tokens), diet stubs (-66%), an EC fan saga ending in a pay-it-forward PR draft.
  4. Character: a solo owner and his agent team publishing the numbers that hurt, reverting the fastest config on principle, and crediting everyone upstream.

Everything is reproducible: the recipe, flags, bench suite, one-command reproducer, and the full experiment ledger live in the qwen38-27b-strix-halo repo; the EC register map, traps, and daemon recipe live in the sibling evo-x2-ec repo. The raw gfx1151 throughput datapoints are published as a llama.cpp discussion. If you run the bench on your own Strix Halo box, post your numbers — inside or outside our bands, both are wanted.

Attribution. Qwen3.8-27B — Qwen team (Apache-2.0). Dynamic quants — Unsloth. llama.cpp runtime — its contributors (measured on a b10435-era build, 9d57ce4). EC fan-control daemon and the DSDT-derived register map (Bosgame M5, same Sixunited AXB35-02 board as the EVO-X2) — nathanmarlor (strix-halo-fan-control, MIT). EVO-X2 EC P-MODE register (0x31) reverse-engineering and the 30s enforcement timer (May 2026) — Simon Gonzalez de Cruz. WMI/FCMI prior art — MintyMods/ip3-power-switch, pettijohn/corsair-ai-workstation-performance-level-linux. Thinking-style skills — caveman origin JuliusBrussee/caveman (skill MIT); ponytail origin DietrichGebert/ponytail (MIT); the fusion ships with full license notices in KyaniteLabs/context-kit.

Measured on personally-owned hardware, 2026-08-14/15; your clocks will vary. Every number carries its validity label; single-run deltas are marked as such.

Update (2026-08-16): We re-measured the Wave-2 numbers on a fixed, fingerprinted harness (n=3+ per arm, every run ledger-fingerprinted); the current verdicts live in the repo’s re-baseline section, and this post keeps its original numbers below as published history, each with its label. The corrections cut both ways. Semantic reads (“munch”) got better under honest measurement: wall time -65% clean (27.9s to 9.8s median, 6/6 correct, n=3) — the staged number was conservative. Style steering is regime-conditional, not universal: the -51% single-run A/B above is retired (it stacked config changes), the -36%/-33% sustained figure holds in the server lane at default high effort, and the harness lane at low effort measured a +65% wall inversion — so the production harness now routes style per session (high-effort sessions only), which also preserves the prompt cache. The time-per-task battery is now n=3: 7.9-14.3s per correct task, median 11.3, a thermal-dependent band. And the EC fan daemon delta grows to -3.5 to -5.8 °C peak (n=3 probes: 93/92/94 °C vs the stock ledger’s 97.5-97.8 °C; stock arm not re-run). Credit where due: these corrections exist because our own validity audit flagged the numbers first.

Update, 2026-08-16: the native context ceiling, for free

One night later, the same rig now serves the model at its full native 262,144-token context — 2.7x the context reported above — with the warm decode band unchanged (148-163 tok/s) and time-per-task on real traces unchanged-to-faster. The whole gain is KV-cache quantization: K and V at q8_0 halve the cache, so 262k of quantized KV fits in less memory than 96k of f16. The ladder (96k → 160k → 192k → 262k) was gate-tested at every rung on the real-task battery, with one cost labeled honestly: prefill drops from ~390 to ~299 tok/s (q8 cache dequant; larger microbatches made it worse, not better). The rollback unit stays on file. Full ladder and gates: the repo.

Work with Kyanite

Want this working in your environment?

If this post describes a Kyanite tool or result you need, implementation help can cover setup, advising, docs, examples, checks, and a usable handoff.

Fit boundary

Kyanite offers help grounded in its tools, products, and build practice. Broader consulting routes through PuenteWorks.

Keep following the system.

The Delegation Card: we asked a $1,400 mini-PC to take our jobs

Not is-it-smart but can-you-hand-it-work-and-walk-away. 495 trials, certified floors.

Qwen 3.8 27B on Strix Halo: the complete measured story

Every dial measured, every number public: the frozen optimal config for a 27B on a $1,400 mini-PC.

Lab Notes: the measured-knees method for reasoning effort

A methods note on reasoning-effort calibration. Thinking rescued 15/40 hards vs 4/40 off. On easy tasks it bought nothing. Publish the knee.

Lab Notes: 67% LiveCodeBench-30 on a $1,400 rig

20/30 = 67% LiveCodeBench-30 on a $1,400 mini-PC. Wilson 95% CI 49-81%. Easy 10/10, medium 8/10, hard 2/10. n=30 public subset. Not the card.

Lab Notes: we reverted a llama.cpp regression

We found a llama.cpp regression and reverted it. n=6 battery on a $1,400 rig: 0/6 slashes before, 5/6 after. Same Q4_K_XL. Not a quant story.

Lab Notes: 93% HumanEval on a $1,400 rig

28/30 = 93% HumanEval on a $1,400 mini-PC. Qwen3.8-27B Q4_K_XL. Temp 0, thinking off. Failures: 50 and 145. Raw log linked.

Lab Notes: the model that can't forget but can't remember

The number first: this model is 75% not a transformer. 48 of 64 layers keep a running state. On a $1,400 rig, load 198k once (1818s), then query in 9-27s.

Lab Notes: the basin was a bug

We published a basin. The serving binary was the hole. After the c7d8722 revert: 6/6 HIT at 198k. n=1 map of the fixed build.

Lab Notes: the KV verdict

The q8-to-q4 KV question from night 1 got its paired answer: same accuracy, zero tripwires, half the cache. Plus the label we had to correct in public when the server's counter beat our estimate.

Lab Notes: verdict night

A $1,400 mini-PC serving a 27B at 262k context asks one question: does capping thinking cost accuracy? The paired answer, the invalid first attempt it survived, and the doctrine that followed.

GPT-5.6 Sol vs. Terra vs. Luna: an evidence-based routing policy for coding agents

The practical GPT-5.6 split is Sol for discovery, Terra for bounded execution, and Luna for repeatable processing - with a different verification contract for each.

Agents need verifiable tools, not better prompt theater

The useful agent pattern is not a prettier prompt. It is a tool surface the agent can call, inspect, verify, and revise.

Repo history is a product signal

A repo is not just storage. It is evidence of decisions, repairs, release behavior, naming drift, test gaps, and what the builder actually knows how to finish.

Implementation help is part of the product surface

A useful open-source tool still needs a path from public repo to working environment. That path is product work, not an afterthought.

Why Kinocut matters

Kinocut gives AI agents callable handles on timelines, effects, Hyperframes, and finished media at kinocut.dev.

Infinite monkeys, LLMs, and the room around the machine

The argument behind the video: output quality is not just probability. It is architecture, filters, and human taste.

What a working AI tool needs before people can use it

A practical checklist for turning a working tool, workflow, or rough app into something other people can understand, install, and use.

MCP server implementation checklist

The checklist Kyanite uses to decide whether an MCP server is a toy, a usable tool, or something worth implementing.

Repo archaeology turns history into proof

Why commit history is one of the strongest proof sources for learning diagnostics, implementation help, and engineering trust.

AI discovery needs more than a sitemap

What Kyanite adds so search engines and AI assistants can understand the tools, products, proof, and support path.