By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_), assisted by GLM-5.3. Night 1 of a nightly cadence: what ran, what broke, and what the numbers said.
The number first: a 512-token thinking cap bought the same measured accuracy as unlimited thinking tonight — at 7.5x faster turns. Its shadow number: left uncapped, the same model thought itself to death on 26 of 50 problems, reasoning until the output ceiling killed the answer. Both came out of one paired run. Neither was the night's real story. The real story is that the first version of this experiment measured nothing at all.
The setup
The rig: a $1,400 GMKtec EVO-X2 mini-PC (Ryzen AI Max+ 395, 96GB unified memory) serving Qwen3.8-27B, a 27B dense model, at its full native 262,144-token context. The question for verdict night is the one every local-model operator eventually faces: if I cap how long the model is allowed to think, what does that cost in accuracy?
The instrument
A paired design. Every problem is asked under all three thinking budgets — uncapped, 1024 tokens, 512 tokens — with cell order rotated per problem so any drift hits all arms equally. Fifty problems, enriched toward difficulty (25 hard / 15 mid / 10 easy), temperature 0, full reasoning traces stored in every row. The statistic is exact McNemar on the discordant pairs: not “which average is higher,” but “where the arms disagreed, did one arm systematically win.” Conditions throughout: Q4 dynamic quant, q8_0 KV cache at 262k context, ROCm build of llama.cpp, package holding 90-92 °C at 120W.
The twist: the first experiment was invalid
The control arm — “uncapped” — did not exist. The champion server carries a server-level default, --reasoning-budget 2048, enforced with the message “Reasoning budget reached. Answer now.” Any request that omits an explicit budget silently inherits it. Our first runner omitted it. The “no cap” cell was a 2048 cell wearing a “none” name tag.
The proof was embarrassingly clean: six byte-identical thinking lengths between the “none” and 2048 cells; maximum thinking length identical in both (9,361 characters); the same hard problem truncating at 7,955 characters in one arm and 7,937 in the other. The archived budget curve was never {none, 2048, 4096, 8192} — it was {2048, 2048, 4096, 8192}, and its “none vs 2048: zero discordants” was a cell compared with itself.
Fixes shipped before the rerun: uncapped arms now send an explicit million-token override; the output ceiling went from 6,000 to 12,000 tokens (the old ceiling killed uncapped answers mid-think — an output-cap confound on the very arm labeled uncapped); and every row now carries its full trace, so the next autopsy reads text instead of inferring from counts.
The verdict
| Thinking budget | Accuracy (n=50) | Median wall per problem |
|---|---|---|
| Uncapped (106 override) | 44% | 367 s |
| 1024 tokens | 40% | — |
| 512 tokens | 40% | 49 s |
All three pairwise exact-McNemar tests are non-significant: p = 0.73 for uncapped-vs-512, with 0.63 and 1.0 on the remaining contrasts. On this benchmark, tonight, caps are statistically free.
And protective. The uncapped arm's median completion hit the full 12,000-token ceiling, and on 26 of 50 problems the thinking never terminated at all — the model reasoned past the output cap and emitted no answer. Those score as wrong (no boxed answer = wrong, no exceptions). The cap doesn't only save time; it forces a conclusion the uncapped model sometimes cannot reach on its own.
Both polarities, stated plainly: this is one benchmark (an Omni-MATH subset), one night, n=50 paired problems. A capped cell measures accuracy under forced early termination — the knee we observed sits at or above the model's true budget knee, so read it conservatively. And on the hardest band (difficulty ≥ 5.0), every cell scored a flat 16%: those problems are capability-bound, not budget-bound. A cap cannot take from you what the model never had.
The envelope it rode in on
The run held the box at its measured thermal edge — 90-92 °C package at 120W, fans at 100% — through hours of sustained generation with zero throttling. That is now codified doctrine: this is the chassis envelope, not a misconfiguration; no ritual cooldown breaks (steady-warm beats heat-cycling); and the always-on product lane will get a power cap rather than the experiment lane's full tilt.
What the harness takes from it
The harness's shipped default — a 1024-token thinking budget for margin, 512 where speed matters — earned its number tonight: the nominal 44%-vs-40% gap is inside what this design can detect, and 512 turned 367-second problems into 49-second problems at the same score. The methodology and the raw data are public alongside this post: METHODOLOGY.md (n-sizes, gates, grading rules — the rows benchmark tables never print) and the full paired JSONL, traces in-row, under results/ in the qwen38-27b-strix-halo repo.
Tonight
Window 2 runs the community-corroborated lane: q4_0 KV cache at 262k context. Every 262k row in the community hardware map we mined runs q4-KV; we serve q8 — so tonight is the paired quality A/B our configuration has been missing, plus whatever KV headroom it frees. Lab notes tomorrow night.
Conditions: GMKtec EVO-X2 (Ryzen AI Max+ 395, 96GB unified memory), Qwen3.8-27B Q4 dynamic quant, llama.cpp ROCm build, 262,144-token context, q8_0 KV cache, temperature 0, Omni-MATH difficulty-enriched subset (25/15/10), n=50 paired problems per arm, exact McNemar on discordants, measured 2026-08-17/18. One benchmark, one night, n=50 — a datapoint, not a law.