Local LLM / Benchmarks

The Delegation Card: we asked a $1,400 mini-PC to take our jobs

Not is-it-smart but can-you-hand-it-work-and-walk-away. 495 trials, certified floors.

2026-08-226 minlocal LLM delegation benchmark certified floors Strix Halo

By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_). 2026-08-22. The complete measured answer.

We ended the 27B story with a frozen, measured machine. The next question was obvious: what work can it actually take over? Not “is it smart” — every benchmark answers that. “Can you hand it a job and walk away?” No published benchmark answers that. So we built one.

The method

Nine job classes from real work. Size ladders. Hidden tests the system never sees. A sabotage cell with a planted bad instruction. Certified floors — exact statistics that say “at least X percent reliable.” A walk-away tier: thirty-five perfect trials before any cell earns the “leave it alone” stamp.

The benchmark is open source. The sealed test sets never publish.

The results

495 trials, 29 cells. Every capability cell passed 20/20 (floors 81.9-86.1%). Eight cells earned walk-away certification at 35/35 or 30/30 (floors 90.5-91.8%).

Code: 20/20 at every size. Debugging: 20/20. Document search: 20/20. Trap-bait summaries: 20/20. Data extraction: 20/20. Translation: 20/20. Chained reasoning: 20/20. Decisions: 20/20. Safety: 20/20. Vision: 15/15 on five of six types.

Walk-away certification

Eight cells each ran fifteen additional untouched confirmation trials. Every single one passed: code 35/35, debugging 35/35, document search 35/35, reasoning 35/35, decisions 35/35, vision charts 30/30, vision terminals 30/30. Certified floors: at least 90.5-91.8 percent.

The judge story

Our first graders failed two cells. Reading the raw answers showed the model was RIGHT — our judges were wrong. One broke on a Spanish accent. One could not tell “mentioned the false number to reject it” from “repeated it as fact.” Both bugs are now permanent regression cases.

The honest misses

Giant documents (130k tokens) are impractical — the ceiling is measured; the product now fast-fails with an estimate. Tiny-text transcription: one letter dropped in long email addresses (12/15, labeled).

What this proves

A complete serving story on hardware anyone can buy, measured through a real product, every number reproducible.

The card is public. The benchmark is open source. Point it at your own machine.

Conditions: $1,400 GMKtec EVO-X2, Qwen3.8-27B Q4_K_XL, llama.cpp ROCm, tokflint/tokpal. github.com/KyaniteLabs/delegation-bench.

Work with Kyanite

Want this working in your environment?

If this post describes a Kyanite tool or result you need, implementation help can cover setup, advising, docs, examples, checks, and a usable handoff.

Start implementation intake See implementation help
Fit boundary

Kyanite offers help grounded in its tools, products, and build practice. Broader consulting routes through PuenteWorks.

More Lab Notes

Keep following the system.

Local LLM / Serving4 min
Qwen 3.8 27B on Strix Halo: the complete measured story

Every dial measured, every number public: the frozen optimal config for a 27B on a $1,400 mini-PC.
Local LLM / Methods3 min
Lab Notes: the measured-knees method for reasoning effort

A methods note on reasoning-effort calibration. Thinking rescued 15/40 hards vs 4/40 off. On easy tasks it bought nothing. Publish the knee.
Local LLM / Benchmarks4 min
Lab Notes: 67% LiveCodeBench-30 on a $1,400 rig

20/30 = 67% LiveCodeBench-30 on a $1,400 mini-PC. Wilson 95% CI 49-81%. Easy 10/10, medium 8/10, hard 2/10. n=30 public subset. Not the card.
Local LLM / Benchmarks3 min
Lab Notes: we reverted a llama.cpp regression

We found a llama.cpp regression and reverted it. n=6 battery on a $1,400 rig: 0/6 slashes before, 5/6 after. Same Q4_K_XL. Not a quant story.
Local LLM / Benchmarks3 min
Lab Notes: 93% HumanEval on a $1,400 rig

28/30 = 93% HumanEval on a $1,400 mini-PC. Qwen3.8-27B Q4_K_XL. Temp 0, thinking off. Failures: 50 and 145. Raw log linked.
Local LLM / Benchmarks6 min
Lab Notes: the model that can't forget but can't remember

The number first: this model is 75% not a transformer. 48 of 64 layers keep a running state. On a $1,400 rig, load 198k once (1818s), then query in 9-27s.
Local LLM / Benchmarks7 min
Lab Notes: the basin was a bug

We published a basin. The serving binary was the hole. After the c7d8722 revert: 6/6 HIT at 198k. n=1 map of the fixed build.
Local LLM / Benchmarks5 min
Lab Notes: the KV verdict

The q8-to-q4 KV question from night 1 got its paired answer: same accuracy, zero tripwires, half the cache. Plus the label we had to correct in public when the server's counter beat our estimate.
Local LLM / Benchmarks5 min
Lab Notes: verdict night

A $1,400 mini-PC serving a 27B at 262k context asks one question: does capping thinking cost accuracy? The paired answer, the invalid first attempt it survived, and the doctrine that followed.
Local LLM / Benchmarks14 min
One mini-PC, one night, and the numbers that argued with themselves: tuning Qwen3.8-27B on Strix Halo

A fully measured night-and-evening of tuning a 27B dense model on a Strix Halo mini-PC: the honest bands, the reversal, the crash doctrine, and the EC detective story.
Model Routing / Agent Systems12 min
GPT-5.6 Sol vs. Terra vs. Luna: an evidence-based routing policy for coding agents

The practical GPT-5.6 split is Sol for discovery, Terra for bounded execution, and Luna for repeatable processing - with a different verification contract for each.
MCP Implementation7 min
Agents need verifiable tools, not better prompt theater

The useful agent pattern is not a prettier prompt. It is a tool surface the agent can call, inspect, verify, and revise.
Repo Intelligence6 min
Repo history is a product signal

A repo is not just storage. It is evidence of decisions, repairs, release behavior, naming drift, test gaps, and what the builder actually knows how to finish.
AI Tool Implementation7 min
Implementation help is part of the product surface

A useful open-source tool still needs a path from public repo to working environment. That path is product work, not an afterthought.
MCP / Video Automation5 min
Why Kinocut matters

Kinocut gives AI agents callable handles on timelines, effects, Hyperframes, and finished media at kinocut.dev.
Agent Systems6 min
Infinite monkeys, LLMs, and the room around the machine

The argument behind the video: output quality is not just probability. It is architecture, filters, and human taste.
Build Notes7 min
What a working AI tool needs before people can use it

A practical checklist for turning a working tool, workflow, or rough app into something other people can understand, install, and use.
MCP Implementation8 min
MCP server implementation checklist

The checklist Kyanite uses to decide whether an MCP server is a toy, a usable tool, or something worth implementing.
Repo Intelligence7 min
Repo archaeology turns history into proof

Why commit history is one of the strongest proof sources for learning diagnostics, implementation help, and engineering trust.
SEO / GEO6 min
AI discovery needs more than a sitemap

What Kyanite adds so search engines and AI assistants can understand the tools, products, proof, and support path.