By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_). 2026-08-24. The case for evals as the core forward deployed skill, proven on a public artifact.
Evals are the one forward deployed engineering skill the market says it cannot find, and almost no job posting lists it. An eval is a test you build that answers the only question a customer eventually asks: can I hand this system real work and walk away? If you can design that test, run it honestly, and report it with exact statistics, you can de-risk any AI deployment. That is the whole job in one skill. Mine is public: delegation-bench, 495 trials, open source. This post walks through what it measures and why the design is the part worth copying.
First, why evals and not prompting or fine-tuning. Those are model-side skills. The forward deployed engineer sits on the other side of the model, where the questions are: which of this customer's jobs can this system take over, how reliably, and what happens when it fails. Vibes cannot answer that. Leaderboards cannot either, because they test the model in the abstract, not inside your workflow. Only an eval composed for the real decision answers it.
What the benchmark measures
delegation-bench asks one question: which of the operator's real jobs can be safely delegated to a local AI, consistently? Nine job classes taken from actual work: code, debugging, document search, summarization, data extraction, translation, chained reasoning, decisions, and safety. Each class runs size ladders, meaning small, medium, and large versions of the task. The tests are hidden: the system under test never sees them. The sealed holdout sets will never be published, and that is what keeps our own green honest.
The design was pre-registered: published before any run, so nothing could be tuned after the fact. If you are building an eval for a customer, pre-registration is the trust move. It converts "trust me" into "I committed in writing first."
The numbers
495 trials across 29 cells. Every capability cell passed 20/20, with certified floors of 81.9 to 86.1 percent. Eight cells earned walk-away certification, meaning 35/35 or 30/30 untouched confirmation trials, with floors of 90.5 to 91.8 percent: code, debugging, document search, reasoning, decisions, vision charts, vision terminals. "Certified floor" means exact Clopper-Pearson statistics: "at least 90.5 percent reliable," not "felt solid."
There is a sabotage cell: a planted bad instruction that the system must refuse. And a decision table where every green carries its floor printed on it, so a non-engineer can read the verdict without knowing what a confidence interval is.
The part everyone skips: judging the judge
The hardest layer of an eval is not testing the model. It is testing your grader. Our first automated judges failed two cells. Reading the raw answers showed the model was right and our judges were wrong. One broke on a Spanish accent. One could not tell "mentioned the false number in order to reject it" from "repeated it as fact." Both bugs are now permanent regression cases in the repo.
This is the skill in one story. A customer's deployment "fails" and the question is: model, pipeline, or measurement? The eval builder is the person who can answer, because they built all three layers and tested each. When long context and vision silently broke on my rig, the eval discipline was what caught it: bisected to one upstream llama.cpp commit, reported as issue 26209, fixed and validated 9/9 paired. Without evals, that failure is invisible until a customer finds it.
The honest misses, published
Giant documents around 130k tokens are impractical on this rig. The ceiling is measured and the product now fast-fails with an estimate instead of hanging. Tiny-text transcription drops one letter in long email addresses: 12/15, labeled as such. Publishing the misses is not a weakness in an eval. It is the eval.
How to start building this skill
Pick one repeated task you actually do. Write ten test cases with answers you would stake money on. Run the system, grade blind, count honestly, and compute a floor (20/20 means at least about 82 percent, 30/30 at least about 90 percent). You now have the seed of everything I described. The full method, the statistics, and the code are in the repo.
Related reading: what a forward deployed engineer actually does and how I became one without a software title.
Need evals like this for a deployment you are responsible for? Implementation intake. Conditions: $1,400 GMKtec EVO-X2, Qwen3.8-27B Q4_K_XL, llama.cpp, tested through the real product (tokflint/tokpal). Exact stats: Clopper-Pearson floors, Holm correction across cells. Everything public except the sealed holdouts.