By Simon Gonzalez de Cruz (follow the build in public on X @KyaniteLabs_). 2026-09-25. Numbers measured 2026-09-24; every artifact on disk.
Our working thesis, stated publicly enough times that it deserved measurement: a small local model plus a deterministic tool layer should match a frontier model working alone. For video editing we now have the numbers, and they are cleaner than the thesis predicted.
The short version
A 3-billion-parameter local model, asked to write raw ffmpeg commands, produced zero working commands out of twelve attempts. The same model, driving Kinocut’s typed surface instead, shipped 4 of 12 — at lower latency than a 35B MoE writing raw commands. At 2B the lift is +80%. At 27B it is +10%. At 35B, parity. The guardrail compensates exactly where the model cannot, and stops mattering where the model is already competent. That curve is the whole product argument, now with cells instead of adjectives.
The rig
Twelve representative edit tasks (trim, resize, audio extract to mp3 and wav, 2× speed, rotate, thumbnail, WebM, center crop, 1fps frame export, animated GIF, self-concat) on one deterministic fixture: testsrc2 + sine, 10 seconds, 1280×720, H.264 + AAC. Two arms per seat. Alone: the model writes a raw ffmpeg command — what an unguarded agent does. + Kinocut: the model drives the kino CLI with its real help text in context, exactly as an MCP client sees the tool schema, and the deterministic engine builds and runs the actual ffmpeg. One verifier, blind to arm: the artifact must exist and pass every ffprobe check (duration, dimensions, codecs, streams, frame count). Commands execute argv-only — no shell — sandboxed, 240-second cap, temperature 0, fixed seed.
| local model | alone (raw ffmpeg) | + Kinocut | latency, median |
|---|---|---|---|
| 3B VL | 0 / 12 | 4 / 12 | 540 ms → 337 ms |
| 2B | 5 / 12 | 9 / 12 (+80% rel.) | 5.6 s → 2.2 s |
| 27B | 10 / 12 | 11 / 12 | 2.2 s → 2.6 s |
| 35B MoE | 11 / 12 | 11 / 12 (parity) | 18.3 s → 9.6 s |
Full tables, method, and the per-cell caveats live at kinocut.dev/bench.
The trim trap
Every seat’s raw-arm “canonical” trim was the command an experienced human writes:
ffmpeg -i input.mp4 -ss 2 -to 5 -c copy out.mp4
It exits zero. It reports the right duration. It silently drops the video stream — the stream-copy and output-seek interaction. A 27B model still writes it. This is the failure class that makes operators distrust agent-driven editing, and it is invisible unless you probe the output file. Our verifier probes the output file; that is the entire point of the blind-verification design.
The bug our own bench caught
The bench also indicted Kinocut itself. The documented absolute-end trim form (kino trim -s 2 -e 5) produced a 5.00-second clip instead of a 2→5 cut: validation treated the end time as absolute while execution — input-seek rebasing timestamps to zero — treated it as relative. Two different seats used the documented form correctly and were scored FAIL by the rubric; the layer’s own bug cost it the task. Fixed in 1.15.2: the engine now converts the absolute end explicitly when input-seeking. With the fix, the capable seats read 10–12/12 on the Kinocut arm. A benchmark that cannot fail your product is marketing; this one filed the defect before any user could.
Why this matters beyond kinocut
The same harness shape — deterministic layer as the independent variable, blind verifier as the judge — now runs across our portfolio (tastecheck’s rubric unlock, checkyourself’s guaranteed floor). The pattern generalizes: stop asking whether models are smart, start measuring what your tool surface does to the competence floor. The frontier row in our tables is empty by policy until a sanctioned credential exists; we do not fabricate reference rows.
Reproducibility
- Fixture:
ffmpeg -f lavfi -i testsrc2... -f lavfi -i sine...10s 1280×720 h264+aac, built by script, checksummed. - Seats: four local models on the lab floor (3B VL, 2B, 27B, 35B MoE), on-disk weights, zero paid APIs, zero spend.
- Scoring: ffprobe-based blind verifier; argv-only execution; 240s cap; temperature 0, seed 42; single-shot (no repair rounds in either arm).
- Artifacts: runner scripts, fixture builder, and per-cell result JSON preserved in the bench directory; the public distillation is kinocut.dev/bench.
Caveats we mean
n=12 tasks, one run per cell; per-cell differences of ±1 are noise. Frontier hosted models were not run (no sanctioned credential; the row stays empty rather than invented). ASR/transcription and GPU-heavy renders were out of scope for the headless loop. The latency column is median single-shot latency under the bench harness, not a serving benchmark. And the K-1 bug means the published table understates the Kinocut arm on capable seats — we are leaving the measured numbers, not the flattering ones, in the table.
Build on it, break it, or show us your hardware’s curve — that’s the conversation. Find us through the lab.