Why self-host at all
The product behind this bench is an “Ask your website” box. The site owner types plain English: change the opening hours, the phone number is wrong, make that heading say this. The platform runs the tool loop server-side and the site changes. No account to create, no API key to paste, nothing to install.
Hosted models already do this well and absurdly cheaply, as the baseline below shows. The reason to self-host is not capability. It is where the words go. For privacy-first customers in the EU and UK, the strongest possible answer is: your customers’ words are processed on dedicated hardware we operate, in EU jurisdiction, and they never leave it. That promise requires the model to run on metal we control.
The standing lesson from every round of our A/B testing: the harness matters more than the model. A small model inside a tight loop, with bounded edit operations and hard verification gates behind it, is safer than a big model free-styling. This bench exists to test exactly how small the model can be.
How the bench works
The corpus is a real five-page website for a small vehicle-testing garage, built end to end by our Discovery AI. Not toy fixtures: full documents of 14–21 KB with the usual mess of real pages.
The protocol
- The engine receives the full document plus one plain-English instruction.
- It must reply with a strict JSON array of
{find, replace}operations. Nothing else. - Every
findstring must match the document exactly once, or the task fails. Blast radius is bounded by construction. - Operations are applied mechanically. No model ever touches the file directly.
- Each task has hard assertions on the result, and the CSS task additionally faces a rendered Playwright check: the fixed element must reach a WCAG contrast ratio of at least 3:1 on the actual painted page.
The ten tasks
- Opening hours: change the hours everywhere they appear (three places).
- Phone number: a number change across visible text and
tel:links (six replacements in total). - Contrast fix: a genuine white-on-white bug. A more specific CSS rule overrides the intended link colour inside a dark call-to-action band, so the phone number renders invisible. Fix the specificity properly, without
!important. Judged on the rendered page. - Price change: a fee changes everywhere it appears (three places).
- Add a card: add a new service card following the exact markup pattern of eight existing cards, in a stated position.
- Accent colour: change a CSS custom property to a new hex value.
- Remove an element: delete one item from a list of place names, leaving the rest untouched.
- Hero copy: replace the page’s
h1text exactly, keeping the tag and attributes. - Footer small print: insert a new sentence at a stated position in the footer.
- Names: a staff change phrased naturally, requiring the same phrase replaced in four places.
Why the rendered gate is non-optional. In the July round, the only real correctness failure across seventy runs passed every textual assertion. The engine had emitted a literal \n inside a CSS replacement, silently breaking every rule that followed it and leaving ten newly invisible elements. Only the rendered check caught it. Text diffs tell you what changed; they do not tell you what the customer now sees.
Every engine runs the same tasks two ways: single-shot, and with one corrective retry in which a mechanical failure (a find string that did not match exactly once, or malformed JSON) is fed back to the model, exactly as our production loop does. Semantic misses never get a retry, because production cannot detect those automatically either.
The hosted baseline
Seven hosted engines ran the bench on 25 July 2026. Costs are per completed edit at each provider’s list prices as fetched that day; check current prices before relying on them.
| Engine | Score | Median | ~Cost per edit | |
|---|---|---|---|---|
| Claude Haiku 4.5 | 10/10 | 2.3s | $0.0070 | |
| Claude Sonnet 5 | 10/10 | 2.5s | $0.0255 | |
| Kimi K3 | 10/10 | 9.8s | $0.0199 | |
| DeepSeek V4-Flash | 10/10 | 12.6s | $0.0011 | |
| GPT-5 mini | 10/10 | 12.9s | n/a | |
| Kimi K2.7 Code (highspeed) | 9/10 | 3.5s | $0.0154 | |
| Gemini Flash Lite | 8/10 | 0.8s | $0.0022 |
Five engines at 10/10, and the cheapest of them at a tenth of a cent per edit. K2.7’s single failure was the render-only CSS break described above. Hosted, this problem is solved; the open question was never capability but data residency.
The self-hosted round
On 28 July we ran four open-weight models through the identical bench, served locally with llama.cpp at 4-bit quantisation. All four are Apache 2.0 licensed and freely downloadable; the Gemma 4 line went Apache in April 2026, which is why it stands in for the older, licence-gated Gemma 3 that prompted this round.
| Model | Single-shot | VRAM (Q4) | Failure profile | |
|---|---|---|---|---|
| Qwen3.6-27B | 9/10 | ~16.5 GB | One non-unique find string; the edits themselves were correct | |
| Gemma 4 12B (QAT) | 8/10 | ~7 GB | Under-counted the two heaviest multi-occurrence tasks | |
| Gemma 4 26B-A4B (QAT) | 6/10 | ~15 GB | Mechanical only: find-uniqueness and JSON escaping slips | |
| Devstral Small 2 24B | 6/10 | ~14 GB | Mechanical slips plus one hallucinated find string |
Two results stand out. First, every one of the four fixed the white-on-white contrast bug properly, on the rendered page, without !important: the task a fast hosted model failed in July. Second, almost every local failure was mechanical rather than semantic: a find string quoted with too little context, an invalid JSON escape. Those are precisely the errors the harness detects deterministically.
The thinking trap. With its default reasoning behaviour, Gemma 4 spent up to ten thousand tokens deliberating about a one-line edit and timed out. With reasoning switched off at the server, the same edit took nineteen tokens and under two seconds. If you deploy a reasoning-capable small model into a tight tool loop, turning thinking off is part of the configuration, not an optimisation.
Latency on this round came from a laptop, not a server, so we publish scores here and leave speed to the forthcoming GPU-server pilot. For reference, generation ran at roughly 13 tokens per second on the 12B and the tasks above completed in 10–100 seconds each.
Second chances: the production-faithful retry
Our production loop does not give a model one blind shot. When an edit operation fails to apply, the error goes back to the model and it corrects itself. The bench mirrors that with exactly one retry, and only for mechanical failures.
| Model | Single-shot | With one retry | |
|---|---|---|---|
| Qwen3.6-27B | 9/10 | 10/10 | CLEARS THE BAR |
| Gemma 4 26B-A4B (QAT) | 6/10 | 6/10 | REPEATS ITS MISTAKES ON RETRY |
| Devstral Small 2 24B | 6/10 | 9/10 | ALL 3 MECHANICAL SLIPS HEALED; 1 SEMANTIC MISS REMAINS |
| Gemma 4 12B (QAT) | 8/10 | 8/10 | SEMANTIC MISSES, NO RETRY POSSIBLE |
Qwen’s single miss illustrates the whole story. Asked to change a recurring phrase in four places, it produced the correct four edits but quoted one find string with too little surrounding context, so the apply step refused it. Shown the error once, it returned the corrected set and passed, rendered gate included. That is not the bench being generous; it is the bench finally behaving like the production loop it feeds.
The dividing line is worth stating plainly: mechanical errors self-heal in a well-built loop. Semantic errors do not. Gemma 4 12B’s two misses were under-counts that no automatic check can flag, so no retry can rescue them. That asymmetry, not raw score, is what should drive model choice for autonomous editing.
What it runs on, and what that costs
Prices below were fetched on 28 July 2026, exclude VAT, and move constantly: treat them as a snapshot. The jurisdiction column matters as much as the price for an EU data-residency promise, because a US-headquartered provider carries US legal reach into its EU regions.
| Option | Card / VRAM | Price | Jurisdiction | Note |
|---|---|---|---|---|
| Hetzner GEX44 | RTX 4000 SFF Ada, 20 GB | Out of stock | EU (DE) | The natural small box, when it returns |
| Hetzner GEX131 | RTX PRO 6000, 96 GB | €889/mo, no setup | EU (DE) | Best price per GB of VRAM for always-on |
| IONOS GPU (A10) | A10, 24 GB | £1.04/h, capped £748/mo | EU (DE) | Billed by the minute, no minimum term |
| IONOS GPU (T4) | T4, 16 GB | £0.88/h, capped £633/mo | EU (DE) | Older silicon, poor value beside the A10 |
| OVH / Scaleway | various small cards | from ~€0.36/h / €0.79/h | EU (FR) | Cheapest EU always-on if shopping wider |
| Vultr (Frankfurt) | A16 slice, 8 GB | $172/mo | US-headquartered | 16/24 GB single-die plans showed no availability |
The fit is comfortable. The winning 27B needs roughly 16.5 GB at 4-bit, so a single 20–24 GB card carries it with headroom for context. The 12B runner-up fits a 16 GB card at around 7 GB. Nothing in this report needs a datacenter-class H100, and nothing needs more than one GPU.
Verdict
Yes: a self-hosted, Apache-licensed model on a single mid-range EU GPU clears the same bar as frontier hosted models, inside this harness.
Qwen3.6-27B went 10/10 with one production-faithful retry, rendered gate included. Gemma 4 12B is a credible runner-up at a third of the VRAM, held back only by semantic under-counts on the heaviest multi-occurrence edits. The protocol, the exactly-once find rule, and the rendered verification gate are doing as much work as the weights are; take the gates away and none of these numbers mean anything.
What happens next. The winner goes onto a rented EU GPU server behind a proper inference stack, where we measure real latency and concurrency rather than laptop numbers, and then into production behind the same gates that judged this bench. Costs a few pounds to find out; the report will be updated when it has.
Every number in this report comes from harness output files, not estimates. Methodology questions are welcome: the protocol is described in full above, and the task set is deliberately small, public and repeatable.