Every model launch arrives with a wall of benchmark scores: MMLU, GPQA, SWE-bench, an alphabet of acronyms that grows with each release. Most are testing something genuinely important. The problem is that each score measures expected performance in some domain rather than performance in yours. A model can top a graduate-level reasoning suite and still fumble the tasks your team actually runs, like extracting a notice period from a tenancy agreement, or declining to follow instructions hidden in a customer review.
The question that matters in production is more specific, because LLMs get expensive at scale. Once something runs millions of times a month, the pressure is to minimise tokens, spend and latency by running the smallest model that still clears your bar, where "good enough" is defined by your application rather than a public leaderboard. Solving that requires your bar to exist as something a machine can check, and quality, cost and latency to be measured on the same run so the trade-off is visible. We made that argument at length in our original GLM-5.2 vs GPT-5.5 comparison; this post is about what happened when we acted on it.
The short answer: Featherbench is the harness — a single Python file that runs 28 fixed tasks against any model and grades them. The Ed-o-meter LLM Leaderboard is what the harness produces. One variable changes between runs, refusals count as failures, every model judges every response blind, and the raw results are public. Three findings came out of it that no vendor chart would have shown, and the mistakes I made building it were more informative than the results.
There is a second reason to run your own. Increasingly the public suites are a question the models have already seen: benchmark contamination is a documented problem, and labs under real competitive pressure have every incentive to optimise for the suites everyone reports. That is Goodhart's law applied to evaluation: a benchmark everyone trains against stops measuring what it was built to measure. A private task set, written after the models shipped and embedding documents that exist nowhere else, is largely immune. Nobody benchmark-maxxed against my tenancy agreement, because until the run it didn't exist. And by the time a public number reaches you it has usually shed its conditions anyway: which harness, which settings, which caveats.
So I ran the measurements myself, on my own tasks, and published enough that anyone can re-derive them. Everyone already vibe-checks a new model, pasting in a few favourite prompts and forming an impression. This isn't more scientific than that so much as it adds a modicum of rigour: the same prompts every time, a written-down pass bar, costs and latencies recorded, and the whole thing repeatable when the next model drops.
One Variable
The design constraint came first: if two numbers on the board differ, the only permissible cause is the model. That means one harness, verbatim prompts, the same checkers, the same latency clock, the same cost arithmetic, for every model. It sounds obvious. It rules out almost every published comparison, where prompts, scaffolding, retry logic and graders all vary between measurements and the reader is left to hope the differences cancel out.
Holding everything constant is unglamorous work — most of the engineering effort in this project went into making sure nothing interesting happens between the prompt and the score.
Why Build a New Eval Harness?
There are good eval frameworks already. Inspect AI, promptfoo and Braintrust are all more capable than what I built, and if you need tracing, dataset versioning or a UI, use one of them.
My requirement was narrower: I wanted to be able to audit the entire measurement path. When a number looks wrong, the distance between "that's odd" and "here's the cause" should be one file read. So Featherbench is a single Python file, around 1,150 lines, with two dependencies. Tasks are JSON, not code, so they diff cleanly in review and a non-engineer can author one. Three provider families run through the same scaffold. It's MIT-licensed, since a result that can't be reproduced is a claim, not a measurement.
A related plumbing decision turned out to matter more than it looked. The models are called through OpenRouter rather than each vendor's API directly. Partly this is laziness with a defensible face: one API key and one billing relationship instead of registering with a dozen providers, which is also what makes adding a model to the panel a one-line change rather than an integration project. But it earns its place on measurement grounds too. Every model, whether a frontier lab's flagship or an open-weight release most shortlists omit, arrives through the same endpoint, the same streaming path and the same latency clock, with no vendor-specific SDK quietly doing its own retries or connection pooling and contaminating the comparison. It also means the panel can include models without discrimination, which is how the board came to include contestants a vendor-by-vendor integration effort would never have got around to. The one risk this introduces, a router silently re-serving a request on a different upstream or a quantized variant, is pinned shut with allow_fallbacks:false, so the model named in the config is the model that was measured.
The trade-off is deliberate. A single file has a scaling ceiling and I'll hit it eventually. Until then, being able to read the whole eval in one sitting is what makes it debuggable in an afternoon, which I ended up needing sooner than expected.
Task Design: Check the Floor, Judge the Ceiling
Most useful answers can't be fully graded by machine. Rather than pretend otherwise, each task splits into two layers.
The floor is the objective minimum a checker can verify: a constraint respected, a fact present, a dangerous tool not called. The technique that makes loosely-specified tasks gradeable is authoring the answer key — because I wrote the tenancy agreement embedded in the prompt, I know the correct notice period, and a checker can assert it. A vegetarian-recipe checker forbids meat words; it makes no attempt to decide whether the recipe is good. Encoding taste in a regex produces false confidence, so the checker stops at the floor.
The ceiling — pacing, tone, sensible trade-offs — goes to an LLM rubric. Here I made the design decision that later proved most useful: every selected model judges every response, blind, and the harness publishes a judge-bias matrix — the mean score each judge gives each contestant. I couldn't eliminate judge bias, so I settled for measuring it. Most published evals use a single judge that is quietly one of the contestants and leave the bias as an assumption.
Three further rules hardened during authoring:
- One capability per task. A task that probes five things is uninterpretable when it fails.
- Labelled sub-checks. A failure should name the specific bar that was missed, not just report a red mark.
- Negative controls for resistance tasks. Seed a canary the model would only emit if it failed, then assert its absence.
How Should a Benchmark Count Refusals?
Models sometimes refuse, and what a benchmark does next quietly determines its results. Retrying on another model attributes one model's output to another. Dropping the refusal from the denominator flatters the refuser. The harness logs a provider-side hard stop as a refusal with its category, pins routing so a provider can't re-serve the request on a different upstream, and counts benign refusals as failures — on the reasoning that, from the user's side, a refused bug fix and a failed bug fix are indistinguishable.
That last choice is contestable, and it materially moves one model's ranking. The raw JSONL is published, so anyone who prefers the other accounting can compute it.
The Decisions That Mattered
Four choices did most of the work. Each bought something specific and cost something specific.
| Decision | Why | What it costs |
|---|---|---|
| Single file, two dependencies | The whole measurement path is auditable in one read, so a surprising number is one file away from an explanation. | A scaling ceiling, and none of the tracing, dataset versioning or UI a real framework gives you. |
| One endpoint for every model | Same streaming path and latency clock for all contestants; adding a model is a config entry, not an integration. | A dependency on the router, and the fallback risk that allow_fallbacks:false exists to close. |
| Check the floor, judge the ceiling | Objective minimums are machine-verifiable; taste isn't. Splitting them keeps false confidence out of the checkers. | Half of every score depends on model judgement, which is why the bias matrix is published alongside it. |
| Benign refusals count as failures | From the user's side, a refused bug fix and a failed bug fix are indistinguishable. | It's a judgement call that moves one model from mid-table to last, so the raw data has to be public. |
Mistakes I Made Building the Harness
The parts of the process I'd actually want to read about, had someone else written this.
Checker false positives cluster where models are careful. My forbidden-term checker for the vegetarian recipe flagged a response that warned "some stock cubes contain animal products" and another that listed what its recipe omitted. The checker read a safety advisory and a negated list as ingredients. I added a negation-aware shield; it caught some phrasings and missed others. The general lesson: a binary checker's errors aren't randomly distributed — they concentrate on exactly the responses that are being most thorough. Validate every checker against a hand-written good answer and a bad one before trusting it, and read the transcripts whenever a result surprises you.
Running model code is a security decision. Coding tasks execute model-generated code against unit tests. The subprocess gets a stripped environment so an answer can't read API keys, and the README tells you to sandbox the whole harness. Any eval framework that runs model output and doesn't document this deserves suspicion.
One trial is a first read. With 28 tasks and a single trial each, the Wilson intervals are wide and overlap. On the LLM Leaderboard they're drawn as whiskers on the pass-rate chart rather than relegated to a footnote, so the presentation looks as uncertain as the data is. Multi-trial runs are the first item on the roadmap.
What the Measurement Produced
Whether the exercise was worth it comes down to whether it surfaced anything the vendor charts don't. Three results suggest it did.
The top rubric score went to a model most shortlists omit, scored blind by a competitor, with a median time-to-first-token of 26.4 seconds — several times longer than anything else on the LLM Leaderboard. Best quality, near-lowest cost, unusable latency for interactive work: that profile only becomes visible when quality, cost and latency are measured on the same run, and no single-axis chart would show it.
The refusal accounting was similarly decisive. One model hard-refused five benign tasks, including a production bug fix, and whether those refusals count as failures or vanish from the denominator moves it from mid-table to last. The full refusal breakdown is in the original write-up. Most leaderboards make that choice silently; making it explicitly, with the raw data published, changed both the result and the conversation about it.
The third finding was the narrowest and the most surprising. Two variants within one model family emitted the jailbreak canary in 10 of 12 security cells, while their sibling and every other family passed cleanly — a split inside a single product line that no aggregate score would reveal.
None of these required a sophisticated benchmark, only a constrained one: everything held constant, refusals counted, bias measured, and results reproducible from a public repository.
What's Next for the Ed-o-meter
- Multi-trial runs to tighten the confidence intervals, which are currently wide enough to overlap across most of the board.
- An independent judge, so no score on the board is self-assigned.
- A fresh run at each major model release, which is the only way the numbers stay worth reading.
The harness, the 28 verbatim prompts and the raw results are on GitHub. If you find a flaw in the method, I'd genuinely rather hear it than not.
Featherbench is MIT-licensed. The Ed-o-meter is re-run on every major model release. To get future posts as they are published, leave your email address.