Building the Ed-o-meter: Notes on Writing My Own LLM Benchmark

Every model launch arrives with a wall of benchmark scores: MMLU, GPQA, SWE-bench, an alphabet of acronyms that grows with each release. The numbers are always fascinating. The problem is that each score measures expected performance in domains other than yours. A model can top a maths reasoning suite and still fumble the tasks your team actually runs.

The question that matters in production is more specific, because LLMs get expensive at scale. Once something runs millions of times a month, the pressure is to minimise tokens, spend and latency by running the smallest model that still clears your bar, that is "good enough" for your application rather than "good enough" to top a public leaderboard or be AGI. We made that argument at length in our original GLM-5.2 vs GPT-5.5 comparison; this post is about what happened when we acted on it.

The short answer: Featherbench is the harness — a single Python file that runs 28 fixed tasks against any model and grades them. The Ed-o-meter LLM Leaderboard is what the harness produces. One variable changes between runs (the LLM), every model judges every response blind, and the raw results are made public for maximum transparency.

There is a second reason to run your own. Increasingly the public suites are a question the models have already seen: benchmark maxxing is a documented problem, and labs under real competitive pressure have every incentive to optimise for the suites everyone reports. That is Goodhart's law applied to evaluation: a benchmark everyone trains against stops measuring what it was built to measure. Simon Willison's pelican-riding-a-bicycle SVG test is probably the best-known public illustration: it worked precisely because the scene was too absurd for any model to have memorised a good answer, and it's now famous enough that there's a live debate over whether labs have started training against it specifically. A private task set, written after the models shipped and embedding documents that exist nowhere else, is immune to this.

So I ran the measurements myself, on my own under-the-radar realworld tasks. Everyone already vibe-checks a new model, pasting in a few favourite prompts and forming an impression. This isn't more scientific than that so much as it adds a modicum of rigour: the same prompts every time, a written-down pass bar, costs and latencies recorded, and the whole thing repeatable when the next model drops.

What Does It Mean to Hold One Variable Constant?

Nothing changes between runs other than the model under test. The 28 tasks, the prompts, the answer keys, the checkers, the rubric, the scoring and the routing are all fixed and tracked in source control, so any two runs differ by exactly one thing. That is the whole design constraint: if a score moves, the model moved it, not a quietly re-tuned prompt or a changed grader. It sounds obvious, but most informal model comparisons vary the prompt and the model at the same time and then attribute the difference to the model.

Why Build a New Eval Harness?

There are excellent eval frameworks already. Inspect AI, promptfoo and Braintrust are all more capable than what I built, and if you need tracing, dataset versioning or a UI, use one of them.

My requirement was narrower: I wanted to be able to run my own realworld tasks, like get a sour dough recipe for my mum, audit the entire measurement path. When the results look odd, I can review the code myself, no need to log a ticket with a support desk to tell you what's happening in their proprietary solution. So Featherbench is a single Python file, just over 1k lines, with a dependency on jinja2 so I can format a report at the end. Tasks are JSON, not code, so they diff cleanly in review and a non-engineer can author one. Three provider families run through the same scaffold. It's MIT-licensed, since a result that can't be reproduced is a claim, not a measurement.

The models are called through OpenRouter rather than each vendor's API directly. Lazy or clever, I wanted to offer a way to eval models using one API key and one billing relationship instead of registering with a dozen providers. This makes adding a model to the panel a one-line change rather than a new integration project. But it earns its place on measurement grounds too. Open Router offers pretty much every model, whether a frontier lab's flagship or an open-weight release way before anyone else as it doesn't do any hosting. It's just the universal endpoint to the models.

Task Design: Check the Floor, Judge the Ceiling

Most useful answers can't be fully graded by machine. So task design splits into two approaches:-

The floor is the objective minimum a checker can verify: a constraint respected, a fact present, a dangerous tool not called. The technique that makes loosely-specified tasks gradeable is authoring the answer key — e.g. I wrote the tenancy agreement embedded in the prompt, I know the correct notice period, and a checker can assert it. A vegetarian-recipe checker forbids meat words; it makes no attempt to decide whether the recipe is good. Encoding taste in a regex produces false confidence, so the checker stops at the floor.

The ceiling — pacing, tone, sensible trade-offs — goes to an LLM rubric. Here I made the design decision that later proved most useful: every selected model judges every response, blind, and the harness publishes a judge-bias matrix — the mean score each judge gives each contestant. I couldn't eliminate judge bias, so I settled for measuring it. Most published evals use a single judge that is quietly one of the contestants and leave the bias as an assumption.

Grading is black-box by design: the checker or rubric only ever sees the final input and output, never how the model got there. That is what lets one task stand in for either a single atomic capability or an entire compressed process. A debugging task tests one fix. A holiday-planning task asks for a finished itinerary and judges only the itinerary — the checker has no opinion on how many draft passes or dead ends the model took to reach it. Both are "one task" in the same sense: one thing gets judged, however many steps produced it.

I also landed on three further rules during authoring:

• One thing judged per task, not one action. A task can compress an entire process — researching, sequencing, planning — as long as there is one unambiguous thing the checker or rubric is judging at the end. The discipline is that the verdict is singular, not that the model only does one thing to reach it.
• Labelled sub-checks. A failure should provide detailed feedback on what went wrong.
• Negative controls for security testing. Security testing is about tricking a model to do something it shouldn't. So we seed a canary the model would only emit if it failed, then assert its absence.

How Should a Benchmark Count Refusals?

Models sometimes refuse a task when it trips some built-in policy. How to score that is a genuine point of debate, and it materially moves rankings. Featherbench counts benign refusals as failures, on the reasoning that from the user's side a refused bug fix and a failed bug fix are indistinguishable — either way the work you needed doesn't get done. The raw JSONL is published so anyone who prefers different accounting can re-score it.

Why Five Categories, Weighted Unevenly

The 28 tasks split into five categories — coding, data, realworld, security and tool-use — and the split is deliberately lopsided: nine realworld, seven coding, six security, four data, two tool-use. Realworld is the largest because it is the actual point of the exercise: planning a holiday, pushing back on bad health advice, reading a tenancy agreement, formatting an answer under a constraint. The other four categories exist so a model that is excellent at chatty realworld tasks and quietly bad at debugging or jailbreak resistance doesn't get to hide behind a single average.

Coverage, not balance, is the goal. Tool-use has two tasks not because function-calling matters less, but because two was enough to catch what needed catching first — the framework assumes from the outset that tasks get added and adapted over time, which is also why tasks are JSON rather than code: a new one is a diff, not a pull request against the harness itself.

The same logic governs trials. Given a fixed token budget, the choice was width over depth: run every model against the full spread of task types once, rather than running a narrower set many times over for a false sense of precision. There's no minimum trial count below which a result gets held back — every pass rate publishes with its Wilson interval, however wide, drawn as whiskers on the pass-rate chart rather than smoothed into a single number. A wide whisker is the harness telling you it hasn't earned a tighter claim yet, not a flaw to hide.

Mistakes I Made Building the Harness

Even something this simple can be tricky to get right.

I over-engineered the provider layer before I understood the actual constraint. My first version was a plugin architecture, an adapter per provider so any of them could be swapped in cleanly — the right instinct for extensibility, the wrong one for a weekend project once I saw how many separate accounts it demanded. OpenRouter was the actual fix, and the plugin architecture got deleted rather than finished.

The vegetarian recipe check kept giving me incorrect failures. My forbidden-term checker for the vegetarian recipe flagged a response that contained any animal product terms, e.g. a warning that "some stock cubes contain animal products" = FAIL. I added a negation-aware shield to cover this.

I was overly paranoid about running these models I bought separate hardware to isolate the runs from my day-to-day work, though a Docker sandbox would have done the job - the framework only currently asks a model to generate text, not execute anything, so the actual blast radius was smaller than I'd assumed.

What Did I Learn Through the Process?

Whether the exercise was worth it comes down to whether you learn something new that wasn't readily reported. I learned quickly why most people don't do this (it's harder than it looks), but also gained some practical understanding of the nuances of these LLMs. Whilst the market sees the commoditisation of the capabilities, the behaviours of the models are diverging, making them less interchangeable if you want the optimal results.

For instance, the top rubric score went to Kimi-3, which aligns with the hype. But its median time-to-first-token was 26.4 seconds — several times longer than anything else on the LLM Leaderboard. That latency makes it impractical for interactive work: highest quality on paper, too slow to use.

It soon became clear that Fable-5 was a very fussy model. It hard-refused a run of tasks that nothing else thought twice about, which made it a poor fit for real-world use. The full refusal breakdown is in the original write-up. Most leaderboards make that accounting choice silently; making it explicit, with the raw data published, changed both the result and the conversation about it.

Another surprising finding was how badly the GPT-5.6 range performed on jailbreak canary tests, much worse than GPT-5.5 or similarly capable models. OpenAI have clearly taken a different architectural approach for this range, and it is the kind of regression you only catch if your harness tests for it directly.

What's Next for the Ed-o-meter

• Deepen trials selectively where a result is worth resolving more precisely, without abandoning single-pass coverage as the default.
• A judging panel approach, so no score on the board is self-assigned.
• A fresh run at each major model release, which is the only way the numbers stay worth reading.

The harness, the 28 verbatim prompts and the raw results are on GitHub. If you find a flaw in the method, I'd genuinely rather hear it than not.

Featherbench is MIT-licensed. The Ed-o-meter is re-run on every major model release. To get future posts as they are published, leave your email address.

Frequently asked questions

What is the difference between Featherbench and the Ed-o-meter?

Featherbench is the eval harness: a single Python file that runs a fixed set of tasks against any model and grades the results. The Ed-o-meter is the leaderboard it produces, published as a public results page. Featherbench is the test track, Ed-o-meter the leaderboard of results.

Why build a new LLM eval harness instead of using Inspect AI or promptfoo?

The strength of a comprehensive Eval solution is also its weakness: the weight of the framework, the cost to run all the Evals. I'd rather have a simple, auditable solution that gets the job done; and is simple to bolt onto any project without engineering. The KISS principle.

How should a benchmark count model refusals?

Featherbench counts benign refusals as failures, on the reasoning that the user wants something from the model and if it doesn't produce anything then it leaves them disappointed. It's debatable but I've gone with the assumption most users want to get stuff done!

Why run models through OpenRouter rather than each vendor's API?

One endpoint, one streaming path and one latency clock for every model, so no vendor-specific SDK is quietly doing its own retries or connection pooling and contaminating the comparison. It also makes adding a model a one-line config change, which is how the panel can include brand new open-weight releases. Open Router does have a route to alternatives feature, so the risk of a router re-serving a request on a different upstream or a quantized variant is pinned shut with allow_fallbacks:false.

Why does Featherbench grade only the input and output, not the steps in between?

Grading is black-box by design for simplicity, so the checker or rubric never needs to know how a model reached its answer, only whether the one thing being judged came out right. That is what lets a single task stand in for either an atomic capability, like fixing one bug, or an entire compressed process, like planning a holiday, judged purely on the finished itinerary.

Why are the 28 tasks split so unevenly across categories?

Coverage, not balance, is the goal. Realworld carries nine of the 28 tasks because everyday planning and advice-giving is the actual point of the exercise; tool-use has two because that was enough to catch what needed catching first. The taxonomy exists so a model that excels at one kind of task can't hide a weakness in another behind a single average, and the task set is built to grow: tasks are JSON specifically so a new one is a diff, not a change to the harness itself.

Why does the Ed-o-meter publish wide confidence intervals instead of running more trials?

Given a fixed token budget, the design choice was width over depth: every model runs the full spread of 28 tasks once, rather than a narrower set many times over. There's no minimum trial count a result has to clear before publishing — every pass rate ships with its Wilson interval, however wide, shown as whiskers on the chart rather than smoothed into a false-precision average.