Methodology
BlindTestBench is a blind human-preference benchmark for frontier AI models. Every model gets the same brief; people judge the work without knowing who made what; the rankings come from a statistical model that treats every vote fairly. This page explains each step.
One brief, every model
Each challenge is a creative brief — a website, a small game, an animation, a data visualization, a tool, or a generative art piece. The exact same prompt goes to every model, and each returns a single self-contained HTML file. Nothing is edited, curated, or retried into shape: what the model produced is what you judge.
Briefs are written like a client brief to a design studio, not a spec to a contractor: they state the goal, the audience, the required content and the emotional register — never pixel sizes, hex colors, or verbatim copy. That is deliberate. The benchmark’s value depends on divergence: prompts that dictate implementation produce a row of near-identical transcriptions, and there is nothing to judge. Prompts that state the goal and leave the design decisions to the model reveal how each one actually thinks.
Blind judging
In the arena, outputs appear under anonymous labels in a random order that changes on every page load — no names, no logos, no prices. You pick the single best. Only after committing a verdict do you see the unmasking: who made what, what it cost to generate, and how long it took. Cost and speed are never visible to voters, so they can’t bias the judgment.
The blindness is enforced, not aspirational. Artifact files and URLs are random identifiers that never encode the model. A sanitizer fails any output in which a model signed its work. Outputs render in a sandboxed frame, and pre-vote pages reference them only through sealed, per-render tokens, so nothing in the page source reveals — or can be used to script votes for — a favourite model. The unmasking itself is only served after a vote or an explicit skip is on record, and a skip permanently closes voting on that challenge, so nobody can peek first and vote “objectively” after.
How the ratings work
Each vote is one choice from a set: the voter saw a specific group of models and picked one winner. Ratings come from a global Plackett–Luce fit over every vote at once, displayed on an Elo-like scale where a 400-point gap means roughly 10:1 odds of being preferred.
We deliberately do not use sequential Elo. Elo accumulates points in arrival order, so a model added later starts at the average and has to climb — underrated the whole way, and deflating the incumbents it beats. A global fit has no starting rating and no order: the number of votes a model has affects the precision of its rating, never its position. A model added today is judged purely on its own record, and old votes are never rewritten to include it.
Every rating carries a 95% confidence interval, computed from the same fit. Overlapping intervals mean the gap between two models isn’t established yet — the leaderboard says so rather than pretending otherwise. Models with too little data are marked provisional. Practice replays (revisiting a challenge you already judged) never count toward ratings, because a replay isn’t blind: you have already seen the unmasking.
What the receipts mean
Alongside preference, the benchmark records what each output actually cost to generate (billed API cost) and how long it took. That is the point of the unmasking: whether the expensive model is actually better is an empirical question, and the answer is often uncomfortable. The rating-versus-cost chart on the leaderboard plots exactly that trade-off.
Limits, honestly
This is a human-preference benchmark, not a correctness suite. It measures which output people prefer when they can’t see the badge — design quality, polish, atmosphere, playability — not factual accuracy or code quality under the hood. Vote counts are still small, which the confidence intervals make visible. And votes come from the people who choose to judge, who skew technical. Read the numbers with those caveats attached.