HOW THE HARNESS WORKS

THE PROCESS

• One fixed registry. Prompts, temperatures, token budgets, timeouts, checks and scoring weights all live in a single file (bench.py). The only thing that changes between runs is the model and its server.
• Atomic per-test results. Every run writes result.json (measurements + objective checks) plus the model's raw output, verbatim, into its own folder. Re-running one test never touches another test's data.
• Objective checks where possible. Fibonacci code is actually executed and verified. JSON output is parsed and schema-checked. The three game tests are rendered headlessly and screenshotted — a broken page scores as broken, not as "looks plausible".
• Subjective scoring where necessary. Reasoning, constraints and creative writing are graded 0–10 by a reviewer reading the actual output; the raw output is linked from every model page so you can grade it yourself.
• No soft caps on quality tests. Thinking models legitimately need thousands of tokens of reasoning before answering; capping that is a harness artifact, not a model property. Only the speed probe uses a fixed budget, because there the budget is the measurement.
• Composite score = weighted mean of the 8 scored tests (weights below). The speed probe is reported but not scored — throughput without quality is meaningless.
• Publishing. Generated pages pass a scrubber (internal hosts replaced by public hardware labels) and a leak gate that refuses the build if any internal IP, hostname or port survives, then ship via CI to GitHub Pages.

THE 9 TESTS

Weights sum to 1.0 across the 8 scored tests.

EVERY MODEL, EVERY TEST

Scores 0–10. Green ≥8, yellow ≥5, red below. Click any model for the full per-test breakdown with raw outputs and game screenshots.