Methodology

BenchLeader is an aggregator. It does not run models, write prompts or grade answers. Once a day it fetches the published results of independent evaluators and arenas, reconciles the different names they use for the same model, and lays the results side by side. The name says what it is: every benchmark, one leaderboard. No single board tells you where a model stands; the picture only becomes clear when all of them are read together.

What counts as a result

• Published by an independent evaluator or a crowdsourced arena. Numbers from a model’s own launch post are not used.
• Machine-readable at the source: a CSV, JSON or dataset the publisher maintains. We do not screen-scrape charts.
• Attributable: every value on this site links to the page it was taken from, with the publisher’s licence noted on the sources list.

One model, many names

GPT-5 (high), openai/gpt-5, gpt-5-2025-08-07 and gpt-5_high are one model evaluated in different configurations. Labels are reduced to a slug, and tokens that describe a run rather than a different model (reasoning effort, thinking budget, dated snapshot, scaffold, quantisation) are split off as the run’s variant. The slug is then matched against a curated registry of aliases. Anything unrecognised becomes a provisional model automatically, so a new release shows up on the day a leaderboard adds it.

When a source lists several runs of the same model, the model page shows the best result and lists the others. For a quality benchmark that is the highest score; for price and speed it is the first-party provider where one exists, otherwise the median across providers.

The BenchLeader Index

Benchmarks report on incompatible scales: percentages, Elo-style ratings, indices with different ceilings. To combine them, each benchmark’s scores are standardised across the models it has evaluated: the population mean becomes 50 and one standard deviation becomes 15 points, clipped at ±3 standard deviations. Only benchmarks with at least five evaluated models are standardised.

Standardised scores are averaged within each category using the weights below, and the BenchLeader Index is the average of a configuration’s category scores, pulled toward 50 by n/(n+3) where n is the number of results behind it. Six results keep two-thirds of their distance from average, twelve keep four-fifths, thirty keep more than nine-tenths. This stops a configuration that was only run on a handful of favourable boards from outranking one measured everywhere. A configuration needs results on at least five benchmarks across at least two categories to receive an index. Weights halve benchmarks that are saturated, narrow, or overlap heavily with another board.

Known limitations. Standardising within each benchmark means a model’s score depends on who else was evaluated there: a board that only tests frontier models compresses the differences between them. A model evaluated on few, easy benchmarks can look better than it is; the coverage column and the minimum of five results exist to make that visible. The index is a summary for orientation. The individual benchmark tables are the evidence.

Contamination. MathArena and Scale both flag a model evaluated after their questions were public, because those questions may already be in its training data. On a fresh competition board most of the field is in that position: 22 of 31 on AIME 2026, 37 of 64 on MASK. The flag is carried through to the benchmark board and the model page, in the publisher’s own words, and those scores still count in the index at full weight — removing them would leave nine models on AIME 2026, too thin to standardise. Read a flagged score as an upper bound.

Reasoning effort

The same model is evaluated at different reasoning-effort settings (none, minimal, low, medium, high, xhigh, max, or an unspecified “thinking” mode), and results can differ by ten points or more. Each setting is therefore its own row, scored only on the benchmarks run at that setting. “Default” means the publisher did not say. Prices and context windows apply to every setting of a model; measured speed and time to first answer stay with the setting they were measured at. Tables show each model’s best-scoring setting by default, with a switch to see them all.

What the charts show by default

A scatter of every configuration is unreadable, so charts open on the frontier set: the models no other model beats on both axes, the top eight by index, and the best model in each band of the horizontal axis, drawn from configurations released in the last eighteen months with at least eight results. The dashed line joins the frontier. “Top 50” and “All” are one click away.

Speed, latency and price

List prices come from models.dev and the LiteLLM price table, cross-checked against Artificial Analysis and OpenRouter’s marketplace. The blended price assumes three input tokens for every output token. Output speed comes from Artificial Analysis’ measured medians where it covers a model, and time to first token from OpenRouter, whose first-token figure leaves out any thinking phase; each fills in where the other is missing. OpenRouter’s figure is the median across healthy providers, each weighted by its request volume over the last 30 minutes. Because that window is short, marketplace readings are smoothed as the median of the last seven daily readings: an hour’s congestion does not move a model, while a real change (a new provider, a faster stack) shows within days. Speed still varies through the day and by provider, so treat it as typical, not guaranteed.

Per-task estimates

Five representative workloads turn per-million prices into numbers you can budget with:

• Chat reply: A short assistant turn: 400 tokens of context in, 300 tokens out.
• Summarize a 30-page report: 12,000 tokens of document in, a 600-token summary out.
• Code edit: 6,000 tokens of files and instructions in, a 1,500-token diff out.
• Agentic coding session: One long tool-using loop: 60,000 tokens in across turns, 4,000 tokens out.
• Structured extraction: 2,000 tokens of source text in, 200 tokens of JSON out.

Cost = input tokens × input price + output tokens × output price. Time = time to first answer (first token where that is all we have) + output tokens ÷ output speed. “With caching” assumes three-quarters of input tokens are served from the prompt cache at the provider’s cache-read price, a typical ratio for agent loops that resend the same context. Reasoning tokens and batch pricing are not modelled.

Method version 1.2

Any change to weights, eligibility or the formula reorders history, so the method is versioned. Trend lines are recomputed from every daily snapshot whenever the version changes, so a chart never mixes two methods.

• v1.2 (2026-09-19): Added CritPt (reasoning), τ²-Bench Banking (agentic) and LMArena Maths (maths) to the index, each at weight 1. The other 104 boards added the same week stay as reference.
• v1.1 (2026-09-09): Rows became model × reasoning-effort configurations. Index shrunk toward 50 by n/(n+3). Added Artificial Analysis, Vals AI, MathArena, Scale SEAL, Terminal-Bench, EQ-Bench, MMMU and Kagi benchmarks with their weights.
• v1.0 (2026-09-08): First release: per-benchmark z-scores mapped to 50 ± 15, averaged per category then across categories; minimum five results in two categories.

Freshness and failure

The pipeline runs once a day, starting at 22:03 UTC. If a source cannot be fetched, its previous values are kept and flagged as stale on the sources list rather than silently disappearing. Every day’s snapshot is versioned, which is what powers the trend lines and the daily digest (also as an Atom feed).

Open data

A summary of the current leaderboard is published as files, rebuilt with every update: the top 100 configurations (model × reasoning effort) with rank, BenchLeader Index and its interval, five category scores, list price, output speed, time to first answer, the data date and the canonical page for each row.

• /data/leaderboard.json (JSON, with field descriptions)
• /data/leaderboard.csv (CSV, same rows)
• /llms-full.txt (plain-text summary for language models: method, top 20 overall and per category, best per use case)

BenchLeader's own derived figures (the BenchLeader Index, its interval and the category scores) are licensed CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/): non-commercial use only, with attribution — credit "BenchLeader" with a link to https://www.benchleader.com (or to the row's url). That licence does not cover anyone else's figures: list prices are the vendors' own published facts, and speed, latency and the benchmark results behind the scores remain under their publishers' terms (credited on each model page and listed under "sources"; Epoch AI's Benchmarking Hub, for example, is CC BY 4.0).

That is all that is published: no raw benchmark results, no history and no model registry. Those stay on the pages, where each value links to its publisher.