A benchmark is a fixed set of tasks with a fixed way of scoring them, so that two models can be compared on the same ground. The best known one for language models, MMLU, was introduced in 2020 with 57 tasks that include elementary mathematics, US history, computer science and law. When it appeared, most models scored near random chance, and the largest GPT-3 model was almost 20 percentage points above it. SWE-bench is built differently: 2,294 software engineering problems drawn from real GitHub issues and their pull requests in 12 popular Python repositories, where the model is handed a codebase and an issue description and must edit the code to resolve it. When its authors published it, the best model they tested, Claude 2, solved 1.96 percent of the issues. Those two early results show the main thing about benchmarks: a score describes how a model did on those questions, in that format, under that scoring rule, on the day someone ran it. How far it carries to anything else is a separate question, and the rest of this lesson is about the reasons it often does not carry far.
The first reason is saturation. When the best models cluster near the top of a test, it stops separating them. The authors of MMLU-Pro wrote that performance on MMLU had begun to plateau, and built a harder version: more questions that need reasoning, and ten answer options instead of four. Accuracy dropped by roughly 16 to 33 percent compared with MMLU, and the score’s sensitivity to how the prompt is worded, measured across 24 prompt styles, fell from about 4 to 5 percent to about 2. A benchmark is a measuring instrument with a range, and a model above the top of the range gets the same reading as one at the top. The second reason is contamination. Training sets are scraped from the public internet, and a benchmark’s questions are on the public internet too. If the test questions, or close paraphrases of them, are in the training data, the model has seen the exam. Yang and colleagues showed how thin the usual defense is: common decontamination removes text that matches the test set word for word, but a paraphrase or a translation gets past it, and a 13-billion-parameter model trained on rephrased test sets reached performance comparable to GPT-4 on MMLU, GSM8k and HumanEval. They recommend stronger decontamination and fresh one-time exams. None of this requires bad faith. A lab can contaminate a benchmark without meaning to, and one defense a reader has is a test written after the model was trained.
The third reason is the setup. A benchmark is code plus a prompt plus a scoring rule, and each of those has choices in it. The team behind Hugging Face’s Open LLM Leaderboard found that three public implementations of the same MMLU questions gave the same model, LLaMA-65B, three different scores: 0.636 in the original implementation, which compares the model’s probabilities for the letters A to D; 0.637 in Stanford’s HELM, which has the model generate text and compares it to the expected answer; and 0.488 in EleutherAI’s evaluation harness as it stood in January 2023, which compares the probabilities of whole answer sequences. The prompts differed by small details, such as a “Question:” prefix, and the scores diverged by almost fifteen points. So when a lab reports its own score, the number is only half of the claim. The other half is the prompt, how the answer was scored, and what the lab chose to include, and a score measured by a third party under a published setup is easier to trust than the same score in the maker’s announcement. The lesson on cherry-picking applies here as well: a table of the benchmarks a lab won is not the table of every benchmark it ran.
A different kind of leaderboard asks people instead of a test set. In Chatbot Arena, a user types any prompt, two anonymous models answer, and the user votes for the better response, or for a tie, or for both being bad, before the names are revealed. The paper behind it reports more than 240,000 votes. It scores models with the Bradley-Terry model, in which the probability that one model beats another is a logistic function of the difference between their scores, and the leaderboard’s first version used Elo ratings, the system from chess, which the team’s first post on it said predicted pairwise win rates relatively well. A rating gap is therefore a win rate in disguise: a model that rates above another does not win every battle, it wins more of them. The same paper lists its own limits. Voters are expected to be mostly hobbyists and researchers, 77 percent of the conversations are in English, the study looks at helpfulness and not safety, and most of the disagreement between the crowd and expert raters came from the crowd missing factual errors. Other researchers argue that the Arena’s numbers are also shaped by how it is used: in “The Leaderboard Illusion”, they say a few providers privately tested many variants before release and withdrew the scores they did not like, and that proprietary models were sampled more often than open-weight ones.
The last gap is the one between a score and the work you want done. One randomized controlled trial of coding assistants, by Becker and colleagues, measured real work instead of a benchmark. In it, 16 experienced open-source developers completed 246 tasks on mature projects they knew well, each task randomly assigned to allow or disallow the AI tools of early 2025. Before the trial the developers expected the tools to cut completion time by 24 percent, and afterwards they estimated that they had cut it by 20 percent. The measured result was the opposite: tasks took 19 percent longer with the tools. It is one study, with 16 people and the tools of one moment, so it does not say what today’s tools do. What it shows is that a person’s impression of a tool and a measurement of the work can disagree by a wide margin, which is a reason to treat a benchmark as a proxy for the work and not as the work. Four questions apply to any score: what was asked, how it was scored, who ran it and under what setup, and whether the model could have seen the test.