Benchmark

What it means

A benchmark is a test that many people agree to use, so they can compare results fairly. In AI, it's usually a fixed set of questions or tasks plus a rule for scoring the answers. When a company says its new model "beats" a rival, it almost always means the model scored higher on one or more benchmarks.

Here's an everyday comparison: a standardized exam for job applicants. Everyone gets the same questions, so the scores can be compared. But a high score shows how someone did on that exam. It doesn't show how they'll handle your customers next Tuesday.

Where the word comes from

The word started in surveying, long before computers. The U.S. National Geodetic Survey still uses it there. One of its staff explained that the agency likes to "reserve the word bench mark for marks that have elevation information."¹ A bench mark was a fixed point that other measurements were taken from.

Computing borrowed the idea for buying machines. Researchers Inioluwa Deborah Raji, Emily M. Bender, and colleagues trace this history. In 1962, industry reports began testing "the performance of machines on specific prototype tasks," so buyers didn't have to weigh each vendor's features one by one. And "By 1976, benchmarking was a fairly common practice in computing," including in U.S. government purchasing.² In AI, they describe "The Common Task Framework, the precursor to modern data benchmarking in AI" as dating to the 1980s.²

Today the word mostly means a public test that AI labs compete on. One well-known example is SWE-bench. Its creators built it from "2,294 software engineering problems drawn from real GitHub issues" (GitHub is a site where programmers share code and track problems).³

How others define it

• Raji, Bender, Paullada, Denton, and Hanna (2021): "we describe a benchmark as a particular combination of a dataset or sets of datasets (at least test data, sometimes also training data), and a metric, conceptualized as representing one or more specific tasks or sets of abilities, picked up by a community of researchers as a shared framework for the comparison of methods."²
• NOAA's National Geodetic Survey (the surveying sense): "reserve the word bench mark for marks that have elevation information."¹
• U.S. National Institute of Standards and Technology (NIST): NIST hasn't published a single definition in the material we could open. Its draft guidance is about "a particular type of evaluation: automated benchmark evaluations."⁴

Where they agree and differ

All of these sources agree that a benchmark is a fixed reference that other things are measured against. They differ on how much a score can tell you. Raji and her coauthors argue that the field tends "to valorize a small collection of influential benchmarks," treating narrow tests as signs of general ability.² Stanford's 2025 AI Index reports that older tests are running out of room. It notes "The saturation of traditional AI benchmarks like MMLU, GSM8K, and HumanEval," meaning top models now score so high on them that they no longer separate the best from the rest.⁵

The strongest opposing view deserves a fair hearing: without shared tests, every claim would just be marketing. Regulators seem to agree. The EU AI Act tells the European Commission to "encourage, as appropriate, the development of benchmarks and measurement methodologies."⁶ NIST's draft sets out "preliminary best practices for evaluating language models and AI agent systems."⁷ The disagreement isn't really about whether to use benchmarks. It's about how far to trust any single score.

Scores can also move fast. Stanford reports that on SWE-bench, "AI systems could solve just 4.4% of coding problems in 2023—a figure that jumped to 71.7% in 2024."⁵

In the news

We found no recent story that changes this picture, so here is the most memorable real example. In April 2025, Meta was criticized for using "an experimental, unreleased version of its Llama 4 Maverick model" to get a high score on LM Arena, a popular benchmark where people vote on which chatbot's answer they prefer. TechCrunch reported that the episode "prompted the maintainers of LM Arena to apologize, change their policies, and score the unmodified, vanilla Maverick," which ranked below rivals.⁸ Meta said the tested model "is a chat optimized version we experimented with that also performs well on LM Arena."⁸ The lesson is about disclosure: a score only means something if it belongs to the product you can actually buy. We're confident in this account because it comes from on-the-record statements reported by TechCrunch.

What it means for your organization

Treat a benchmark score like a reference on a résumé. It's useful, but it isn't the job. Check three things: whether the score was earned by the exact version you'll use, whether the test looks like your work, and whether the test is so old that every model now aces it. The surest benchmark is your own: a small set of real tasks from your organization, scored the same way each time, with a person reviewing the results. That's how an organization becomes [known by its record](/definitions/known-by-its-record) rather than by a vendor's chart.

One question to ask your team: "Which of our own tasks would we use to test an AI agent before we trust it, and who checks the results?"