A score is a result.
The setup gives it meaning.

What does 80% actually mean?

A benchmark is a test of specified tasks under specified conditions. Its score is evidence about that performance.

Suppose a model answers 80 of 100 questions correctly. Its accuracy on that run is 80 ÷ 100 = 80%. That does not mean it is “80% intelligent” or will get 80% of your questions right. To make the second claim, you would need evidence that the test resembles your work.

Every evaluation has four ingredients. Changing any one can change the result, even when the model stays the same.

1. 01

   The tasks

   What problems, languages and difficulty levels were sampled? Which dataset version and test split? This defines the territory the score covers.

2. 02

   The setup

   The exact model, prompts, tools, time and reasoning budget. An agent’s harness controls how it works; a score may describe the whole system.

3. 03

   The grader

   Tests, reference answers, a model judge or human ratings. Passing tests is useful evidence, but incomplete tests can miss a bad patch. Human and model judgments can also be biased.

4. 04

   The summary

   Accuracy, task success, a preference rating or a weighted index? What gets averaged, and how are failures or skipped tasks counted? Not every score is a percentage.

One attempt is not the same as “one answer somewhere”

Pass@1 estimates success for a single attempt; averaging repeated independent attempts can estimate it more reliably. Pass@k instead asks whether at least one of k attempts succeeds. It does not tell you how to select that answer in practice. HumanEval paper ↗

A real selector—tests, a verifier or a vote—can choose among attempts. Its selected-answer accuracy and total cost are what a deployed workflow experiences. Multiple attempts are legitimate; leaving out the selection method or budget makes the claim incomplete.

Illustration: each attempt on the task independently succeeds with probability 50%. At least one success: 1 − 0.5⁴. All four: 0.5⁴. Real tasks have different success probabilities, so substituting a benchmark’s average into these formulas need not give its pass@4 or pass^4. Repeated-success metrics in τ²-bench ↗

Check what gets averaged

Imagine 20 answers, each with five required rules: 100 checks in total. If 95 checks pass but the five failures occur in five different answers, only 15 answers satisfy every rule. The same run scores 95% per rule and 75% per complete answer. If your workflow needs every rule followed, the second number is the relevant one.

Ask first: 80% of what, with which setup, judged how?

Four ways a useful score can mislead

These are reasons to investigate a result. None, by itself, proves a model or a benchmark is worthless.

For measured examples of exposure risk, see LiveCodeBench. For judge biases such as verbosity and answer order, see the MT-Bench study.

How much difference is enough?

There is no reliable “within 5% means a tie” rule. The sample size, which tasks differ, repeated-run variation and grading quality all matter. An 80% score on 20 tasks is much less precise as an estimate of similar-task performance than 80% on 2,000.

Calculated Wilson intervals, rounded to whole percentages. They assume independent, representative pass/fail observations; they do not capture dataset bias or every source of model variability. NIST method ↗

For models tested on the same questions, compare their paired outcomes: which cases one passed and the other failed. A confidence interval for that difference is more useful than eyeballing whether two separate intervals overlap. Overlap alone does not establish a tie. Why interval overlap is insufficient ↗

Then ask a different question: would this difference matter? Moving from 98% to 99% is one percentage point but halves the observed error rate. Whether that is worth paying for depends on the consequences of those errors.

Benchmarks age. They do not all age alike.

A fixed set can remain useful. A newly updated leaderboard can still be testing old questions.

Public benchmarks can become easier as models improve, more exposed as their questions circulate, or less relevant as applications change. These are separate processes. A narrow score spread does not prove contamination, and a successor’s release does not invalidate every use of the original.

1. A test is introduced.It targets a capability its authors want to measure.
2. It is reused.Comparable runs help track progress; repeated tuning may also weaken generalisation.
3. Its usefulness changes.Ask whether remaining errors and differences still answer your question.
4. It is revised or complemented.New tasks, better grading or a different scope can add signal. Preserve version labels.

Two questions that should not be collapsed into one label

What is exposed? A benchmark may publish its questions and answers, keep some answers hidden, or retain entirely private tasks. Keeping answers hidden is different from keeping questions hidden. HLE and FACTS Grounding combine public material with held-out evaluation material. HLE ↗ FACTS Grounding ↗

What gets refreshed? New problems, new model results, new votes and new index weights are four different updates. AIME has new exams by year, but each named exam is fixed. An index can be recalculated without replacing its component tests.

Private tasks reduce exposure risk; they do not eliminate leaks, close variants or overfitting through repeated feedback. Recent tasks help when they genuinely postdate the evaluated model’s training and tuning. Verify the date window: “live” is not a guarantee. LiveBench, for example, is explicitly designed to limit contamination through fresh questions. LiveBench paper ↗

Choose the family before the benchmark

A task can need several abilities. Treat these families as lenses, not mutually exclusive boxes.

Knowledge & reasoning

GPQA Diamond and MMLU-Pro test answering academic questions. HLE includes difficult expert questions. Useful evidence for technical problem solving, with limited coverage of complete workflows.

Math & abstract reasoning

AIME tests competition maths; FrontierMath covers advanced maths; ARC-AGI-2 uses grid transformations. Their problem types, grading and compute budgets differ.

Coding

SWE-bench variants test repository issue resolution. LiveCodeBench tests contest-style programming; HumanEval tests short functions. They probe different parts of software work.

Agents & tools

Terminal-Bench, τ²-bench, GAIA, BFCL and WebArena test different tool-mediated tasks. Results belong to a particular model–agent–environment combination.

Factuality & grounding

SimpleQA asks for short factual answers. FACTS Grounding tests support from supplied documents. Knowing facts and staying faithful to a source are distinct abilities.

Instruction following

IFEval checks verifiable constraints such as required words or response structure. Passing a format check does not prove the answer is useful or follows a complex policy.

Long context & vision

RULER and LongBench v2 probe use of long inputs. MMMU tests reasoning over academic images and text. Test your own document lengths, scans and screenshots as well.

Preference & composites

Arena captures comparative human preference. An index combines chosen evaluations and weights. Both help you explore candidates; neither is a universal quality scale.

The reference catalogue links each named test to its original paper, dataset or official documentation.

A composite inherits someone else’s priorities

Consider two fictional tests with comparable pass-rate scoring. Each model is evaluated on the same coding tasks and the same writing tasks.

Illustrative weighted averages, not real model scores. For A under the second mix: 0.9 × 90 + 0.1 × 60 = 87. Real indexes may normalise and combine unlike metrics, so their point totals may not be percentages at all.

B leads the equally weighted comparison; A leads the coding-heavy one. Neither model changed. The question changed. Read the components and weights before applying an overall ranking to your work.

Read the footnote with the headline

A result can be accurate and still answer a different question from the one a chart suggests.

Start by identifying who ran the evaluation. A developer’s report, an independent rerun and a third-party reproduction are different kinds of evidence. “Verified” can also be part of a dataset’s name: SWE-bench Verified does not mean every submitted score was independently verified. SWE-bench datasets ↗

Find evidence for your use case

Choose a task to see useful starting points, their limits and what to test yourself.

What a headline score leaves out

Some specialised evaluations measure these factors. One capability score rarely covers them all.

Cost per useful result. Divide total workflow cost by the number of outputs that meet your acceptance criteria. Include failed attempts, retries, tools and review time, using actual billed token usage without counting reasoning tokens twice. For a subscription, include the limits and features you actually receive.

End-to-end speed. Time to first token affects responsiveness; tokens per second affects generation time. Tool calls and hidden reasoning can dominate total completion time. Measure the full job and slow runs, not just an average streaming rate.

Usable context. The advertised window is a capacity limit. It does not guarantee reliable use of all the content. Check retrieval, cross-document reasoning and answer support at your actual input length. RULER study ↗

Reliability and failure severity. An average can hide malformed outputs, policy violations, missed facts or occasional destructive actions. Count these separately and repeat important cases. A minor formatting miss and an invented refund entitlement should not cancel each other out.

Product and deployment fit. Test the actual app or agent: file handling, search, tools, permissions and review controls affect results. If you need self-hosting, inspect the weights and licence, hardware needs and operational cost. Downloadable weights alone do not establish privacy, low cost or suitability.

Choose against your requirements: acceptable failures, latency, cost and workflow support—not a universal rank.

A seven-step method

Use public evidence to narrow the field, then test the configurations you can actually use.

1. Define the job and unacceptable errors

   “Draft support replies using our policy documents, for a human to approve.” Name what would make an answer unusable, and your cost and response-time limits.

2. Shortlist feasible candidates

   Start with a few models or products you can access and afford. Use broad boards for discovery and record exact versions; rank one is not a requirement.

3. Read task-matched evidence

   Pick the relevant families. Examine actual tasks and failure cases, not just an aggregate. A product may need grounding, instruction following and tool use together.

4. Identify missing evidence

   Look for relevant independent evaluations. Treat an unreported result as unknown; request it or cover the gap in your own pilot.

5. Normalise the comparison

   Match versions, splits, metrics and budgets. Compare uncertainty where available. For product selection, record meaningful differences in the surrounding agent.

6. Measure the practical trade-offs

   Estimate total task cost and measure full response time and editing effort. Decide which quality differences would justify the extra spend.

7. Pilot, inspect failures and expand

   Start with 20 real tasks if that is manageable. Grade blind with a fixed rubric. Treat the pilot as a way to find problems; use more representative held-out cases when you need a reliable rate or a close comparison.

Worked example: support email drafts

An invented example. All prices and outcomes below are illustrative, not current model quotes or observed results.

A small company handles 400 emails a day. It wants drafts grounded in its refund and shipping policies, with a person approving every message. The task is accurate drafting from supplied facts; autonomously changing an order would be a different job.

Start with the failure, then choose the evidence

The most serious errors are inventing order details and promising something outside policy. FACTS Grounding is relevant to using supplied sources, and IFEval gives narrower evidence about explicit constraints. τ²-bench becomes more relevant if the workflow includes policy-bound tool actions; it is not a direct substitute for evaluating email drafts.

Two accessible candidates survive the shortlist. Their public setups differ, so the team avoids claiming a precise quality lead. An unreported factuality score remains unknown. Both candidates proceed to the same local drafting task.

Make the cost calculation inspectable

Assume this volume every day for 30 days: 400 × 30 = 12,000 drafts. For this calculation only, each draft uses 1,000 input tokens and 250 billed output tokens, with no retries or separate tool charges.

For A: (1,000 × $2 + 250 × $8) ÷ 1,000,000 = $0.004 per draft. Actual usage, caching, reasoning, taxes and vendor billing can change the total.

The difference is $36 per 30 days under these assumptions. If B requires an extra 30 seconds of editing per draft, that adds 100 hours of review across 12,000 drafts. Measuring editing effort matters far more here than extrapolating from a small benchmark lead.

Use the pilot to find what to test next

The team tries 20 past emails, including conflicting policies, missing order information, an angry customer and a multilingual request. Each draft is checked for policy compliance, support for factual claims, a useful next step and editing effort.

Suppose A produces 18 acceptable drafts and B produces 19, but B’s remaining draft invents a refund entitlement. That error deserves investigation. One failure in 20 does not establish a 5% production failure rate, especially when the pilot deliberately includes difficult cases.

Start with twenty. Know what twenty can tell you.

A small evaluation is excellent for discovering obvious mismatches. It is weak evidence about rare failures or small differences.

Your own tasks improve relevance, but do not make evaluation problems disappear. An unrepresentative sample can mislead; repeated tuning can overfit it; a subjective rubric can drift. Keep the task set private where appropriate and follow the data-handling rules that apply to your work.

Separate typical work from stress tests

Sample routine cases in roughly the mix you expect to see, then keep a separate collection of hard or high-consequence cases. For an initial 20-case exploration, 15 typical cases plus 5 stress cases is one workable starting point—not a statistical standard. Report the groups separately, rather than treating an oversampled stress set as your production failure rate.

Write the rubric before viewing the answers

Define three or four observable checks, with examples of passing and failing. Separate hard failures from tone or formatting preferences. For support drafts: follows policy, supports factual claims with supplied evidence, requests missing information appropriately, and needs little editing. Record editing time as well as pass/fail.

Hold something back

Use one set to improve prompts and tools; reserve different cases for the comparison. If you tune after seeing those results, they are now development evidence too. Obtain fresh held-out cases before treating the next score as an independent test.

Run and grade fairly

Use the same inputs, record settings and count timeouts and tool failures. Hide model names and randomise output order for human grading. Use mechanical checks where appropriate, and audit any model judge against your rubric. Repeat important cases; many reruns of one case do not provide the same coverage as many different cases.

Save enough to reproduce the decision

Keep task IDs, expected outcomes, exact model and agent versions, prompts, tool settings, raw outputs, rubric decisions, latency and cost. Record why you selected the configuration and what would make you reconsider. Add newly observed failure cases as the work changes.

Decide what would be enough evidence

Set your acceptance criteria and review point before running the comparison. For daily personal use, a small pilot may be enough to choose a comfortable fit. For a repeated business workflow, require representative held-out results, acceptable failure severity and measured cost and latency. If both candidates meet the bar and the remaining quality uncertainty would not change your choice, choose on the practical factors that do. Increase evaluation when unresolved uncertainty could change the decision.

Benchmark catalogue

28 selected references, including established editions you will encounter in model reports. This is not an exhaustive list or a claim that each is the newest edition.

Descriptions checked 9 September 2026. Public/private composition and versions can change; open the source for the exact result you are reading.

28 of 28 references shown.

Where to find live numbers

Use the board for discovery and its methods for interpretation. Record the date, category and version you compared.

Glossary

Short definitions for the terms used in the guide. Open a term to read it.

No terms match. Clear the search to see all definitions.

Sources & scope

Each catalogue entry links to primary documentation. Links beside explanations identify the relevant paper or method. The numbers in the teaching examples are calculated illustrations, and the model-selection method is practical guidance rather than a guarantee.

• Evaluating Large Language Models Trained on CodeHumanEval and pass@k.
• Judging LLM-as-a-JudgeEvidence on judge biases and human preference evaluation.
• NIST: confidence intervals for proportionsMethods behind the interval examples.
• LiveCodeBenchTask release dates and contamination checks.
• LiveBenchRefreshing task sets to limit exposure.
• Artificial Analysis methodologyIndex composition, evaluation settings and version changes.

Reviewed 9 September 2026. Benchmark editions, splits, leaderboard methods and URLs may change. This guide teaches how to read the evidence; it does not maintain a live model ranking.