How to read AI benchmarks

AI model benchmarks are everywhere and are read poorly almost always. It is not the fault of the reader: they are presented as sports rankings —a winner, of numbers, a ranking —when in fact they are measures of specific tasks under specific conditions.

This guide explains what they really tell, why different vendor numbers don't add up, and how a company can use them to decide instead of suffering them. The data cited are all from official sources, with date, because in this field a number with no date is worth little.

What a benchmark really measures

A benchmark is a set of tests with an answer considered correct. It measures the percentage of tests passed. This means two things that in the rankings disappear.

Measure that task. A programming test tells how much a model solves programming problems similar to those of the test. He does not say if he writes well in Italian, if he reasons on long documents, if he makes little mistake when he does not know.

Measure in a configuration. The same model, with more time for reasoning or with tools available, gets different results. In Claude Opus 5's announcement of July 24, 2026, Anthropic explicitly shows how performance changes as the level of effort changes — not a single number, but a curve.

Why Vendor Numbers Don't Add Up

It is the misunderstanding that generates the most wrong decisions. Anthropic, OpenAI and Google each publish their own results, and it seems natural to put them in a table. It is not.

• Each vendor chooses which benchmarks publish, and publish those in which it goes
• Each vendor uses their own scaffolding: prompts, tools, number of attempts
• The versions benchmarks change — Frontier-Bench v0.1 and CursorBench 3.2 are precise versions, not generic names
• Some results are on internal benchmarks not reproducible by third parties

Anthropic itself, in the announcement of Opus 5, cites improvements "on our internal benchmark" for organic chemistry evaluations. It is honest and useful information, but by definition not comparable with the numbers of another vendor.

The practical consequence: tables that line up different vendor scores are almost always poorly constructed, even when the individual numbers are correct. The theme is deepened in because vendor benchmarks are not comparable.

The Independent Charts

There are ratings not produced by vendors, such as LMArena, that compare models on preferences expressed by people and publish separate rankings by category — chat, agents, code, images, documents.

They have one advantage: the measure is the same for everyone. And one limit: they measure human preference over generic conversations, which is not the same thing as surrender over a specific process of yours.

The cost per task: the measure that matters

Here is the most important difference between reading a passionate benchmark and reading it from a company.

The published price is per million tokens. As of the date of this article, on the official price lists:

Sources: platform.claude.com, developers.openai.com, ai.google.dev — lists consulted on August 19, 2026. Gemini prices referred to the Standard plan.

That price, however, does not say how much it costs solving a problem. A model more dear to the token that comes to the solution in three steps can cost less than an economic one that takes fifteen. It is exactly the reasoning that Anthropic sets out in the Opus 5 announcement, where it talks about cost per task and not price per token.

The complete reasoning is in cost per task or price per token.

The three things that make an honest comparison

1. The date. Models change in weeks. As of the date of this article Opus 5 has been out for less than a month and Google has Gemini 3.6, 3.5 and 3.1 listed at the same time.
2. The configuration. What level of reasoning, what tools, how many attempts.
3. The source. Who produced the number and whether it is reproducible by third parties.

A comparison that states these three things can also be used in six months, because the reader knows what he is looking at. One that omits them is a photograph without a caption.

How a company really chooses

None of the information above replaces the evidence on your case. The method that works is shorter than it looks.

1. Take ten real examples the task you are interested in, including difficult cases.
2. Try two or three models on the same examples, with the same instruction.
3. You evaluate the quality: are these answers you would use?
4. Measure the total cost to get to the result, not the price to the token.
5. It also checks the yield in Italian, which international benchmarks measure little.

Step 5 counts more than rankings suggest: most public benchmarks are in English, and for an Italian company, quality in its own language is a real selection criterion. The theme is taken up in Best AI Model for Italian.

Practical examples

Study that must summarize long documents. Coding benchmarks are irrelevant. They count context window and hold on long texts. Three models, ten real documents, summaries comparison: the decision is made in half a day.

Company that wants to automate customer responses. Here the criterion is not the maximum score but constancy: a model that rarely fails in a serious way beats one that excels and occasionally invents. It is a quality that the rankings do not measure well, and that you can only see by trying.

High-volume project. With thousands of requests per day the price difference between bands becomes dominant. It is worth checking if a light model covers the task, as discussed in when an economic model is enough.

Mistakes to Avoid

• Putting different vendor numbers in the table. Different scaffolds, invalid comparison.
• Ignoring the level of reasoning. Change the results of the same model.
• Comparing different pricing plans. Standard versus Batch is not a comparison.
• Reading undated charts. In this field they age in weeks.
• Choosing the first in the general classification. The general classification is not your use case.
• Not testing on your own data. It's the only measure that really describes your situation.

How to apply it in the company

When you read a comparison between models, ask yourself three questions: when is it, with what configuration the numbers were obtained, and who produced them. If the article does not respond, treat it as an indication of principle and not as a basis for deciding.

Then, before you choose, test your ten cases. It's half a day's work and replaces weeks of chart reading, because it measures exactly the task you're interested in instead of a task that looks like yours.

If you want to understand what processes in your company can be automated with AI, Giallo Studio can help you turn the problem into a real workflow. The framework on choice criteria is in how to choose an AI model, the cost in cost of AI API.

The six-step method

Everything said so far is condensed into a procedure that an SME can perform in half a day, without special technical skills and without spending on consulting before having understood what is needed.

First, write the task in a sentence. We don't "want to use AI", but "classify incoming emails into five categories" or "extract eight fields from supplier invoices". If you can't write it in a sentence, the project is not yet defined and no benchmark will help you.

Second, collect ten real cases. Taken from real work, with the correct outcome already known. At least three must be difficult cases: badly written document, missing information, abnormal format. They are the ones that distinguish the models.

Third, choose two or three candidates. Here benchmarks are needed, and that's the only time they're needed. Look at tests relevant to your task type and narrow down the field.

Fourth: Run the same test on everyone. Same material, same instructions, same configuration. Write down the outcome, time and consumption for each case.

Fifth, let those who know about the job evaluate. Not to those who know AI. The person who does that job every day recognizes in two seconds an error that to a technician seems a reasonable answer.

Sixth: Look how he's wrong. It is the step that almost everyone skips and is the most informative, for the reason explained later.

Five hours in all, and they produce more useful information than weeks spent comparing rankings.

Why Mistakes Matter More Than Success

Two models with the same error rate may have completely different value, and the difference lies in the type of error.

A model that, in the face of an incomplete document, answers "the field is not present" did nothing wrong: it did its job. A model that invents a plausible value has produced an error that no one will verify, because it does not look like an error.

This criterion does not appear in any public ranking, because it requires looking at the answers one by one. And that is exactly why the internal test remains irreplaceable.

How to apply it in the company

The practical way to use this is to turn it into a fixed procedure that is repeated at each evaluation, instead of a reasoning to be repeated each time.

Before you look at any ranking, write the task in one sentence and collect the ten real cases. It is the only work that requires attention, and it must be done by those who know the process. Then use benchmarks to narrow to two or three comparable-range candidates, according to the criterion described in which AI benchmarks really matter.

The real test comes later, and it's the one that decides. Same material for all candidates, same configuration, outcomes evaluated by those who do that job every day. Add two measurements that hardly anyone does: how much each case really costs, with the method of cost per task or price per token, and how much time passes between request and response.

The last step is the one that protects over time: a calendar revision every six months, not when a new model comes out. Revisions made on the wave of an ad start already oriented upwards; fixed-date ones also look downwards, where the saving lies.

If you want this assessment to be made by someone with you, on your documents and your processes, Giallo Studio can help you turn the problem into a real workflow.

Three readings to close the picture. If you are still choosing the band, the operating criterion is in Mixed model strategy: light and powerful. For the calculation of real expenditure —the one that includes repeated context, failed attempts and human review —the reference is when it is best to change the AI model. And when traffic is uneven, the setting that gives the best relationship between quality and cost is described in the best AI model depends on the use case.

The three questions to ask to those who propose a model

Whether the proposal comes from a supplier, a consultant or a colleague, three questions reveal in a matter of minutes how much work is behind it.

"With what configuration was this number obtained?" Benchmark version, scaffolding, reasoning level. A precise answer indicates that the comparison was indeed made; a vague answer indicates that the number was copied from a communiqué.

"Have you tried the underbelly?" It is the question that saves the most. The commercial temptation pushes up, and those who propose a lower band explaining why just is working in your interest.

"What happens when the model can't answer?" If there is no precise answer, the system will safely close even cases it has not understood — and those produce errors that no one will verify.

How to Build a Ten Case That Really Serves

Not all test stands are the same, and the difference is immediately seen in production.

The third line is the one that hardly anyone inserts and is the most informative of all. Giving the model a document that lacks the required field and asking it produces an answer that says everything: declaring absence is correct, returning an empty field is acceptable, inventing a plausible value is disqualifying at any price.

Finally, it is worth keeping the testbed in a file, with the correct outcomes and results of each model tested. Become the permanent reference: at the next evaluation, in six months, the comparison costs an hour instead of five, and you can put the new model next to all those already tested on the same identical cases.

Can a benchmark demonstrate superintelligence?

The name of a test does not alone certify a category of intelligence. A result must be linked to the adopted definition, test conditions and excluded tasks. The guide on AI, AGI and ASI separates breadth, capacity level and autonomy; analysis Does superintelligence already exist? distinguishes documented progress from conclusions that the data do not support.

Comparing a model without tools with a system equipped with search, code or multiple attempts requires declaring those differences. The components of technologies for superintelligence help to read the rated system. Also AlphaEvolve and self-improvement show why an evaluator prepared for a specific problem does not demonstrate a general ability to improve without limits.

Conclusion

AI benchmarks are useful when read for what they are: specific task measurements, in specific configurations, at a specific time. They become misleading when they turn into absolute rankings.

For a company the practical rule is only one: public numbers are used to narrow the field to two or three candidates, the choice is made on their own data. And any figures you read, including those in this article, should be looked at along with its date.