Why Vendor Benchmarks Are Not Comparable
Why Anthropic, OpenAI and Google benchmark figures cannot be compared directly: differences in measurement and how to interpret the results.
Reading time: 9 min
Putting Claude's score, GPT's score, and Gemini's score in a table on the same benchmark seems like the most natural thing in the world. It's also almost always wrong, and not because the individual numbers are false: because they were obtained under different conditions.
This article explains what those conditions are, why no one declares them all, and how you read a vendor ad without drawing conclusions that it does not hold.
In summary
- No need for bad faith: enough different methodological choices to make the numbers non-overlapping.
- The variables that matter: effort, attempts, tools, test version, system prompt.
- Comparisons interior of a vendor are solid; those crossed between vendors are not.
- The internal benchmarks are not verifiable by third parties: they must be read as such.
- Independent rankings mitigate the problem but measure other.
- The only comparison under your control is what you do about your data.
The five variables that move the result
The level of reasoning. Current models have configurable effort settings. A score obtained at maximum effort and an average effort are not the same size, and the difference can be wide.
The number of attempts. One result "on the first try" and one "best out of ten attempts" are different things. Both legitimate, both publishable, not comparable.
The instruments granted. A model that can run code or search the web faces the test differently than one that responds from memory.
The version of the benchmark. There is no "the" benchmark: there is Frontier-Bench v0.1, CursorBench 3.2, ARC-AGI 3. Versions change and numbers do not transfer.
The system prompt. The instruction with which the model faces the test is part of the scaffolding, and is rarely published in full.
Where the numbers are solid
There is a case where vendor data is reliable and useful: comparison between two models of the same vendor.
When Anthropic states that Opus 5 more than doubles the performance of Opus 4.8 on Frontier-Bench v0.1 at a lower cost per task, it is measuring two of its models with the same scaffolding, the same test, the same method. That comparison holds up.
Reliable Comparisons
- New vs previous model of the same vendor
- Same model at different effort
- Progress on later versions of the same test
Fragile Comparisons
- Scores of different vendors side by side
- Numbers without declared configuration
- Tables mixing versions of the benchmark
- Internal benchmarks put next to public ones
This is why the best-built comparative articles compare Price ranges and verifiable features — context, cost, latency, availability — and use benchmarks as a qualitative indication, not a ranking.
Internal benchmarks
They deserve a separate note because they appear often and are misread.
A vendor can build their own assessments on specific domains. Anthropic, also in the Opus 5 announcement, reports measured improvements on internal benchmarks for life sciences — for example on organic chemistry.
There is nothing wrong: the company declares that it is an evaluation of its own. But such a number is not verifiable by third parties and cannot be placed next to a competitor's public result. It is an indication of the company on itself, and as such should be cited.
What to state in an honest comparison
- Who produced the number and when.
- What an exact version of the benchmark.
- With what level of reasoning.
- With how many attempts and what tools.
- Whether the benchmark is public or internal.
An article declaring these five elements remains useful even after months, because the reader can place the data. One that omits them ages badly and misleads.
Practical examples
Comparative table found online. Line up three models on a coding test, with no dates or configurations. It is not usable to decide: it could compare a model at maximum effort with one at average effort, or different versions of the test.
Announcement of a vendor. It says the new model outperforms the previous one on a specific test at lower cost. It's solid information to decide whether to update, and it's exactly the case where vendor data is needed.
Independent ranking. Useful for narrowing the field down to two or three candidates. Not enough to choose, because it measures generic preferences and not your process.
Mistakes to Avoid
- Building cross-vendor tables with published numbers. It's the most common mistake.
- Ignoring the version of the benchmark. v0.1 and 3.2 are not details.
- Treating internal benchmarks as public. They are not verifiable.
- Quoting numbers without date. In this field they age in weeks.
- Deduct "the best" from a single test. Measure one task, not all.
- Not testing on your own data. It's the only confrontation you really control.
How to apply it in the company
When you receive a proposal that relies on benchmarks —from a provider, a consultant, an article. ask for the five information listed above. If they are not there, the number is not a topic: it is an illustration.
Then make the verification that no ranking can do for you: ten real examples of your task, the same for each model, with the same instruction. The complete method is in how to read AI benchmarks and the criteria of choice in how to choose an AI model.
As you read an ad, line by line
Let's take a real case. In the announcement of Claude Opus 5 of July 24, 2026, Anthropic states that the model achieves state of the art on Frontier-Bench and GDPval-AA, and that on Frontier-Bench v0.1 more than doubles the result of Opus 4.8 at a lower cost per task.
It's a well-constructed phrase, and it's worth taking apart to figure out what it really contains.
- The version is declared (v0.1): good sign, allows match
- The comparison is with the previous model of the same vendor: legitimate and verifiable
- He is quoted on cost per task, not just the score: rare and useful
- The data remains self-reported: is not an independent verification
- It is not said which scaffolding produced the number
The same announcement also declares the reverse of the coin: on cybersecurity the model remains behind a competitor. It is a detail worth noting, because an ad that admits a limit is on average more reliable than one that admits none.
The Three Structural Asymmetries
Beyond intentions, there are three reasons why a number produced by a vendor cannot be neutral. It's not about honesty, it's about method.
The selection of tests. Each vendor chooses which benchmarks to quote. If one model goes well on eight tests and bad on three, eight o'clock will appear in the announcement. It is not a lie, it is a selection —and it produces a portrait systematically more favorable than the real.
The scaffolding optimization. Who built the model knows how to get the most out of it: which prompt format works best, which level of reasoning suits, how to structure the attempts. On the competing model this knowledge is lacking, and the resulting gap is partly artificial.
The choice of the moment. The comparison is published when your model has just come out and the competitor's is a few months old. It is the most favorable photograph possible, and in an industry that moves so fast it takes just a few weeks to change the picture.
| Asymmetry | Effect on number | How to defend yourself |
|---|---|---|
| Selection of tests | More favorable portrait | Search for unquoted tests |
| Optimized scaffolding | Amplified gap | Check the configuration |
| Time of comparison | Dated comparison | Check release dates |
None of the three require bad faith to produce themselves. They are structural consequences of the fact that the one who measures is also the one who sells, and that is why these numbers serve to orient but not to decide.
What to do, in practice
The conclusion is not to ignore vendor numbers: they are often the only source available on a newly released model. It is to use them for what they are.
- Read them as hypothesis to be verified, not as conclusions
- Search for the Internal comparison (new vs. previous): it is the most reliable
- Be wary of confrontations with competition without declared configuration
- Check if there is a independent source on the same test
- Use numbers to narrow to two or three candidates, never to choose
The second point deserves a note. When a vendor compares their new model with their previous model, much of the asymmetries disappear: same scaffolding, same expertise in optimization, same interest in showing real progress. The limits of auto-carriage remain, but the comparison is much more solid than that against a competitor.
Practical examples
SMEs evaluating two models for customer support. It reads two ads: each provider claims to be ahead on their favorite test. Neither number is usable, because the tests are different and the configurations undeclared. The decision must be made by trying both out of fifty real messages already arrived from customers.
Company that must justify the choice to the management. Bringing a benchmark score to a meeting is fragile: at the first question about how it was obtained, the number deflates. Much more solid to present the result on their ten cases, with verifiable outcomes from anyone in the company.
Technical team comparing two models on the code. Find two very different scores on the same benchmark. The cause is almost always scaffolding: one of them could perform tests and correct himself, the other could not. The comparison must be done again with equal instrument, as explained in SWE-bench and coding benchmarks.
How to apply it in the company
The operating rule is one: uses vendor numbers as a hypothesis to be verified, never as conclusions.
It basically means three things. First: you prefer internal comparisons —new model versus previous model from the same supplier —because they eliminate much of the asymmetries. Second: When an ad does not declare version, scaffold, and level of reasoning, it discards the number instead of interpreting it. Third: Use the surviving numbers only to narrow to two or three candidates.
Then the real work, which is always the same: ten real cases, the same material for everyone, evaluation entrusted to those who know the process. The complete method is in how to read AI benchmarks.
If you want to understand what processes in your company can be automated with AI, Giallo Studio can help you turn the problem into a real workflow.
From here, three useful insights. To orient yourself among the price ranges, the complete picture is in when an economic model is enough. The correct way to read the numbers declared by the manufacturers is in cost per task or price per token. And to understand if your task really requires the band you are paying, verification is in Mixed model strategy: light and powerful.
Conclusion
Vendor numbers are not false: they are measures produced with different scaffolds, which stop being comparable the moment they are side by side.
They remain valuable to one thing, and that's what they're meant to be: understand how much a new model improves over the previous one of the same vendor. For everything else you need your test on your data.
If you want to understand what processes in your company can be automated with AI, Giallo Studio can help you turn the problem into a real workflow.
Giallo Studio guides: methodology and sources
How we apply it in the company
Related resources
FAQ
Do vendors cheat on benchmarks?
There is no need to cheat to produce numbers that cannot be compared. Just choose which tests to publish, with which configuration and with which scaffolding: these are legitimate choices that, however, make the results not comparable to those of a competitor measured differently.
What changes between measures?
The level of reasoning set, the number of attempts granted, the tools the model can use, the exact version of the benchmark and the system prompt. Each of these moves the result, and they are rarely declared all.
Are vendor figures therefore useless?
No, they are useful for internal comparisons: when Anthropic says that Opus 5 more than doubles Opus 4.8 on Frontier-Bench v0.1, it is comparing two of its models with the same scaffold, and it is solid information. It becomes brittle when you put that number next to that of another vendor.
What are internal benchmarks?
Evaluations built by the vendor and not public. Anthropic cites some in the Opus 5 announcement, for example on organic chemistry. They are honest information but by definition not verifiable by third parties, and should be read as indications of the company about itself.
Do independent rankings solve the problem?
They reduce it because they apply the same standard to everyone. But they measure what they measure, often human preference in generic conversations, which does not match performance on a specific business task.
How do I compare two models honestly?
Trying them yourself, on the same ten real examples, with the same instruction and configuration. It's the only comparison where you control all the variables, and for a business decision it's also the only one that matters.