SWE-bench and coding benchmarks

SWE-bench is the benchmark test when it comes to AI models that write code, and is also one of the few benchmarks built on authentic work instead of invented exercises. That is why its numbers appear in almost all ads of new models.

Understanding how it works serves two things: reading those numbers without getting fooled, and knowing when they don't say anything useful to your case.

How it works

The mechanism is more concrete than the name suggests. It starts with real open source projects and problems that someone had actually reported. For each case we take the description of the problem and the status of the code before to be resolved.

The model receives those two elements and must produce a modification. The change is then applied and subjected to automatic project testing: if they pass, the case is solved; if not, it has failed. There is no subjective evaluation, the criterion is binary.

This makes it an unusually honest test compared to many others. No one judges whether the answer "seems good": either the code works or does not work.

Why the Verified Version Exists

The original set also contained problematic cases: descriptions too vague to be solved, tests that required information not present in the material provided, situations where multiple solutions were legitimate but only one passed.

Such cases lower everyone's score arbitrarily and make the comparison less informative. Hence the version Verified: a subset revised by hand, in which each problem was checked to ensure that it was actually resolvable with what is given to the model.

When you find a cited number, it is worth checking which variant it refers to. A score on the complete set and one on the verified subset are not comparable, and it is a mistake that is often seen against quickly made.

The scaffolding problem

It is the point that makes comparisons more fragile, and it is the same as described in because vendor benchmarks are not comparable.

The model does not work on its own: there is a system around it that decides how The problem is presented to him, and those choices change the outcome substantially.

• How much project code is shown, and how it is selected
• If the model can run tests before giving the answer
• How many attempts has available
• Whether he can browse files or only receives an extract
• The level of reasoning set
• If multiple responses are generated and then filtered

A scaffold that allows you to run tests and try again produces scores much higher than the exact same model-problem pair without that possibility. Not because the model is better: because the system around it allows it to correct itself.

What the test doesn't measure

Here lies the most important limit to keep in mind, especially if the result must orient a real choice.

The right column is the one that determines whether the product code will be an aid or a problem in six months. No automatic benchmark covers it, because it requires human judgment.

Then there is a second, thinner limit: the projects used are public, so there is a possibility that some of that material ended up in the training data. Curated versions of the test try to limit the phenomenon, but caution remains justified.

When relevant

The general criterion for understanding which tests to look at based on your case is in which AI benchmarks really matter.

Practical examples

Software house evaluating an assistant for the team. The benchmark is relevant and useful for narrowing the field to two or three models. The decision, however, must be made by trying them on its own code base, which has conventions and structure different from any open source project.

Company that wants to automate report generation. The test has nothing to do with it: the task is to produce text from data, not to modify software.

SMEs evaluating an agent for placing orders. Not even here. The relevant criterion concerns agent duties and reliability on actual documents, not programming.

Mistakes to Avoid

• Comparing different variants of the set without saying it.
• Ignoring scaffolding used to get the number.
• Deducting its quality of the product code.
• Transferring it to other domains: doesn't say anything about the rest.
• Neglecting the Cost for task solved.
• Stopping at the score without trying on its own code.

How to use it in practice

The correct way is to treat it as an initial filter. Shrink to two or three candidates using results obtained with the same scaffold, then move on to the test that really matters.

That test is simple: take ten real problems already solved by your team, give every model the same material and compare the solutions with the actual ones. In half a day, you learn much more than rankings will ever tell you, because you are measuring behaviour on your own code, with your conventions and quirks accumulated over the years.

Finally, it is worth looking at the cost per problem solved instead of the price per token: a model that costs twice as much but solves in one attempt instead of four agrees, and it is the reasoning developed in cost per task or price per token.

Why the scores went up so fast

Those who have been following these numbers for a few years notice an obvious acceleration: results that seemed distant have become ordinary within a few cycles of models. There are three reasons and it is better to distinguish them, because only one concerns the model.

The first is genuine: the models are improved in understanding existing code and locating the right intervention. The second is about scaffolding, which has become much more sophisticated — being able to run tests and correct yourself radically changes the outcome. The third is the maturation of the set itself, cleaned up by the impossible cases that kept everyone's score artificially low.

Three different causes, one number going up. When an ad attributes the entire improvement to the model, it's simplifying: part of the merit belongs to the system built around it.

What changes for those who have to choose

This has a concrete and often overlooked consequence. If you adopt a model that excels in ranking but use it in a tool that won't allow it to run tests and try again, you won't get that behavior. The result you read included capabilities that your environment does not provide.

It is therefore worth evaluating the Instrument-Model Pair, exactly as it happens in the agent benchmarks described in Frontier-Bench: what it is and what it measures. Changing one or the other produces different results, and in production it counts what you will really use.

The hidden cost of multiple attempts

There is one economic aspect that the rankings never show. A model that solves the problem on the fourth attempt solved the problem, and in the ranking counts as success. But in production those four attempts pay them all.

The last line is the most annoying: failed attempts are paid anyway. For this reason the sensible comparison is not on the price per token but on the cost per problem actually solved, which holds in both the unit price and the number of steps needed.

A more expensive model that closes on the first shot can cost less than a cheap one that takes five, and it happens more often than you imagine when you only look at the price list.

How to apply it in the company

If you develop software, the correct use of this benchmark is as an initial filter: narrow to two or three candidates using results obtained with the same scaffold, then go to the test that counts.

That test is simple and takes half a day. Take ten real problems already solved by your team, including at least three interventions on existing code and not just new files. Give each model the same material and tool that you will really use in production, then compare the solutions with those adopted by the team.

Evaluate two things separately: if the code works and if it is code you would accept in revision. They are different judgments, and only the first one measures it a benchmark.

One last trick that applies to any model: always ask for explain the modification and to declare the assumptions made. A model that writes "I assumed that this field is always present" is giving you the information to verify; one that does not state it has made the same assumption in silence.

The complete guide to choice for development is in best AI model for programming.

If you want to integrate AI into your company's development flow, Giallo Studio designs tailor-made solutions and puts them into production.

A Note on Contamination

A bottom line that is worth keeping in mind on any benchmark built on public material: the projects used are accessible to everyone, so there is a possibility that some of that code ended up in the training data of the evaluated models.

Curated versions of the test try to limit the phenomenon, and newer projects are born to circumvent it. But caution remains justified: a very high score on a set of public and dated tasks says less than a more modest one on tasks built on purpose and renewed over time.

Conclusion

SWE-bench is one of the best built benchmarks around, because it uses real work and has an objective success criterion. But it measures one specific thing —the modification passes the tests —and must be read along with the scaffolding that produced the number.

If you develop software, it is the right starting point to narrow the field. If you do not develop software, it does not concern you: look benchmarks that matter to your case.

If you want to figure out which model makes sense for your process, Giallo Studio designs custom AI solutions and puts them into production.