ARC-AGI: what it is and why it is talked about

ARC-AGI is the benchmark that appears in every discussion about ‘how much an AI model really reasons.' It is also the most misunderstood: it does not measure how useful a model is, it measures how well it can solve problems it has never seen.

It is worth understanding, because it is the test that best explains the difference between know and understand in an AI — system and because it matters much less to a company than it seems.

How it works, in concrete

The format is simple to explain and difficult to solve. Some pairs of colored grids are shown: one starting and one finishing. The answerer must understand which rule it turns the first into the second, and then apply it to a new grid.

The rule is never stated. It can be "fill the closed forms", "repeat the pattern to the edge", "exchange the two most frequent colors". Each task has its own, and each task is different.

Because it's designed to be difficult

Many benchmarks have a known problem: a model that saw similar content during training can respond well without reasoning. It is the phenomenon of contamination, and undermines the validity of any test built on public material.

ARC-AGI circumvents the problem in a structural way. The tasks are built on purpose, the answer is not recoverable from any source, and the rule must be derived. If a model succeeds, that result says something real about its ability to generalize.

And that is why progress on this test is closely followed: they are considered a cleaner signal than the scores that go up on tests now much represented in the training data.

What the vendors say

In the announcement of Claude Opus 5 of July 24, 2026, Anthropic states that the model gets on ARC-AGI 3 a score of approximately three times that of the second best model. It is a data stated by the vendor, referring to that version of the test and to that date.

As with all self-reported numbers, it should be read for what it is: meaningful, but produced by those who have an interest in showing it. The complete reasoning is in because vendor benchmarks are not comparable.

How Much It Matters for a Company

Little, and it's important to say it without a word. ARC-AGI is a search indicator: it says where the technology is coming, not which model will solve your problem.

The right column contains everything you need to decide. The criterion for choosing what to really look at is in which AI benchmarks really matter.

Practical examples

SMEs evaluating an assistant on internal procedures. ARC-AGI is irrelevant: the task is to retrieve and rephrase information in documents, not to infer abstract rules.

Company that wants to automate the extraction from invoices. Here, too, the test doesn't help. You need to try on your own PDFs, including those that are wrong.

Team building an agent on open and variable tasks. Here the signal is a little more relevant, because the ability to deal with unexpected situations matters. It still remains secondary compared to field tests.

Mistakes to Avoid

• Read it as a measure of general intelligence. It measures a family of problems.
• Use it to choose a provider. It doesn't predict the outcome on your case.
• Comparing different versions. The task set changes.
• Ignoring the source of the number. Many are self-reported.
• Infer the quality of writing. It has nothing to do with it.
• Neglecting cost and latency. They weigh more in production.

How to use it, if you really want to use it

Treat it as an industry thermometer, not as a product board. If you need to understand how much the reasoning ability of the models is advancing, it is one of the few tests that says it honestly.

When instead you have to choose, the method is another: ten real cases of your process, the same for each model, evaluated by those who know the work. It is described step by step in how to read AI benchmarks.

The history of the test, in short

ARC-AGI comes from a precise idea: if we want to measure the intelligence of a system, we cannot measure how much it knows, because knowledge can accumulate. We have to measure how much learn quickly from a few examples, which is a different property.

This approach has an uncomfortable consequence for the industry. A model can be trained on a huge amount of text and get excellent results on most of the tests in circulation, remaining weak right on ARC-AGI. For years, exactly this has happened: the gap between human and model performance on this test has remained much wider than elsewhere.

The fact that that gap has narrowed in recent cycles is, in itself, the news. Not because it serves something in the company tomorrow morning, but because it indicates that models are becoming less dependent on what they have already seen.

Why the versions follow each other

Every time the models come close to saturating a version of the test, the test loses informational power: if everyone takes a high score, it no longer distinguishes anything. For this the task set is renewed and made more difficult.

It's a healthy mechanism, but it introduces a trap for the reader: numbers of different versions do not compare. A result on a newer and more difficult version may look worse while representing a better model. When you find a quoted digit with no version, that digit is not usable.

What should not be confused with ARC-AGI

There is some confusion between this test and other things that only resemble it in name or aura. It is worth separating them.

The last three lines are those that enter into a business decision. The first line concerns those who follow the state of the art. The extended comparison is in which AI benchmarks really matter.

The Risk of Reading It Badly

The most expensive mistake is not overestimating ARC-AGI: it is using it as a shortcut to avoid testing. It happens more often than it seems, and the dynamic is always the same. We read that a model dominates the most difficult test in circulation, it is deduced that it is the best ever, it is adopted everywhere.

Then in production the real problems emerge, which are of a completely different nature: the model costs too much for the volume it needs, or it is too slow for a real-time interface, or it is wrong exactly on the most frequent class of documents in the company because no one has ever put it before it.

The cost variable also weighs on this, which benchmarks almost always ignore and which instead decides whether a project is sustainable. The difference between price per token and actual cost per task is explained in cost per task or price per token.

A question worth more than the score

Before looking at any ranking, it is best to answer only one question: Does my problem require you to infer new rules, or to apply already known rules?

In the vast majority of business cases the answer is the second. Classify an invoice, respond to a recurring request, summarize a contract, fill out a field from a document: these are tasks in which the rule already exists and must be applied well, consistently, on high volumes.

For those tasks the quality that is needed is not creativity in inferring, but the reliability in repeating. And it is a property that ARC-AGI does not measure at all, while they measure it much better ten proofs on your true documents, including those written badly, incomplete or out of format.

If, on the other hand, your case really requires you to deal with always different situations —exploratory analysis, open problems, diagnosis on unexpected scenarios —then the signal becomes more relevant, while remaining a clue and not a test.

How to apply it in the company

The honest answer is that this benchmark doesn't fit into a business decision, but the reasoning around it does, and it's worth taking it with you.

The useful question that ARC-AGI suggests is this: Does my task require you to infer new rules, or to apply rules already known? In the vast majority of business cases it is the second, and then the quality that is needed is not creativity in inferring but reliability in repeating on high volumes.

That property does not measure it any ranking: they measure it ten tests on your true documents, including those written badly, incomplete or out of format. The method is in how to read AI benchmarks.

In practice: schedule a six-month review of the projects you had set aside because the technology wasn't enough. That's where progress on benchmarks like this becomes relevant to a company, and it's a control that costs half an hour.

If you want to understand what processes in your company can be automated with AI today, Giallo Studio can help you turn the problem into a real workflow.

Three readings to close the picture. If you are still choosing the band, the operating criterion is in when an economic model is enough. For the calculation of real expenditure —the one that includes repeated context, failed attempts and human review —the reference is Mixed model strategy: light and powerful. And when traffic is uneven, the setting that gives the best relationship between quality and cost is described in when it is best to change the AI model.

Conclusion

ARC-AGI is the most interesting test to follow and the least useful to decide on. It measures a real and difficult thing —to solve new problems without ever having seen them —but it doesn't coincide with "it works well on my work".

Keep an eye on him to see where the technology is going. benchmarks that matter and especially the tests on your data.

If you want to figure out which model makes sense for your process without getting lost in scores, Giallo Studio designs and puts into production custom AI solutions.