Frontier-Bench: what it is and what it measures
What Frontier-Bench is: the agentic task benchmark succeeding Terminal-Bench, and why it measures the agent and model as a pair.
Reading time: 8 min
Frontier-Bench is one of the benchmarks cited most often in recent model ads, and is also one of the least understood. It does not measure how well a model writes: it measures as much as a pair Agent Plus Model carries out real work tasks in an isolated environment.
This distinction changes the way you read the results, and for a company that evaluates automations it is more relevant than any text generation test.
In summary
- It is the successor to Terminal-Bench, with tasks that evolve over time.
- Measurement of tasks agentici run in isolated environment, not text generation.
- The result describes the pair agent + model, not the model alone.
- The task set changes: the version It's part of the data.
- It is an open project with review of the proposed tasks.
- Relevant if you evaluate systems that act, not that they answer.
What it measures, concretely
According to the project's official repository, Frontier-Bench is designed to "follow the frontier with a set of diverse, difficult and high-quality tasks that evolve over time." Tasks run in the sandbox environment, and the project recommends validating them by running a reference solution several times before considering the execution environment reliable.
Translated: it is not a closed-ended questionnaire. It is a set of tasks that go completed, and the score reflects how many are solved.
The point that changes everything: agent plus model
In the documentation of the project, to perform an evaluation you must indicate both the agent and the model. A result therefore does not say "this model is worth X": it says "this model, guided by this agent, with this configuration, has solved X".
The practical consequences are two and it is worth keeping them in mind every time you read a number on this benchmark.
- The same model with different agents gets different results
- An improvement may come from the agent, not the model
- Comparing two models requires the the same agent and the same configuration
- The level of reasoning set is part of the result
It is the same scaffolding problem described in because vendor benchmarks are not comparable, here made explicit by the very nature of the instrument.
Why the version counts
The project states that the task set evolves over time: new tasks are proposed, reviewed and added. This means that a score on one version is not comparable to one on another, because the test has changed.
That is why the precise version appears in the ads. In Claude Opus 5's announcement of July 24, 2026, Anthropic cites Frontier-Bench v0.1 and states that Opus 5 outperforms all other models on that test, more than doubling the result of Opus 4.8 at a lower cost per task. It is a data stated by the vendor, referring to that version and to that date.
When relevant to a company
Look at him if...
- Evaluate agents performing multi-step tasks
- The system must use tools, not just respond
- Are you interested in the ability to bring to a conclusion
- You are choosing between models for an agent project
Ignore him if...
- Your case is to write or summarize
- You need to extract data from documents
- The system just responds
- The task is single-step
The right column covers many common business cases, for which this benchmark says nothing useful. The selection criterion is in which AI benchmarks really matter.
Practical examples
Company evaluating an agent for the passive cycle. The task is multi-step — reads, checks, proposes, records — so the agent family is relevant. However, it remains to be proven on your documents, because the benchmark uses generic tasks.
Studio that wants an assistant on procedures. Single-step task: Answering a question. Frontier-Bench is not the right criterion.
Software house that chooses a model for its development tool. Here the benchmark is very relevant, and it is useful to compare results obtained with the same agent.
Mistakes to Avoid
- Read it as a model score. Measure agent plus model.
- Comparing different versions. The task set changes.
- Ignoring the agent used. It changes the outcome substantially.
- Use it for non-agent tasks. He doesn't describe that job.
- Forgetting the level of effort. It is part of the configuration.
- Assuming transferability to your case. Tasks are generic.
How to apply it in the company
If you are evaluating an agent project, use this benchmark to narrow the field to two or three models, verifying that the cited results use the same agent.
Then do the test that counts: ten real tasks of your process, with the agent you're really going to use. It is the only way to know how the pair behaves that will end up in production, and it is the method described in how to read AI benchmarks.
Why the results fluctuate
Those who look at a ranking of agent benchmarks for the first time immediately notice one thing: numbers move more than you would expect. It's not noise, it's the characteristics of the method.
An agentic task involves many steps, and each step can go wrong independently. The model can choose the wrong tool, misinterpret an error message, insist on a road that leads nowhere, or stop too soon convinced that it is finished. A weak link is enough to lose the whole task, even when much of the work was correct.
Hence the practice of repeating each task several times and considering the average. It is why some rankings talk about average reward on repeated attempts instead of a simple success rate.
What affects the result in addition to the model
- The Agent used and its repetition logic after an error
- The level of reasoning set
- The number of attempts granted by task
- The limit of steps or time for execution
- The tools available in the environment
- The version the Task Set
Six variables, and the model is just one of them. When an ad reports a number without specifying the other five, that number tells a configuration, not a capacity. The mechanism is the same as described in because vendor benchmarks are not comparable.
How it translates to real work
The useful question is not which model leads the ranking, but what a success rate means on a business process.
| Success rate | What it implies in production |
|---|---|
| Very high | The agent works, we need sample control. |
| Middle-high | Useful with human review on each outcome |
| Medium | Useful just to prepare work, not to close it |
| Low | Not ready for that task |
The point is that almost no business process tolerates a silent error. If one agent misses one accounting record out of twenty and no one notices, the cost of recovery outweighs the savings. For this reason the value of an agent depends as much on the success rate as on the Possibility of noticing mistakes.
What to watch alternatively
If your goal is to put an automation into production, the benchmark only serves to narrow the field. The decision is made on something else.
Criteria that decide
- Outcome on ten real tasks of your process
- Behavior on Limit Cases and Bad Documents
- Cost per completed task, not per token
- Ability to stop and report uncertainty
Distracting Criteria
- Position in general classification
- Number of parameters or declared size
- Scores on tasks away from yours
- Announcements without version or configuration
The method of constructing those ten tasks is described in how to read AI benchmarks; the comparison of real costs is in cost per task or price per token.
The relationship with Terminal-Bench
Frontier-Bench is the Direct successor to Terminal-Bench, and this continuity helps to understand its setting. The starting idea was to measure how much a model can work in a command-line environment: read, execute, correct, arrive at a verifiable result.
The limit of that approach, like any fixed set of tasks, is that it gets older. As the models improve, the tasks become easy and the test stops distinguishing. The answer was to build a project in which the set evolves: new tasks are proposed, revised and added over time.
This has an advantage and a cost. The advantage is that the benchmark remains informative. The cost is that it loses stability over time: two results a few months apart may refer to different tests. It is why the version should always be quoted, and it is also why these numbers should be used to orient themselves and not to decide for themselves.
How to apply it in the company
If you are evaluating an agent project, this benchmark has a precise and limited use: narrow the field to two or three models, verifying that the cited results were obtained with the same agent.
From then on, only your process matters. Take ten complete real tasks — not isolated requests — and run them with the agent you will actually use in production. Record every step, not just the final outcome: the steps show where the chain breaks.
The three numbers to be collected are always the same: how many tasks come to an end without intervention, how much each task completed according to the method of cost per task or price per token, and how long it takes the whole execution.
Finally, he designs from the beginning the human level: a small percentage of cases must reach a person, with the context already prepared and an indication of what is doubtful. The complete criterion is in Best AI Model for Agents.
If you want to go from testing with ChatGPT to an AI system integrated into business processes, Giallo Studio custom designed agents and automations.
Three references that complete the framework. The criterion of choice by type of task is in when an economic model is enough. For the economic part the reference is Mixed model strategy: light and powerful. And if you are considering reviewing a configuration already in production, the path is in when it is best to change the AI model.
A Note on Versions
A final practical warning for those who read these numbers over time. Since the project states that the task set evolves, a score only meant along with the version it refers to.
This has an annoying but inevitable consequence: a result on a newer and more difficult version may seem worse while representing a better model. When you find a quoted digit with no version, that digit is not usable — and it is worth mistrusting the source that reported it that way.
Conclusion
Frontier-Bench measures something closer to real work than traditional tests: difficult, multi-step tasks, carried out in an isolated environment. That's why it's often cited when it comes to agents.
The thing to remember when reading a number is only one: describes an agent-model pair in a configuration, not a model. Changing agent changes the result.
If you want to go from testing with ChatGPT to an AI system integrated into business processes, Giallo Studio custom designed agents and automations.
Giallo Studio guides: methodology and sources
How we apply it in the company
Related resources
FAQ
What is Frontier-Bench?
It is an agentic-task benchmark and the successor to the Terminal-Bench project. According to the project’s official repository, it is designed to track the frontier of agent capabilities through a set of difficult, high-quality tasks that evolves over time.
What exactly does it measure?
The ability to complete agentic work tasks in an isolated environment, typically through a terminal. Tasks run in sandboxes, and the project validates each task by running a reference solution several times before accepting it as valid.
Does it measure the model or the agent?
The combination of both, which is the most important point to understand. The project documentation requires both an agent and a model to be specified for a run: a result describes that pair, rather than the model in isolation.
Why is it relevant to a company?
Because it measures something closer to real work than tests of text generation alone: multi-step tasks completed using tools. If you are evaluating automations that take action as well as answer, this is the relevant family of benchmarks.
Why does the version number matter?
Because the authors state that the task set evolves. A score on one version cannot be compared with a score on another, which is why announcements cite specific versions such as v0.1 rather than the generic name.
Who maintains it?
It is an open project that accepts external contributions: the repository describes a review process for proposed tasks and lists third-party organizations among task authors. Proposals undergo automated checks and review before acceptance.