Gemini 4 Hydrogen benchmarks: how it compares with Claude Sonnet 5.5 and GPT-6.1 Terra
Gemini 4 Hydrogen benchmarks: the 10 model card tests against Claude Sonnet 5.5 and GPT-6.1 Terra, where it wins, where it loses, and how to rerun them.

Gemini 4 Hydrogen, in the tests Google published on 12 November 2026, scores highest in 7 of the 10 rows of the table against Claude Sonnet 5.5 and GPT-6.1 Terra, while in the other 3 Claude Sonnet 5.5 is ahead. The numbers come from the DeepMind model card, the evaluation and methodology and the Google announcement, all accessed on 14 November 2026.
These are results declared by Google: Giallo has not rerun them. In this guide you will find the full table translated, what Google says about each test, who calculated the values and how to build a test on your own cases. The model’s introduction is in Gemini 4 Hydrogen: what it is.
In summary
- The Gemini 4 Hydrogen model card reports 10 tests, split into six areas, and compares the model with Claude Sonnet 5.5 and GPT-6.1 Terra.
- Hydrogen is first in SWE-sweep, Agents’ Last Exam v1.1, AutomationBench-AA, Vending-Bench 2, MKA, Chartography and HLE-Diamond.
- Claude Sonnet 5.5 is first on Terminal-Bench 4.1, OSWorld-2.1 and Terminal-Bench-Science 0.1.
- For seven tests Hydrogen is measured by Google; the competitors’ numbers are mostly declared by the vendors or taken from public leaderboards.
- On safety, Google reports improvements over Gemini 3.8 Flash, but also more unwarranted refusals (+5.57 points), and the full report is not out yet.
- For an SME, the choice of model is made with a test on its own cases: benchmarks help decide which models to try.
What does the Gemini 4 Hydrogen benchmark table say?
The model card table is “as of November, 2026” and compares three models across ten tests. Below you will find it translated, with a column showing what Google says about each test. Where Google does not explain a test, we give only the category and the table’s note, without adding our own interpretation. The highest value in each row is in bold.
| Test (area) | What Google says | Gemini 4 Hydrogen | Claude Sonnet 5.5 | GPT-6.1 Terra | Source |
|---|---|---|---|---|---|
| SWE-sweep (agentic coding) | Find and fix bugs without breaking existing code (announcement) | 13,5% | 3,1% | 5,8% | model card |
| Terminal-Bench 4.1 (agentic coding) | Note: “cheating-adjusted” (literally, corrected for cheating) | 57,0% | 63,9% | not available | model card |
| Agents’ Last Exam v1.1 (knowledge work) | Complete professional workflows, average score per criterion | 71,6% | 60,4% | 69,8% | model card |
| AutomationBench-AA (knowledge work) | No description; Google only writes that Hydrogen leads its class of models | 76,1% | 71,8% | 60,2% | model card |
| Vending-Bench 2 (knowledge work) | Note: bank balance in dollars | $14,549 | $11,165 | $9,107 | model card |
| OSWorld-2.1 (computer use) | Note: offline subset of 82 tasks | 77,6% | 83,9% | 74,2% | model card |
| MKA (multimodal understanding) | Agentic knowledge work on complex visual files (announcement); note “Climb WSR accuracy” | 53,4% | 52,0% | 43,8% | model card |
| Chartography (multimodal understanding) | No description in the pages consulted | 66,6% | 61,6% | 63,2% | model card |
| Terminal-Bench-Science 0.1 (science and math) | No description in the pages consulted | 51,4% | 59,9% | 58,3% | model card |
| HLE-Diamond (reasoning) | Reasoning and analysis in specialized domains (announcement) | 58,8% | 53,3% | 47,3% | model card |
The descriptions marked “announcement” come from the Google post of 12 November 2026; the notes are the model card’s labels, translated; the number of OSWorld tasks comes from the methodology document. On Terminal-Bench 4.1 the GPT-6.1 Terra cell shows a dash: Google does not publish that figure.
Where does Gemini 4 Hydrogen win and lose against Claude Sonnet 5.5 and GPT-6.1 Terra?
Hydrogen is first in seven tests out of ten, and in four it has wide margins over at least one of the two competitors. On the data from the model card, with gaps in percentage points calculated by us:
- AutomationBench-AA: +4.3 over Sonnet 5.5 and +15.9 over Terra.
- HLE-Diamond: +5.5 over Sonnet 5.5 and +11.5 over Terra.
- Agents’ Last Exam v1.1: +11.2 over Sonnet 5.5 and +1.8 over Terra.
- SWE-sweep: 13.5% against 3.1% and 5.8% (model card). These are low values in absolute terms for all three: the order matters more than the distance.
Where it loses, the gaps are not small (model card). On Terminal-Bench 4.1 Sonnet 5.5 is ahead by 6.9 points, on OSWorld-2.1 by 6.3 and on Terminal-Bench-Science 0.1 by 8.5; in the last one Hydrogen is also behind Terra (58.3%), so it comes last of the three. On OSWorld-2.1 Hydrogen stays ahead of Terra, with a 3.4-point margin.
On MKA (+1.4 over Sonnet 5.5) and Chartography (+3.4 over Terra) the margins are small: with a single attempt (pass@1) and values from different sources, a gap that small does not let you say that either model is better at your work.
The reading that emerges from the numbers fits in one sentence: on knowledge-work and reasoning tests Hydrogen is ahead, while on terminal, computer and science tests Sonnet 5.5 is. It is a direction, not a verdict. For the wider picture with other models, the guide Claude, GPT and Gemini in comparison explains how to think about the differences between model families.
Who calculated these numbers, and why is the comparison not entirely level?
The most important point is the methodology. In the evaluation document Google writes that Hydrogen’s scores are pass@1, meaning a single attempt, unless stated otherwise, with no majority voting and no parallel test-time compute, and that the Gemini results use the highest reasoning level unless stated otherwise (for Terminal-Bench 4.1 the level is high). Results for non-Gemini models, on the other hand, are those declared by the vendors, except where noted; for Sonnet 5.5 and Terra Google reports the highest reasoning settings available, and where none exist it uses the best results it could find.
The row-by-row detail, from the evaluation document:
Who calculated what, test by test
- SWE-sweep: Hydrogen calculated by Google with the Antigravity-CLI system version 1.4.8 and safety filters on; Sonnet 5.5 and Terra from the official public leaderboard.
- Terminal-Bench 4.1: Hydrogen calculated by Google with mini-swe-agent, average of 5 attempts per task and a high effort level; Sonnet 5.5 from the official leaderboard. All results use “cheating-adjusted” accuracy.
- Agents’ Last Exam v1.1: 122 tasks and a 5-hour window, with safety filters. Hydrogen is calculated with Antigravity-CLI 1.4.8, Sonnet 5.5 with the ALE-Claw harness in the same configuration; Terra comes from the official leaderboard. Responses blocked by the filters are returned as an empty string.
- AutomationBench-AA and HLE-Diamond: all values from the Artificial Analysis leaderboard.
- Vending-Bench 2: values from the public Andon Labs leaderboard.
- OSWorld-2.1: 82 offline tasks, with no internet. Hydrogen and Sonnet 5.5 calculated by Google, taking the best of three runs; Terra from the published system card. The partial score is reported.
- MKA: Hydrogen calculated by Google; the other values from the Artificial Analysis leaderboard.
- Chartography: Hydrogen and Terra calculated by Google, without tools; Sonnet 5.5 from the published system card.
- Terminal-Bench-Science 0.1: Hydrogen calculated by Google as an average of 5 attempts; the other values from the official leaderboard.
Two practical consequences. The first: where Google measures its own model with a harness it chose, and the competitor is taken from elsewhere, part of the gap may come from the harness and not from the model. The second: where the value comes from a public leaderboard, as for AutomationBench-AA and HLE-Diamond, all models are at least judged by the same referee.
What does Google’s announcement add beyond the table?
The announcement contains other figures, all declared by Google and not verified by us. You will find them in the post of 12 November 2026, which has since been updated on 13 November:
- Artificial Analysis Intelligence Index (from the announcement): 51 at the default reasoning level (
high) against 50 for Gemini 4 Argon, and 54 at the highest level (xhigh). - Agent Arena on arena.ai: fifth place.
- Vals Tax Agent Bench: first, with 52.3%. Vals Legal Research Bench: joint first, with 55.8%.
- Compared with Gemini 3.8 Flash: tasks finished 2.5 times faster at
high; Terminal-Bench 4.1 at 57% against 18.2%; made-up claims reduced by more than 5 times. - Orchestrator in Google Antigravity: on Terminal-Bench 4.0 it reaches success rates close to Carbon with 24% less quota, finishing the work 19% faster. This is version 4.0 of the test, different from the 4.1 in the table.
The comparison between Hydrogen and its sibling models, Carbon and Argon, with role, price and reasoning levels, is in Gemini 4 Hydrogen vs Carbon and Argon. The tables for the two “frontier” models are in Gemini 4 Carbon benchmarks and Gemini 4 Argon benchmarks.
What do the Gemini 4 Hydrogen safety numbers say?
Google presents safety as a measure to read together with performance, and the picture has two sides. In the announcement the comparison is with Gemini 3.8 Flash: up to 81% fewer out-of-scope actions (“overreach”), up to 79% fewer half-served tasks (“under-service”), misaligned actions and shortcuts reduced 12-fold in tool and coding evaluations, 94% honesty in the internal agent evaluation, and more than 2.3 times greater resistance to indirect prompt injection on the Gray Swan IPI benchmark. Unwarranted refusals on HealthBench Pro stay below 0.4%.
The model card is less triumphant. In the automated evaluations against 3.8 Flash, text safety (+0.80 points) and multilingual safety (+1.10) remain, according to Google, similar within the confidence intervals; image-to-text safety rises by 7.00 points. Refusal tone (-0.65) is judged similar, while unwarranted refusals rise by 5.57 points, especially with multimodal prompts: in practice, the model refuses legitimate requests more often. Google adds that the full “Safety & Security” report will come later, and that known limitations include hallucinations, occasional slowdowns and higher token consumption at high reasoning levels.
On extreme risks, the card says that Hydrogen shows no material increase in capability over Argon, that the critical thresholds for biological risk and cybersecurity are not reached but the alert thresholds are, and that the mitigations remain active. For an agent in a business, two points matter most: resistance to instructions hidden in documents, and behavior when facing a blocked task. These are measures declared by Google, and they should be checked with hallucination tests and with the edge cases of your own process.
What do MKA and Chartography mean for an administrative office?
Take an example, declared as such: an administrative office that every month extracts amounts, dates and codes from supplier PDFs, bank statements and report charts. The two multimodal understanding tests are the closest to that work, but not in the same way.
Google describes MKA as a new test of agentic knowledge work on complex visual files and reports 53.4% for Hydrogen, against 52.0% for Sonnet 5.5 and 43.8% for Terra (announcement and model card). On Chartography Hydrogen scores 66.6%, Sonnet 5.5 61.6% and Terra 63.2%, but in the pages consulted on 14 November 2026 Google does not explain what it measures: we only know that it places it under multimodal understanding. For this reason we do not read it as “chart reading” or as anything else: it is a category hint.
An office that needs to extract data from visual documents takes one message from these rows: Hydrogen deserves a place in the test, alongside Sonnet 5.5. Which of the two makes fewer mistakes on your PDFs, with your formats and your signatures in the bottom right corner, only the test can say. For the basics of the topic, the guide Multimodal AI explains what it means to have a model read text, images and tables together.
How much does it cost to get these results?
Google lists Hydrogen at $1.50 per million input tokens and $7.50 per million output tokens, with a flat price across the whole one-million-token window (announcement and price list). Reasoning tokens are billed as output, and the table results are obtained, except for cases like Terminal-Bench 4.1 (level high), at the highest reasoning level: at a lower level, cost and scores may change. The full calculation, with examples, is in Gemini 4 Hydrogen pricing. The competitors’ price lists are not in Google’s sources and we have not reviewed them here.
How do you rerun a test on your own cases in an SME?
A benchmark answers the question “how does this model do on other people’s tasks?” You need to answer “how does it do on mine?” The method works for any pair of models, including Hydrogen and Sonnet 5.5, and does not require made-up numbers.
- Choose a real process: something repetitive and verifiable, such as extracting data from invoices or summarizing tender specifications.
- Collect about twenty cases already closed: with the correct outcome known, including two or three messy ones, with crooked scans or missing data.
- Set the error criteria before testing: which mistake makes the result unusable, which one is tolerable, and how long the correction takes.
- Use the same instructions for all models: same prompt, same documents, same response format; change one variable at a time, for example the reasoning level.
- Count errors, review time and cost: a model that makes fewer mistakes but costs twice as much may not be worth it, and the reverse is also true.
- Repeat the test at every model change: the test bench is an asset of the company, the model can be replaced.
If you already have live integrations, you will find the technical steps in migrating to Gemini 4 Hydrogen; for product and privacy choices, the guide Gemini 4 Hydrogen for business shows which surface to start from.
What mistakes do benchmark comparisons most often make?
Four mistakes come up every time a new table is released.
- Adding up the rows and declaring a winner. Seven tests out of ten do not make Hydrogen better at every task: for terminal and computer use, the table backs Sonnet 5.5.
- Confusing versions of the same test. Terminal-Bench 4.0 and 4.1 are different tests, and even the announcement cites them in different contexts.
- Ignoring who measured. A value calculated by the vendor with its own system and a value from a third-party leaderboard carry different weight.
- Taking a one-point difference for a fact. With values almost always from a single attempt and data partly declared by the vendors, margins of a point or two are within the noise.
What should you do with these numbers?
The Gemini 4 Hydrogen table says that Google has a model, which it describes as fast, that in its own tests is ahead for knowledge work, reasoning and visual files, and that on terminal and computer use the model to beat remains Claude Sonnet 5.5. On everything else Google itself points to limitations, possible hallucinations and a safety report still to be published. Anyone choosing today can use these numbers to decide which models to test, not to close the decision; the general picture on Claude is in Claude AI: what it is and when to use it.
Giallo Studio guides: methodology and sources
Want to apply this to your business?In 30 minutes, free of charge: the first process to automate and its estimated return, in writing.
Book the free auditHow we apply it in the company
Related resources
Gemini 4 Hydrogen: What It Is, How It Works, Where to Use It
Gemini 4 Hydrogen pricing: how much the API costs and how to estimate spend
Gemini 4 Hydrogen vs Carbon and Argon: which Gemini model to choose
Gemini 4 Hydrogen for business: use cases, costs and data privacy for an SME
Migrate to Gemini 4 Hydrogen: Checklist From Gemini 3.8 Flash and 2.5 Before 1 December
Gemini 4 Carbon benchmarks: the full table and how to read it
Gemini 4 Argon benchmarks: how to read the LLM comparison
Claude, GPT and Gemini in comparisonFAQ
Is Gemini 4 Hydrogen better than Claude Sonnet 5.5?
It depends on the test. In Google’s model card table, published on 12 November 2026, Hydrogen is ahead of Sonnet 5.5 in 7 of 10 tests, but it stays behind on Terminal-Bench 4.1, OSWorld-2.1 and Terminal-Bench-Science 0.1. The numbers are declared by Google, and there is no test on your own documents.
Is Gemini 4 Hydrogen better than GPT-6.1 Terra?
In Google’s table, Hydrogen beats GPT-6.1 Terra in 8 of 9 tests, because the Terra figure for Terminal-Bench 4.1 is missing. The only test where Terra is ahead is Terminal-Bench-Science 0.1, with 58.3% against 51.4%.
Which tests is Gemini 4 Hydrogen behind Claude Sonnet 5.5 in?
Three: Terminal-Bench 4.1 (57.0% against 63.9%), OSWorld-2.1 (77.6% against 83.9%) and Terminal-Bench-Science 0.1 (51.4% against 59.9%). They are tests of terminal coding, computer use, and science and math.
Who calculated the Gemini 4 Hydrogen benchmarks?
For seven tests, the Hydrogen results are calculated by Google. For AutomationBench-AA and HLE-Diamond they come from the Artificial Analysis leaderboard, and for Vending-Bench 2 from the Andon Labs one. The competitors’ numbers are largely declared by the vendors or taken from public leaderboards, so test conditions do not all match.
What is the difference between Terminal-Bench 4.0 and 4.1?
They are two different tests. The model card table uses 4.1, where Hydrogen scores 57.0%. The announcement cites 4.0 for the Antigravity Orchestrator and 4.1 for the comparison with Gemini 3.8 Flash: they should not be confused.
Is Gemini 4 Hydrogen safer than Gemini 3.8 Flash?
According to Google, yes, on many agentic measures, but not all of them. The model card reports more unwarranted refusals than 3.8 Flash (+5.57 percentage points), especially with multimodal prompts. As of 14 November 2026, the full safety report is not yet published.
Can I choose an AI model for my business by looking at benchmarks?
No, not on their own. Benchmarks tell you which categories a model starts ahead in, but none of them uses your documents, your customers or your error criteria. A test on about twenty real cases, with the correct outcome already known, tells you more than any table.