İçeriğe geç / Skip to content / Zum Inhalt

Gemini 4 Hydrogen vs Claude Sonnet 5.5 vs GPT-6.1 Terra

Ahmet Balaman

13 min read

Vibe CodingGemini 4 HydrogenClaude Sonnet 5.5GPT-6.1 TerraGemini 4BenchmarkAI
Gemini 4 Hydrogen vs Claude Sonnet 5.5 vs GPT-6.1 Terra

Gemini 4 Hydrogen is not ahead everywhere. It is first in 7 of the 10 rows of Google's own table, but a closer look at that table and at the independent measurements shows this:

  • On Terminal-Bench 4.1, Claude Sonnet 5.5 scores 63.9% and Hydrogen 57.0%.
  • On OSWorld-2.1, which measures computer use, Sonnet 5.5 scores 83.9% and Hydrogen 77.6%.
  • In Artificial Analysis's own Terminal-Bench run, Hydrogen (about 54%) also trails Sonnet 5.5 (64%).
  • On the Artificial Analysis index, the order depends on the thinking level: Hydrogen scores 51 at its default setting, Sonnet 5.5 scores 55 at its highest.
  • I could not find an independent index row for GPT-6.1 Terra; every number for it here comes from Google's table.

For the model itself, start with What Is Gemini 4 Hydrogen?. Free use and access are in a separate post, and the rival's side is in the Claude Sonnet 5.5 post.

This post is a news roundup and compilation; I have not tested the model, and I have not run the three models side by side on the same tasks. The numbers come from Google's November 12, 2026 announcement, the Google DeepMind model page, Artificial Analysis and news sites. Sources are at the end.

Side by side: Google's own comparison

Google compares Hydrogen with Claude Sonnet 5.5 and GPT-6.1 Terra. The table in the announcement is an image; I took the values below from the text table on the Google DeepMind model page (published with the note "as of November 2026").

Benchmark Hydrogen Claude Sonnet 5.5 GPT-6.1 Terra
SWE-sweep (find and repair bugs without breaking existing code) 13.5% 3.1% 5.8%
Terminal-Bench 4.1 (cheating-adjusted) 57.0% 63.9% not given
Agents' Last Exam v1.1 (end-to-end professional workflows) 71.6% 60.4% 69.8%
AutomationBench-AA 76.1% 71.8% 60.2%
Vending-Bench 2 (bank balance) $14,549 $11,165 $9,107
OSWorld-2.1 (offline subset, computer use) 77.6% 83.9% 74.2%
MKA (Multimodal Knowledge Agent) 53.4% 52.0% 43.8%
Chartography 66.6% 61.6% 63.2%
Terminal-Bench-Science 0.1 51.4% 59.9% 58.3%
HLE-Diamond 58.8% 53.3% 47.3%

Hydrogen leads 7 of the 10 rows. Sonnet 5.5 leads three: Terminal-Bench 4.1, OSWorld-2.1 and Terminal-Bench-Science. Terra is first in none; it is ahead of Hydrogen on Terminal-Bench-Science and within two points of it on Agents' Last Exam.

Four things to keep in mind when reading the table:

  • The vendor chose the benchmarks. This is the table Google prepared to introduce Hydrogen. It is good that the three rows where it trails are included, but Google decided which benchmarks made the table.
  • Most rival numbers are not Google's own runs. According to Google's evaluation methodology document, Hydrogen's results are single-attempt (pass@1) scores obtained through the Gemini API, at the highest thinking setting unless stated otherwise. The numbers for non-Google models are, unless stated otherwise, the vendors' self-reported values; for Sonnet 5.5 and Terra Google uses the result at the maximum thinking setting, or the best available result where that is missing.
  • Row-by-row sourcing. On SWE-sweep, Terminal-Bench 4.1 and Terminal-Bench-Science, Google ran Hydrogen itself and took the rivals' numbers from the official leaderboards. AutomationBench-AA and HLE-Diamond come from the Artificial Analysis leaderboard, Vending-Bench 2 from the Andon Labs leaderboard. On Agents' Last Exam Google ran Hydrogen and Sonnet 5.5 itself and took Terra from the leaderboard. On Chartography Google ran Hydrogen and Terra itself and took Sonnet 5.5 from its system card. On MKA Hydrogen is Google's run and the others come from Artificial Analysis.
  • In two rows the conditions are not equal. On Terminal-Bench 4.1 Hydrogen was run at high (not at xhigh, its highest level), averaged over five attempts per task. On OSWorld-2.1 the best of three runs was taken for Hydrogen and Sonnet 5.5, while Terra's value comes from its system card. These are two of the rows Sonnet 5.5 leads. As you will see below, the thinking level changes the result a lot.
  • The Terminal-Bench version is different. This table uses Terminal-Bench 4.1 (cheating-adjusted). The table in the Carbon comparison used Terminal-Bench 4.0 (Carbon 65.2%, Argon 57.1%). Do not put numbers from the two versions in one column.
  • Everyone is low on SWE-sweep. Hydrogen's 13.5% is several times its rivals' scores, but in absolute terms none of the three models solves most of the tasks.

Independent measurements: the thinking level changes everything

Artificial Analysis measures models separately at different thinking levels. The values on the Intelligence Index v5.0.1 page (checked November 14, 2026):

Model (thinking level) Index Cost per index task Time per task
Hydrogen xhigh 54 $3.16 14.1 min
Hydrogen high (default) 51 $1.87 7.8 min
Hydrogen medium 41 $0.80 3.7 min
Hydrogen low 27 $0.29 1.3 min
Claude Sonnet 5.5 max 55 $10.02 34.0 min
Claude Sonnet 5.5 xhigh 50 $4.09 14.6 min
Claude Sonnet 5.5 high 42 $1.64 6.8 min
GPT-6.1 Sol max 55 $1.27 n/a

Gemini 4 Hydrogen's Artificial Analysis index score by thinking level: low 27, medium 41, high 51, xhigh 54, with the cost per task at each level

What this table shows:

  • At the same level, Hydrogen is ahead. At high, Hydrogen scores 51 and Sonnet 5.5 scores 42. Their costs per task are close ($1.87 and $1.64).
  • At the top setting, Sonnet 5.5 is one point ahead. Sonnet 5.5 max scores 55, Hydrogen xhigh 54. But Sonnet gets there at $10.02 per task, roughly three times Hydrogen's $3.16.
  • Similar score, different bill. Hydrogen high (51) and Sonnet 5.5 xhigh (50) are almost level. Hydrogen does it at less than half the cost and in about half the time.
  • Lowering Hydrogen's default is costly in quality. Going from high to medium takes the score from 51 to 41, and low drops it to 27. If you lower the level to save money, you are effectively using a different model.
  • There is no Terra row. I could not find a row for GPT-6.1 Terra on Artificial Analysis. The OpenAI model in this table is GPT-6.1 Sol; the one in Google's table is Terra. They are different models, so do not substitute one's number for the other's.

For terminal work, the independent data points the same way as Google's table. As DataCamp reports, in Artificial Analysis's own Terminal-Bench 4.0 run Hydrogen scores about 54% (BenchLM lists 53.5%), Claude Sonnet 5.5 64% and GPT-6.1 Sol 56%. DataCamp's conclusion is that Hydrogen is "not the best model available for terminal-heavy work."

Where does Hydrogen sit inside the Gemini 4 family?

On the same Artificial Analysis page, at the high level, Gemini 4 Carbon scores 60, Hydrogen 51 and Gemini 4 Argon 50.

Model (high) Index Cost per index task Time per task
Gemini 4 Carbon 60 $5.48 29.0 min
Gemini 4 Hydrogen 51 $1.87 7.8 min
Gemini 4 Argon 50 $4.97 30.0 min

So Hydrogen is in the same score band as Argon, but clearly cheaper and faster per task. Carbon is nine points ahead and asks roughly three times the cost per task in return. (The per-task costs relayed by NeuralTrust in my Carbon comparison were higher; the ones here are the values on the index's current v5.0.1 page.)

Google's own tables show a similar picture, but the Hydrogen and Carbon tables are separate tables; read each one on its own. On Agents' Last Exam v1.1, Carbon scores 77.4%, Hydrogen 71.6% and Argon 69.0%. On OSWorld-2.1, Carbon 81.5%, Hydrogen 77.6%, Argon 74.7%. On Chartography, Hydrogen (66.6%) is behind both Carbon (72.5%) and Argon (71.6%). The one reverse example is SWE-sweep: Hydrogen 13.5%, Carbon 9.6%. Carbon's comparison with its own rivals is in a separate post.

Hallucination, speed and verbosity

Hallucination rate. According to the AA-Omniscience values BenchLM gives, citing Artificial Analysis, Hydrogen's hallucination rate is 8.5% and its accuracy 50.5% (DataCamp writes the rate as "8%"). This rate is how often the model gives a wrong answer instead of saying it does not know; lower is better. Next to the values in my earlier posts, Hydrogen falls between Argon (4%) and Carbon (11%). I have no value from this benchmark for Sonnet 5.5 or Terra. Google, for its part, says fabricated claims were cut by more than 5x compared with Gemini 3.8 Flash; that is Google's own measurement, and the comparison is with the older Flash model, not with the rivals.

Speed. The Artificial Analysis model page shows 182 output tokens per second at xhigh. Time to the first answer token is 42.9 seconds at xhigh and 16.3 seconds at high. The page describes the model as slower than average per task. Google's claim that it finishes tasks 2.5x faster than Gemini 3.8 Flash at the default high setting is its own measurement, and the method has not been published.

Verbosity. Hydrogen spent 304 million output tokens to complete the index at xhigh; the median is 77.3 million. Artificial Analysis calls this "very verbose." The price per token is low, but the number of tokens spent is high; that is the reason to look at the cost-per-task table.

Price

Model Input (1M tokens) Output (1M tokens) Source
Gemini 4 Hydrogen (standard) $1.50 $7.50 Google pricing page
Gemini 4 Hydrogen (Batch / Flex) $0.75 $3.75 Google pricing page
Claude Sonnet 5.5 $2 $10 Third-party list
GPT-6.1 Terra $1 $5 Third-party list
GPT-6.1 Sol $2 $10 Third-party list
Gemini 4 Argon and Carbon $2 ($4 from December 1) $10 ($20 from December 1) Google pricing page

I checked the Hydrogen prices on Google's pricing page on November 14, 2026. Hydrogen also has a rate-limited free API tier; Argon and Carbon do not. The Claude and GPT prices come from third-party lists and I have not verified them on the vendors' own pages; check the official page before you decide.

By list price Terra is the cheapest, then Hydrogen. But Hydrogen's thinking tokens are billed at the output price, and models spend different numbers of tokens on the same job. So the list price alone is not enough; the cost per task above is a more realistic measure.

Which one, and when?

This part is my interpretation: general conclusions I draw from the numbers above.

  • Starting for free, or a small budget: Hydrogen. It has a free API tier and a low cost per task at the default setting.
  • Agents that work in a terminal: Sonnet 5.5. Google's table and Artificial Analysis's run say the same thing.
  • Computer use (OSWorld): Sonnet 5.5.
  • End-to-end workflows and automation: Hydrogen leads in Google's table (Agents' Last Exam, AutomationBench-AA), but I have not seen independent confirmation for these rows.
  • Charts and multimodal questions: Hydrogen by a small margin (Chartography, MKA).
  • If you need the highest score: look one class up instead of at these three. There is one point between Sonnet 5.5 max and Hydrogen xhigh; Carbon is at 60.
  • GPT-6.1 Terra: the lowest list price, and close to Hydrogen on Agents' Last Exam. Because I found no independent index data, I cannot say more.

How to read benchmarks

  1. A vendor's table is the vendor's choice of benchmarks. Speed, verbosity and hallucination rate are not in Google's table; they are on third-party pages.
  2. Ask for the thinking level. "Hydrogen scored 51" only means something once you add "at high." Check whether the two models being compared are at the same level.
  3. The agent harness changes the result. The same model scores differently with a different set of tools. I explained this in the post on what a harness is.
  4. Measure your own task. Five of your own tasks teach you more than fifty of someone else's.

To see where the models one class up stand on the same kind of benchmarks, see the Claude Fable 5.2 comparison.

Frequently Asked Questions

Is Gemini 4 Hydrogen better than Claude Sonnet 5.5?

It depends on the kind of work. Hydrogen leads 7 of 10 rows in Google's table; Sonnet 5.5 leads terminal work and computer use. On the Artificial Analysis index, Hydrogen is ahead at the same thinking level, and Sonnet 5.5 is one point ahead at its highest setting.

Is Hydrogen better than GPT-6.1 Terra?

In Google's table Hydrogen leads eight of the nine rows that have a Terra value; Terra is above Hydrogen only on Terminal-Bench-Science. I could not find an independent Artificial Analysis row for Terra, so this comparison rests on a single source.

Which one is better for coding?

For agents that run commands in a terminal, the measurements point to Sonnet 5.5. On SWE-sweep, which measures repairing bugs without breaking existing code, Hydrogen leads, but the absolute values there are low. Do not decide before trying it on your own codebase.

Which one is cheaper?

By list price: Terra ($1 / $5, third-party list), then Hydrogen ($1.50 / $7.50), then Sonnet 5.5 ($2 / $10, third-party list). By Artificial Analysis's cost per task, Hydrogen high is $1.87 and Sonnet 5.5 high is $1.64; at the upper settings Sonnet is clearly more expensive.

Does Hydrogen hallucinate more?

On AA-Omniscience its hallucination rate is 8.5%: higher than Argon (4%), lower than Carbon (11%). I have no value from the same benchmark for Sonnet 5.5 or Terra.

Why does the thinking level matter so much?

Because the same model scores very differently at different levels: Hydrogen scores 27 at low, 51 at high and 54 at xhigh. Cost and time rise with the level too.

Sources

Comments