Gemini 4 Argon loses its points in the shell game

Google's Gemini 4 Argon is the first Gemini model in our grounded video QA series that is not a Flash model. We ran it through the same pipeline and the same 300 questions as GPT-6 Astra and Claude Opus 5.5, and compared it question by question with the two OpenAI models that lead the benchmark.

The headline number is unremarkable: Argon scores 0.5933 HOTA, level with Opus and behind both GPT-6 models. The breakdown is more useful. Argon does not lose a little everywhere. On most questions it matches GPT-6.1 Sol, and on some question types it is ahead. It loses its points in one place, the shell game, and for two reasons that we can show frame by frame.

The short version: Gemini 4 Argon scores 0.593 HOTA, against 0.616 for GPT-6.1 Sol and 0.622 for GPT-6 Astra, for $30.72 per 300-question run. On 246 of the 300 questions it is slightly ahead of Sol on average. The whole gap comes from 54 questions, mostly shell games, where Argon tracks the cup instead of the hidden object, or re-anchors the tracker on an identical cup.

The setup

The pipeline, the 300-question sample from the Perception Test training split, and the official HOTA scorer are unchanged from the first post. Stage 3 refers to objects by number, as introduced in the Astra post. Argon had no earlier run to reuse, so it ran all three model stages on that pipeline, and SAM 3 carried its boxes on an RTX PRO 6000 Blackwell, as for the other v3 runs.

Two details are new with Argon, and both matter if you call it yourself:

• Argon takes a thinking level. The Flash models could not turn thinking down. Argon accepts low, medium and high (it rejects minimal). With no level set, it thinks about as much as at medium. On a short logic puzzle it used 150 to 210 thought tokens by default, none at low, and about 1,000 at high. Opus and the GPT-6 models ran at high, so Argon did too.
• At high, thinking can use the whole output budget. On 5 of 300 stage-1 calls, Argon spent all of its 8,192 output tokens on thinking and returned no answer. Those calls produced nothing, so we ran them again with a 32,768-token cap, and all five answered. If you set a cap sized for the answer, leave room for the thought.

Results

All models on the same pipeline, 300 questions. Cost is the API bill for one full run. *Sol reused its stages 1 and 2 from an earlier run, so its cost is an estimate, as on our VLM tracker. Argon's price is an introductory rate of $2 / $10 per million input / output tokens. It doubles on 1 December 2026, which makes the run $61.44.

Against Gemini 3.7 Flash, Argon is +0.023 HOTA, 95% CI [−0.001, +0.047], p = 0.07: probably better, but not clearly so. Against Opus it is a tie (−0.001, CI [−0.020, +0.018]) at 37% of Opus's cost.

Against GPT-6.1 Sol, which costs about the same per token, Argon is −0.023, CI [−0.044, −0.003]. The interval excludes zero, but the signed-rank test does not separate the two (p = 0.25). Argon wins 124 questions and loses 139. A mean gap without a consistent per-question gap means that a few large losses carry it, and that is what we found.

Where the gap comes from

We sorted every question by the reason Argon's score differs from Sol's:

The three causes add up to more than the whole gap. Averaged across the remaining 246 questions, Argon is slightly ahead of Sol. The same breakdown against Astra gives +0.004 on the remaining questions.

By question family, only shell games cost Argon a clear amount (−0.049, CI [−0.079, −0.021], 104 questions). The Astra post showed that shell games are hard for every model. Argon finds them harder in two specific ways.

Failure 1: it tracks the cup, not what the cup hides

A shell game question asks for one of two things: the object that occludes, or the object that is occluded. On 13 of the 44 questions that ask for the occluded object, Argon answered with the occluder: "the plastic cup that covers the coin". Sol and Astra did this on none of them.

This is a reading error, not a vision error. Argon's description mentions the right coin. It just picks the wrong noun in its own sentence as the answer. All 13 scored below 0.16 HOTA, with a mean of 0.06. Our right-object check counts 8 of them as right, because the cup's box contains the coin, so Argon's 92.3% right-object rate is slightly generous.

Failure 2: it re-anchors the tracker on an identical cup

Stage 2 picks several keyframes, stage 3 draws a box on each one, and every box re-anchors the same SAM 3 track. If one keyframe points at an identical cup, the tracker follows the wrong cup from there.

This was not one bad video. In the 104 shell games, 8.7% of Argon's keyframe boxes landed on a different object, against 0.8% for Sol and 0.6% for Astra. Argon's boxes for one object changed identity on 27 games, against 12 for Sol and 8 for Astra. On those 27, Argon scored 0.355. On the other 77, it scored 0.548, the same as Sol's 0.546 on its consistent games.

One habit may make this worse. Argon almost never relies on a single keyframe: it did so on 1 question in 300, against 18 for Sol and 14 for Astra. More keyframes give the tracker more chances to recover, but in a shell game each extra box is another chance to point at the twin. We did not test this directly.

Where Argon is ahead

Two question families favour Argon. On containers and pouring (19 questions) it is +0.034 against Sol, CI [+0.011, +0.063], with 7 wins and no losses. On questions about what changed while the camera looked away (23 questions) it is +0.055, CI [−0.002, +0.144]: a large gain, but on a small sample.

The second case shows the other side of the keyframe habit. Sol and Astra both asked for one frame, the frame at 9 seconds, and found no notebook there, so the question scored zero. Argon spread five keyframes over the second half of the clip and found it.

The box follows the name

Argon's object names are short: "the notebook", "the water bottle", "the paper towel roll". Its median name is 4 words, and 41% of its names are 3 words or fewer. Sol and Astra have a median of 6 words, and only 16% to 17% of their names are that short. Sometimes the extra words change the box. In the example below, both models chose the same five keyframes. Sol named "the paper towel roll on its holder" and boxed the holder as well. Argon named the roll and boxed the roll.

We expected this to show up as tighter boxes overall. It does not. On the 165 single-object questions where both models chose the right object, Argon's box overlap is 0.015 lower than Sol's on average. Its median overlap is slightly higher, 0.942 against 0.935. Argon draws some boxes much better and some worse; it does not draw better boxes in general.

Where the money goes

Argon reads the clip as video and the GPT-6 models read it as a sequence of images, so a full Argon run sent about half as many input tokens as a full Sol run (7.6 million against 14.6 million). At thinking level high, Argon wrote about twice as many output tokens (1.5 million against 0.75 million). At the same per-token price, the two effects nearly cancel: $16 against $19 per hour of video.

That makes Argon the cheapest model in the table that ties Opus, at 37% of its cost. It does not make Argon the best value overall. For about the same price, Sol is more accurate. For under a quarter of the price, Gemini 3.7 Flash scores only 0.023 less.

What this means in practice

If your questions are about which of several identical objects did something, use one of the GPT-6 models. Argon's shell game errors are the kind that a downstream count or alert cannot detect: the track is smooth, it is just on the wrong cup.

If your questions are about single, distinct objects, such as what was removed, what was poured, or what appeared, Argon is as good as Sol and costs a little less, until the introductory price ends. A hybrid worth testing is Sol for stages 1 and 2, Argon for stage 3. That could help with the initial object choice, but stage 3 draws the boxes that switch to an identical cup, so keeping Argon there could retain that failure. We have not run this hybrid or isolated the contribution of each stage.

Limitations

• Every figure comes from the same 300 training questions as the earlier posts. Family results with fewer than 30 questions are indicative only.
• Five Argon questions ran twice after a call returned no answer. The failed calls reported no token use, so about $0.41 of thinking is missing from the cost.
• Argon ran all three stages on the current pipeline. Sol and Astra reused stages 1 and 2 from their earlier runs and re-ran stage 3 only. Stage 3 samples each box again on every run, so part of every difference is run-to-run variation.

Agnify answers questions like these from the cameras already on a production floor. To see the pipeline on your own footage, read about Agnify for manufacturing or book a demo.

FAQ

How does Gemini 4 Argon compare with GPT-6.1 Sol on grounded video QA?

Argon scores 0.593 HOTA against 0.616 for Sol, a paired difference of −0.023 (95% CI [−0.044, −0.003]). Argon wins 124 questions and loses 139. The net score deficit is concentrated in 54 questions, mostly shell games, where it tracks the occluding cup instead of the hidden object, switches to an identical object between keyframes, or makes other wrong-object picks. Argon is slightly ahead on average across the remaining 246 questions.

Is Gemini 4 Argon better than Gemini 3.7 Flash?

Probably, but not clearly. Argon is +0.023 HOTA (95% CI [−0.001, +0.047], p = 0.07) and costs about four times as much: $30.72 per 300-question run against $7.08 at current prices.

What thinking level should I use with Gemini 4 Argon?

We used high, the same effort level as the Opus and GPT-6 runs. With no level set, Argon thinks about as much as at medium. At high, leave a large output cap: 5 of our 300 first-stage calls used all 8,192 output tokens on thinking and returned no answer, and all five answered with a 32,768-token cap.

How much does Gemini 4 Argon cost for video?

On this benchmark, $16 per hour of video at the introductory rate of $2 / $10 per million input / output tokens, against $19 for GPT-6.1 Sol and $105 for GPT-6 Astra. The rate doubles on 1 December 2026.