Gemini vs self-hosted Qwen: the pipeline mattered more

We keep running into the same build decision at Agnify: should a visual reasoning step call a frontier model over an API, or run open weights on our own GPUs?

The usual benchmarks did not answer it for us. Our difficult cases are temporal. Two identical pallets can sit side by side, and the one that matters is the one a specific machine loaded. A single frame cannot tell you that.

So we built one shared pipeline, ran three model configurations through it on 300 questions, and scored the finished object tracks with the benchmark's official metric.

TL;DR: Gemini 3.7 Flash scored 0.5694 HOTA. Self-hosted Qwen3-VL 30B reached 0.4770, or 84% of the Gemini score, after retries. Gemini 3.8 did not produce a statistically distinguishable improvement over 3.7, despite costing 1.8 times as much. A pipeline change was worth +0.0400 HOTA. The model upgrade was worth -0.0041 as run.

The Task: Answer a Question with a Track

We used the grounded video question answering track of Google DeepMind's Perception Test. The input is a video and a question in plain language. The output is not a sentence. It is one or more object tracks: a bounding box on every graded frame, following the object that answers the question.

One question might ask for “the object that occludes the smaller object.” To answer it, a system must:

1. Understand what happened across the clip.
2. Decide which physical object the question refers to.
3. Locate that object in a useful frame.
4. Keep its identity consistent while tracking it through time.

That is close to the work our production systems do. A customer asking for “the forklift that entered through the north gate” needs that forklift's path, not a paragraph about forklifts.

The benchmark includes deliberate shell games. A person hides an object under one of several identical cups, then moves the cups around. At any instant the cups look interchangeable. Only their history separates the right cup from the wrong ones.

In the 586 training clips, 116 have the benchmark's occlusion-game flag. Those clips contain a median of 14 annotated look-alike objects, with a range from 8 to 32. We sampled the same 300 training questions for every result in this post.

How We Scored It

The benchmark uses Higher Order Tracking Accuracy, or HOTA:

HOTA = sqrt(DetA * AssA)

DetA: did the system find the object in each frame?
AssA: did it keep calling that track the same object?

HOTA is strict in the right way for this problem. A system can draw a perfect box on every frame and still score badly if it switches identities. That makes the metric useful for scenes full of visually similar objects.

We used the official Perception Test scoring code. Before comparing any new arm, we ran the scorer against the existing Gemini 3.7 tracks and reproduced HOTA, DetA and AssA exactly. Questions with no answer score zero. A model does not improve its average by declining hard cases.

Every number below is from the training split. The pipeline and prompts were tuned there, and the challenge rules forbid tuning on validation or test. We do not mix these measurements with the earlier 0.3500 test score from a different version of the pipeline.

One Pipeline, Four Stages

We did not fine-tune Gemini or Qwen. Both models ran the same four-stage design. Only the model transport changed: Google's API for Gemini, and vLLM on an NVIDIA H200 for Qwen3-VL 30B.

• Plan: watch the video, name the answer objects, and say whether each is recognised by appearance, by its role in an event, or only by remembering where it went.
• Keyframe: choose up to five moments where those objects can be identified. For event-defined objects, also return the moment that settles their identity.
• Point: return one mask and box for each object. Appearance-defined objects use a frame. Event-defined objects use a short clip that includes both the event and the target frame.
• Carry: seed SAM3 with those boxes and propagate each identity through the video.

The example above asks for the cup that covers a green bottle cap. The model first describes that role, then locates the covering event at 18.04 seconds. The pointing call sees a clip spanning the event and the frame it must mark. Its red box lands on the same cup as the green annotation.

An earlier version passed identity from stage to stage as a text label. That failed on this exact question. “The cup” describes every cup on the table. Passing the event description and the clip itself into the pointing stage fixed the ambiguity.

End-to-End Results

Here is the full comparison over the same 300 questions. HOTA includes unanswered questions as zero.

model and policy	answered	HOTA	DetA	AssA	measured API cost
Gemini 3.8 Flash + retry	299/300	0.5774	0.5657	0.5947	$17.57
Gemini 3.7 Flash	296/300	0.5694	0.5605	0.5827	$9.29
Gemini 3.8 Flash, as run	291/300	0.5654	0.5536	0.5826	$16.70
Gemini 3.7 Flash, previous pipeline	296/300	0.5295	0.5204	0.5434	not measurable
Qwen3-VL 30B + retry	299/300	0.4770	0.4593	0.5050	self-hosted
Qwen3-VL 30B, no retry	257/300	0.4233	0.4079	0.4472	self-hosted

The self-hosted model reaches 84% of Gemini 3.7's HOTA once retries recover its missing answers. That is close enough to make Qwen useful, but not close enough to call the models equivalent. Paired question by question, the Qwen retry arm trails Gemini 3.7 by 0.0925 HOTA, with a 95% bootstrap interval of [-0.1229, -0.0621].

The qualitative split is useful. Across the first three stages, Gemini named the right object on 88.0% of questions against Qwen's 78.0%. On the 216 questions where both selected the same correct object, their boxes were much closer: median overlap was 0.939 for Gemini and 0.915 for Qwen.

Two points of box quality separate the models once they agree on the target. Ten points separate them on what the target is. For our systems, question interpretation is where a frontier API earns its keep. Localisation is a much stronger candidate for self-hosting.

The Pipeline Beat the Model Upgrade

The most important comparison is not Gemini against Qwen. It is Gemini 3.7 against itself.

Changing how identity travels between stages moved Gemini 3.7 from 0.5295 to 0.5694 HOTA, a gain of 0.0400 after rounding. Paired per question, the result is significant at p = 0.017.

Then we changed only the API model from Gemini 3.7 Flash to Gemini 3.8 Flash. The tracker, propagation stride, extracted frames and prompts stayed fixed. As run, HOTA moved from 0.5694 to 0.5654: a mean change of -0.0041, with a 95% bootstrap interval of [-0.0230, +0.0145]. The paired Wilcoxon test returned p = 0.82.

In other words, the two Gemini versions are indistinguishable here. The confidence interval rules out a gain above 0.026 HOTA, still smaller than the improvement from the pipeline change.

This is why benchmark work should preserve the whole harness. A new model name is easy to notice. A better contract between two old stages is easy to miss, even when it moves the metric more.

Gemini 3.8 Was Faster, but Cost 1.8 Times More

The 3.8 run did almost exactly the same amount of work. API calls, keyframes and tracker seeds stayed within 2% of the 3.7 run. Token use did not.

300-question run	Gemini 3.7	Gemini 3.8	ratio
input tokens	4,972,627	5,450,210	1.10×
output tokens	1,483,083	3,362,672	2.27×
API calls	1,949	1,945	1.00×
wall time	44,814 s	22,936 s	0.51×
API cost	$9.29	$16.70	1.80×

Gemini 3.8 generated more than twice as many output tokens, where thinking tokens are billed. It was also much faster: median time per question fell from 75 to 50 seconds, and p90 fell from 481 to 166 seconds.

That creates a real operational trade. Gemini 3.8 buys latency, not accuracy, on this pipeline. At the measured rates, cost per HOTA point rises from $16.32 for 3.7 to $29.53 for 3.8.

A Declined Answer Is a Lottery, Not a Limit

Qwen originally answered only 257 of 300 questions. In 37 of the 43 missing cases, it hit the output cap without closing its reasoning and our client treated that as a permanent failure.

The obvious fix seemed to be a larger token budget. The data rejected it. Every successful call we measured used fewer than 9,516 of the 22,000 tokens already available. One question looped until a 28,000-token cap, then answered in 2,173 tokens on the next identical attempt.

Changing one policy, from “empty at the length limit means failed” to “empty at the length limit means retry,” recovered 42 of 43 answers. Qwen moved from 0.4233 to 0.4770 HOTA, a gain of 0.0537. That is larger than the entire pipeline improvement and far larger than the Gemini model change.

Gemini 3.8 showed the same structure. It initially declined nine questions. Re-running each one three times recovered eight, lifting HOTA from 0.5654 to 0.5774 for another $0.87.

There is a metric trap here. Among answered questions only, Qwen's average fell from 0.4941 to 0.4786 after retrying because the recovered cases were difficult. Across all 300 questions, the score rose. A dashboard that filters out failures would have recommended reverting a successful change.

Three Engineering Lessons

Enforce Structured Output at Decode Time

Our first Qwen comparison was invalid. Gemini received a real response schema through its API. Qwen received the same schema pasted into the prompt.

In 19 of 35 inspected calls, Qwen copied the schema instead of filling it in. Switching vLLM to constrained decoding through response_format changed one line and moved right-object accuracy from 42.3% to 78.0%.

A schema described in prose is a request. A schema enforced during decoding is a constraint. Comparing those two setups measures the harness, not the models.

Give Temporal Identity to the Pointing Stage

On shell games, “the cup” is not an identity. We first asked the model to choose the right event, but then showed the pointing stage only a still image. The information needed to distinguish the cups had disappeared between calls.

Sending a short clip around the deciding event increased correct-instance results on paired, single-answer occlusion games from 27 to 40. Across the full sample, the complete identity-preserving change moved correct-instance results from 119 to 139.

Size Token Budgets for the Thought, Not the JSON

One Gemini keyframe call had a 1,024-token cap. It spent 1,010 tokens reasoning, returned 23 characters of visible output, failed to parse, and silently fell back to the middle frame. The same call answered normally at a 4,096-token cap.

That truncation affected 26% of occlusion-game questions in a pipeline we had been measuring for weeks. The fix was a larger cap and a raised exception. If a provider counts reasoning against the output limit, a budget sized for the final JSON is a budget sized for failure.

What We Would Deploy

There is no universal winner here.

Gemini 3.7 was the best measured price-to-accuracy choice. Gemini 3.8 cut wall time in half but did not improve accuracy. Self-hosted Qwen reached 84% of Gemini 3.7's end-to-end score, and its localisation quality was close once both models agreed on the object.

The architecture we would carry into production is therefore modular:

• Spend the frontier API where temporal language and object identity are hardest.
• Keep localisation and propagation on owned infrastructure where possible.
• Make incomplete reasoning explicitly retryable.
• Record unanswered cases as zero in every aggregate.
• Version prompts, schemas, frame sampling and tracker settings alongside the model name.

The broader lesson is simple. On a difficult visual reasoning task, the newest model was not our biggest lever. Preserving identity between stages and handling stochastic failure correctly moved the result much more.

FAQ

Did Gemini beat self-hosted Qwen3-VL?

Yes. Gemini 3.7 Flash scored 0.5694 HOTA against 0.4770 for self-hosted Qwen3-VL 30B with retries. Qwen reached about 84% of the Gemini score and drew nearly comparable boxes when both models chose the same correct object, but it interpreted the questions less accurately.

Was Gemini 3.8 better than Gemini 3.7?

Not on accuracy in this experiment. Gemini 3.8 scored 0.5654 as run against 0.5694 for Gemini 3.7, with no statistically significant difference. It completed the workload in roughly half the wall time but cost 1.8 times as much because it generated 2.27 times as many output tokens.

Why use HOTA instead of box IoU?

Box IoU measures localisation on one frame. HOTA also measures detection coverage and identity consistency over time. That matters when several objects look alike and the correct answer is defined by what happened earlier in the video.

Are these held-out benchmark results?

No. Every result in this post uses the same 300-question sample from the Perception Test training split because the prompts and pipeline were tuned there. We keep these numbers separate from validation and test results from earlier pipeline versions.