Detailed comparison of Gemini and Qwen on complex visual grounding tasks

At Agnify we keep returning to the same infrastructure decision: should a difficult visual reasoning step call a frontier model over an API, or should it run open weights on our own GPUs?

Agnify supports both deployment models. Choosing between them requires understanding where the quality of a frontier API justifies its price and where a self-hosted model can provide comparable results with greater control over data, latency and marginal cost. We therefore measured the quality gap on video tasks representative of our use cases.

Standard evaluations do not always capture this distinction. A recent study of widely used video-QA benchmarks, such as MVBench, found that many tasks could be solved from static information in a single frame. It also found that reversing videos on MVBench often failed to cause the performance drop expected when a model depends on temporal order.^[1]

Many of our customer problems require temporal information. We solve action-recognition tasks where understanding what happened depends on the sequence itself: no single frame, and sometimes no unordered set of frames, contains the answer. Two identical pallets can sit side by side, and one of them has just been labelled by an operator. Identifying the relevant pallet requires observing the earlier event.

In many cases, answering the question is only the first step. We also need to detect the object in question and track that same physical object through the video. Google DeepMind's Perception Test includes a grounded video question-answering task that measures this combination of temporal reasoning and spatial grounding.^[2]

The Gemini Flash family was our frontier baseline because it represented the current state of the art in visual reasoning. Google reports Gemini 3.7 Flash at 85.4% on LVBench and documents native agentic video understanding in both 3.7 and 3.8 Flash.^[9][10] We chose Qwen3.8-27B as the self-hosted point because it fit comfortably on a single 141 GB H200 in our setup and provided a strong quality-to-size trade-off.^[3][11] We then ran the three models through the same four-stage visual grounding pipeline on 300 questions.

The short version: Gemini 3.7 Flash scored higher than Qwen3.8-27B end to end. Self-hosted Qwen reached 84% of Gemini's HOTA score. Moving from Gemini 3.7 to 3.8 reduced the score by 0.4%, while improving the Gemini pipeline's identity handoff raised HOTA by 4.0 percentage points. In the Qwen pipeline, retrying incomplete inference calls raised HOTA by 5.3 percentage points.

A Temporal Visual Grounding Task

Perception Test is a diagnostic benchmark for multimodal video models, covering tasks such as video QA, object and point tracking, and action and sound localisation. We used its grounded video-QA task. The input is a video and a question, and the output is a sequence of bounding boxes highlighting the requested object or objects.

One question asks the system to track “the object that occludes the smaller object.” Solving it requires several different capabilities:

1. Understand what happened across the clip.
2. Decide which physical object the question refers to.
3. Locate that object in a useful moment.
4. Preserve its identity while tracking it through time.

The benchmark includes shell-game examples. A person places a small object on a table, covers it with one of several identical cups, and moves the cups around. The cups are visually indistinguishable in the final frame, so the target can be identified only from the preceding sequence.

This structure is representative of industrial use cases we address. A customer asking for “the forklift that entered through the north gate” expects us to reconstruct that forklift's path and distinguish it from the other forklifts currently visible.

The Pipeline

An effective pipeline for this task must handle the full range of questions in the dataset without relying on prompts so specific that they fail to generalise to out-of-distribution videos and questions. After several iterations, we settled on the following four-stage design:

• Plan: watch the video, name the answer objects and state how each can be recognised: by appearance, by its role in an event or only through its history.
• Keyframe: choose moments where the object is visible, including the event that establishes its identity.
• Point: return one box for each named object. Appearance-defined objects can use a still frame; event-defined objects receive a short clip spanning the deciding event.
• Carry: seed a SAM 3 video tracker and propagate that object identity through the clip.^[5]

The stages intentionally separate language reasoning from geometry. That let us measure whether a failure came from interpreting the question, choosing evidence, drawing the box or maintaining the track.

The analysis also identified a limitation in the original design: identity was passed between stages only as a noun. On a table of identical cups, “the cup” describes every candidate. As a result, the model could interpret the event correctly but select the wrong physical object when the next stage received only a still image.

Making the Comparison Fair

Both models received the same 300 questions from the Perception Test training split.^[*] They used the same video and frame sampling, the same stage prompts, the same propagation model and the same official scoring implementation. Gemini ran through Google's API. Qwen3.8-27B ran through vLLM on an NVIDIA H200.

^* We selected this 300-question subsample from the challenge training set, stratified to include all video and question categories in approximately the same proportions as the validation set, then checked it by measuring the baseline score on the subset. We used the same sample to calculate every metric in this report and run all ablation studies, limiting compute use and API cost.

Structured output required an equivalent implementation for both models. Gemini received a schema enforced by its API.^[6] In the first Qwen experiment, the schema was included only in the prompt, and the model sometimes reproduced it rather than filling it in. We aligned the comparison by using vLLM's constrained decoding through response_format.^[7]

Before reporting the results, we validated the scoring path by re-scoring the stored Gemini 3.7 tracks. This reproduced HOTA, DetA and AssA exactly.

HOTA, or Higher Order Tracking Accuracy, is the geometric mean of detection and association accuracy:^[8]

HOTA = sqrt(DetA × AssA)

DetA: did the system locate the object across the graded frames?
AssA: did the track preserve the same identity through time?

The association term measures a separate source of error: a system can produce accurate boxes on individual frames while assigning them to different, visually identical objects over time.

End-to-End Results

Gemini 3.7 Flash scored 0.5694 HOTA. Self-hosted Qwen3.8-27B scored 0.4759, reaching approximately 84% of Gemini's end-to-end score.

Qwen is not an accuracy-equivalent replacement for Gemini on this task. However, reaching 84% of Gemini's score makes it a practical self-hosted option when control, privacy, throughput or marginal cost are important deployment requirements.

The aggregate HOTA score does not tell us where that trade is being made. To understand it, we separated target selection from localisation.

Error Breakdown: Target Selection and Localisation

Gemini selected the correct object on 89.7% of the 300 questions, against 81.7% for Qwen.

The category breakdown shows where Qwen had the most difficulty. It identified every requested object correctly on 11 of 17 memory questions (64.7%), 10 of 13 abstraction questions (76.9%) and 224 of 270 physics questions (83.0%). The memory and abstraction subsets are small, so these rates should be treated as directional. A broader pattern appears in answer cardinality: Qwen scored 76.0% on the 121 multi-object questions and 85.5% on the 179 single-object questions. On the 86 cup-game questions, Qwen reached 81.4%, compared with 95.3% for Gemini; outside the cup-game subset, the corresponding rates were 81.8% and 87.4%.

Using “all requested objects correctly identified” as the question-level criterion, the models had the same outcome on 250 of 300 questions and differed on 50. Both were correct on 232 questions and both missed at least one requested object on 18. Among the 50 disagreements, Gemini alone was correct on 37, while Qwen alone was correct on 13.

We then isolated the 460 objects, across 258 questions, that both systems identified correctly. On that subset, Gemini's median box overlap was 0.937, while Qwen reached 0.887.

These results indicate that more of Qwen's performance gap comes from selecting the target object than from localising an object that it has already identified. This suggests that temporal interpretation and localisation can be evaluated as separate components of the architecture.

Pipeline Ablation and Model Version Comparison

We next separated the effect of the pipeline change from the effect of changing the Gemini model version.

The previous pipeline moved object identity from stage to stage as a textual name and pointed from one still frame. With Gemini 3.7 Flash it scored 0.5295 HOTA.

The identity-preserving pipeline carried the deciding event forward and showed the pointing stage a clip spanning that event. With the same Gemini 3.7 model, HOTA rose to 0.5694. The paired improvement was +4.0 percentage points, with p = 0.017.

The Qwen pipeline showed a separate improvement in inference reliability. Retrying calls that ended without a completed response raised HOTA from 0.4233 to 0.4759, a gain of +5.3 percentage points. This change addressed incomplete inference calls rather than the object-identity handoff, so it is reported separately from the Gemini ablation.

Changing only the API model from Gemini 3.7 to Gemini 3.8 then produced 0.5654 HOTA, a paired change of −0.4 percentage points with p = 0.82. On this experiment, the two model versions were statistically indistinguishable.

In this experiment, changing the inter-stage representation of object identity had a larger effect on HOTA than changing from Gemini 3.7 to Gemini 3.8.

Preserving Identity Between Pipeline Stages

The matched example below illustrates the difference. Both versions interpreted the question as referring to a cup involved in an occlusion event, and both returned a tight box. In the older pipeline, the pointing stage received the object name without the event history needed to distinguish between the identical cups, and selected the wrong one.

The revised handoff contains three pieces of information:

• the name of the target;
• the event that distinguishes it;
• a clip in which that event and the target frame are both visible.

Together, these fields specify a physical object rather than only its category. In process-control tasks, a failure may leave the affected object visually indistinguishable from the others. Identifying it therefore requires locating the failure event, such as an assembly mistake, in time and carrying that event context into the visual grounding step.

Gemini 3.8: Lower Latency, Similar Accuracy

Gemini 3.8 did not improve HOTA in this pipeline, but it changed the operational profile substantially.

300-question run	Gemini 3.7	Gemini 3.8	ratio
input tokens	4,972,627	5,450,210	1.10×
output tokens	1,483,083	3,362,672	2.27×
API calls	1,949	1,945	1.00×
wall time	44,814 s	22,936 s	0.51×
API cost	$9.29	$16.70	1.80×

The 3.8 run completed in roughly half the wall time, with median time per question falling from 75 to 50 seconds. It also generated 2.27 times as many output tokens, pushing measured API cost from $9.29 to $16.70.

For this workload, Gemini 3.8 reduced latency without improving accuracy. Its suitability therefore depends on the relative importance of response time and inference cost in production.

We do not present a dollar-for-dollar Gemini versus Qwen comparison. The Qwen run used owned GPU time rather than API credits, and a useful cost comparison would need equivalent utilisation, amortisation and concurrency assumptions.

Production Implications

The preferred deployment depends on which constraint is most important for the use case.

Gemini remained stronger end to end, particularly where the question referred to an object through an event. Qwen was closer once the target had already been identified. The pipeline itself gained more from preserving temporal identity than from moving to the newer Gemini version.

That suggests a modular design:

• Use the strongest temporal reasoning where choosing the correct physical object is the bottleneck.
• Keep localisation and propagation on controlled infrastructure where the target is already well specified.
• Preserve event context between stages. When possible, forward a short clip spanning the deciding event together with its timestamps or keyframes, rather than reducing the target to a name or a single final frame.
• Add targeted retries for clearly defined nondeterministic inference failures, such as truncated or incomplete responses. This simple policy improved Qwen's end-to-end score in our experiment, but the retry conditions should be fixed in advance so valid model decisions are not rerolled.
• Enforce response schemas during generation, especially when reasoning outputs are long. In our initial Qwen setup, longer reasoning was more likely to drift into describing the requested schema instead of returning a valid instance. Constrained decoding removed this failure mode.
• Treat prompts, schemas, evidence selection, context windows and stage handoffs as versioned parts of the model system.
• Evaluate the full pipeline whenever a model or interface changes.

The diagram is an interpretation of the measurements, not a separately benchmarked hybrid configuration. A follow-up experiment should measure this split directly by using frontier-model inference only for the stages where it provides a measurable benefit.

Limitations

All model-comparison and ablation results in this post come from the same 300-question sample of the Perception Test training split, stratified to match the validation split's categories. We used this fixed sample throughout the analysis to keep every comparison consistent and control compute and API costs.

The public ECCV challenge score is a held-out test result from an earlier pipeline, so it should not be compared numerically with the training-sample figures above. It is included only to provide external context from a contemporary public evaluation.

The experiment measures one pipeline on one particularly temporal grounding task. A different task distribution, inference server, precision or prompting strategy could change the ranking and economics.

FAQ

How did Gemini compare with self-hosted Qwen3.8-27B?

Gemini 3.7 Flash scored 0.5694 HOTA against 0.4759 for self-hosted Qwen3.8-27B. Qwen reached about 84% of Gemini's score, with the larger gap appearing in selecting the correct object rather than localising an object both models identified correctly.

Was Gemini 3.8 more accurate than Gemini 3.7?

Not in this experiment. Gemini 3.8 scored 0.5654 against 0.5694 for Gemini 3.7, and the paired difference was not statistically significant. Gemini 3.8 completed the workload faster but cost 1.8 times as much at the measured API rates.

Why use HOTA instead of box IoU?

Box IoU measures localisation on one frame. HOTA also measures detection across time and whether a track preserves the same object identity. That makes it better suited to questions where several objects look alike.

Are these results comparable with the ECCV leaderboard?

No. The article's model comparisons use a 300-question sample of the training split. The challenge leaderboard uses the held-out test split. Our public test submission came from an earlier pipeline and is included only as external context.

References and Attributions

1. Cores, D., Dorkenwald, M., Mucientes, M., Snoek, C. G. M. and Asano, Y. M. “TVBench: Redesigning Video-Language Evaluation.” arXiv:2410.07752, 2024. Paper · Official repository
2. Pătrăucean, V. et al. “Perception Test: A Diagnostic Benchmark for Multimodal Video Models.” arXiv:2305.13786, 2023. Paper · Official benchmark repository
3. Qwen. “Qwen3.8-27B.” Official model card. Model card
4. Google DeepMind. “Perception Test Challenge 2026: Grounded VideoQA.” EvalAI, 2026. Challenge overview and leaderboard
5. Carion, N. et al. “SAM 3: Segment Anything with Concepts.” Meta AI, 2025. Publication page
6. Google AI for Developers. “Structured outputs.” Gemini API documentation. Documentation
7. Kwon, W. et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th ACM Symposium on Operating Systems Principles, 2023. Paper · vLLM repository
8. Luiten, J. et al. “HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking.” International Journal of Computer Vision 129, 548–578, 2021. Paper
9. Google DeepMind. “Gemini 3.7 Flash: Model Card.” August 2026. Model card and benchmark results
10. Google AI for Developers. “Video understanding.” Gemini API documentation. Documentation
11. NVIDIA. “NVIDIA H200 Tensor Core GPU.” Product specifications

All result charts, pipeline diagrams and qualitative comparison graphics in this article were created by Agnify from its experiment outputs. Benchmark task definitions and examples are derived from Perception Test and are attributed to its authors in reference 2.