When a car climbs a wall, only SAM3 still sees a car

Tesla's recent Roadster demo has the car drive up a vertical wall, fly off it on thrusters, land in a cloud of smoke and do it again. It is a fun video. It is also a very good test for an object detector, because the object never changes - it is a car in every shot - but almost nothing about how the car looks matches the cars a detector was trained on.

So we ran four models on every frame of the 46-second clip and counted one thing: on how many frames did the model find a car?

TL;DR: On frames where the car is on the ground, the COCO specialists are fine - D-FINE-seg X finds the car on 95% of frames, at about 9 ms per frame. Once the car is on the wall or in the air, D-FINE X and D-FINE-seg X drop to 23% and 32%. SAM3, prompted with the single word "car", finds it on 97% of those frames. It is about 11x slower.

The setup

• The video: 1,094 frames, 3840x2160, 24 fps. Every frame went to every model - no sampling, no tracker, no smoothing between frames.
• The specialists: three closed-set detectors trained on COCO, where "car" is one of 80 classes. For the results we enable only the car class, so a model cannot spend its box on another class. D-FINE X is the original D-FINE COCO detection checkpoint. D-FINE-seg X is the instance segmentation checkpoint trained with D-FINE-seg. Both run through D-FINE-seg's TensorRT inference code, at confidence 0.5.
• RF-DETR L runs at confidence 0.35. That is RF-DETR's F1-optimal threshold from our Cityscapes benchmark.
• The generalist: SAM3, an open-vocabulary segmentation model, with the text prompt car and confidence and presence thresholds of 0.5.
• The metric: there is a car in (almost) every frame, so a frame is a hit if the model returned at least one car box. That is image-level recall: the share of frames on which the model found a car. We did not hand-label the boxes, so it says nothing about precision - more on that below.

All four models ran on our production inference box (one NVIDIA H200), with the same preprocessing our platform uses.

Results

One result we did not expect: the instance segmentation model finds more cars than the detection model - D-FINE-seg X is 7 points ahead of D-FINE X overall, 11 on the ground and 9 on the wall (only in smoke is D-FINE X ahead). Both are X-size models with the same inference code, but different checkpoints: the segmentation model is also trained to predict a mask for every object. A guess is that this extra task teaches it more about object shape, which helps when the car looks unusual. The training runs also differ in other ways, so we have not tested this.

The total hides the interesting part. Split the video by what the car is doing, and the story is clear:

On the ground, D-FINE-seg X is almost as good as SAM3 and 11x faster. This is the domain it was trained for: a car, wheels down, seen from the side or the front. Put the same car on a wall and the specialists lose 68-79% of the frames. SAM3 loses about one in thirty.

Where the specialists lose the car

Here is one frame from the top of the wall. The car is large, sharp and well lit. SAM3 finds it with a score of 0.95. None of the three specialists returns a car box at all:

The clip at the top of the post shows six seconds of the climb and the jump. For the first second the car is still on the ground, and all four models find it. Once it turns onto the wall, D-FINE X does not find it again for the rest of the clip, and D-FINE-seg X and RF-DETR L find it only now and then. SAM3 finds it on all but 3 of the 144 frames.

Why SAM3 holds up

SAM3 is not a better-tuned car detector. It does not have a fixed list of classes at all. You give it a short text prompt, and it finds and segments every instance of that concept in the image. It learned this from a very large and very diverse training set, so its idea of "car" is not limited to cars on roads seen from the street.

That is the whole difference. For the specialists, a car on a vertical wall at night with rocket exhaust is a large shift from their training data. For SAM3 it is a small one. The same property lets SAM3 handle the new car design itself - the shape of this car is not in COCO either.

The price is speed. At 95 ms per frame, SAM3 is 11-16x slower than the specialists. On one GPU that is the difference between about 10 frames per second and more than 100.

So which one should you use?

Both, for different jobs:

• Specialists for a domain you know and control - a fixed camera on a road, a production line, a parking lot. There they are accurate and very cheap to run.
• SAM3 when the scene can change in ways you cannot predict, or when you need a new class without training anything.
• SAM3 to train the specialist when you move into a new domain. The usual fix for the specialists on this video is not a new architecture - it is labeled frames of cars on walls. SAM3 can make those labels, and a D-FINE-seg model fine-tuned on them runs at 9 ms per frame again.

Keeping it honest

• This is image-level recall, not a benchmark. A frame counts as a hit if any box is labeled car. We did not check that the box is on the right car, and we did not count false positives, so there is no precision number. In the wide shots (the last 15 seconds) there is another car on the ground, so a model can score a frame from the easy car and miss the one on the wall. This makes the frame count kind to the specialists.
• Not every frame has a visible car. For about 2 seconds (frames 592-639) the car is fully inside a smoke cloud. No model can find it there, and SAM3 hits only 3 of those 48 frames. Without them, SAM3 is at 97.6%. On a few other frames all four models miss a small car at the top of the wall.
• RF-DETR L is under-confident on this video, not blind. Its best box on the car has a median score of 0.24 on the ground, against 0.65-0.72 for the D-FINE models, so a 0.35 threshold drops most of its correct boxes. The 0.35 came from a Cityscapes-finetuned RF-DETR, and a threshold does not transfer between checkpoints and footage. If we hold false positives equal instead (at most one stray car box per frame, scored against SAM3's boxes), RF-DETR L at 0.13 finds the car on 71% of frames, D-FINE X on 68% and D-FINE-seg X on 78%. On the wall all three stay at 58% or lower. RF-DETR's own torch predict() and our TensorRT engines give the same results, for all three specialists.
• How we enabled only car. The D-FINE-seg inference code runs class-agnostic NMS after its top-k step, so with all classes on, a box of another class can suppress a car box from the same query. We therefore ran the D-FINE X and D-FINE-seg X TensorRT engines with only the car class kept before NMS. RF-DETR has no NMS step, so keeping only its car boxes gives the same result as enabling only car.
• Latency is round-trip, not engine time. It is the median of 40 calls from the same machine, on a GPU shared with production traffic, and it includes JPEG decoding and the response. SAM3 gets a 1008x1008 input; the specialists get 640 or 704.
• One video, one run. The outputs are deterministic - a second run of the same frames gave the same boxes and scores - but 46 seconds of one stunt is not a dataset.

FAQ

What is domain shift in object detection?

Domain shift is the gap between the images a model was trained on and the images it sees in production. A COCO detector learned cars mostly from street photos. A car driving up a vertical wall at night is far outside that, so the detector's recall drops, even though the object is still a car.

Is SAM3 better than D-FINE or RF-DETR for car detection?

Not in general. On normal driving footage in this test, D-FINE-seg X found the car on 95% of frames against SAM3's 100%, at about 11x lower latency. SAM3's advantage is robustness: on the wall and in the air it stayed at 97% while the COCO detectors fell to 21-32%.

How fast is SAM3 compared to D-FINE and RF-DETR?

On an NVIDIA H200, the median round trip for one frame was 94.6 ms for SAM3, 8.6 ms for D-FINE-seg X, 8.4 ms for D-FINE X and 5.8 ms for RF-DETR L.

Can SAM3 detect objects it was not trained on?

SAM3 is prompted with text, so it has no fixed class list. It found the car in this video zero-shot, with no fine-tuning and only the prompt "car", including a new car design in poses that are rare in any training data.