D-FINE-seg gets a semantic segmentation head

I recently added semantic segmentation to D-FINE-seg, extending the framework beyond object detection and instance segmentation. At Agnify, we have integrated D-FINE-seg into our platform, where its efficient training, export, and inference capabilities support production computer vision applications.

D-FINE-seg is a framework for 3 vision tasks: object detection, instance segmentation and now semantic segmentation too. It allows you to preprocess your data, train the model, export and quantize to various formats (like TensorRT, OpenVINO, etc.) and run efficient inference (maxing out the GPU usage). In this article I am sharing how I added the semantic segmentation task to a DETR-family model - and how the result stacks up on Cityscapes against the two frameworks most people reach for today.

Benchmarks first

I'll get to how the semantic head actually works further down. But let me start with the thing you probably care about most: does it hold up? I trained and benchmarked on Cityscapes, comparing D-FINE-seg in all three tasks against the two frameworks people reach for today - YOLO26 and RF-DETR. Short version: at real-time latency it matches or beats both, from a model with a fraction of the parameters. If you count end-to-end latency (image resize and normalization, forward pass, and postprocessing of boxes/masks), D-FINE-seg is faster across the board - and on accuracy it wins on detection and instance segmentation, and takes the top mIoU on semantic segmentation. The numbers are below. The full protocol and every asymmetry I'm aware of live in the benchmark repo.

A few ground rules first, because they're the whole point:

• This is a production benchmark. I report F1 at each model's own best confidence threshold, not mAP. The reason is simple: in production you pick one threshold and ship it - nobody serves a model at conf 0.05. F1 at the deployed threshold is the number that predicts what you'll actually get. To keep it fair, each model is scored at its own F1-optimal threshold (YOLO lands at 0.25, D-FINE-seg at 0.50, RF-DETR at 0.35), and accuracy and latency come from the same pass at that threshold. Semantic segmentation has no threshold - it's a dense argmax - so there I report mIoU.
• Every framework runs its own shipped inference code. YOLO goes through YOLO(engine).predict(), RF-DETR through its own TRTInference + PostProcess, D-FINE-seg through its own TensorRT wrapper.
• One scorer for everyone. All three are scored by D-FINE-seg's own validator against the same ground truth at original 2048x1024 resolution.
• Sizes are matched on latency, not parameters. YOLO and RF-DETR are never under-resourced - they carry 2-3x the params. I report two latency columns: end-to-end (what you feel, including each framework's CPU preprocessing) and pure engine time (the architecture-only number). They can disagree, and I show both.

Everything below is TensorRT 10.13 FP16, batch 1, on 500 Cityscapes val images at original resolution, on an idle RTX 5070 Ti.

Instance segmentation

I'll start here, because it's where the gap is widest and it's the hardest of the three tasks.

D-FINE-seg-S lands F1 0.661 - about 6 points over YOLO26-M and 4 over RF-DETR-seg-medium - with under half YOLO26-M's parameters and a third of RF-DETR's, at comparable engine latency.

Detection

Here D-FINE-seg-S is the fastest engine and the top F1 (0.703). I'd read this as "a touch more accurate and meaningfully faster".

Semantic segmentation

RF-DETR doesn't offer semantic segmentation yet, so this one is D-FINE vs YOLO26.

D-FINE-seg-M takes the top mIoU (0.753), and across the board the two frameworks trade blows within a point or two - D-FINE-seg is right in the mix. I won't oversell it: at the very smallest sizes YOLO26-M is a hair faster on the engine, so this is "competitive, and it leads on mIoU," not a clean sweep. The bigger point is that the task is there at all - same framework, same config, a third capability sitting on top of the detection and instance-seg machinery I already had.

On raw engine latency YOLO26-M edges D-FINE-seg-S. But end-to-end, both D-FINE-seg sizes are faster than both YOLO sizes - and D-FINE-seg-M is also the most accurate (a bigger YOLO26-L still lands below it on mIoU).

Keeping it honest

• These are numbers on a single dataset, single-seed runs. So numbers can fluctuate in other usecases. In my own use on other domains the ranking held.
• Getting RF-DETR onto TensorRT took real work. Two bugs, both still in rfdetr 1.8.3: their shipped engine runner imports the wrong module, so it always raises "TensorRT is not installed" on a machine where it is; and their own transform hands TensorRT a non-contiguous tensor, so the engine reads scrambled memory - scores collapse, F1 reads ~0.00, no error raised. The numbers above are RF-DETR after both fixes, verified against the checkpoint's own training-time validation metrics.

Now, how it actually works.

Architecture

Quick look at DETRs:

• Backbone to extract visual features (in this case it's a CNN, HGNetV2 specifically);
• Hybrid Encoder - applies self-attention to the coarsest feature map, then fuses the scales with convolutional FPN (top-down) + PAN (bottom-up);
• Decoder (with cross attention) to output each box. Hungarian matcher assigns each predicted box to ground truth box one to one during training, so model doesn't have to use NMS.

As you can see, DETR-like models are object-centric: sparse queries, matcher, NMS-free. Semantic segmentation on the other hand is a dense, per pixel classification. The good thing is that we have the encoder - it already outputs rich, fully fused multi-scale dense features. Detection head stacks queries on top, but for semantic segmentation we are going to attach a dense head (fully convolutional) straight onto those features and drop the existing decoder entirely. Matcher also is not needed.

What's even better, we already support the instance segmentation task, which means D-FINE-seg already has a lightweight MaskDecoder - a small module that fuses the encoder's features into a single quarter-resolution (stride 4) feature map. The new semantic segmentation head reuses this fuser and adds only a tiny neck (a couple of conv layers) and a 1x1 classifier that labels every pixel, then bilinearly upsamples the logits x4 back to the input resolution. During training we also attach an auxiliary head on the finest encoder feature for deep supervision - it helps learning and costs nothing during the inference.

One more nice trick is that our pretrained instance segmentation fuser weights are transferred directly to the new semantic segmentation head. So you can finetune from instance segmentation weights and only couple of conv layers will be initialized from scratch.

So, the forward path for the segmentation task is: image -> backbone -> hybrid encoder -> (fuser -> neck -> 1x1 classifier -> x4 upsample) -> per-pixel labels

Training and export

Losses. Three dense per-pixel terms, weighted 1 / 1 / 0.4:

• Cross-entropy - carries the bulk of the pixels.
• Multi-class soft Dice - keeps thin and rare classes from being washed out by the common ones.
• Auxiliary cross-entropy on the deep-supervision head - sharpens learning, free at inference.

Void/unlabeled pixels (ignore_index) are dropped from every term, and imbalanced datasets can add an optional per-class weight. No matcher, no query loss - that whole machinery is skipped here.

Training. You fine-tune straight from the instance-segmentation checkpoint - the mask-fuser weights transfer directly, and only the small neck, classifier and aux head start from scratch. From there:

• Shared recipe. 640x640 input, bf16 mixed precision, EMA weights, OneCycleLR with a short warmup, mosaic (0.5, down from detection's 0.8), flips and light photometric jitter.
• One rule for dense labels. Masks are only ever resized nearest-neighbor, and any padding an augmentation adds is filled with ignore_index - so you never invent a fake class at a border.
• mIoU at original resolution. Predictions are argmaxed at 640, nearest-upsampled back to the full 2048x1024, and scored against the untouched ground-truth mask. Scoring at network resolution would flatter the number.

Export. The whole postprocess folds into the graph, so every backend ships the same thing - a single sem_seg output, shape [B, H, W], int32:

• In the graph: quarter-res logits -> bilinear x4 upsample -> argmax -> int32 label map. No NMS, no detection postprocessor. (int32 because TensorRT dislikes int64, and an HxW label map is far cheaper on output bandwidth than a CxHxW logit tensor.)
• In the inference wrapper: one nearest resize of the label map back to the original image size. That's it.

That one graph serves ONNX, TensorRT, OpenVINO, CoreML and LiteRT identically - semantic segmentation is actually simpler to export than detection, whose postprocessor OpenVINO can't fully absorb. INT8 works, and the export runs a parity check on per-pixel argmax agreement against PyTorch.

One deliberate call: I argmax at network resolution and resize the labels, rather than upsampling the logits to full resolution first. The alternative gives smoother edges (+0.004 mIoU on Cityscapes) for ~20% more TensorRT latency - not a trade worth making in production.

Visualization

Finally, let's see couple of examples of D-FINE-seg-M inference on the Cityscapes dataset.