cs.CVNov 19, 2026

Backbone Matters: Benchmarking Encoder-Decoder Networks for Semantic Segmentation of Driving Scenes on BDD100K

Authors: Ehsan Rassekh

Organizations: Institute for Advanced Studies in Basic Sciences (IASBS), Zanjan, Iran

Abstract

Semantic segmentation gives an autonomous vehicle a dense, pixel-level description of its surroundings and is a core part of scene understanding. Most segmentation networks consist of an encoder (the backbone) and a decoder, yet the backbone is often chosen without a systematic comparison. This work studies this choice on the BDD100K driving dataset. Four widely used decoders (U-Net, FPN, LinkNet and PSPNet) are combined with 13 backbones from six families (ResNet, VGG, DenseNet, Inception, MobileNet and EfficientNet), and all 52 models are trained with one common protocol. The best model, an FPN decoder with an EfficientNet-B4 backbone, reaches 62.6% mean intersection over union (mIoU), 89.8% accuracy and a loss of 0.092. Only two of the 52 models, both FPN with EfficientNet, exceed the 56.9% mIoU reported with the BDD100K benchmark. The effect of the backbone depends strongly on the decoder: the same EfficientNet-B4 gives 42.6% mIoU with PSPNet but 62.6% with FPN, and the gap between the best and the worst backbone ranges from 12.0 points (U-Net) to 42.5 points (FPN). The results and the scope of this study are discussed.

Figures & tables

Explore similar work

Oct 30, 2025

AD-SAM: Adapting the Segment Anything Model for Semantic Segmentation in Autonomous Driving

This paper presents the Autonomous Driving Segment Anything Model (AD-SAM), a foundation-model adaptation framework for semantic segmentation in autonomous driving. AD-SAM combines a frozen Segment Anything Model (SAM) Vision Transformer (ViT-H) encoder with a trainable ResNet-50 encoder to integrate general visual representations with multi-scale, domain-specific spatial features. Features from the two encoders are integrated through deformable convolution and channel attention, followed by a multi-stage deformable decoder for semantic prediction. Training employs a hybrid objective combining Focal, Dice, Lovász-Softmax, and Surface losses. Experiments on Cityscapes and Berkeley DeepDrive 100K (BDD100K) show that AD-SAM outperforms SAM, Generalized SAM (G-SAM), and DeepLabV3 under a controlled training protocol. AD-SAM achieves 76.27% mIoU on Cityscapes and 64.74% on BDD100K, exceeding DeepLabV3 by 3.45 and 5.00 percentage points, respectively, with larger gains over the SAM-based baselines. Sample-size experiments reveal dataset-dependent behavior. In particular, AD-SAM performs strongly across training sizes on Cityscapes, while its advantage on the more heterogeneous BDD100K becomes pronounced with increased training data. When trained on Cityscapes and directly evaluated on BDD100K, AD-SAM achieves the highest cross-dataset retention (84.88%) among the evaluated models. AD-SAM also converges rapidly, while precomputed frozen SAM embeddings reduce training memory requirements. These findings demonstrate the potential of combining general foundation-model representations with domain-specific multi-scale features for accurate and robust autonomous-driving semantic segmentation.
Jun 22, 2026

Technical Report for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Pretraining-Diverse Ensemble of Foundation Vision Encoders for Robust Outdoor Scene Understanding

This report presents our solution for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge, which requires parsing unstructured outdoor scenes from four camera platforms into 56 fine-grained categories. Our approach pairs foundation vision encoders (including DINOv3, SigLIP2, and InternImage) with a Mask2Former decoder, and trains them with a strong recipe including long training schedules, exponential moving average, a larger crop size, and multi-scale plus flip test-time augmentation. The three encoders, chosen for their complementary pretraining objectives, are combined into a pretraining-diverse ensemble through per-class validation-IoU weighting. Evaluated on the official GOOSE test set, our submission achieves 75.40% composite mIoU and wins the second place of the challenge. Our study further shows that the encoder's pretraining recipe, rather than its parameter count or the decoder design, is the dominant factor for accuracy on this benchmark.
Sep 2, 2026

What Does the Encoder Actually Decide? A Controlled Comparison of Vision Backbones on Joint Tree Segmentation and Stereo Depth

A robot pruning trees needs two facts per pixel: whether it belongs to a tree, and its distance. Both are usually obtained via task heads attached to a vision backbone chosen by reputation rather than measurement. Holding dataset, decoders, losses, schedule, and evaluation fixed, we ask: how much does the encoder choice change joint semantic segmentation and stereo depth on thin vegetation? We build a hard parameter-sharing network with one encoder feeding both branches, swapping only the encoder without downstream retuning. We evaluate [N] encoders across [M] architecture families (CNNs, transformers, hybrids, MLP-mixers, state-space models) near a ~25M budget, trained from scratch. Depth is evaluated on tree pixels only; segmentation uses boundary F1 and background IoU to prevent "label-everything-tree" shortcuts. Three findings stand out. First, the strongest encoders are convolutional and hybrid, not transformers: [BestEncoder] leads with [MIoU] segmentation mIoU and [Delta] depth δ1δ_1, while [X] of [Y] plain vision transformers collapse when trained from scratch. Second, parameter count does not predict quality --- [SmallEncoder] at only [P]M parameters outranks models two orders of magnitude larger. Third, segmentation and depth rankings agree strongly (Spearman ρρ = [RhoValue]), showing no task conflict. Finally, [K] of [N] encoders collapse to degenerate all-tree segmentation --- exposed by boundary F1 but hidden by region IoU.