Semantic Segmentation

Follow this topic Ask about this topic

Semantic segmentation predicts a category for each pixel, supporting applications such as autonomous driving, medical imaging, and remote sensing. Common approaches include CNN and vision-transformer encoders with multi-scale decoders, while foundation models, vision-language prompts, and diffusion features are increasingly adapted for dense prediction. To reduce annotation demands, researchers use weak or sparse labels, pseudo-labeling, synthetic data, few-shot learning, and domain adaptation; RGB-D, thermal, radar, and other sensors provide complementary cues. Key challenges are reliable transfer across domains and conditions, accurate boundaries and rare classes, and balancing accuracy with calibration and inference cost.

Latest papers

416 papers33 papers in the last four weeks, down 20% on the four weeks before. 0.2% of all new papers.

Feb 25, 2026

Tokenizing Semantic Segmentation with Run Length Encoding

This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks, and adapt the Pix2Seq framework to learn autoregressive models to output these tokens. We propose novel tokenization strategies to compress the lengths of the token sequences to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our models on two domain-specific datasets to demonstrate their competitiveness with the state of the art in certain scenarios, in spite of being severely bottlenecked by our limited computational resources. We supplement these analyses by proposing several promising approaches to foster future competitiveness in general-purpose applications, and facilitate this by making our code and models publicly available.

Abhineet Singh, Justin Rozeboom, Nilanjan Ray

Department of Computing Science, University of Alberta, Edmonton, Canada

Semantic Segmentation Image Tokenization Visual Token Compression

Feb 24, 2026

Region of Interest Segmentation and Morphological Analysis for Membranes in Cryo-Electron Tomography

Cryo-electron tomography (cryo-ET) enables high resolution, three-dimensional reconstruction of biological structures, including membranes and membrane proteins. Identification of regions of interest (ROIs) is central to scientific imaging, as it enables isolation and quantitative analysis of specific structural features within complex datasets. In practice, however, ROIs are typically derived indirectly through full structure segmentation followed by post hoc analysis. This limitation is especially apparent for continuous and geometrically complex structures such as membranes, which are segmented as single entities. Here, we developed TomoROIS-SurfORA, a two step framework for direct, shape-agnostic ROI segmentation and morphological surface analysis. TomoROIS performs deep learning-based ROI segmentation and can be trained from scratch using small annotated datasets, enabling practical application across diverse imaging data. SurfORA processes segmented structures as point clouds and surface meshes to extract quantitative morphological features, including inter-membrane distances, curvature, and surface roughness. It supports both closed and open surfaces, with specific considerations for open surfaces, which are common in cryo-ET due to the missing wedge effect. We demonstrate both tools using in vitro reconstituted membrane systems containing deformable vesicles with complex geometries, enabling automatic quantitative analysis of membrane contact sites and remodeling events such as invagination. While demonstrated here on cryo-ET membrane data, the combined approach is applicable to ROI detection and surface analysis in broader scientific imaging contexts.

Xingyi Cheng, Julien Maufront, Aurélie Di Cicco +3

Institut Curie, CNRS UMR168, Université PSL, Sorbonne Université, Paris, France · Inria Center at University of Rennes, SAIRPICO team, Rennes, France · Leiden Institute of Advanced Computer Science (LIACS), Universiteit Leiden, the Netherlands

Image Segmentation Electron Microscopy Cryo-Electron Microscopy Semantic Segmentation

Feb 24, 2026

DA-Cal: Towards Cross-Domain Calibration in Semantic Segmentation

While existing unsupervised domain adaptation (UDA) methods greatly enhance target domain performance in semantic segmentation, they often neglect network calibration quality, resulting in misalignment between prediction confidence and actual accuracy---a significant risk in safety-critical applications. Our key insight emerges from observing that performance degrades substantially when soft pseudo-labels replace hard pseudo-labels in cross-domain scenarios due to poor calibration, despite the theoretical equivalence of perfectly calibrated soft pseudo-labels to hard pseudo-labels. Based on this finding, we propose DA-Cal, a dedicated cross-domain calibration framework that transforms target domain calibration into soft pseudo-label optimization. DA-Cal introduces a Meta Temperature Network to generate pixel-level calibration parameters and employs bi-level optimization to establish the relationship between soft pseudo-labels and UDA supervision, while utilizing complementary domain-mixing strategies to prevent overfitting and reduce domain discrepancies. Experiments demonstrate that DA-Cal seamlessly integrates with existing self-training frameworks across multiple UDA segmentation benchmarks, significantly improving target domain calibration while delivering performance gains without inference overhead.

Wangkai Li, Rui Sun, Zhaoyang Li +2

Deep Space Exploration Laboratory, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei 230026, China

Model Calibration Semantic Segmentation Unsupervised Domain Adaptation Pseudo-Labeling

Feb 24, 2026

Onboard-Targeted Segmentation of Straylight in Space Camera Sensors

This study details an artificial intelligence (AI)-based methodology for the semantic segmentation of space camera faults. Specifically, we address the segmentation of straylight effects induced by solar presence around the camera's Field of View (FoV). Anomalous images are sourced from our published dataset. Our approach emphasizes generalization across diverse flare textures, leveraging pre-training on a public dataset (Flare7k++) including flares in various non-space contexts to mitigate the scarcity of realistic space-specific data. A DeepLabV3 model with MobileNetV3 backbone performs the segmentation task. The model design targets deployment in spacecraft resource-constrained hardware. Finally, based on a proposed interface between our model and the onboard navigation pipeline, we develop custom metrics to assess the model's performance in the system-level context.

Riccardo Gallon, Fabian Schiemenz, Alessandra Menicucci +1

Department of Space Systems Engineering, Faculty of Aerospace Engineering, Kluyverweg 1, HS Delft, 2629, Netherlands · Airbus Defence and Space GmbH, Claude-Dornier Straße, Immenstaad am Bodensee, 88090, Germany

Efficient Semantic Segmentation Image Segmentation Semantic Segmentation

Feb 24, 2026

Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing

A fine-grained understanding of egocentric human-environment interactions is crucial for developing next-generation embodied agents. One fundamental challenge in this area involves accurately parsing hands and active objects. While transformer-based architectures have demonstrated considerable potential for such tasks, several key limitations remain unaddressed: 1) existing query initialization mechanisms rely primarily on semantic cues or learnable parameters, demonstrating limited adaptability to changing active objects across varying input scenes; 2) previous transformer-based methods utilize pixel-level semantic features to iteratively refine queries during mask generation, which may introduce interaction-irrelevant content into the final embeddings; and 3) prevailing models are susceptible to "interaction illusion", producing physically inconsistent predictions. To address these issues, we propose an end-to-end Interaction-aware Transformer (InterFormer), which integrates three key components, i.e., a Dynamic Query Generator (DQG), a Dual-context Feature Selector (DFS), and the Conditional Co-occurrence (CoCo) loss. The DQG explicitly grounds query initialization in the spatial dynamics of hand-object contact, enabling targeted generation of interaction-aware queries for hands and various active objects. The DFS fuses coarse interactive cues with semantic features, thereby suppressing interaction-irrelevant noise and emphasizing the learning of interactive relationships. The CoCo loss incorporates hand-object relationship constraints to enhance physical consistency in prediction. Our model achieves state-of-the-art performance on both the EgoHOS and the challenging out-of-distribution mini-HOI4D datasets, demonstrating its effectiveness and strong generalization ability. Code and models are publicly available at https://github.com/yuggiehk/InterFormer.

Yuejiao Su, Yi Wang, Lei Yao +2

The Hong Kong Polytechnic University

Human-Object Interaction Egocentric Video Understanding Semantic Segmentation

Feb 23, 2026

Segmenting Low-Contrast XCTs of Concrete: An Unsupervised Approach

X-Ray Computed Tomography (XCT) is a compelling tool in experimental mechanics, capable of non-destructively extracting information pertaining to the internal morphology of materials. For materials with random heterogeneous morphology such as concrete, such information is of particular relevance since it allows for studies of morphology-related behaviour and for predictive modelling. Nevertheless, XCT images require semantic segmentation for practical usage. Here, concrete poses a unique challenge due to the similar X-ray attenuation coefficients of aggregates and mortar, which result in low contrast between the two phases in the ensuing XCT images. As such, purely intensity-dependent semantic segmentation tools remain unfeasible. While vision transformers (ViTs) and convolutional neural networks (CNNs) are proven techniques for semantic segmentation in such challenging cases, they typically require labelled training data, which is often unavailable for concrete or resource-intensive to obtain, thereby limiting their relevance. To address this challenge, a self-annotation technique is presented here that leverages superpixel algorithms to identify perceptually similar local regions in an image and relates them to the global context by utilizing the receptive field of a CNN-based model. This enables the model to learn a global-local relationship in the images and facilitates the identification of semantically similar structures. When evaluated against manually annotated ground truth on out-of-distribution data, the proposed methodology consistently outperformed direct greyscale thresholding across all pertinent metrics, demonstrating improved discernibility between aggregates and mortar, and providing the most favourable balance of sensitivity and precision for aggregate-phase identification.

Kaustav Das, Gaston Rauchs, Jan Sykora +1

Department of Mechanics, Faculty of Civil Engineering, Czech Technical University in Prague, Thákurova 7, 166 29 Prague 6, Czech Republic · Structural Composites Unit, Luxembourg Institute of Science and Technology (LIST), 5 avenue des Hauts-Fourneaux L-4362 Esch-sur-Alzette, Luxembourg

Image Segmentation Superpixel Segmentation Computed Tomography Semantic Segmentation Unsupervised Image Segmentation

Feb 21, 2026

Enhancing 3D LiDAR Segmentation by Shaping Dense and Accurate 2D Semantic Predictions

Semantic segmentation of 3D LiDAR point clouds is important in urban remote sensing for understanding real-world street environments. This task, by projecting LiDAR point clouds and 3D semantic labels as sparse maps, can be reformulated as a 2D problem. However, the intrinsic sparsity of the projected LiDAR and label maps can result in sparse and inaccurate intermediate 2D semantic predictions, which in return limits the final 3D accuracy. To address this issue, we enhance this task by shaping dense and accurate 2D predictions. Specifically, we develop a multi-modal segmentation model, MM2D3D. By leveraging camera images as auxiliary data, we introduce cross-modal guided filtering to overcome label map sparsity by constraining intermediate 2D semantic predictions with dense semantic relations derived from the camera images; and we introduce dynamic cross pseudo supervision to overcome LiDAR map sparsity by encouraging the 2D predictions to emulate the dense distribution of the semantic predictions from the camera images. Experiments show that our techniques enable our model to achieve intermediate 2D semantic predictions with dense distribution and higher accuracy, which effectively enhances the final 3D accuracy. Comparisons with previous methods demonstrate our superior performance in both 2D and 3D spaces.

Xiaoyu Dong, Tiankui Xian, Wanshui Gan +1

The University of Tokyo, Tokyo, Japan · RIKEN AIP, Tokyo, Japan

Multimodal Semantic Segmentation 3D Semantic Segmentation Semantic Segmentation Camera-LiDAR Fusion

Feb 20, 2026

Faster Training, Fewer Labels: Self-Supervised Pretraining for Fine-Grained BEV Segmentation

Dense Bird's Eye View (BEV) semantic maps are central to autonomous driving, yet current multi-camera methods depend on costly, inconsistently annotated BEV ground truth. We address this limitation with a two-phase training strategy for fine-grained road marking segmentation that removes full supervision during pretraining and halves the amount of training data during fine-tuning while still outperforming the comparable supervised baseline model. During the self-supervised pretraining, BEVFormer predictions are differentiably reprojected into the image plane and trained against multi-view semantic pseudo-labels generated by the widely used semantic segmentation model Mask2Former. A temporal loss encourages consistency across frames. The subsequent supervised fine-tuning phase requires only 50% of the dataset and significantly less training time. With our method, the fine-tuning benefits from rich priors learned during pretraining boosting the performance and BEV segmentation quality (up to +2.5pp mIoU over the fully supervised baseline) on nuScenes. It simultaneously halves the usage of annotation data and reduces total training time by up to two thirds. The results demonstrate that differentiable reprojection plus camera perspective pseudo labels yields transferable BEV features and a scalable path toward reduced-label autonomous perception.

Daniel Busch, Christian Bohn, Thomas Kurbiel +3

University of Wuppertal · APTIV

Bird's-Eye-View Segmentation Autonomous Driving Perception Semantic Segmentation Self-Supervised Pre-Training

Feb 16, 2026

SAILS: Segment Anything with Incrementally Learned Semantics for Task-Invariant and Training-Free Continual Learning

Continual learning remains constrained by the need for repeated retraining, high computational costs, and the persistent challenge of forgetting. These factors significantly limit the applicability of continuous learning in real-world settings, as iterative model updates require significant computational resources and inherently exacerbate forgetting. We present SAILS -- Segment Anything with Incrementally Learned Semantics, a training-free framework for Class-Incremental Semantic Segmentation (CISS) that sidesteps these challenges entirely. SAILS leverages foundational models to decouple CISS into two stages: Zero-shot region extraction using Segment Anything Model (SAM), followed by semantic association through prototypes in a fixed feature space. SAILS incorporates selective intra-class clustering, resulting in multiple prototypes per class to better model intra-class variability. Our results demonstrate that, despite requiring no incremental training, SAILS typically surpasses the performance of existing training-based approaches on standard CISS datasets, particularly in long and challenging task sequences where forgetting tends to be most severe. By avoiding parameter updates, SAILS completely eliminates forgetting and maintains consistent, task-invariant performance. Furthermore, SAILS exhibits positive backward transfer, where the introduction of new classes can enhance performance on previous classes.

Shishir Muralidhara, Didier Stricker, René Schuster

German Research Center for Artificial Intelligence (DFKI) · RPTU – University of Kaiserslautern-Landau, Kaiserslautern

Task-Free Continual Learning Continual Learning Class-Incremental Learning Semantic Segmentation

Feb 12, 2026

Modeling The Object Representations Underlying Human Physical Reasoning

Humans appear to represent objects when reasoning about physics with coarse, volumetric "bodies" that smooth concavities, trading fine visual detail for efficient physical predictions. Yet, the structure of these representations remains largely unknown. Segmentation models, in contrast, are trained for pixel-accurate masks that may misalign with such bodies. We ask whether and when these models nonetheless acquire human-like object representations. Using a time-to-collision (TTC) and change detection (CD) behavioral task with data from 178 and 50 human participants, respectively, we introduce a pipeline and an alignment metric to compare the visual representations of segmentation models to those of humans. We do this systematically on multiple architectures (DINOv2, SegFormer, DeepLabV3+, and UPerNet), varying their size and training time. We find that briefly trained models segment objects too coarsely, aligning poorly with humans, while fully trained models segment objects too finely. For each model, there is an intermediate training regime that best matches the coarse bodies observed in human behaviour, and larger models tend to reach it earlier. We show these bodies emerge under resource constraints in general-purpose vision models, providing computational support to resource-rational accounts of human cognition. This work provides a foundational framework for testing alignment between vision models and humans and shows there is a growing gap between the state-of-the-art in artificial intelligence and human cognition, driven by scaling model size and training.

Andrey Gizdov, Andrea Procopio, Lorenzo Caputi +4

Harvard University · Weizmann Institute of Science · Bocconi University

Physical Reasoning Computational Cognitive Modeling Object-Centric Representation Learning Representation Alignment Semantic Segmentation

Feb 9, 2026

GaussianCaR: Gaussian Splatting for Efficient Camera-Radar Fusion

Robust and accurate perception of dynamic objects and map elements is crucial for autonomous vehicles performing safe navigation in complex traffic scenarios. While vision-only methods have become the de facto standard due to their technical advances, they can benefit from effective and cost-efficient fusion with radar measurements. In this work, we advance fusion methods by repurposing Gaussian Splatting as an efficient universal view transformer that bridges the view disparity gap, mapping both image pixels and radar points into a common Bird's-Eye View (BEV) representation. Our main contribution is GaussianCaR, an end-to-end network for BEV segmentation that, unlike prior BEV fusion methods, leverages Gaussian Splatting to map raw sensor information into latent features for efficient camera-radar fusion. Our architecture combines multi-scale fusion with a transformer decoder to efficiently extract BEV features. Experimental results demonstrate that our approach achieves performance on par with, or even surpassing, the state of the art on BEV segmentation tasks (57.3%, 82.9%, and 50.1% IoU for vehicles, roads, and lane dividers) on the nuScenes dataset, while maintaining a 3.2x faster inference runtime. Code and project page are available online.

Santiago Montiel-Marín, Miguel Antunes-García, Fabio Sánchez-García +3

Department of Electronics. University of Alcalá, Spain. · Department of Cognitive Robotics. Delft University of Technology, The Netherlands.

Camera-Radar Fusion Autonomous Driving Perception Bird's-Eye-View Segmentation Gaussian Splatting Semantic Segmentation Feature Fusion

Feb 8, 2026

Multi-encoder ConvNeXt Network with Smooth Attentional Feature Fusion for Multispectral Semantic Segmentation

This work proposes MeCSAFNet, a multi-branch encoder-decoder architecture for land cover segmentation in multispectral imagery. The model separately processes visible and non-visible channels through dual ConvNeXt encoders, followed by individual decoders that reconstruct spatial information. A dedicated fusion decoder integrates intermediate features at multiple scales, combining fine spatial cues with high-level spectral representations. The feature fusion is further enhanced with CBAM attention, and the ASAU activation function contributes to stable and efficient optimization. The model is designed to process different spectral configurations, including a 4-channel (4c) input combining RGB and NIR bands, as well as a 6-channel (6c) input incorporating NDVI and NDWI indices. Experiments on the Five-Billion-Pixels (FBP) and Potsdam datasets demonstrate significant performance gains. On FBP, MeCSAFNet-base (6c) surpasses U-Net (4c) by +19.21%, U-Net (6c) by +14.72%, SegFormer (4c) by +19.62%, and SegFormer (6c) by +14.74% in mIoU. On Potsdam, MeCSAFNet-large (4c) improves over DeepLabV3+ (4c) by +6.48%, DeepLabV3+ (6c) by +5.85%, SegFormer (4c) by +9.11%, and SegFormer (6c) by +4.80% in mIoU. The model also achieves consistent gains over several recent state-of-the-art approaches. Moreover, compact variants of MeCSAFNet deliver notable performance with lower training time and reduced inference cost, supporting their deployment in resource-constrained environments. Model code is available at: https://github.com/Leo-Thomas/mecsafnet

Leo Thomas Ramos, Angel D. Sappa

Computer Vision Center, Universitat Autònoma de Barcelona, Barcelona, 08193, Spain · ESPOL Polytechnic University, Guayaquil, 090112, Ecuador

Semantic Segmentation Remote Sensing Image Segmentation Convolutional Neural Networks Land Cover Classification Multi-Scale Feature Fusion

Jan 2, 2026

Learning to Segment Liquids in Real-world Images

Liquids like water, wine and medicine are everywhere. However, limited attention has been given to the task of segmenting liquids, hindering the ability of robots to safely avoid and interact with them. The segmentation of liquids is difficult because liquids come in diverse appearances and shapes; moreover, they can be both transparent or reflective, taking on arbitrary objects and scenes from their background and surroundings. To take on this challenge, we construct a liquid dataset, LQDS, consisting of 5000 real-world images annotated into 14 distinct classes, and design a novel liquid detection model, LQDM, which leverages cross-attention between a dedicated boundary branch and the main segmentation branch to enhance mask predictions. Extensive experiments demonstrate the effectiveness of LQDM on the testing set of LQDS, outperforming state-of-the-art methods to establish a strong baseline for the semantic segmentation of liquids. We believe that LQDS and LQDM will facilitate future research in liquid segmentation and enable practical applications in robotics. Our dataset and code is released at https://lonaslee.github.io/LQDM/.

Jonas Li, Michelle Li, Luke Liu +2

Department of Computer Science and Engineering, University of North Texas, Denton, TX 76203, USA

Image Segmentation Boundary-Aware Segmentation Robotic Perception Semantic Segmentation

Nov 11, 2025

Exploring the Underwater World Segmentation without Extra Training

Accurate segmentation of marine organisms is vital for biodiversity monitoring and ecological assessment, yet existing datasets and models remain largely limited to terrestrial scenes. To bridge this gap, we introduce AquaOV255, the first large-scale and fine-grained underwater segmentation dataset containing 255 categories and over 20K images, covering diverse categories for open-vocabulary (OV) evaluation. Furthermore, we establish the first underwater OV segmentation benchmark, UOVSBench, by integrating AquaOV255 with five additional underwater datasets to enable comprehensive evaluation. Alongside, we present Earth2Ocean, a training-free OV segmentation framework that transfers terrestrial vision--language models (VLMs) to underwater domains without any additional underwater training. Earth2Ocean consists of two core components: a Geometric-guided Visual Mask Generator (GMG) that refines visual features via self-similarity geometric priors for local structure perception, and a Category-visual Semantic Alignment (CSA) module that enhances text embeddings through multimodal large language model reasoning and scene-aware template construction. Extensive experiments on the UOVSBench benchmark demonstrate that Earth2Ocean achieves significant performance improvement on average while maintaining efficient inference.

Bingyu Li, Tao Huo, Da Zhang +3

Institute of Artificial Intelligence (TeleAI), China Telecom, China · University of Science and Technology of China, China · Northwestern Polytechnical University, China

Image Segmentation Open-Vocabulary Semantic Segmentation Semantic Segmentation Transfer Learning

Nov 11, 2025

DWFF-Net: A Multi-Scale Farmland System Habitat Identification Method with Adaptive Dynamic Weight Feature Fusion

To address insufficient accuracy in multi-scale segmentation for agricultural habitat recognition, this study proposes a Dynamic Weighted Feature Fusion Network (DWFF-Net). Its encoder uses frozen DINOv3 to extract basic features and introduces a data-level adaptive dynamic weighting strategy based on relationships between image categories and feature maps. The decoder employs a dynamic weight calculation network for deep fusion of multi-level features and a hybrid loss for optimization. Statistical analysis shows that weight entropy tends to decrease as habitat category count increases, indicating adaptive adjustment of fusion strategy according to scene complexity. Experiments on a previously constructed agricultural habitat dataset validate DWFF-Net. Ablations yield mIoU 0.6979 and mF1 0.8049, exceeding the Static Weighted Feature Fusion Network by 1.82% and 1.54%, respectively, confirming that dynamic weighting improves multi-level feature utilization. Compared with U-Net, DeepLabv3+, SegFormer, and DPT, DWFF-Net improves mIoU by 16.32%, 6.49%, 4.17%, and 3.18%, respectively. For tiny features like scattered trees, IoU reaches 0.2707, outperforming those models by 99.85%, 11.45%, 22.24%, and 19.32%, verifying effectiveness in tiny habitat segmentation. This framework enables low-cost, high-precision habitat mapping and supports refined monitoring in agricultural landscapes.

Kesong Zheng, Zhi Song, Peizhou Li +4

National Engineering Research Center of Geographic Information System, China University of Geosciences, Wuhan, China · College of Science, Shenyang Agricultural University, Liaoning, China · College of Engineering, Shenyang Agricultural University, Liaoning, China +2

Agricultural Image Analysis Multi-Scale Feature Fusion Semantic Segmentation

Oct 30, 2025

AD-SAM: Adapting the Segment Anything Model for Semantic Segmentation in Autonomous Driving

This paper presents the Autonomous Driving Segment Anything Model (AD-SAM), a foundation-model adaptation framework for semantic segmentation in autonomous driving. AD-SAM combines a frozen Segment Anything Model (SAM) Vision Transformer (ViT-H) encoder with a trainable ResNet-50 encoder to integrate general visual representations with multi-scale, domain-specific spatial features. Features from the two encoders are integrated through deformable convolution and channel attention, followed by a multi-stage deformable decoder for semantic prediction. Training employs a hybrid objective combining Focal, Dice, Lovász-Softmax, and Surface losses. Experiments on Cityscapes and Berkeley DeepDrive 100K (BDD100K) show that AD-SAM outperforms SAM, Generalized SAM (G-SAM), and DeepLabV3 under a controlled training protocol. AD-SAM achieves 76.27% mIoU on Cityscapes and 64.74% on BDD100K, exceeding DeepLabV3 by 3.45 and 5.00 percentage points, respectively, with larger gains over the SAM-based baselines. Sample-size experiments reveal dataset-dependent behavior. In particular, AD-SAM performs strongly across training sizes on Cityscapes, while its advantage on the more heterogeneous BDD100K becomes pronounced with increased training data. When trained on Cityscapes and directly evaluated on BDD100K, AD-SAM achieves the highest cross-dataset retention (84.88%) among the evaluated models. AD-SAM also converges rapidly, while precomputed frozen SAM embeddings reduce training memory requirements. These findings demonstrate the potential of combining general foundation-model representations with domain-specific multi-scale features for accurate and robust autonomous-driving semantic segmentation.

Het Patel, Mario Camarena, Fatemeh Nazari +3

Department of Computer Science, The University of Texas Rio Grande Valley, Edinburg, TX 78539 USA · Department of Civil Engineering, The University of Texas Rio Grande Valley, Edinburg, TX 78539 USA · Department of Computer Science and Engineering, University of California Riverside, Riverside, CA 92521 USA

Vision Foundation Model Adaptation Autonomous Driving Perception Semantic Segmentation

Oct 24, 2025

AURASeg: Attention-Guided Upsampling with Residual-Assisted Boundary Refinement for Drivable-Area Segmentation

Free-space segmentation is essential for autonomous robots to identify drivable regions and navigate safely across indoor, outdoor, and road-scene environments. However, conventional encoder-decoder models often recover coarse region masks while losing the fine spatial information needed to localize drivable-area boundaries accurately. We propose Attention-Guided Upsampling with Residual-Assisted Boundary Refinement (AURASeg), a segmentation framework designed to preserve region-level accuracy while improving boundary quality. Built on a ResNet-18 encoder, AURASeg introduces an Attention Progressive Upsampling Decoder (APUD) that progressively combines semantic context with high-resolution spatial detail, together with a Residual Boundary Refinement Module (RBRM) that explicitly refines contour-sensitive features before final prediction. We evaluate AURASeg across indoor simulation, ground-robot imagery, and road-driving benchmarks. The results show that our proposed model remains competitive with established segmentation models on region-level metrics while providing particularly strong boundary localization, including in comparison with boundary-focused methods. Detailed ablations further demonstrate the role of the proposed decoding and refinement modules.

Narendhiran Vijayakumar

Autonomous Driving Perception Boundary-Aware Segmentation Semantic Segmentation

Sep 29, 2025

Improved Robustness from Biologically Inspired Sparse Contrast Representations

Deep neural networks surpass humans on many vision benchmarks, yet remain far less robust to distribution shifts such as illumination and weather changes. Existing approaches address this challenge by additional training data, extensive augmentation, architectural modifications, or test-time adaptation. In this work, we explore a complementary direction: inspired by the human retina, we propose a fixed, model-agnostic preprocessing module that extracts signals that are more stable with respect to variations of illumination. Our method combines color remapping with local contrast extraction, producing sparse representations that emphasize structural features. We study its impact on semantic segmentation by training on Cityscapes and evaluating generalization under adverse conditions on Dark Zurich and ACDC. Our results show that the biologically inspired preprocessing preserves in-distribution performance while consistently improving robustness in challenging lighting scenarios, such as nighttime, where annotated training data are scarce. Moreover, the segmentation accuracy remains stable even when the contrast-based representation is sparsified by up to 70%. These gains suggest that rethinking the input representation itself can improve robustness while also opening opportunities for lower-latency, transmission-aware imaging sensors when sparsity can be exploited close to acquisition.

Lorena Stracke, Lia Nimmermann, Shashank Agnihotri +3

Media Systems, University of Siegen, Germany · Data and Web Science Group, University of Mannheim, Germany · Analogue Circuits and Image Sensors, University of Siegen, Germany +1

Distribution Shift Robustness Image Segmentation Image Corruption Robustness Visual Representation Learning Neural Network Robustness Semantic Segmentation

Aug 27, 2025

JVLGS: Joint Vision-Language Gas Leak Segmentation

Gas leaks pose severe risks to human health and industrial safety. However, accurate and timely monitoring of gas leaks remains a major challenge. Existing vision-based methods using infrared (IR) imagery are limited by the inherently blurry and non-rigid nature of leak plumes, which reduces detection reliability and precision. To overcome these limitations, this paper proposes a Joint Vision-Language Gas leak Segmentation (JVLGS) framework that integrates the complementary strengths of visual and textual modalities to enhance gas leak segmentation. Recognizing that gas leaks are sporadic and many video frames contain no leakage, JVLGS incorporates an adaptive postprocessing module to effectively suppress false positives caused by noise and non-target objects-a common limitation of existing approaches. Extensive experiments across diverse industrial scenarios demonstrate that JVLGS significantly outperforms state-of-the-art gas leak segmentation methods. Furthermore, it achieves consistently strong performance under both supervised and few-shot learning settings, whereas competing methods typically perform well in only one setting or underperform in both.

Xinlong Zhao, Qixiang Pang, Shan Du

University of British Columbia - Okanagan · University of Central Missouri

Semantic Segmentation

Aug 22, 2025

Through the Looking Glass: A Dual Perspective on Weakly-Supervised Few-Shot Segmentation

Meta-learning aims to uniformly sample homogeneous support-query pairs, characterized by the same categories and similar attributes, and extract useful inductive biases through identical network architectures. However, this identical network design results in over-semantic homogenization. To address this, we propose a novel homologous but heterogeneous network. By treating support-query pairs as dual perspectives, we introduce heterogeneous visual aggregation (HA) modules to enhance complementarity while preserving semantic commonality. To further reduce semantic noise and amplify the uniqueness of heterogeneous semantics, we design a heterogeneous transfer (HT) module. Finally, we propose heterogeneous CLIP (HC) textual information to enhance the generalization capability of multimodal models. In the weakly-supervised few-shot semantic segmentation (WFSS) task, with only 1/24 of the parameters of existing state-of-the-art models, TLG achieves a 13.2% improvement on Pascal-5\textsuperscript{i} and a 9.7% improvement on COCO-20\textsuperscript{i}. To the best of our knowledge, TLG is also the first weakly supervised (image-level) model that outperforms fully supervised (pixel-level) models under the same backbone architectures. The code is available at https://github.com/jarch-ma/TLG.

Jiaqi Ma, Guo-Sen Xie, Fang Zhao +1

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China

Weakly Supervised Image Segmentation Weakly Supervised Semantic Segmentation Few-Shot Learning Few-Shot Semantic Segmentation Weakly Supervised Learning Semantic Segmentation

Aug 8, 2025

SynSeg: Feature Synergy for Multi-Category Contrastive Learning in End-to-End Open-Vocabulary Semantic Segmentation

Semantic segmentation in open-vocabulary scenarios presents significant challenges due to the wide range and granularity of semantic categories. Existing weakly-supervised methods often rely on category-specific supervision and ill-suited feature construction methods for contrastive learning, leading to semantic misalignment and poor performance. In this work, we introduce a novel weakly-supervised approach, SynSeg, to address the challenges. SynSeg performs Multi-Category Contrastive Learning (MCCL) as a stronger training signal which robustly injecting intra- and inter-category knowledge during training. We also propose a new feature reconstruction framework named Feature Synergy Structure (FSS). FSS reconstructs discriminative features for contrastive learning through prior fusion and semantic-activation-map enhancement, effectively avoiding the foreground bias introduced by the visual encoder. Furthermore, SynSeg is a lightweight end-to-end solution capable for real-time inference. In general, SynSeg effectively improves the abilities in semantic localization and discrimination under weak supervision in an efficient manner. Extensive experiments on benchmarks demonstrate that our method outperforms state-of-the-art (SOTA) performance, with mIoU score gains ranging from 0.6% up to 8.9% across all reported benchmarks.

Weichen Zhang, Kebin Liu, Fan Dang +3

Tsinghua University, 100084, Beijing, China · Beijing Jiaotong University, 100044, Beijing, China

Contrastive Learning Weakly Supervised Image Segmentation Weakly Supervised Semantic Segmentation Weakly Supervised Learning Open-Vocabulary Semantic Segmentation Semantic Segmentation

May 21, 2025

From Pixels to Images: A Structural Survey of Deep Learning Paradigms in Remote Sensing Image Semantic Segmentation

Remote sensing images (RSIs) capture both natural and human-induced changes on the Earth's surface. Semantic segmentation (SS) of RSIs enables the fine-grained interpretation of surface features, making it a critical task in RS analysis. With the increasing diversity and volume of RSIs collected by sensors on various platforms, traditional processing methods struggle to maintain efficiency and accuracy. In response, deep learning (DL) has emerged as a transformative approach, enabling substantial advances in remote sensing image semantic segmentation (RSISS). As researchers continue to explore end-to-end SS, DL-based RSISS has undergone a structural evolution from pixel-level and patch-based classification to tile-level and image-level segmentation. However, existing reviews often focus on individual components, such as supervision strategies or fusion stages, and lack a unified operational perspective aligned with segmentation granularity and the training/inference pipeline. This paper provides a comprehensive review by organizing DL-based RSISS into a pixel-patch-tile-image hierarchy, covering early pixel-based methods, prevailing patch-based and tile-based techniques, and emerging image-based approaches. Specifically, the survey analyzes four supervision strategies, eleven feature extraction strategies, and six information fusion strategies, revealing the field's progression from local to global feature extraction, from traditional DL architectures to foundation models, and from unimodal to multimodal segmentation. This review offers a holistic and structured understanding of DL-based RSISS, highlighting representative datasets, comparative insights, and open challenges related to data scale, model efficiency, domain robustness, and multimodal integration. Furthermore, to facilitate reproducible research, curated code collections are provided at: https://github.com/quanweiliu/RSISS.

Quanwei Liu, Tao Huang, Jiaqi Yang +1

College of Science and Engineering, James Cook University, Cairns QLD 4878, Australia · Center for AI and Data Science Innovation, James Cook University, Cairns QLD 4878, Australia · Department of Forest and Wildlife Ecology, University of Wisconsin-Madison, Madison, WI 53706 USA +1

Remote Sensing Image Segmentation Multimodal Semantic Segmentation Semantic Segmentation

May 6, 2025

Panoramic Out-of-Distribution Segmentation

Panoramic imaging enables capturing 360° images with an ultra-wide Field-of-View (FoV) for dense omnidirectional perception, which is critical to applications such as autonomous driving and augmented reality. However, current panoramic semantic segmentation methods fail to identify outliers, and pinhole out-of-distribution segmentation (pinhole-OoS) models perform unsatisfactorily in the panoramic domain due to pixel distortions and background clutter. To address these issues, we introduce a new task, Panoramic Out-of-Distribution Segmentation (PanOoS), with the aim of achieving comprehensive and safe scene understanding. Furthermore, we propose the first solution, POS, which adapts to the characteristics of panoramic images through text-guided prompt distribution learning. Specifically, POS integrates a disentanglement strategy designed to materialize the cross-domain generalization capability of CLIP. The proposed Prompt-based Restoration Attention (PRA) optimizes semantic decoding by prompt guidance and self-adaptive correction, while Bilevel Prompt Distribution Learning (BPDL) refines the manifold of per-pixel mask embeddings via semantic prototype supervision. Besides, to compensate for the scarcity of PanOoS datasets, we establish two benchmarks: DenseOoS, which features diverse outliers in complex environments, and QuadOoS, captured by a quadruped robot with a panoramic annular lens system. Extensive experiments demonstrate the effectiveness of POS, with AuPRC increasing and FPR95 decreasing by 34.25 and 21.42 percentage points, respectively, compared with RbA on DenseOoS, outperforming state-of-the-art pinhole-OoS methods. Moreover, POS achieves leading closed-set segmentation capabilities and advances the development of panoramic understanding. Code and datasets will be available at https://github.com/MengfeiD/PanOoS.

Mengfei Duan, Yuheng Zhang, Yihong Cao +5

The authors are with the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China

OOD Detection Semantic Segmentation Prompt Learning

May 4, 2025

SARTM: Segment Any RGB Thermal Model with Language aided Distillation

The recent Segment Anything Model (SAM) demonstrates strong instance segmentation performance across various downstream tasks. However, SAM is trained solely on RGB data, limiting its direct applicability to RGB-thermal (RGB-T) semantic segmentation. Given that RGB-T provides a robust solution for scene understanding in adverse weather and lighting conditions, such as low light and overexposure, we propose a novel framework, SARTM, which customizes the powerful SAM for RGB-T semantic segmentation. Our key idea is to unleash the potential of SAM while introduce semantic understanding modules for RGB-T data pairs. Specifically, our framework first involves fine-tuning the original SAM by adding extra LoRA layers, aiming at preserving SAM's strong generalization and segmentation capabilities for downstream tasks. Secondly, we introduce language information as guidance for training our SARTM. To address cross-modal inconsistencies, we introduce a Cross-Modal Knowledge Distillation(CMKD) module that effectively achieves modality adaptation while maintaining its generalization capabilities. This semantic module enables the minimization of modality gaps and alleviates semantic ambiguity, facilitating the combination of any modality under any visual conditions. Furthermore, we enhance the segmentation performance by adjusting the segmentation head of SAM and incorporating an auxiliary semantic segmentation head, which integrates multi-scale features for effective fusion. Extensive experiments are conducted across three multi-modal RGBT semantic segmentation benchmarks: MFNET, PST900, and FMB. Both quantitative and qualitative results consistently demonstrate that the proposed SARTM significantly outperforms state-of-the-art approaches across a variety of conditions. Code and pre-trained weights can be found at https://github.com/wahaha-debug/SARTM.

Dong Xing, Jinhe Zhang, Hang Yang +1

Changchun Institute of Optics, Fine Mechanicsand Physics, Chinese Academy of Sciences University

Vision Foundation Model Adaptation Semantic Segmentation Multimodal Semantic Segmentation

Mar 19, 2025

Asynchronous Federated Continual Segmentation with Evolving Clients and Label Spaces

Federated learning seeks to foster collaboration among distributed clients while preserving the privacy of their local data. Traditional federated learning methods typically assume a fixed setting, where participating clients, client data, and learning objectives remain unchanged. However, in real-world scenarios, a federation may evolve over time, with changes in both its client composition and target label space. In this evolving federated setting, conventional round-wise model aggregation becomes inflexible, as each federation update requires repeated communication, repeated local computation, and synchronized participation from all accumulated clients. To address this limitation, we propose CA-MMDS, a continual multiple-model distillation framework for federated continual segmentation with asynchronous clients and evolving label spaces. Instead of repeatedly aggregating model parameters from all clients, CA-MMDS maintains a server-side archive of client models and updates the global model through proxy-based distillation from multiple archived local models. When new clients join or existing clients evolve, only the newly added or updated local models need to be uploaded, while unchanged clients can remain offline and continue to contribute through their archived models. This design substantially reduces communication and computation costs while enabling flexible asynchronous cooperation among evolving clients. Using multi-class 3D abdominal CT segmentation as an application task, we demonstrate that CA-MMDS efficiently incorporates evolving client knowledge while achieving competitive segmentation performance.

Can Peng, Qianhui Men, Pramit Saha +5

University of Oxford, Oxford, United Kingdom · University of Bristol, Bristol, United Kingdom

Federated Learning Aggregation Federated Continual Learning Asynchronous Federated Learning Medical Image Segmentation Semantic Segmentation Federated Learning

Dec 20, 2024

SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

Improving the reliability and completeness of colonoscopic inspection is critical for reducing missed lesions and improving colorectal cancer prevention. Reliable scene understanding is essential for navigation, reconstruction, and assessment of inspection completeness. Anatomical structures such as mucosal folds provide stable geometric cues for endoscope localization, while surgical instruments introduce dynamic occlusions that complicate visual interpretation. However, existing gastrointestinal endoscopy datasets largely focus on disease detection or artifact segmentation, leaving a gap in precise annotations of structural landmarks and instruments. We introduce SegCol, a dataset and benchmark for semantic segmentation of colon fold edges and surgical instruments derived from the EndoMapper dataset. SegCol provides manually annotated pixel-level masks for three instrument classes and thin fold-edge structures across temporally consistent image sequences. It forms the basis of the SegCol Challenge, organized as part of the EndoVis Challenge at MICCAI 2024, evaluating both supervised segmentation and annotation-efficient active learning. We further study segmentation metrics, including Dice, ODS/OIS, AP, and CLDice, under structural perturbations and different object geometries, and analyze participating methods, architectural choices, and active learning strategies. Our findings show that metric behavior strongly depends on target structure, highlighting the need for carefully selected evaluation protocols in endoscopic segmentation. Details are available at https://www.synapse.org/Synapse:syn54124209/wiki/626563, and code at https://github.com/surgical-vision/segcol_challenge.

Xinwei Ju, Rema Daher, Razvan Caramalau +7

UCL Hawkes Institute, University College London, London, UK · Department of Computer Science, University College London, London, UK · Medtronic, London, UK +3

Endoscopy Image Segmentation Active Learning Medical Image Segmentation Semantic Segmentation Medical Image Benchmarks