Semantic Segmentation

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 24% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 255

Oct 7, 2026cs.CV

Insights from Autoresearch for Solar Panel Segmentation

This paper investigates AutoResearch, a protocol in which a coding language model edits a training program under a one-hour GPU budget and retains a change only if validation IoU improves. The protocol is applied to photovoltaic panel segmentation on a frozen real-image split, with DeepLabV3--ResNet-50 held fixed. Three campaigns of 24 experiments, using Gemma~4 12B, Qwen3-8B all improve their one-hour baselines, but retained modifications do not transfer across hardware. The Qwen3-8B configuration, trained on real images only, reaches a test IoU of 0.836 versus 0.833 for the reference GAN-augmented schedule. Research repository https://github.com/VU-AIML/automl4eo-autoresearch-segmentation.

Justinas Lekavicius, Kursat Komurcu, Valentas Gruzauskas +1

Vilnius University, Institute of Computer Science, Artificial Intelligence Methods Lab · IRISA, Université Bretagne Sud · European Commission Joint Research Centre

Large Language Model-Guided Optimization Semantic Segmentation

Oct 7, 2026cs.RO

MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency

We introduce MultiFly, a real-world, low-altitude UAV dataset for semantic perception across RGB, thermal, LiDAR, and radar modalities. MultiFly provides 17,272 synchronized samples from four suburban scenes with frame-wise annotations for 15 semantic classes, together with calibration and GNSS-RTK/IMU measurements. To avoid costly and inconsistent modality-specific annotation, we propagate labels from only 115 manually annotated RGB images through shared geometric representations to all four modalities. This approach generates semantic labels for 17,157 additional RGB images, 17,272 thermal images, 840M LiDAR points, and 3.4M radar points. Transferred annotations achieve 89.93% average agreement with held-out manual annotations, and 90.94% average semantic consistency across all six modality pairs. We further establish semantic segmentation benchmarks for all four modalities, revealing distinct architectural behavior for dense LiDAR and sparse radar data. Taken together, MultiFly provides a scalable foundation for multimodal aerial perception and, to the best of our knowledge, the first public real-world low-altitude aerial benchmark that combines consistent frame-wise semantic annotations for RGB, thermal, LiDAR, and radar. Data at https://github.com/markus-42/multifly.

Markus Gross, Andreas Greiner, Taehyoung Kim +12

Autonomous Aerial Systems, Fraunhofer Institute IVI · Computer Vision Group, Technical University of Munich · Munich Center for Machine Learning (MCML) +2

Remote Sensing Benchmarks Semantic Segmentation Aerial Robotics

Oct 7, 2026cs.CV

A Probabilistic Perspective on Wasserstein-Based Evidential Uncertainty for Out-of-Distribution Segmentation

Semantic segmentation networks operate on a fixed set of classes and therefore fail when out-of-distribution (OOD) objects appear during deployment, a critical limitation for safety-critical applications such as autonomous driving. Reliably identifying OOD objects requires well-calibrated epistemic uncertainty, yet common softmax-based confidence scores remain overconfident, while Bayesian alternatives such as Monte Carlo dropout or deep ensembles require costly repeated forward passes. Evidential Deep Learning (EDL) offers an efficient alternative by modeling class probabilities as a Dirichlet distribution learned from a single deterministic forward pass. Existing EDL formulations rely on Euclidean objectives that push predictions towards the simplex vertices, encouraging overconfidence rather than preserving uncertainty for unfamiliar inputs. We instead employ Wasserstein-based objectives, which respect the geometry of the probability simplex, and study the influence of the Wasserstein order on segmentation accuracy and OOD detection within a unified evidential framework. We evaluate this framework on a convolutional (DeepLabV3+) and a transformer-based (SegFormer) architecture on the SegmentMeIfYouCan benchmark, including LostAndFound, RoadObstacle21, RoadAnomaly21, and Fishyscapes. Our results show the optimal Wasserstein order is architecture-dependent: second-order objectives dominate on the convolutional backbone, third-order objectives on the transformer backbone, and our framework surpasses comparable baselines on most metrics, with a single deterministic forward pass.

Arnold Brosch, Abdelrahman Eldesokey, Michael Felsberg +1

Heinrich-Heine-University Düsseldorf, Universitätsstraße 1, 40225, Düsseldorf, Germany · Hamad Bin Khalifa University (HBKU), Education City, Gate 8, P.O. Box 5825, Doha, Qatar · Linköping University, SE-581 83, Linköping, Sweden

Segmentation Uncertainty Estimation Evidential Deep Learning Semantic Segmentation

Oct 6, 2026cs.CV

RBMatch: Dual-Level Class Rebalancing for Semi-Supervised Building Footprint Extraction

Accurate building footprint extraction from high-resolution remote sensing imagery is essential for urban planning, disaster response, and environmental monitoring. However, obtaining dense pixel-level annotations is costly, motivating the use of semi-supervised learning (SSL) to leverage unlabeled imagery. In remote sensing, severe foreground--background imbalance poses a particular challenge for self-training, as it can bias pseudo-label generation and the resulting unsupervised optimization toward the majority background class. We show that addressing this imbalance at only one stage is insufficient: balancing pseudo-label selection alone does not prevent background bias from re-emerging during unsupervised loss optimization, a failure mode we term \emph{imbalance leak}. To address this issue, we propose \textbf{RBMatch}, a dual-level class-rebalancing framework that jointly regulates pseudo-label generation and unsupervised optimization. RBMatch combines a supervised learning pathway with a self-training module comprising three components: adaptive class-specific thresholding (ACT) for balanced pseudo-label selection, confidence-aware class-balanced reweighting (CACBR) for mitigating class bias in the unsupervised loss, and distribution alignment (DAL) for matching the predicted unlabeled-data distribution to the labeled-data prior. Experiments on the WHU, INRIA, and Massachusetts building footprint datasets across labeled ratios of 1%--10% show that RBMatch consistently achieves the best building IoU and F1-score among the evaluated methods. The improvement is most pronounced on the highly imbalanced Massachusetts dataset, where RBMatch improves IoU by 1.37 points over the strongest baseline at a 1% labeling ratio and is the only method to outperform the fully supervised baseline across all twelve dataset--ratio settings.

Akil Ahmad Taki, Shaikh Anowarul Fattah

Department of EEE, Bangladesh University of Engineering and Technology(BUET) · Department of EEE, University of Asia Pacific

Class-Imbalanced Learning Remote Sensing Image Segmentation Building Footprint Extraction Self-Training Semantic Segmentation

Oct 5, 2026cs.CV

FrontVeg V2: A Training-Free Software Framework for Foreground-Aware Zero-Shot Plant Trait Segmentation in High-Resolution Images of Trellised Crops

FrontVeg V2 is an open-source, training-free software framework for foregroundaware zero-shot segmentation of plant traits in high-resolution images of trellised crops. The pipeline combines monocular depth estimation, automatic foreground extraction using Valley-Aware Depth Thresholding, tiled zero-shot segmentation, Graph-Based Mask Assembly, and geometry-aware fusion. This design enables plant organs and disease symptoms to be segmented while reducing detections arising from neighboring vegetation rows. The current implementation integrates Depth Anything V2 (DAV2) and SAM3 and can be used through both command-line batch processing and a Napari graphical interface. FrontVeg V2 provides a reusable framework for multi-crop, multi-trait digital phenotyping without task-specific model retraining.

Abdoul Djalil Ousseini Hamza, Herearii Metuarea, Corentin Lothod{é} +6

IRHS-IMHORPHEN,DPT SPE · GAFL · SVQV,BAP +2

Image Segmentation Plant Phenotyping Semantic Segmentation

Oct 5, 2026cs.CV

Vision Transformer Ensembles for Panoramic Street Segmentation

Semantic segmentation of street panoramas can support detailed descriptions of urban environments, yet small datasets and unequal training costs make model selection difficult. This paper presents the system used for a first place submission to the PalmCity challenge in the leaderboard snapshot dated 5 October 2026. Nine pretrained segmentation systems are compared using approximately equal computation budgets. The candidates include DeepLabV3+, SegFormer, UPerNet, Mask2Former, DINOv3 with a linear decoder, and an Encoder only Mask Transformer using DINOv3. The two leading candidates are trained independently with three random seeds and longer budgets. Equal averaging of class probabilities from the three Encoder only Mask Transformer models, evaluated at three image scales with horizontal reflection, produces 60.95% mean intersection over union and 71.16% mean F1 on the 84 image public validation split. The submitted predictions receive 57.08% mean intersection over union and 67.96% mean F1 on the hidden test leaderboard. Producing all 249 test masks takes 251.49 seconds including model initialization and provenance checks on one NVIDIA RTX 5090. Peak allocated GPU memory is 2.70 GiB. The study reports all eligible models, all inference variants, class level errors, source conditions, and reproducibility checks, providing a documented challenge workflow with existing architectures.

Yunus Serhat Bıçakçı

Department of Artificial Intelligence and Machine Learning, Faculty of Applied Sciences, Marmara University, Istanbul, Türkiye · Geospatial Data Science Group, School of Geographical and Earth Sciences, University of Glasgow, Glasgow, UK

Ensemble Learning Vision Transformer Semantic Segmentation

Sep 30, 2026cs.CV

On the Relaxation of Conditional Independence Assumption for Image Segmentation

In semantic segmentation, a recent line of RankSEG methods directly optimizes Dice/IoU scores at inference time, improving alignment with evaluation metrics without modifying model training. Despite its theoretical and empirical success, RankSEG relies on the restrictive Conditional Independence Assumption (CIA), which ignores crucial label correlations and therefore degrades performance in ambiguous or low-contrast scenarios. However, accounting for full label dependence is computationally prohibitive, requiring

\mathcal{O}(d^3)

time. To address this, we replace the CIA with a Spatially Localized Dependence (SLD) structure that captures local label correlations while keeping the dependence model tractable. We further overcome the remaining computational bottleneck via a Reciprocal Moment Approximation coupled with a novel fixed-point optimization strategy that eliminates exhaustive search. The proposed algorithm achieves a highly practical

\mathcal{O}(d \log d)

complexity and consistently outperforms conventional argmax and CIA-based RankSEG across diverse segmentation benchmarks. Improvements are significant in low-contrast or small-object scenarios, where label dependence offers valuable signals complementary to image information for accurate segmentation. The code of experiments is available at https://github.com/ZixunWang/RankSEG-DEP.

Zixun Wang, Ben Dai

Department of Statistics and Data Science The Chinese University of Hong Kong

Semantic Segmentation

Sep 29, 2026cs.CV

GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion

Transparent surfaces are ubiquitous in built environments, yet they remain a persistent failure case for robotic perception. RGB cameras perceive the background behind glass rather than the surface itself, while depth sensors such as LiDAR, time-of-flight, and RGB-D often return invalid or background measurements in transparent regions. As a result, systems that rely solely on optical sensing may misinterpret glass walls, doors, or mirrors as free space, compromising safe and reliable navigation. Existing glass segmentation approaches address this by learning visual cues such as reflections, boundaries, and semantic context from RGB images. While effective under favourable lighting and viewing conditions, these cues degrade in low-light environments, under glare, or when glass surfaces are featureless or partially occluded. In this work, we propose a multimodal framework that fuses millimetre-wave radar with RGB-D sensing for real-time transparent surface segmentation. Radar reflects strongly off glass surfaces, providing a geometric cue that remains reliable precisely where vision and depth fail. We exploit this cross-modal inconsistency to generate a radar-guided spatial prior, which is integrated into a lightweight transformer-based segmentation network, GlassFormer, via cross-modal attention. We report results on a mixed-condition test split covering all scene types and a dedicated low-light split designed to stress vision-only methods. GlassFormer achieves 0.88 mIoU on the mixed split, and 0.59 mIoU on the low light split, demonstrating substantial robustness gains over vision-only baselines while maintaining real-time performance on resource-constrained platforms.

Suhani Grover, Astik Srivastava, Viswas Dinesh +2

Robotics Research Center, IIIT Hyderabad, India

Robotic Perception Semantic Segmentation Multimodal Sensor Fusion

Sep 27, 2026cs.CV

When Noise Meets Long-Tail: Feature-Threshold Dual Calibration for Robust Pseudo-Labeling

Pseudo-labeling has become a cornerstone of learning from unlabeled data in semantic segmentation. Yet its effectiveness drops sharply in real-world scenarios where strong imaging noise and long-tailed class distributions occur together. We trace this failure to a vicious cycle of pseudo-label degradation. Imaging noise entangles foreground and background features, lowering prediction confidence across all classes, while long-tailed distributions leave tail classes with far fewer training samples and inherently lower confidence. Under fixed high-threshold filtering, these tail-class predictions are systematically filtered out, so they receive no supervision from unlabeled data and thus features keep degrading in subsequent iterations. Critically, noise and long-tail are not independent obstacles but mutually amplifying ones, and addressing either alone is insufficient. To break this cycle, we propose FTC-Seg, a Feature-Threshold dual-Calibration framework built on a standard teacher-student framework. At the feature level, Orthogonal Prototype Reconstruction (OPR) uses a set of learnable orthogonal prototypes to residually purify pixel-wise features, widening the margin between weak foreground targets and noisy backgrounds. At the threshold level, Adaptive Threshold Calibration (ATC) dynamically adjusts class-specific thresholds based on learning difficulty and prediction-distribution bias, rescuing low-confidence pseudo-labels of tail classes from systematic exclusion. Extensive experiments on four public benchmarks spanning three distinct noise modalities show that FTC-Seg achieves strong performance against state-of-the-art methods, with particularly substantial gains on tail classes. Our results establish that jointly calibrating features and thresholds is essential for robust pseudo-labeling under compounded noise and class imbalance.

Ping Guo, Zhiqi Huang, Xinran Li

Graduate School of Information, Production and Systems, Waseda University, Kitakyushu, Japan · School of Software, Dalian University of Technology, Dalian, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China

Image Segmentation Long-Tail Learning Pseudo-Labeling Semantic Segmentation

Sep 27, 2026cs.CV

TC-ADA: One-Shot Active Domain Adaptation for Semantic Segmentation

Manual dense annotation remains a major obstacle to deploying semantic segmentation models in new driving environments. Active domain adaptation (ADA) seeks label-efficient transfer by annotating only a selected portion of the target domain. Existing ADA methods commonly implement this process through multiple rounds of acquisition, annotation, and retraining. We study a practical one-shot image-level setting that selects and densely annotates a fixed target subset in a single round, followed by uninterrupted adaptation. Within this setting, we develop Target-Calibrated Active Domain Adaptation (TC-ADA) as a joint design of complete-image acquisition and target-calibrated adaptation. Stage1 uses visual representations from a vision foundation model (VFM) together with semantic predictions from a fixed unsupervised domain adaptation model to select representative and informative target images without target annotations. Stage2 jointly uses labeled source data, labeled target data, and the remaining unlabeled target data, while calibrating source and target supervision under limited target labels. Extensive experiments across five synthetic-to-real and real-to-real driving transfers show consistent improvements over representative ADA baselines. With only 23 to 46 labeled target images on four transfers and 140 on Mapillary, TC-ADA stays within 1.9 mean intersection over union (mIoU) points of target-only full supervision. Code will be available at https://github.com/ywher/TC-ADA.

Weihao Yan, Yeqiang Qian, Yueyuan Li +3

School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, 200240, China

Synthetic-to-Real Domain Adaptation Autonomous Driving Semantic Segmentation

Sep 24, 2026cs.CV

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassing abundant categories and high-quality annotations. RGBD20K possesses several attractive properties: (1) Expanded Semantic Space. In particular, it covers 160 fine-grained categories, largely surpassing the category diversity of existing popular RGB-D benchmarks (e.g., NYUv2 with 40 classes and SUN RGB-D with 37 classes). With such enriched semantic coverage, we expect to promote the learning of more generalizable segmentation models. (2) Larger Scale. Compared with current benchmarks, RGBD20K offers 20,000 RGB-D image pairs, providing a substantially larger training resource that benefits the development of more powerful deep models. (3) High-Fidelity Annotation. We perform rigorous re-evaluation and correction of existing labels to resolve long-standing annotation noise, resulting in a clean and reliable ground-truth foundation. Furthermore, we propose a novel score-purified fusion (SPF) method, which achieves state-of-the-art performance across all evaluated benchmarks, demonstrating the effectiveness of our approach in leveraging high-quality multimodal information for RGB-D semantic segmentation. The dataset is here: https://github.com/ShaohuaDong2021/RGBD20K/.

Shaohua Dong, Zexuan Meng, Haiyan Sun +6

University of North Texas. Denton, TX 76207, USA.

Multimodal Fusion Semantic Segmentation

Sep 23, 2026cs.CV

Privacy-Preserving Semantic Segmentation from High-Resolution Depth and Ultra-Low-Resolution RGB

As mobile robots become increasingly integrated into everyday environments, privacy risks arising from onboard cameras have become a growing concern. Ultra-low-resolution (ULR) RGB can mitigate visual privacy exposure at the source, but ULR appearance alone substantially limits semantic and spatial understanding. We therefore introduce a privacy-preserving asymmetric sensing setting that combines high-resolution (HR) depth with ULR RGB, preserving dense geometry while restricting fine-grained visual information. To address the severe information imbalance between HR depth and ULR RGB, we propose a joint 2D framework using HR geometry to guide semantic-oriented RGB reconstruction and RGB-D segmentation. Despite reliable frame-level predictions, consistent scene-level understanding remains challenging under the asymmetric HR depth--ULR RGB setting. We therefore develop an end-to-end 2D-to-3D pipeline that consolidates 2D semantic features for 3D segmentation. Experiments on ScanNet show that our method achieves the best 2D and 3D segmentation performance among privacy-preserving approaches and delivers the strongest zero-shot transfer to SUN RGB-D and SceneNN. Privacy recoverability analysis shows that our proposed HR depth--ULR RGB input reduces the recoverability of sensitive data, and real-robot experiments demonstrate the utility of the resulting 3D semantics for object-goal navigation.

Xuying Huang, Swithinraj Moses Daniel, Sicong Pan +2

Fraunhofer Institute for Intelligent Analysis and Information Systems

3D Point Cloud Segmentation 3D Semantic Segmentation Privacy-Preserving ML Robotic Perception Semantic Segmentation

Sep 23, 2026cs.CV

ICM: Intra-class Mixing for Domain Adaptation in Adverse Weather

Unsupervised domain adaptation (UDA) for semantic segmentation remains challenging under adverse weather conditions because severe appearance changes enlarge the domain gap and degrade the reliability of pseudo labels in the target domain. To address this problem, we propose an Intra-Class Mixing Consistency (ICM) framework that enforces prediction consistency between an intra-class mixed image and its original counterpart. Unlike previous mixing-based consistency methods that combine regions across different images or domains and may introduce unrealistic semantic inconsistencies, ICM performs mixing within the same image and semantic class, preserving realistic semantic layout for consistency regularization. With ICM, we establish a new state-of-the-art performance for clear-to-adverse-weather unsupervised domain adaptation (UDA) in semantic segmentation. On the Cityscapes

\rightarrow

ACDC benchmark, our method achieves 75.7% mIoU, outperforming the previous state of the art by +1.9 pp, demonstrating its effectiveness in mitigating class confusion under challenging environmental conditions. The code is provided in the supplementary material.

Boying Li, Chang Liu, Britta Ayano Wilde +4

Unsupervised Domain Adaptation Domain Adaptation Consistency Regularization Semantic Segmentation Unsupervised Image Segmentation

Sep 22, 2026cs.CV

LiFR v2: Completion-Augmented Event Propagation for High-Rate Dense Prediction

High-rate dense perception in dynamic environments is limited by the low update rate of RGB cameras, as rapid scene changes can occur between frames. Event cameras offer temporally dense but spatially sparse measurements, complementary to spatially dense RGB observations. Direct fusion cannot fully exploit this complementarity, while event-guided propagation fails on newly appearing or disoccluded regions without valid RGB support. We present LiFR v2, a unified propagation-completion-memory framework for causal anytime and streaming dense prediction from an RGB keyframe and events. LiFR v2 introduces an Event-Guided Completion Module (EGCM) to recover task-relevant representations where propagation is unsupported, and a History Retrieval Module (HRM) to reuse completed representations across successive queries. The framework supports semantic segmentation, monocular depth estimation, and multi-task dense prediction, and we further introduce SHF-Emerge to evaluate rapid object emergence and disocclusion. LiFR v2 achieves 74.37% mIoU on DSEC and 56.13% on SHF-Emerge, improving LiFR-Seg by 1.85 percentage points on the latter, while reducing SHF-Emerge depth RMSE from 1.564 m to 1.118 m over the propagation baseline. It also exceeds 100 FPS for both segmentation and depth, demonstrating accurate and efficient high-rate perception beyond RGB frame rates.

Tao Wan, Xiaoshan Wu, Yifei Yu +6

School of Microelectronics, Southern University of Science and Technology, Shenzhen 518055, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong, China

Event-Based Vision Monocular Depth Estimation Dense Prediction Semantic Segmentation

Sep 21, 2026cs.CV

What Survives on Real Drawings: Active Sampling, Connectome Wiring, and Matched Baselines in Architectural Document Vision

A connectome-constrained model of the fly visual system, optimized for motion and then frozen, can be driven over architectural drawings by prescribed motion and used as a texture representation. We compare it with information-matched baselines that see the same 721 photoreceptor samples. On clean synthetic data the frozen model transfers but loses to task training: 0.857 area-weighted accuracy in one-shot hatch matching versus 0.959 for a 5,888-parameter CNN, and 0.619 IoU in wall segmentation versus 0.905 for a matched network. Under scan noise and thickened strokes, the trained networks lose up to 0.188 accuracy while the frozen pipeline loses 0.030. On fourteen production sheets, opened once, a 1,876-parameter fly model reaches 0.505 average precision versus 0.415 for a network two hundred times larger. A preregistered held-out split confirms the clean-data ordering: 0.835 for the circuit, 0.894 for receptors only, and 0.971-0.980 for trained CNNs. Rewiring the connectome while preserving degrees or type pairs and transmitter signs costs 0.271-0.356 accuracy across three seeds, so the exact wiring is load-bearing. Yet the intact circuit does not beat its moving retina, and T4/T5 silencing leaves both tasks intact. Longer observations reverse the circuit-receptor ordering once the stimulus spans a period, but not through T4/T5. Thus active sampling and exact structure matter, while clean-data practical performance remains dominated by task-trained networks and the useful transfer margin is largely retinal.

Dmitry Kuklev

Independent Researcher

Visual Representation Learning Image Matching Active Visual Perception Semantic Segmentation

Sep 17, 2026cs.CV

PerSeM: Persistent Semantic Memory for Long-Horizon Open-Vocabulary UAV Mapping

Open-vocabulary segmentation enables rich semantic perception for UAVs, but frame-wise predictions can remain temporally inconsistent across repeated observations and changing viewpoints. We present PerSeM, a training-free persistent semantic memory framework for long-horizon open-vocabulary UAV mapping. PerSeM associates frame-wise semantic observations with persistent world-space voxels and constructs a majority-based semantic memory, which is conservatively refined through history-preserving spatial refinement, trust-aware replay, and context-guided verification. Experiments on the Forest and UAVScenes benchmarks show that persistent 3D memory provides substantial gains in semantic correctness and temporal stability over frame-wise predictions. Beyond this strong persistent-memory baseline, PerSeM provides consistent additional improvements, improving both semantic accuracy and temporal stability across all five evaluated UAVScenes sequences. Analysis using regions identified independently of the final PerSeM predictions further shows that these gains are concentrated in semantically difficult and temporally unstable regions, where majority-based memory is most likely to remain uncertain. These results demonstrate that persistent 3D aggregation provides a strong foundation for long-horizon semantic mapping, while conservative refinement of uncertain memory states can provide additional improvements without retraining or additional neural-network inference.

Saurbh Singh Jamwal, Ganesh Ramakrishnan

Department of Computer Science and Engineering Indian Institute of Technology Bombay

Open-Vocabulary 3D Segmentation Semantic Occupancy Mapping Semantic Mapping UAV Remote Sensing Semantic Segmentation

Sep 15, 2026cs.CV

PiPS: Post-Hoc Prototypical Explanations for Interpretable Semantic Segmentation

With the increasing deployment of deep neural networks in critical systems, such as medical diagnostics and autonomous vehicles, ensuring their interpretability is crucial to building trust in decision-making systems. In the field of explainable artificial intelligence, prototype-based reasoning has gained particular popularity, as it mimics human cognitive processes by explaining model decisions based on visual similarity under the looks like this paradigm. While this paradigm has been thoroughly investigated in the context of global image classification, the interpretability of dense predictions, particularly semantic segmentation, remains largely unexplored despite its immense importance in tasks requiring precise object localization. Existing prototype-based interpretable segmentation models rely on ante-hoc architectures, which entails significant limitations because they require costly training from scratch and modifications to the network structure, ultimately leading to a noticeable drop in predictive performance compared to standard black-box models. To address this issue, we propose PiPS (Post-hoc interpretable Prototypical Segmentation), the first fully post-hoc solution for generating prototypical explanations for semantic segmentation models. Our method enables the extraction of intuitive, spatially localized explanations from any pre-trained network without modification or fine-tuning, thereby preserving 100% of the model's original predictive performance. This approach opens a new avenue for the safe and cost-effective deployment of transparent systems in advanced computer vision tasks. Codebase available at https://github.com/gmum/PIPS.

Miłosz Adamczyk, Tymoteusz Zapala, Piotr Borycki +1

Jagiellonian University · Wrocław University of Science and Technology · IDEAS Research Institute

Explainable Artificial Intelligence Semantic Segmentation

Sep 15, 2026cs.CV

TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation

Query-based segmentation methods have shown promising potential for surgical instrument segmentation and recognition, which is essential for scene understanding and downstream tasks in computer assisted surgery. However, most existing approaches predominantly rely on per-frame predictions and overlook cross-frame temporal priors as well as temporal-consistency constraints. This limitation often leads to unstable query representations and suboptimal category recognition. In this paper, we propose TEDi, a Temporal memory-Enhanced and Denoising transformer for surgical instrument segmentation that addresses these is sues through Memory Search Enhancement and Temporal Consistency Denoising. The former introduces a query-level memory bank and a memory search enhancement encoder to retrieve discriminative representations from historical frames, enriching current-frame features. The latter constructs a temporally consistent reference as a cross-frame semantic anchor to suppress temporally unstable predictions and promote semantic coherence across frames. Extensive experiments on two benchmark datasets, EndoVis 2017 and EndoVis 2018, demonstrate that TEDi consistently outperforms state-of-the-art methods, highlighting its potential to further advance computer-assisted surgery. Our code is available at github.com/argon-xixi/TEDi.

Jiahong Yuan, Weiming Mi, Tao Zhang +1

Department of Automation, Tsinghua University, Beijing, China · Surgical Planning Laboratory, Brigham and Women’s Hospital, Harvard Medical School, United States

Video Object Segmentation Medical Image Segmentation Semantic Segmentation

Sep 14, 2026cs.CV

CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs

In order to design conservation and restoration strategies to counter the global decline of coral reefs, ecological monitoring of reefs needs to be scaled up dramatically. Computer vision methods are increasingly used to tackle the vast amount of data: as the paradigm of data collection in reefs shifts from highly standardized and constrained survey images to unconstrained imagery on scalable platforms, it is necessary to design machine learning methods that help to get a fine-grained understanding of reefs from general-purpose reef imagery. This paper provides CoralscapesV2, an extension of the Coralscapes dataset for general-purpose visual scene understanding in reefs. CoralscapesV2 increases the dataset size, scope, label completeness and quality for semantic segmentation, and extends the number of classes from 39 to 95 fine-grained visual categories. Furthermore, CoralscapesV2 provides 65k exhaustive fish instance mask annotations, meticulously annotated to completeness by using the video, revealing that annotation of fish based on only static images is insufficient. CoralscapesV2 is the first dataset for panoptic segmentation in coral reefs, capturing a wide range of scenarios in the wild, posing a challenging benchmark for contemporary semantic segmentation and instance segmentation models. CoralscapesV2 is an important step towards general-purpose panoptic segmentation in coral reefs, which has substantial implications for scaling up coral reef monitoring, as it can be employed in a wide range of applications from benthic cover mapping from robot or handheld videos to designing methods for automated quantification and understanding of fish behavior and fish-reef interactions.

Jonathan Sauder, Thomas Ruckli, Gabrielė Strodomskytė +17

MIT · EPFL · University of Djibouti +7

Panoptic Segmentation Instance Segmentation Semantic Segmentation Fine-Grained Visual Perception

Sep 11, 2026cs.CV

R4Tun: LLM-guided adaptive segmental tunnel lining segmentation in point clouds

Automated inspection of segmental tunnel linings requires adaptive segmentation from 3D point clouds, yet expert-tuned pipelines often degrade when tunnel conditions vary. This paper presents R4Tun, a large language model (LLM)-driven adaptation framework that extends an expert-designed pipeline (SAM4Tun) with bounded parameter tuning informed by structured context: memory (

m

), state (

s

), and knowledge (

k

). Evaluated on 30 selected Seg2Tunnel subsets (13 regular, 17 complex) across three LLMs, the full

m+s+k

design raised mean Intersection-over-Union (mIoU) from 0.18 to 0.43--0.48 and overall accuracy (OA) from 0.42 to 0.59--0.65 relative to the static SAM4Tun baseline, with the near-reference regular (staggered) subsets reaching mIoU 0.784--0.796 across LLMs. Across 270 (30 tunnels

\times

3 different LLMs

\times

3 context settings) runs, the LLMs showed similar parameter-adjustment trends (with overlapping 95% CIs on mean gains) and consistently adjusted a shared set of critical parameters. These results support R4Tun as a controlled, label-free, cross-LLM adaptation mechanism in the tested SAM4Tun--Seg2Tunnel setting, demonstrating consistent accuracy gains; we position R4Tun as a mechanism contribution rather than a deployable final-inspection system, in which each bounded parameter change is auditable via logged rationales.

Xinghui Tao, Zehao Ye, Guangming Wang +2

Construction Engineering, University of Cambridge, Trumpington Street, Cambridge, CB2 1PZ, United Kingdom · Department of Engineering, Durham University, Stockton Road, Durham, DH1 3LE, United Kingdom

3D Point Cloud Segmentation Large Language Model-Guided Optimization Semantic Segmentation

Sep 10, 2026cs.CV

HSI-Road Relabeled: Surface-Aware Road-Scene Segmentation

The HSI-Road dataset provides paired RGB and 25-channel NIR (600-960nm) images with binary masks but no surface-level labels. This paper introduces a manually labeled six-class taxonomy: Background, Asphalt, Concrete, Dirt, Water, and Grass, and an RGB-to-NIR registration pipeline with corresponding annotations. Six semantic-segmentation models are evaluated under four input configurations: original-resolution RGB, registered low-resolution RGB (RGB

_{\text{reg}}

), NIR, and channel-stacked RGB

_{\text{reg}}

-NIR (RGBN

_{\text{stk}}

). The comparison quantifies the effect of spatial-resolution reduction on RGB, along with evaluation of NIR and RGBN

_{\text{stk}}

, with results reported using per-class and mean IoU and F1 scores. The original-resolution RGB achieves the highest overall performance but contains 12

\times

more pixels and incurs a 15.5-20.6

\%

latency penalty compared to the reduced-resolution inputs. At the common 192

\times

384 resolution, RGBN

_{\text{stk}}

outperforms NIR for all six models and RGB

_{\text{reg}}

for five of six, with the most consistent gains for the Water class. These results highlight the importance of spatial resolution while showing that NIR provides complementary information to RGB.

Imad Ali Shah, Imran Mehmood, Enda Ward +3

School of Engineering and Ryan Institute, University of Galway, Ireland · Valeo Vision Systems, Tuam, Ireland

Remote Sensing Image Segmentation Semantic Segmentation

Sep 9, 2026cs.CV

Enhanced Deformable Convolution with Center-invariant Offset and Edge-aware Mask

Deformable convolution networks have recently become popular for many computer vision tasks, especially for semantic segmentation, because of their exceptional capabilities in dynamic spatial modeling. However, due to the dense deformable offsets and the lack of longer-range dependencies, they can not fully adopt proper and precise deformations for feature representations. To tackle the issues, in this paper, we propose Enhanced Deformable ConvNets (EDCN) for semantic segmentation. Specifically, a novel Enhanced Deformable Convolution (EDC) is exploited in the decoder, which integrates the Center-invariant Offset Module (COM) and Edge-aware Mask Module (EMM). The COM employs larger kernels and eliminates deformations at the kernel center, obtaining offsets that are more in line with the target from richer spatial information. Concurrently, the EMM obtains the significance of image content via Sobel edge detection, then selectively applies deformations based on the content significance, minimizing unnecessary deformations associated with relatively less important information, thereby avoiding impact from less informative regions. Experiments show that EDC outperforms state-of-the-art deformable convolution variants, including Deformable ConvNets V1-V4 and Entire Deformable ConvNets, across mainstream segmentation datasets with various decoder settings. Moreover, ablation studies confirm the effectiveness of each component. In addition, visualizations illustrate that EDC enhances spatial adaptation and target focus. We further analyze the extendibility of EDC to larger kernels on the image classification benchmark. Code will be publicly released.

Yixiao Li, Xiaoyuan Yang, Jin Jiang +5

School of Mathematical Sciences, Beihang University, Beijing, China · Shanghai Academy of Spaceflight Technology, Shanghai Radio Equipment Research Institute, Shanghai, China · School of Computational and Mathematical Sciences, Cardiff University, Cardiff, United Kingdom +3

Convolutional Neural Networks Deformable Convolution Semantic Segmentation

Sep 9, 2026cs.CV

CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids

Semantic segmentation for autonomous driving requires reliable detection of vulnerable road users (VRUs) despite heavy class imbalance. We introduce CLFTv2, a hierarchical camera-LiDAR fusion framework replacing global ViT attention with a Swin-based multi-scale encoder and a lightweight FPN-style residual decoder. Operating in the 2D perspective domain, CLFTv2 integrates multi-scale geometric cues through shifted-window attention and per-scale residual fusion, avoiding the computational overhead of query-matching decoders. Across three driving datasets, CLFTv2 consistently improves VRU recall. On ZOD, CLFTv2-Large achieves 53.5% mIoU, improving pedestrian IoU from 35.5% to 44.9% over the prior CLFT model. On Waymo, CLFTv2 reaches 61.7% mIoU. Additionally, a modality-isolation study suggests ViT's global receptive field yields stronger fusion gains only under dense LiDAR returns. Compared to a Swin-based Mask2Former adaptation, CLFTv2 requires 1.4

\times

fewer GFLOPs and delivers 2.2

\times

higher throughput, while achieving comparable overall accuracy. These results demonstrate that hierarchical local-attention fusion offers an efficient, scalable alternative to global-attention and query-based decoders for real-time on-vehicle perception in intelligent transportation systems. Source code is publicly available.

Toomas Tahves, Mauro Bellone, Raivo Sell

Department of Mechanical and Industrial Engineering, Tallinn University of Technology, Tallinn, Estonia · FinEst Centre for Smart Cities, Tallinn University of Technology, Tallinn, Estonia · Universitas Mercatorum, Rome, Italy

Camera-LiDAR Fusion Autonomous Driving Perception Multi-Scale Feature Fusion Efficient ViTs Semantic Segmentation

Sep 8, 2026cs.CV

Data-Efficient Crosswalk Segmentation from Overhead CCTV via Confidence- and Geometry-Guided Pseudo-Labeling

Pixel-level annotation of fixed traffic-camera imagery is expensive, while crosswalk models trained from street-level imagery face a substantial viewpoint and appearance shift when applied to elevated CCTV. We investigate a data-efficient target-domain pipeline using 241 manually annotated CCTV images and 5,926 unlabeled CCTV frames. A source-domain experiment trains a 31.0M-parameter custom U-Net on 3,300 first-person-view (FPV) images and obtains 93.05% IoU on its 330-image FPV test split. This result is a source baseline, not transferred performance: the released CCTV notebook instantiates a 42.0M-parameter DeepLabV3-ResNet50 from torchvision weights, and no compatible mapping from the U-Net checkpoint is implemented. Training on 201 manual CCTV images and selecting on 40 held-out manual masks yields 88.91% IoU. The model then predicts all unlabeled frames; image-level certainty and a largest-component area prior rank the candidates, and the top 1,000 attain mean certainty 0.976 and mean combined score 0.988. A repository audit shows that the reported second-stage 98.52% IoU was measured on a 150-image split containing only teacher-generated pseudo-masks. Because of a directory-layout mismatch, the executed combined-data loader found zero manual samples and split 1,000 pseudo-labeled samples into 850 training and 150 evaluation samples. We therefore report 98.52% as internal pseudo-label agreement rather than human-ground-truth accuracy. The defensible target-domain result is 88.91% IoU on the 40 manual validation images. Batch-one FP32 inference at 512 x 512 requires 12.98 ms, corresponding to 77.03 FPS, on an NVIDIA RTX A6000 48 GB GPU. These findings support the practicality of confidence-and-geometry filtering while also showing why pseudo-label evaluation must remain isolated from the labels used for self-training.

Abdirashid Omar, Jonghyuk Park

Department of Data Science, Graduate School of Kookmin University · Seoul 02707, Republic of Korea

Pseudo-Labeling Semantic Segmentation Video Surveillance

Sep 8, 2026cs.CV

CoordFormer: Give Me Any Coordinates and I Will Give You Labels

Semantic segmentation on very-high-resolution images remains challenging due to the high computational cost and the difficulty of capturing fine-grained details. We propose CoordFormer, a novel coordinate-based architecture for semantic segmentation that predicts labels at arbitrary spatial locations through a Coordinate Decoder equipped with a Localized Cross-Attention mechanism. The decoder combines coordinate embeddings with high-resolution local patch features and interacts with global tokens extracted from a downsampled image processed by a ViT foundation encoder, enabling rich semantic context while preserving pixel-level precision. This design enables flexible inference at arbitrary resolutions while keeping memory low on very-high-resolution inputs, and supports an efficient semantic-edge-focused strategy that concentrates computation along boundaries, maintaining fine-grained accuracy while reducing latency and computational cost. CoordFormer achieves state-of-the-art performance on MaSS13K and outperforms comparably sized and higher-parameter methods on DIS5K and KPIs, demonstrating its effectiveness for high-quality, very-high-resolution semantic segmentation.

Iacopo Curti, Pierluigi Zama Ramirez, Alioscia Petrelli +1

CVLab, University of Bologna · Ca’ Foscari University of Venice · SINA, company

Vision Transformer Semantic Segmentation

Sep 8, 2026cs.CV

MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentation

Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in zero-shot transfer but often struggles with dense prediction tasks due to low spatial resolution and the loss of structural information. To address these limitations, we propose MARS-CLIP (Multi-resolution and Attention Refined Segmentation for CLIP), a novel framework for zero-shot semantic segmentation. Our approach introduces two key strategies: (i) a multi-resolution feature extraction module that fuses local fine-grained features with global context to overcome input resolution constraints, and (ii) an attention refinement mechanism that injects spatial and color biases from intermediate layers into the final self-attention block to accurately restore object boundaries. A set of experiments on six public datasets demonstrates that MARS-CLIP significantly outperforms state-of-the-art methods.

Nagito Saito, Shintaro Ito, Koichi Ito +1

Graduate School of Information Sciences, Tohoku University, Japan.

Multi-Scale Feature Fusion Boundary-Aware Segmentation Semantic Segmentation

Sep 7, 2026cs.CV

TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding

Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only

\sim

590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20

\to

18.11 mIoU, COCO-Stuff 7.35

\to

24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP

\,

2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04

\to

22.51 mIoU).

Harsha Patnala, Debopriyo Banerjee, Ayush Sunil Munot +1

Eightfold AI · Inception42 · Indian Institute of Technology Kharagpur

Visual Reasoning Vision-Language Alignment Semantic Segmentation Vision-Language Pretraining

Sep 3, 2026cs.CV

Efficient Semantic Understanding from Digital Foveation

Dense semantic segmentation allocates computational resources uniformly across the entire image, regardless of scene complexity or task relevance. Inspired by biological vision, we investigate whether semantic understanding can be achieved more efficiently through digital foveated perception. We introduce a lightweight active-vision pipeline that combines saliency-driven fixation selection, high-resolution foveal observations, low-resolution contextual information, semantic accumulation, and adaptive computation. Beyond conventional dense prediction metrics, we use object-level evaluation to measure semantic understanding under sparse observations. On ADE20K-Object, a single foveated observation achieves 95.9% of the baseline Top-1 accuracy and 96.9% of the baseline Top-3 accuracy while requiring only 4.7% of the computational cost. At the scene level, semantic accumulation recovers 90.6% of the baseline object recall while using 58.6% of the computation. These results suggest that substantial semantic understanding can emerge from sparse observations when computation is allocated selectively, highlighting active vision as an efficient alternative to uniform dense processing and motivating evaluation protocols beyond conventional pixel-wise segmentation metrics.

Caterina Caccavella, Vittorio Fra, Andreas Ziegler +2

Zurich University of Applied Sciences (ZHAW), Wädenswil, Switzerland · ETH Zürich, Zürich, Switzerland · Politecnico di Torino, Turin, Italy +2

Active Visual Perception Semantic Segmentation Adaptive Computation

Sep 2, 2026cs.RO

Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

LiDAR-based semantic segmentation is a core perception module for autonomous vehicles and mobile robots. Despite the strong performance of recent state-of-the-art methods on standard benchmarks, existing evaluation protocols remain focused on clean, single-domain settings and fine-grained label taxonomies, leaving deployment readiness largely unassessed. Real-world systems must handle safety-critical label semantics, degraded sensing conditions, and cross-domain variability, yet no unified protocol currently addresses all three aspects together. In this paper, we propose a structured evaluation protocol that assesses the deployment readiness of LiDAR semantic segmentation models along three complementary dimensions: (i) coarse-label evaluation aligned with autonomous driving safety priorities, revealing how label granularity affects different methods; (ii) robustness under eight types of LiDAR corruptions designed to emulate real-world atmospheric, geometric, and sensor degradations; and (iii) domain generalization across datasets without adaptation. The evaluation includes inference speed measured on an embedded Jetson AGX Orin platform, directly reflecting deployment constraints. Our results show that fine-grained benchmark rankings do not always reflect safety-relevant performance, that all methods experience substantial degradation under corruptions with architecture-dependent robustness characteristics, and that current domain generalization remains insufficient for reliable deployment. These findings expose concrete gaps between benchmark performance and deployment readiness, and provide a reference protocol for more practically grounded evaluation of LiDAR semantic segmentation.

Samir Abou Haidar, Alexandre Chariot, Mehdi Darouich +2

Mines Paris, PSL University, Centre for Robotics (CAOR), 75006 Paris, France · Paris-Saclay University, CEA, List, F-91120, Palaiseau, France

Semantic Segmentation 3D Semantic Segmentation Corruption Robustness Domain Generalization Autonomous Driving

Sep 2, 2026cs.CV

InsightSeg: Reusing Correction Insights for Guideline-Consistent Segmentation

Guideline-consistent semantic segmentation requires more than category recognition, as real-world labeling policies demand fine-grained, task-specific decisions. Recent multi-agent refinement systems improve compliance with such textual guidelines by detecting and correcting errors. However, they are stateless: feedback from the critiquing agent is discarded, causing the same guideline-specific mistakes to be repeatedly rediscovered and corrected across the dataset at the cost of additional refinement. We introduce InsightSeg, an episodic memory mechanism that converts successful correction episodes into reusable, visually grounded insights. A meta-analyzer distills each qualifying episode into directive natural-language insights and anchors them to the local image regions that caused the error using patch-level visual concept vectors. On subsequent images, these concepts are matched against dense patch embeddings to retrieve relevant insights, which condition the segmenting agent before making its first prediction. This shifts the system from correcting recurring errors to preventing them, improving segmentation quality before any refinement occurs. Across Waymo and Cityscapes, InsightSeg improves both first-pass and final guideline-consistent segmentation performance while requiring fewer refinement steps, demonstrating that multi-agent refinement can become more accurate and efficient by drawing on past correction experience.

Vanshika Vats, Ashwani Rathee, James Davis

University of California, Santa Cruz, CA, USA

Semantic Segmentation Memory-Augmented Agents