Semantic Segmentation

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 175

May 12, 2026cs.CV

Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation

Semantic segmentation is essential for analysing anatomical features in biomedical research, yet a performance gap remains for Vision Transformers (ViTs) in the field, particularly for sparse, fine-structured, and low signal-to-noise targets. We attribute this challenge in part to the lightweight pixel decoders commonly used in promptable ViT models, who may lack the local inductive bias needed for high-precision biomedical masks. We bridge this gap by introducing ViTC-UNet, which conditions a UNet on frozen pre-trained ViT representations through learnable tokens and a two-way attention decoder. This combines ViT global visual priors with the local inductive bias and high-resolution decoding capacity of UNets, while avoiding end-to-end ViT fine-tuning even in cross-domain settings. ViTC-UNet outperforms baseline results in semantic segmentation tasks across MRI and CT modalities, demonstrating that structure-conditioned UNet decoding can efficiently adapt large-scale visual priors to high-complexity biomedical segmentation.

Joel Valdivia Ortega, Tingying Peng, Marion Jasnin

Helmholtz AI, Helmoltz Munich, Neuherberg, Germany · Helmholtz Pioneer Campus, Helmholtz Munich, Neuherberg, Germany · School of Computation, Information and Technology, TUM, Garching, Germany +1

Semi-Supervised Medical Image Segmentation Self-Supervised Vision Transformers U-Net Vision Transformer Semantic Segmentation

May 12, 2026cs.SE

SemChunk-C: Semantic Segmentation for C Code

Semantic segmentation of code written in a C-family language remains a challenging problem, due to the language's complex syntax, macro expansion, and irregular structural patterns. Existing chunking methods, such as fixed-sized windows, heuristic splitting, and syntax-based tools, often fail to capture meaningful functional units, limiting the efficacy of retrieval and other downstream LLM driven tasks. In this paper, we address the problem of chunking in C-related languages. First, we define a set of code chunk categories. Second, we train an LLM-based classifier to a) identify chunk boundaries, and b) assign each chunk a descriptive functional attribute (a category), which can be useful for downstream tasks. By leveraging the LLM's ability to capture semantic context within the code, we assume flexible chunk boundaries, allowing to adapt to the specific structure and context of each instance. Third, we introduce SemChunk-C, a family of lightweight language models for semantic chunking of C-related files (.c, .cpp, .h, .cs, etc.). These models are based on the first four Ettin encoders [1] with 17M, 32M, 68M, and 150M parameters. Despite their relatively small size, they are capable of identifying cohesive code units, such as data structures, interface blocks, and other components. Furthermore, we demonstrate the robustness of our approach on real-world code, including challenging constructs such as nested definitions and macros. We test our approach on various datasets, and show that it achieves high boundary accuracy and semantic coherence, matching or outperforming chunkers that are based on much larger code-oriented LLMs. We also validate the improved performance of the downstream tasks on a few curated benchmarks.

Boris Nazarov, Darya Frolova, Shaked Leibzirer +1

Chunk Document Segmentation Semantic Segmentation

May 11, 2026cs.CV

Can Graphs Help Vision SSMs See Better?

Vision state space models inherit the efficiency and long-range modeling ability of Mamba-style selective scans. However, their performance depends critically on the representation of two-dimensional visual features as one-dimensional token sequences. Existing scan operators range from predefined geometric traversals to dynamic coordinate-based samplers that reroute tokens through predicted offsets and interpolation. While effective, these mechanisms primarily adapt paths or sampling locations, rather than explicitly modeling which local patches should exchange information before global state-space mixing. This motivates a simple question: \emph{can graphs help vision state space models see better?} We introduce \textbf{GraphScan}, a graph-induced dynamic scanning operator for Vision SSMs. For each token, GraphScan constructs a spatially bounded local graph, learns feature-conditioned affinities with relative positional bias, and produces the output token by one-step message passing over its semantic neighborhood. The resulting tokens are locally grounded before being processed by the selective SSM for global aggregation. GraphScan preserves token count and linear scaling in image size, while replacing coordinate-conditioned interpolation with feature-conditioned semantic routing. Integrated into a hierarchical backbone, \textbf{GraphScan-Mamba} achieves state-of-the-art performance among Vision SSMs across image classification, object detection, instance segmentation, and semantic segmentation, with modest computational overhead. Our analysis further shows that GraphScan induces interpretable displacement fields over the token lattice, providing a semantic and spatially grounded view of dynamic scanning. These results suggest that future Vision SSMs should treat scanning not merely as geometric serialization, but as learned local semantic routing before global state-space modeling.

Dhruv Parikh, Anvitha Ramachandran, Haoyang Fan +3

USC · UT Austin · DEVCOM ARL Army Research Office, USA

Vision State Space Models Semantic Segmentation

May 11, 2026cs.CV

DA-SegFormer: Damage-Aware Semantic Segmentation for Fine-Grained Disaster Assessment

Rapid and accurate damage assessment following natural disasters is critical for effective emergency response. However, identifying fine-grained damage levels (e.g., distinguishing minor from major roof damage) in UAV imagery remains challenging due to the degradation of texture cues during resizing and extreme class imbalance. We propose DA-SegFormer, a damage-aware adaptation of the SegFormer architecture optimized for high-resolution disaster imagery. Our method introduces a Class-Aware Sampling strategy to guarantee exposure to rare damage features, and it integrates Online Hard Example Mining (OHEM) with Dice Loss to dynamically focus on underrepresented classes. In addition, we employ a resolution-preserving inference protocol that maintains native texture details. Evaluated on the RescueNet dataset, DA-SegFormer achieves 74.61% mIoU, outperforming the baseline by 2.55%. Notably, our improvements yield double-digit gains in critical damage classes: Minor Damage (+11.7%) and Major Damage (+21.3%).

Kevin Zhu, William Tang, Raphael Hay Tene +3

Bina Labs, Lehigh University

Post-Disaster Damage Assessment Structural Damage Aerial Imagery Semantic Segmentation

May 10, 2026cs.CV

AtteConDA: Attention-Based Conflict Suppression in Multi-Condition Diffusion Models and Synthetic Data Augmentation

Recent conditional image generation methods can improve controllability by generating images that are faithful to conditions such as sketches, human poses, segmentation maps, and depth. By applying these techniques to image augmentation while preserving annotations, generated images can be used as additional training data and can improve recognition performance. However, for high-level driving tasks such as traffic-rule extraction and driving-behavior understanding, simply using annotations as conditions is insufficient. Instead, images must be augmented while preserving the detailed high-level structure of the original scene. One possible solution is to use multiple conditions so that generated images retain diverse structural cues after generation. However, when multiple conditions are used, conflicts among conditions can prevent reliable structure preservation. In this work, we input semantic segmentation, depth, and edges extracted from the original image into a multi-condition image generation model, thereby providing rich structural information as conditions. We further propose a modeling approach for handling conflicts among multiple conditions and show that it enables image generation with stronger structural preservation. We also build a generation framework and evaluation protocol for driving tasks, establishing a basis for comparison with prior and future models. As a result, this work contributes to image generation research by addressing condition conflicts in multi-condition generation and provides an important step toward mitigating data scarcity in high-level autonomous-driving tasks.

Shogo Noguchi

Gunma University

Text-Conditioned Diffusion Model Data Augmentation Semantic Segmentation Robotwin

May 9, 2026cs.CV

Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation requires adapting image-level vision-language models such as CLIP to dense pixel-level prediction, which is challenging due to the mismatch between hierarchical structure and semantic alignment in the embedding space. While recent works leverage hyperbolic geometry to model hierarchical relationships, they align embeddings across hierarchical levels but overlook semantic misalignment among embeddings within the same level. In this work, we propose HyRo, a hyperbolic fine-tuning framework that decouples hierarchical and semantic alignment in the Poincaré ball model. HyRo aligns hierarchical levels by adjusting the hyperbolic radius and refines semantic relationships through angular alignment using an orthogonal transformation that theoretically preserves the hyperbolic radius. Experiments on standard open-vocabulary semantic segmentation benchmarks demonstrate that HyRo achieves state-of-the-art performance over prior methods.

Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh

Fulbright University Vietnam

Hyperbolic Space Semantic Geometry Semantic Segmentation Open-Vocabulary Dense Prediction Continual Fine-Tuning

May 8, 2026cs.CV

Geometric Flood Depth Estimation: Fusing Transformer-Based Segmentation with Digital Elevation Models

Post-disaster situational awareness relies heavily on understanding both the extent and the volume of floodwaters. While 2D semantic segmentation provides accurate flood masking, it lacks the vertical dimension required to assess navigability and structural risk. This paper presents a geometric "Water Surface Elevation" approach for estimating flood depth from monocular aerial imagery. Our pipeline utilizes Mask2Former, a state-of-the-art transformer-based segmentation model, to generate precise 2D flood masks. These masks are fused with Digital Elevation Models (DEMs) to identify the water-land boundary, calculate a global water surface elevation (

Z_{water}

), and compute per-pixel depth based on the principle of local hydrostatic equilibrium. We evaluate this workflow using the FloodNet and CRASAR-U-DROIDS datasets, demonstrating how high-performance segmentation can be leveraged to extract 3D volumetric data from 2D imagery without the latency of hydrodynamic simulations.

Nhut Le, Ehsan Karimi, Maryam Rahnemoonfar

Lehigh University

Flood Detection Digital Elevation Models Aerial Imagery Semantic Segmentation Sea-Surface Temperature Data

May 8, 2026cs.CV

Explainable Part-Based Vehicle Classifier with Spatial Awareness

In the area of Intelligent Transportation Systems (ITS), fine-grained vehicle classification systems play an essential role. Recently, the authors have presented a novel vision-based classification approach in which standard end-to-end Convolutional Neural Networks (CNNs) have been decomposed into 1) a CNN-based detector for semantically strong vehicle parts, followed by 2) feature construction and 3) final classification by a decision tree. In contrast to conventional CNNs, this allows both easy extensibility to new vehicle categories - without the need to fully retrain the part detector - and an important step towards the interpretability of the model, removing partially the black-box nature inherent to CNNs. Here we present an important extension of this approach that now incorporates spatial awareness of the vehicle parts: while the feature construction 2) of the previous approach used a binary decision for each feature (present vs. absent), now a full spatial probability map is constructed to condition the presence of each individual part with respect to a given vehicle category. The classification is performed using a softmax regression approach for the overall vehicle probabilities. This method shows a considerably improved robustness against false (part-)detections, a point that is crucial for practical application. Comparative analyses with a state-of-the-art end-to-end CNN indicate that our part-based methods achieve comparable accuracy, effectively challenging the presumed trade-off between accuracy and explainability. This research represents a significant advance in vehicle classification for ITS and forms the basis for systems that combine high accuracy with intuitive interpretability.

Andreas Caduff, Klaus Zahn, Jonas Hofstetter +2

Competence Center for Intelligent Sensors and Networks, Lucerne University of Applied Science and Art · Competence Center for Intelligent Sensors and Networks, Lucerne University of Applied Science and Art, Technikumstr. 21, Horw, 6048, Switzerland. · SICK AG +1

Automotive Detection Semantic Segmentation Computer Vision Interpretability Spatial Reasoning

May 8, 2026cs.CV

UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition

Semantic segmentation of large-scale 3D point clouds is crucial for applications such as autonomous driving and urban digital twins. However, the sparse sampling pattern of LiDAR and the view-dependent geometric distortion in image observations complicate cross-modal alignment and hinder stable fusion. Inspired by the fact that 2D images captured by cameras are representations of the 3D world, we recognize that the features learned from 2D and 3D segmentation share some common semantics, while other aspects remain modality-specific. This insight motivates a unified multimodal framework for joint 2D-3D semantic segmentation. We combine a SAM-based vision encoder with a SPTNet-based geometric encoder to extract complementary semantic and geometric representations. The resulting features from both modalities are explicitly decomposed into shared and private subspaces, where the shared components summarize semantic factors common to both domains, and the private components preserve properties that are unique to each modality. A lightweight attention-based fusion module aggregates the shared features into a consistent cross-modal representation, and a regularized training objective ensures both semantic alignment and subspace independence. Experiments on the SemanticKITTI and nuScenes benchmarks demonstrate consistent improvements in segmentation accuracy over representative multimodal baselines, accompanied by competitive computational efficiency. Cross-domain evaluation on nuScenes USA-Singapore shows stable performance under distribution shifts, demonstrating strong generalization. The implementation code is publicly available at: https://github.com/shuaizhang69/UniD-Shift.

Shuai Zhang, Zhecheng Shi, Zhuxiao Li +4

1HKUST(GZ) · 2HKUST

Point Clouds Semantic Segmentation Cross-Modal Vision Encoders

May 7, 2026cs.CL

Linear Semantic Segmentation for Low-Resource Spoken Dialects

Semantic segmentation is a core component of discourse analysis, yet existing models are primarily developed and evaluated on high-resource written text, limiting their effectiveness on low-resource spoken varieties. In particular, dialectal Arabic exhibits informal syntax, code-switching, and weakly marked discourse structure that challenge standard segmentation approaches. In this paper, we introduce a new multi-genre benchmark (more than 1000 samples) for semantic segmentation in conversational Arabic, focusing on dialectal discourse. The benchmark covers transcribed casual telephone conversations, code-switched podcasts, broadcast news, and expressive dialogue from novels, and was annotated and validated by native Arabic annotators. Using this benchmark, we show that segmentation models performing well on MSA news genres degrade on dialectal transcribed speech. We further propose a segmentation model that targets local semantic coherence and robustness to discourse discontinuities, consistently outperforming strong baselines on dialectal non-news genres. The benchmark and approach generalize to other low-resource spoken languages.

Kirill Chirkunov, Younes Samih, Abed Alhakim Freihat +1

Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI

Dialects Arabic Multilingual Automatic Speech Recognition Low-Resource Languages Multilingual Benchmark Efficient Semantic Segmentation+3

May 7, 2026cs.CV

Leveraging Image Generators to Address Training Data Scarcity: The Gen4Regen Dataset for Forest Regeneration Mapping

Sustainable forest management relies on precise species composition mapping, yet traditional ground surveys are labour-intensive and geographically constrained. While Uncrewed Aerial Vehicles (UAVs) offer scalable data collection, the transition to deep learning-based interpretation is bottlenecked by the severe scarcity of expert-annotated imagery, particularly in complex, visually heterogeneous regeneration zones. This paper addresses the dual challenges of data scarcity and extreme class imbalance in the semantic segmentation of fine-grained forest regeneration species by providing a scalable framework that reduces reliance on manual photo-interpretation for high-resolution, millimetre-level aerial imagery. Importantly, we leverage the large-scale vision-language Nano Banana Pro model to simultaneously generate high-fidelity images and their corresponding pixel-aligned semantic masks from prompts. We introduce WilDReF-Q-V2, an expansion of a natural forest dataset with 13 977 new unlabelled and 50 labelled real images, as well as the Gen4Regen dataset, featuring 2101 pairs of synthetic images and semantic masks. Our methodology integrates real-world data with AI-generated images, highlighting that AI-generated data is highly complementary to real-world data, with unified training yielding an F1 score improvement of over 15 %pt compared to purely supervised baselines. Furthermore, we demonstrate that even small quantities of prompt-generated data significantly improve performance for underrepresented species, some of which saw per-species F1 score gains of up to 30 %pt. We conclude that vision-language models can serve as agile data generators, effectively bootstrapping perception tasks for niche AI domains where expert labels are scarce or unavailable. Our datasets, source code, and models will be available at https://norlab-ulaval.github.io/gen4regen.

Gabriel Jeanson, David-Alexandre Duclos, William Larrivée-Hardy +5

1Northern Robotics Laboratory, Université Laval, Québec, QC, G1V 0A6, Canada · 2Département d’informatique et de génie logiciel, Université Laval, Québec, QC, G1V 0A6, Canada

Vision Datasets Forest Biomass Image Generation Semantic Segmentation Training Data Yolo26

May 6, 2026cs.CV

DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation

Weakly Supervised Semantic Segmentation (WSSS) with image-level labels typically leverages Class Activation Maps (CAMs) to achieve pixel-level predictions. Recently, Contrastive Language-Image Pre-training (CLIP) has been introduced to generate CAMs in WSSS. However, previous WSSS methods solely adopt CLIP's vision-language paired property for dense localization, neglecting its inherently limited dense knowledge across both visual and text modalities, which renders CAM generation suboptimal. In this work, we propose DiCLIP, a novel WSSS framework that leverages the generative diffusion model to enhance CLIP's dense knowledge across two modalities. Specifically, Visual Correlation Enhancement (VCE) and Text Semantic Augmentation (TSA) modules are proposed for dense prediction enhancement. To improve the spatial awareness of visual features, our VCE module utilizes diffusion's reliable spatial consistency to mitigate the over-smoothing issue in CLIP's attention. It designs the Attention Clustering Refinement (ACR) module to reliably extract diverse correlation maps from the diffusion model. The correlation maps act as a diversity bias for CLIP's self-attention, recursively pushing its visual features towards a more discriminative dense distribution. To augment the semantics of text embeddings, our TSA module argues that a single text modality is insufficient to encompass the variability of visual categories. Thus, we leverage diffusion's generative power to maintain a dynamic key-value cache model, shifting CAM generation from a patch-text matching mechanism to a novel visual knowledge retrieval paradigm. With these enhancements, DiCLIP not only outperforms state-of-the-art methods on PASCAL VOC and MS COCO but also significantly reduces training costs. Code is publicly available at https://github.com/zwyang6/DiCLIP.

Zhiwei Yang, Pengfei Song, Yucong Meng +3

Academy for Engineering and Technology, Fudan University, Shanghai 200433, China · Shandong Computer Science Center, China · Digital Medical Research Center, School of Basic Medical Science, Fudan University, and also with Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China +1

Semantic Segmentation Class Activation Mapping Image Enhancement Diffusion Models

May 5, 2026cs.CV

RD-ViT: Recurrent-Depth Vision Transformer for Semantic Segmentation with Reduced Data Dependence Extending the Recurrent-Depth Transformer Architecture to Dense Prediction

Vision Transformers (ViTs) achieve state-of-the-art segmentation accuracy but require large training datasets because each layer has unique parameters that must be learned independently. We present RD-ViT, a Recurrent-Depth Vision Transformer that adapts the Recurrent-Depth Transformer (RDT) architecture to dense prediction tasks, supporting both 2D and 3D inputs. RD-ViT replaces the deep stack of unique transformer blocks with a single shared block looped T times, augmented with LTI-stable state injection for guaranteed convergence, Adaptive Computation Time (ACT) for spatial compute allocation, depth-wise LoRA adaptation, and optional Mixture-of-Experts (MoE) feed-forward networks for category-specific specialization. We evaluate on the ACDC cardiac MRI segmentation benchmark in both 2D slice-level and 3D volumetric settings with exclusively real experiments executed in Google Colab. In 2D, RD-ViT outperforms standard ViT at 10% training data (Dice 0.774 vs 0.762) and at full data (0.882 vs 0.872). In 3D, RD-ViT with MoE achieves Dice 0.812 with 3.0M parameters, reaching 99.4% of standard ViT performance (0.817) at 53% of the parameter count. MoE expert utilization analysis reveals that different experts spontaneously specialize for different cardiac structures (RV, MYO, LV) without explicit routing supervision. ACT halting maps show higher compute allocation at cardiac boundaries, and the mean ponder time decreases from 2.6 to 1.4 iterations during training, demonstrating learned computational efficiency. Depth extrapolation enables inference with more loops than training without degradation. All code, notebooks, and results are publicly released.

Renjie He

Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA

Self-Supervised Vision Transformers Dense Prediction Semantic Segmentation Transformer Architectures

May 5, 2026cs.CV

MILE: Mixture of Incremental LoRA Experts for Continual Semantic Segmentation across Domains and Modalities

Continual semantic segmentation requires models to adapt to new domains or modalities without sacrificing performance on previously learned tasks. Expert-based learning, in which task-specific modules specialize in different domains, has proven effective in mitigating forgetting. These methods include dynamic expansion, which suffers from scalability issues, or parameter isolation, which constrains the ability to learn new tasks. We introduce Mixture of Incremental LoRA Experts (MILE), a modular and parameter-efficient framework for continual segmentation across both domains and modalities. MILE leverages Low-Rank Adaptation (LoRA) to instantiate lightweight experts for each new task while keeping the pretrained base network frozen. Each expert is trained exclusively on its task data, thus avoids overwriting previously learned information. A prototype-guided gating mechanism dynamically selects the most appropriate expert at inference. MILE achieves the benefits of expert-based learning while overcoming its scalability limitations. It requires only a marginal parameter increase per task and tens of LoRA adapters are needed before matching the size of a single full model, making it highly efficient in both training and storage. Across domain- and modality-incremental benchmarks, MILE achieves strong performance while ensuring better stability, plasticity, and scalability.

Shishir Muralidhara, Didier Stricker, René Schuster

German Research Center for Artificial Intelligence (DFKI), Trippstadter Str. 122, 67663 Kaiserslautern, Germany · University of Kaiserslautern-Landau (RPTU), Gottlieb-Daimler-Str. 47, 67663 Kaiserslautern, Germany

Efficient Semantic Segmentation Semantic Segmentation Multimodal Benchmarks Mixtures Discontinuous Gating

May 5, 2026cs.CV

TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation

Attacking semantic segmentation models is significantly harder than image classification models because an attacker must flip thousands of pixel predictions simultaneously. Standard pixel-wise cross-entropy (CE) is ill-suited to this setting: it tends to overemphasize already-misclassified pixels, which slows optimization and overstates model robustness. To address these issues, we introduce TsallisPGD, an adversarial attack built on the Tsallis cross-entropy, a generalization of CE parameterized by

q

, which adaptively reshapes the gradient landscape by controlling gradient concentration across pixels. By varying

q

, we steer the attack toward pixels at different confidence levels. We first show that no single fixed-

q

is universally optimal, as its effectiveness depends on the dataset, model architecture, and perturbation budget. Motivated by this, we propose a dynamic

q

-schedule that sweeps

q

during optimization. Extensive experiments on Cityscapes, Pascal VOC, and ADE20K show that TsallisPGD, using a single validation-selected schedule, achieves the best average attack rank across all evaluated settings and improves over CEPGD, SegPGD, CosPGD, JSPGD, and MaskedPGD in reducing accuracy and mIoU on both standard and robust models.

Alexander Matyasko, Xin Lou, Indriyati Atmosukarto +1

Singapore Institute of Technology, Singapore

Efficient Semantic Segmentation Semantic Segmentation Pixel-Level Prediction Cross-Entropy Losses Weight Decay Optimization Landscape

May 5, 2026cs.CV

Dual-Foundation Models for Unsupervised Domain Adaptation

Semantic segmentation provides pixel-level scene understanding essential for autonomous driving and fine-grained perception tasks. However, training segmentation models requires costly, labor-intensive annotations on real-world datasets. Unsupervised Domain Adaptation (UDA) addresses this by training models on labeled synthetic data and adapting them to unlabeled real images. While conceptually simple, adaptation is challenging due to the domain gap, i.e., differences in visual appearance and scene structure between synthetic and real data. Prior approaches bridge this gap through pixel-level mixing or feature-level contrastive learning. Yet, these techniques suffer from two major limitations: (1) reliance on high-confidence pseudo-labels restricts learning to a subset of the target domain, and (2) prototype-based contrastive methods initialize class prototypes from source-trained models, yielding biased and unstable anchors during adaptation. To address these issues, we propose a dual-foundation UDA framework that leverages two complementary foundation models. First, we employ the Segment Anything Model (SAM) with superpixel-guided prompting to enable learning from a broader range of target pixels beyond high-confidence predictions. Second, we incorporate DINOv3 to construct stable, domain-invariant class prototypes through its robust representation learning. Our method achieves consistent improvements of +1.3% and +1.4% mIoU over strong UDA baselines on GTA-to-Cityscapes and SYNTHIA-to-Cityscapes, respectively.

Yerin Cheon, Aruna Balasubramanian, Francois Rameau

Stony Brook University, Stony Brook, NY, USA · SUNY Korea, The State University of New York, Korea

Domain Adaptation Semantic Segmentation Recent Vision Foundation Models Wireless Foundation Models Fine-Grained Perception Pixels

May 4, 2026cs.CV

DINO Soars: DINOv3 for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagery

The remote sensing (RS) domain suffers from a lack of densely labeled datasets, which are costly to obtain. Thus, models that can segment RS imagery well without supervised fine-tuning are valuable, but existing solutions fall behind supervised methods. Recently, DINOv3 surpassed SOTA RS foundation models on the GEO-bench segmentation benchmark without pre-training on RS data. Additionally, DINO.txt has enabled open vocabulary semantic segmentation (OVSS) with the DINOv3 backbone. We leverage these developments to form an OVSS model for RS imagery, free of RS-domain fine-tuning. Our model, CAFe-DINO (Cost Aggregation + Feature Upsampling with DINO) exploits the strong OVSS performance of DINOv3 for RS imagery via cost aggregation and training-free upsampling of text-image similarity scores. The robust latent of the DINOv3 backbone eliminates the need for fine-tuning on RS imagery; we instead fine-tune our model on a RS-targeted subset of COCO-Stuff. CAFe-DINO achieves state-of-the-art performance on key RS segmentation datasets, outperforming OVSS methods fine-tuned on RS data. Our code and data are publicly available at https://github.com/rfaulk/DINO_Soars.

Ryan Faulkenberry, Saurabh Prasad

University of Houston

Remote Sensing Image Segmentation Dinov3 Remote Sensing Semantic Segmentation Recent Vision Foundation Models Dino+1

May 4, 2026cs.CV

FoR-Net: Focus-on-Regions Network for Semantic Segmentation

This paper presents Focus-on-Regions Network (FoR-Net), an efficient semantic segmentation framework that explicitly focuses on hard regions through a selector-driven Top-K mechanism. Instead of relying on heavy global modeling, FoR-Net selectively enhances structurally informative regions. Multi-scale reasoning branches are introduced to aggregate spatial context efficiently. Experiments on the Cityscapes dataset demonstrate that FoR-Net achieves competitive performance while maintaining a lightweight architecture.

Sheng-Wei Chan, Hsin-Jui Pan, Jen-Shiun Chiang

Department of Electrical and Computer Engineering, Tamkang University, No.151, Yingzhuan Rd., Tamsui Dist., New Taipei City, 251301, Taiwan

Efficient Semantic Segmentation Semantic Segmentation Semantic Region Receptive Fields Hard

May 3, 2026cs.CV

EAPFusion: Intrinsic Evolving Auxiliary Prior Guidance for Infrared and Visible Image Fusion

Infrared-visible image fusion aims to create an information-rich fused image by integrating the complementary thermal saliency from infrared sensing and fine textures from visible imaging. Such accurate fusion is essential for real-world perception applications in complex scenes, including nighttime autonomous driving, search and rescue, and surveillance, and can further benefit downstream tasks such as semantic segmentation. However, most existing fusion methods rely upon static trained weights that cannot adapt to scene-specific content at inference time, and often suffer from a granularity mismatch when coarse auxiliary semantics are injected, which makes it difficult to simultaneously highlight targets and preserve details. In this work, we propose EAPFusion to address these issues by using self-evolving intrinsic priors instead of relying on external auxiliary models. Concretely, EAPFusion maintains a compact set of intrinsic priors and progressively updates them across scales. These evolved priors are utilized to dynamically generate convolutional kernels, shifting the paradigm from fixed, pre-trained filters to instance-adaptive parameters via prior-conditioned dynamic convolution. Furthermore, we design a channel-level fusion module that shuffles and interleaves infrared and visible channels, applying local channel mixing to boost cross-modal complementarity. Experiments on different datasets, including cross-dataset evaluation and semantic segmentation, show that the proposed method achieves state-of-the-art quantitative and qualitative fusion results, and consistently boosts downstream performance. Code is coming soon.

Zhenyu Sun, Luobin Zhang, Axi Niu +2

Northwestern Polytechnical University, Xi’an, China

Infrared-Visible Image Fusion Visible Image Fusion Adm-Fusion Infrared Semantic Segmentation

Apr 30, 2026cs.LG

OTSS: Output-Targeted Soft Segmentation for Contextual Decision-Weight Learning

Many machine learning systems make constrained decisions by optimizing factorized objectives, but the context-specific objective is often treated as fixed. We study contextual decision-weight learning: from logged decisions and proxy outputs, learn an optimizer-facing weight vector w(x) over interpretable decision factors z(x,d), rather than a direct policy or generic predictive score. We propose OTSS, an output-targeted soft-segmentation model that deploys the personalized decision-ready weight vector. At the function-class level, the theory highlights a hard-versus-soft distinction. Hard partitions incur an approximation-estimation tradeoff under overlap, while a realizable fixed-K soft class removes the hard-partition approximation floor and attains a parametric rate. We evaluate OTSS in controlled benchmarks with finite evaluation libraries, where the true weight vector and downstream regret can be computed exactly. In the representative overlap setting, OTSS attains the lowest mean regret among the comparators, including EM mixture regression, the strongest soft-mixture baseline in our comparison; it matches EM on coefficient recovery while running about two orders of magnitude faster. In a matched K=5 benchmark, OTSS remains competitive under hard-routed truth and improves as heterogeneity becomes softer and sample size grows. On a fixed Complete Journey retail anchor with real household covariates and action geometry, OTSS again achieves the lowest mean-regret point estimate.

Renjun Hu, Hyun-Soo Ahn

University of Michigan, Ross School of Business

Objectives Semantic Segmentation Hard

Apr 30, 2026cs.CV

Noise2Map: End-to-End Diffusion Model for Semantic Segmentation and Change Detection

Semantic segmentation and change detection are two fundamental challenges in remote sensing, requiring models to capture either spatial semantics or temporal differences from satellite imagery. Existing deep learning models often struggle with temporal inconsistencies or in capturing fine-grained spatial structures, require extensive pretraining, and offer limited interpretability - especially in real-world remote sensing scenarios. Recent advances in diffusion models show that Gaussian noise can be systematically leveraged to learn expressive data representations through denoising. Motivated by this, we investigate whether the noise process in diffusion models can be effectively utilized for discriminative tasks. We propose Noise2Map, a unified diffusion-based framework that repurposes the denoising process for fast, end-to-end discriminative learning. Unlike prior work that uses diffusion only for generation or feature extraction, Noise2Map directly predicts semantic or change maps using task-specific noise schedules and timestep conditioning, avoiding the costly sampling procedures of traditional diffusion models. The model is pretrained via self-supervised denoising and fine-tuned with supervision, enabling both interpretability and robustness. Our architecture supports both tasks (SS and CD) through a shared backbone and task-specific noise schedulers. Extensive evaluations on the SpaceNet7, WHU, and xView2 buildings damaged by wildfires datasets demonstrate that Noise2Map ranks on average 1st among seven models on semantic segmentation and 1st on change detection by a cross-dataset rank metric (average F1 primary, IoU tie-break). Ablation studies highlight the robustness of our model against different training noise schedulers and timestep control in the diffusion process, as well as the ability of the model to perform multi-task learning.

Ali Shibli, Andrea Nascetti, Yifang Ban

Division of Geoinformatics, School of Architecture and Built Environment, KTH Royal Institute of Technology, 11428 Stockholm, Sweden

Change Detection Semantic Segmentation Diffusion Models Iterative Denoising Methods Satellite Imagery

Apr 30, 2026cs.CV

A generalised pre-training strategy for deep learning networks in semantic segmentation of remotely sensed images

In the segmentation of remotely sensed images, deep learning models are typically pre-trained using large image databases like ImageNet before fine-tuned on domain-specific datasets. However, the performance of these fine-tuned models is often hindered by the large domain gaps (i.e., differences in scenes and modalities) between ImageNet's images and remotely sensed images being processed. Therefore, many researchers have undertaken efforts to establish large-scale domain-specific image datasets for pre-training, aiming to enhance model performance. However, establishing such datasets is often challenging, requiring significant effort, and these datasets often exhibit limited generaliza-bility to other application scenarios. To address these issues, this study introduces a novel yet simple pre-training strategy designed to guide a model away from learning domain-specific features in a pre-training dataset during pre-training, thereby improving the generalisation ability of the pre-trained model. To evaluate the strategy's effectiveness, deep learning models are pre-trained on ImageNet and subsequently fine-tuned on four semantic segmentation datasets with diverse scenes and modalities, including iSAID, MFNet, PST900 and Potsdam. Experimental results show that the proposed pre-training strategy led to state-of-the-art accuracies on all four datasets, namely 67.4% mIoU for iSAID, 56.9% mIoU for MFNet, 84.22% mIoU for PST900, 91.88% mF1 for Potsdam. This research lays the groundwork for developing a unified foundation model applicable to both computer vision and remote sensing applications.

Yuan Fang, Yuanzhi Cai, Jagannath Aryal +4

Department of Civil Engineering, Xi’an Jiaotong-Liverpool University, Suzhou 215000, China · CSIRO Mineral Resources, Kensington, WA 6151, Australia · Department of Infrastructure Engineering, The University of Melbourne, Melbourne VIC 3010, Australia

Remote Sensing Image Segmentation Remote Sensing Semantic Segmentation Pretraining Imagenet Large-Scale Image Datasets

Apr 29, 2026cs.CV

Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and A Large-scale Benchmark

Fine-grained RGBT image semantic segmentation is crucial for all-weather unmanned aerial vehicle (UAV) scene understanding. However, UAV RGBT image semantic segmentation faces two coupled challenges: cross-modal spatial misalignment caused by sensor parallax and platform vibration, and severe semantic confusion among fine-grained ground objects under top-down aerial views. To address these issues, we propose a Graph-based Semantic Calibration Network (GSCNet) for unaligned UAV RGBT image semantic segmentation. Specifically, we design a Feature Decoupling and Alignment Module (FDAM) that decouples each modality into shared structural and private perceptual components and performs deformable alignment in the shared subspace, enabling robust spatial correction with reduced modality appearance interference. Moreover, we propose a Semantic Graph Calibration Module (SGCM) that explicitly encodes the hierarchical taxonomy and co-occurrence regularities among ground-object categories in UAV scenes into a structured category graph, and incorporates these priors into graph-attention reasoning to calibrate predictions of visually similar and rare categories. In addition, we construct the Unaligned RGB-Thermal Fine-grained (URTF) benchmark, to the best of our knowledge, the largest and most fine-grained benchmark for unaligned UAV RGBT image semantic segmentation, containing over 25,000 image pairs across 61 semantic categories with realistic cross-modal misalignment. Extensive experiments on URTF demonstrate that GSCNet significantly outperforms state-of-the-art methods, with notable gains on fine-grained categories. The dataset is available at https://github.com/mmic-lcl/Datasets-and-benchmark-code.

Fangqiang Fan, Zhicheng Zhao, Xiaoliang Ma +2

Key Laboratory of Intelligent Computing & Signal Processing (Anhui University), Ministry of Education, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Artificial Intelligence, Anhui University, Hefei 230601, China · School of Computer Science and Technology, Anhui University, Hefei 230601, China · GEOVIS Earth Technology Co., Ltd., Hefei 230088, China +1

Aerial Imagery Semantic Segmentation U-Net

Apr 29, 2026cs.CV

Cross-Domain Transfer of Hyperspectral Foundation Models

Hyperspectral imaging (HSI) semantic segmentation typically relies on in-domain training, but limited data availability often restricts model performance in real-world applications. Current approaches to leverage foundation models in proximal sensing use cross-modality techniques, bridging RGB and HSI to exploit vision foundation models. However, these methods either discard spectral information or introduce architectural complexity. We propose cross-domain transfer as an alternative, reusing HSI foundation models - originally trained in remote sensing - for proximal sensing applications. By eliminating the need to bridge modality gaps, our approach preserves spectral information while maintaining a simple architecture. Using the HS3-Bench benchmark, we systematically evaluate and compare conventional in-domain, in-modality training, cross-modality transfer and cross-domain transfer strategies. Our results demonstrate that cross-domain transfer achieves large performance improvements over in-domain, in-modality training, reduces the performance gap to cross-modality approaches and maintains strong performance in limited data settings. Thus, this work advances more effective HSI semantic segmentation in diverse applications.

Nick Theisen, Peer Neubert

Intelligent Autonomous Systems, University of Koblenz, Germany

Hyperspectral Image Cross-Modality Recent Vision Foundation Models Semantic Segmentation In-Domain Accuracy Foundation Model

Apr 29, 2026cs.CV

Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation

Open-vocabulary semantic segmentation (OVSS) in remote sensing images is a promising task that employs textual descriptions for identifying undefined land cover categories. Despite notable advances, existing methods typically employ a static inference paradigm, overlooking the distinct distribution of each scene, resulting in semantic ambiguity in diverse land covers and incomplete foreground activation. Motivated by this, we propose Seeking Consensus, termed SeeCo, a plug-and-play framework to boost the performance of training-free OVSS models in remote sensing images, which recalibrates arbitrary OVSS models on-the-fly by seeking dual consensus: geometric consensus learning (GCL) through multi-view consistent observations and semantic consensus learning (SCL) via textual description adaptive calibration, which assists collaborative recalibration of visual and textual semantics. The two consensus are injected via an online consensus injector (OCI), effectively alleviating the under-activation and semantic bias. SeeCo requires no specific training process, yet recalibrates semantic-geometric alignment for each unique scene during inference. Extensive experiments on eight remote sensing OVSS benchmarks show consistent gains, proving its effectiveness and universality.

Guanchun Wang, Chenxiao Wu, Xiangrong Zhang +4

School of Artificial Intelligence, Xidian University · MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University

Remote Sensing Image Segmentation Semantic Segmentation Open-Vocabulary Land Cover

Apr 28, 2026cs.CV

The Surprising Effectiveness of Canonical Knowledge Distillation for Semantic Segmentation

Recent knowledge distillation (KD) methods for semantic segmentation introduce increasingly complex hand-crafted objectives, yet are typically evaluated under fixed iteration schedules. These objectives substantially increase per-iteration cost, meaning equal iteration counts do not correspond to equal training budgets. It is therefore unclear whether reported gains reflect stronger distillation signals or simply greater compute. We show that iteration-based comparisons are misleading: when wall-clock compute is matched, canonical logit- and feature-based KD outperform recent segmentation-specific methods. Under extended training, feature-based distillation achieves state-of-the-art ResNet-18 performance on Cityscapes and ADE20K. A PSPNet ResNet-18 student closely approaches its ResNet-101 teacher despite using only one quarter of the parameters, reaching 99% of the teacher's mIoU on Cityscapes (79.0 vs 79.8) and 92% on ADE20K. Our results challenge the prevailing assumption that KD for segmentation requires task-specific mechanisms and suggest that scaling, rather than complex hand-crafted objectives, should guide future method design.

Muhammad Ali, Kevin Alexander Laube, Madan Ravi Ganesh +3

University of Freiburg · Bosch Center for Artificial Intelligence · Aleph Alpha Research +1

Knowledge Distillation Semantic Segmentation Probe-Logit Distillation Resnet-18

Apr 27, 2026eess.IV

Semantic Segmentation for Histopathology using Learned Regularization based on Global Proportions

In pathology, the spatial distribution and proportions of tissue types are key indicators of disease progression, and are more readily available than fine-grained annotations. However, these assessments are rarely mapped to pixel-wise segmentation. The task is fundamentally underdetermined, as many spatially distinct segmentations can satisfy the same global proportions in the absence of pixel-wise constraints. To address this, we introduce Variational Segmentation from Label Proportions (VSLP), a two-stage framework that infers dense segmentations from global label proportions, without any pixel-level annotations. This framework first leverages a pre-trained transformer model with test-time augmentation to produce a pixel-wise confidence estimate. In the second stage, these estimates are fused by solving a variational optimization problem that incorporates a Wasserstein data fidelity term alongside a learned regularizer. Unlike end-to-end networks, our variational method can visualize the fidelity-regularization energy, resulting in more interpretable segmentation. We validate our approach on two public datasets, achieving superior performance over existing weakly supervised and unsupervised methods. For one of these datasets, proportions have been estimated by an experienced pathologist to provide a realistic benchmark to the community. Furthermore, the method scales to an in-house dataset with noisy pathologist labels, severely outperforming state-of-the-art methods, thereby demonstrating practical applicability. The code and data will be made publicly available upon acceptance at https://github.com/xiaoliangpi/VSLP.

Yangping Li, Thomas Pinetz, Michael Hölzel +2

Institute of Pathology, University Hospital Bonn, Sigmund-Freud-Str. 25, 53127 Bonn, Germany · Institute for Applied Mathematics, University of Bonn, Endenicher Allee 60, 53115 Bonn, Germany · Institute of Experimental Oncology, University Hospital Bonn, Sigmund-Freud-Str. 25, 53127 Bonn, Germany

Digital Pathology Semantic Segmentation Pixel-Level Prediction Regularization

Apr 27, 2026cs.CV

Instance Awareness of Multi-class Semantic Segmentation Loss Functions

Instance-sensitive losses for semantic segmentation such as blob loss and CC loss were designed to address instance imbalance, ensuring small lesions generate the same gradient as large ones, but operate only on single-class segmentation. In multi-class settings, class imbalance poses an additional problem: rare classes with few instances receive a disproportionately small share of the training signal. We show that extending instance-sensitive losses to multi-class segmentation via a one-vs-rest class decomposition repurposes them to also address class imbalance, as uniform averaging over classes ensures each class contributes equally regardless of frequency. We further show that inverse-size weighting, which destabilizes training when applied globally due to weight imbalances across rare and common classes, becomes effective when integrated within the per-component loss, confining the reweighting to each component's spatial context. On the BraTS-METS 2025 dataset (260 test cases), multi-class CC loss improves foreground Dice (0.64 +/- 0.26 vs. 0.59 +/- 0.27 baseline) and rare-class Dice, while maintaining Panoptic Quality at DSC threshold 0.5. Multi-class blob loss achieves the best Panoptic Quality at threshold 0.5 (0.40 +/- 0.24 vs. 0.38 +/- 0.25 baseline) and recognition quality (0.53 +/- 0.29 vs. 0.49 +/- 0.30). Integrating inverse-size weighting within the per-component loss increases rare-class Dice to 0.44 +/- 0.36 at the cost of reduced detection quality.

Soumya Snigdha Kundu, Florian Kofler, Marina Ivory +3

1King’s College London · University of Tübingen · 3Technical University of Munich +1

Amodal Segmentation Multiple Instance Learning Semantic Segmentation Loss Function Detection Performance Similarity and Metrics

Apr 27, 2026cs.CV

Graph-augmented Segmentation of Complex Shapes in Laser Powder bed Fusion for Enhanced In Situ Inspection

The technological maturity of in situ inspection and monitoring methods in additive manufacturing is steadily increasing, enabling more efficient and practical qualification procedures. In this context, image segmentation of powder bed images in Laser Powder Bed Fusion (L-PBF) has been investigated by various authors, leveraging both edge detection and machine learning approaches to identify deviations from nominal geometry. Despite these developments, several challenges remain, including the sensitivity of segmentation performance to industrial illumination conditions and layer-to-layer variability in pixel intensity patterns. The study addresses these limitations by proposing a graph-augmented segmentation approach. The underlying principle consists of preserving the geometrical information at a global level rather than at pixel-wise level, modeling dependencies and relational information among spatial regions with a Graph Neural Network bottleneck embedded into a U-Net architecture. This allows enhancing the consistency and accuracy of the geometry reconstruction in the presence of spatial and layer-wise photometric variability systematically faced in real data. The method is evaluated against benchmark techniques for the in situ reconstruction of lattice structures produced by L-PBF, demonstrating its potential as a scalable solution for robust in situ inspection and geometric verification in industrial environments.

Stefano Raimondo, Matteo Bugatti, Marco Grasso

Department of Mechanical Engineering, Politecnico di Milano, Milan, Italy · Department of Mechanical Engineering, Politecnico di Milano, Via La Masa 1, 20156 Milan, Italy

Inspection Semantic Segmentation Enhancement

Apr 27, 2026cs.CV

Open-Vocabulary Semantic Segmentation Network Integrating Object-Level Label and Scene-Level Semantic Features for Multimodal Remote Sensing Images

Semantic segmentation of multi-modal remote sensing imagery plays a pivotal role in land use/land cover (LULC) mapping, environmental monitoring, and precision earth observation. Current multi-modal approaches mainly focus on integrating complementary visual modalities, yet neglect the incorporating of non-visual textual data - a rich source of knowledge that can bridge semantic gaps between visual patterns and real-world concepts. To address this limitation, we propose TSMNet, a text supervised multi-modal open vocabulary semantic segmentation network that synergistically integrates textual supervision with visual representation for open-vocabulary semantic segmentation. Unlike conventional multi-modal segmentation frameworks, TSMNet introduces a dual-branch text encoder to extract both scene-level semantic and object-level label information from various textual data, enabling dynamic cross-modal fusion. These text-derived features dynamically interact with visual embeddings through the proposed text-guided visual semantic fusion module, enabling domain-aware feature refinement and human-interpretable decision-making. To verify our method, we innovatively construct two new multi-modal datasets, and carry out extensive experiments to make a comprehensive comparison between the proposed method and other state-of-the-art (SOTA) semantic segmentation models. Results demonstrate that TSMNet achieves superior segmentation accuracy while exhibiting robust generalization capabilities across diverse geographical and sensor-specific scenarios. This work establishes a new paradigm for explainable remote sensing analysis, demonstrating that textual knowledge integration significantly enhances model generalizability. The source code will be available at https://github.com/yeyuanxin110/TSMNet

Jinkun Dai, Yuanxin Ye, Peng Tang +4

Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · State-Province Joint Engineering Laboratory of Spatial Information Technology for High-Speed Railway Safety, Southwest Jiaotong University, Chengdu 611756, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China +1

Remote Sensing Image Segmentation Remote Sensing Semantic Segmentation Land Cover Multimodal Dataset

Apr 25, 2026cs.CV

Breaking the Resource Wall: Geometry-Guided Sequence Modeling for Efficient Semantic Segmentation

High-performance semantic segmentation has achieved significant progress in recent years, often driven by increasingly large backbones and higher computational budgets. While effective, such approaches introduce substantial computational overhead and limit accessibility under constrained hardware settings. In this paper, we propose DGM-Net (Directional Geometric Mamba Network), an efficient architecture that improves modeling capability through structural design rather than increasing model capacity. We introduce Directional Geometric Mamba (G-Mamba), a linear-complexity O(N) operator as an alternative to conventional context modeling modules such as ASPP and PPM. To further enhance structural awareness in state space model (SSM)-based modeling, we design the DGM-Module, which extracts centripetal flow fields and topological skeletons to guide the scanning process and improve boundary preservation. Without relying on large-scale pretraining or heavy backbone scaling, DGM-Net achieves 80.8% mIoU within 28k iterations, 82.3% mIoU on Cityscapes test set, and 45.24% mIoU on ADE20K. In addition, the model maintains stable performance under constrained hardware settings (e.g., batch size of 2 on 8GB VRAM), highlighting its efficiency and practicality. These results demonstrate that incorporating geometric guidance into SSM-based architectures provides an effective and resource-efficient direction for semantic segmentation.

Sheng-Wei Chan, Hsin-Jui Pan, Chun-Po Shen +3

Department of Electrical and Computer Engineering, Tamkang University

Efficient Semantic Segmentation Semantic Segmentation Sequence Modeling Boundaries

Apr 20, 2026cs.CV

Feasibility of Indoor Frame-Wise Lidar Semantic Segmentation via Distillation from Visual Foundation Model

Frame-wise semantic segmentation of indoor lidar scans is a fundamental step toward higher-level 3D scene understanding and mapping applications. However, acquiring frame-wise ground truth for training deep learning models is costly and time-consuming. This challenge is largely addressed, for imagery, by Visual Foundation Models (VFMs) which segment image frames. The same VFMs may be used to train a lidar scan frame segmentation model via a 2D-to-3D distillation pipeline. The success of such distillation has been shown for autonomous driving scenes, but not yet for indoor scenes. Here, we study the feasibility of repeating this success for indoor scenes, in a frame-wise distillation manner by coupling each lidar scan with a VFM-processed camera image. The evaluation is done using indoor SLAM datasets, where pseudo-labels are used for downstream evaluation. Also, a small manually annotated lidar dataset is provided for validation, as there are no other lidar frame-wise indoor datasets with semantics. Results show that the distilled model achieves up to 56% mIoU under pseudo-label evaluation and around 36% mIoU with real-label, demonstrating the feasibility of cross-modal distillation for indoor lidar semantic segmentation without manual annotations.

Haiyang Wu, Juan J. Gonzales Torres, George Vosselman +1

Department of Earth Observation Science, Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, 7522 NB Enschede, The Netherlands

Lidar Semantic Segmentation Vision Foundation Models Dataset Distillation

Apr 18, 2026cs.CV

Lorentz Framework for Semantic Segmentation

Semantic segmentation in hyperbolic space enables compact modeling of hierarchical structure while providing inherent uncertainty quantification. Prior approaches predominantly rely on the Poincaré ball model, which suffers from numerical instability, optimization, and computational challenges. We propose a novel, tractable, architecture-agnostic semantic segmentation framework (pixel-wise and mask classification) in the hyperbolic Lorentz model. We employ text embeddings with semantic and visual cues to guide hierarchical pixel-level representations in Lorentz space. This enables stable and efficient optimization without requiring a Riemannian optimizer, and easily integrates with existing Euclidean architectures. Beyond segmentation, our approach yields free uncertainty estimation, confidence map, boundary delineation, hierarchical and text-based retrieval, and zero-shot performance, reaching generalized flatter minima. We introduce a novel uncertainty and confidence indicator in Lorentz cone embeddings. Further, we provide analytical and empirical insights into Lorentz optimization via gradient analysis. Extensive experiments on ADE20K, COCO-Stuff-164k, Pascal-VOC, and Cityscapes, utilizing state-of-the-art per-pixel classification models (DeepLabV3 and SegFormer) and mask classification models (mask2former and maskformer), validate the effectiveness and generality of our approach. Our results demonstrate the potential of hyperbolic Lorentz embeddings for robust and uncertainty-aware semantic segmentation. Code is available at https://github.com/mxahan/Lorentz_semantic_segmentation.

Zahid Hasan, Masud Ahmed, Nirmalya Roy

Department of Information Systems, University of Maryland, Baltimore County

Efficient Semantic Segmentation Semantic Segmentation Hyperbolic Space Semantic Geometry Pixel-Level Prediction

Apr 17, 2026cs.CV

SENSE: Stereo OpEN Vocabulary SEmantic Segmentation

Open-vocabulary semantic segmentation enables models to segment objects or image regions beyond fixed class sets, offering flexibility in dynamic environments. However, existing methods often rely on single-view images and struggle with spatial precision, especially under occlusions and near object boundaries. We propose SENSE, the first work on Stereo OpEN Vocabulary SEmantic Segmentation, which leverages stereo vision and vision-language models to enhance open-vocabulary semantic segmentation. By incorporating stereo image pairs, we introduce geometric cues that improve spatial reasoning and segmentation accuracy. Trained on the PhraseStereo dataset, our approach achieves strong performance in phrase-grounded tasks and demonstrates generalization in zero-shot settings. On PhraseStereo, we show a +2.9% improvement in Average Precision over the baseline method and +0.76% over the best competing method. SENSE also provides a relative improvement of +3.5% mIoU on Cityscapes and +18% on KITTI compared to the baseline work. By jointly reasoning over semantics and geometry, SENSE supports accurate scene understanding from natural language, essential for autonomous robots and Intelligent Transportation Systems.

Thomas Campagnolo, Ezio Malis, Philippe Martinet +1

ACENTAURI · Centre Inria d’Universite Cote d’Azur, France · NXP Semiconductors, France

Open-Vocabulary Segmentation Semantic Segmentation Scene Reasoning Video Object Segmentation Geometric Cues

Apr 17, 2026cs.CV

Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection

Multimodal remote sensing data provide complementary information for semantic segmentation, but in real-world deployments, some modalities may be unavailable due to sensor failures, acquisition issues, or challenging atmospheric conditions. Existing multimodal segmentation models typically address missing modalities by learning a shared representation across inputs. However, this approach can introduce a trade-off by compromising modality-specific complementary information and reducing performance when all modalities are available. In this paper, we tackle this limitation with CBC-SLP, a multimodal semantic segmentation model designed to preserve both modality-invariant and modality-specific information. Inspired by the theoretical results on modality alignment, which state that perfectly aligned multimodal representations can lead to sub-optimal performance in downstream prediction tasks, we propose a novel structured latent projection approach as an architectural inductive bias. Rather than enforcing this strategy through a loss term, we incorporate it directly into the architecture. In particular, to use the complementary information effectively while maintaining robustness under random modality dropout, we structure the latent representations into shared and modality-specific components and adaptively transfer them to the decoder according to the random modality availability mask. Extensive experiments on three multimodal remote sensing image sets demonstrate that CBC-SLP consistently outperforms state-of-the-art multimodal models across full and missing modality scenarios. Besides, we empirically demonstrate that the proposed strategy can recover the complementary information that may not be preserved in a shared representation. The code is available at https://github.com/iremulku/Multispectral-Semantic-Segmentation-via-Structured-Latent-Projection-CBC-SLP-.

Irem Ulku, Erdem Akagündüz, Ömer Özgür Tanrıöver

Department of Computer Engineering, Ankara University, Ankara, Türkiye · Department of Modeling and Simulation, Graduate School of Informatics, METU, Ankara, Türkiye

Amodal Segmentation Complementary Modalities Semantic Segmentation Modalities Large Multimodal Models Dropout

Apr 17, 2026cs.CV

SegMix:Shuffle-based Feedback Learning for Semantic Segmentation of Pathology Images

Segmentation is a critical task in computational pathology, as it identifies areas affected by disease or abnormal growth and is essential for diagnosis and treatment. However, acquiring high-quality pixel-level supervised segmentation data requires significant workload demands from experienced pathologists, limiting the application of deep learning. To overcome this challenge, relaxing the label conditions to image-level classification labels allows for more data to be used and more scenarios to be enabled. One approach is to leverage Class Activation Map (CAM) to generate pseudo pixel-level annotations for semantic segmentation with only image-level labels. However, this method fails to thoroughly explore the essential characteristics of pathology images, thus identifying only small areas that are insufficient for pseudo masking. In this paper, we propose a novel shuffle-based feedback learning method inspired by curriculum learning to generate higher-quality pseudo-semantic segmentation masks. Specifically, we perform patch level shuffle of pathology images, with the model adaptively adjusting the shuffle strategy based on feedback from previous learning. Experimental results demonstrate that our proposed approach outperforms state-of-the-arts on three different datasets.

Zhiling Yan, Sicheng Chen, Tianyi Zhang +3

JD AI Research, Beijing, China · School of Microelectronics, Xi’an Jiaotong University · Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University

Computational Pathology Semantic Segmentation Class Activation Mapping

Apr 16, 2026cs.CV

Privacy-Preserving Semantic Segmentation without Key Management

This paper proposes a novel privacy-preserving semantic segmentation method that can use independent keys for each client and image. In the proposed method, the model creator and each client encrypt images using locally generated keys, and model training and inference are conducted on the encrypted images. To mitigate performance degradation, an image encryption method is applied to model training in addition to the generation of test images. In experiments, the effectiveness of the proposed method is confirmed on the Cityscapes dataset under the use of a vision transformer-based model, called SETR.

Mare Hirose, Shoko Imaizumi, Hitoshi Kiya

Chiba University · Tokyo Metropolitan University

Efficient Semantic Segmentation Semantic Segmentation

Apr 16, 2026cs.CV

From Boundaries to Semantics: Prompt-Guided Multi-Task Learning for Petrographic Thin-section Segmentation

Grain-edge segmentation (GES) and lithology semantic segmentation (LSS) are two pivotal tasks for quantifying rock fabric and composition. However, these two tasks are often treated separately, and the segmentation quality is implausible albeit expensive, time-consuming, and expert-annotated datasets have been used. Recently, foundation models, especially the Segment Anything Model (SAM), have demonstrated impressive robustness for boundary alignment. However, directly adapting SAM to joint GES and LSS is nontrivial due to 1) severe domain gap induced by extinction-dependent color variations and ultra-fine grain boundaries, and 2) lacking novel modules for joint learning on multi-angle petrographic image stacks. In this paper, we propose Petro-SAM, a novel two-stage, multi-task framework that can achieve high-quality joint GES and LSS on petrographic images. Specifically, based on SAM, we introduce a Merge Block to integrate seven polarized views, effectively solving the extinction issue. Moreover, we introduce multi-scale feature fusion and color-entropy priors to refine the detection.

Yili Ren, Shiqi Wen, Li Hou +8

Research Institute of Petroleum Exploration and Development (RIPED) · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology +1

Lithology Classification Semantic Segmentation Porous Media Boundaries

Apr 8, 2026cs.CV

ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation

Weakly supervised semantic segmentation aims to achieve pixel-level predictions using image-level labels. Existing methods typically entangle semantic recognition and object localization, which often leads models to focus exclusively on sparse discriminative regions. Although foundation models show immense potential, many approaches still follow the tightly coupled optimization paradigm, struggling to effectively alleviate pseudo-label noise and often relying on time-consuming multi-stage retraining or unstable end-to-end joint optimization. To address the above challenges, we present ModuSeg, a training-free weakly supervised semantic segmentation framework centered on explicitly decoupling object discovery and semantic assignment. Specifically, we integrate a general mask proposer to extract geometric proposals with reliable boundaries, while leveraging semantic foundation models to construct an offline feature bank, transforming segmentation into a non-parametric feature retrieval process. Furthermore, we propose semantic boundary purification and soft-masked feature aggregation strategies to effectively mitigate boundary ambiguity and quantization errors, thereby extracting high-quality category prototypes. Extensive experiments demonstrate that the proposed decoupled architecture better preserves fine boundaries without parameter fine-tuning and achieves highly competitive performance on standard benchmark datasets. Code is available at https://github.com/Autumnair007/ModuSeg.

Qingze He, Fagui Liu, Dengke Zhang +2

South China University of Technology, Guangzhou, China · Pengcheng Laboratory, Shenzhen, China

Semantic Segmentation High Confidence Pseudo-Labels Feature Pyramid Training-Free Discovery

Mar 18, 2026cs.CV

SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale

Semantic segmentation for uncrewed aerial vehicles (UAVs) is fundamental for aerial scene understanding, yet existing RGB and RGB-T datasets remain limited in scale, diversity, and annotation efficiency due to the high cost of manual labeling and the difficulties of accurate RGB-T alignment on off-the-shelf UAVs. To address these challenges, we propose a scalable geometry-driven 2D-3D-2D paradigm that leverages multi-view redundancy in high-overlap aerial imagery to automatically propagate labels from a small subset of manually annotated RGB images to both RGB and thermal modalities within a unified framework. By lifting less than 3% of RGB images into a semantic 3D point cloud and rendering it into all views, our approach enables dense pseudo ground-truth generation across large image collections, automatically producing 97% of RGB labels and 100% of thermal labels while achieving 91% and 88% annotation accuracy without any 2D manual refinement. We further extend this 2D-3D-2D paradigm to cross-modal image registration, using 3D geometry as an intermediate alignment space to obtain fully automatic, strong pixel-level RGB-T alignment with 87% registration accuracy and no hardware-level synchronization. Applying our framework to existing geo-referenced aerial imagery, we construct SegFly, a large-scale benchmark with over 20,000 high-resolution RGB images and more than 15,000 geometrically aligned RGB-T pairs spanning diverse urban, industrial, and rural environments across multiple altitudes and seasons. On SegFly, we establish the Firefly baseline for RGB and thermal semantic segmentation and show that both conventional architectures and vision foundation models benefit substantially from SegFly supervision, highlighting the potential of geometry-driven 2D-3D-2D pipelines for scalable multi-modal aerial scene understanding. Data and Code available at https://github.com/markus-42/SegFly.

Markus Gross, Sai Bharadhwaj Matha, Rui Song +4

Fraunhofer Institute IVI · TU Munich · MCML +2

Aerial Dataset Efficient Semantic Segmentation Semantic Segmentation Rgbt

Mar 11, 2026cs.CV

SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning

Sign Language Production (SLP) faces a fundamental trade-off: direct text-to-pose models suffer from regression-to-the-mean effects, while dictionary-retrieval methods produce disjointed transitions. To resolve this, we propose a novel training paradigm that leverages sparse keyframes to capture the underlying kinematic distribution of human signing. By generating dense motion from discrete anchors, our approach mitigates regression-to-the-mean while ensuring fluid articulation. To achieve this at scale, we introduce FAST, an ultra-efficient sign segmentation model that automatically mines precise temporal boundaries. We then present SignSparK, a Conditional Flow Matching (CFM) framework that utilizes these temporal anchors to synthesize 3D signing sequences. This keyframe-driven formulation also unlocks Keyframe-to-Pose (KF2P) generation, making precise spatiotemporal editing of signing sequences possible. Furthermore, SignSparK scales across four distinct sign languages, constituting the largest multilingual SLP framework to date, and integrates 3D Gaussian Splatting for photorealistic rendering. Extensive evaluations demonstrate that SignSparK achieves state-of-the-art across diverse SLP tasks and multilingual benchmarks. Our code is available at https://github.com/JianHe0628/SignSparK.

Jianhe Low, Alexandre Symeonidis-Herzig, Maksym Ivashechkin +2

CVSSP, University of Surrey, United Kingdom

Sign Language Translation Keyframe Selection Semantic Segmentation

Mar 6, 2026cs.CV

JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas

Semantic segmentation across visual modalities such as 3D point clouds and panoramic images remains a challenging task, primarily due to the scarcity of annotated data and the limited adaptability of fixed-label models. In this paper, we present JOPP-3D, an open-vocabulary segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding. We convert RGB-D panoramic images into their corresponding wide field-of-view tangential perspectives and 3D point clouds, then use these modalities to extract and align foundational vision-language features. This allows natural language querying to generate semantic masks on both input modalities. Experimental evaluation on the Stanford-2D-3D-s and ToF-360 datasets demonstrates the capability of JOPP-3D to produce coherent and semantically meaningful segmentations across panoramic and 3D domains. Our proposed method achieves a significant improvement compared to the SOTA in open and closed vocabulary 2D and 3D semantic segmentation.

Sandeep Inuganti, Hideaki Kanayama, Kanta Shimizu +4

German Research Center for Artificial Intelligence, DFKI, Germany · RPTU Kaiserslautern, Germany · Ricoh Company, Ltd. Japan +1

Open-Vocabulary 3D Segmentation Open-Vocabulary Segmentation Point Clouds Semantic Segmentation Omnidirectional Images

Feb 23, 2026cs.CV

Segmenting Low-Contrast XCTs of Concrete: An Unsupervised Approach

X-Ray Computed Tomography (XCT) is a compelling tool in experimental mechanics, capable of non-destructively extracting information pertaining to the internal morphology of materials. For materials with random heterogeneous morphology such as concrete, such information is of particular relevance since it allows for studies of morphology-related behaviour and for predictive modelling. Nevertheless, XCT images require semantic segmentation for practical usage. Here, concrete poses a unique challenge due to the similar X-ray attenuation coefficients of aggregates and mortar, which result in low contrast between the two phases in the ensuing XCT images. As such, purely intensity-dependent semantic segmentation tools remain unfeasible. While vision transformers (ViTs) and convolutional neural networks (CNNs) are proven techniques for semantic segmentation in such challenging cases, they typically require labelled training data, which is often unavailable for concrete or resource-intensive to obtain, thereby limiting their relevance. To address this challenge, a self-annotation technique is presented here that leverages superpixel algorithms to identify perceptually similar local regions in an image and relates them to the global context by utilizing the receptive field of a CNN-based model. This enables the model to learn a global-local relationship in the images and facilitates the identification of semantically similar structures. When evaluated against manually annotated ground truth on out-of-distribution data, the proposed methodology consistently outperformed direct greyscale thresholding across all pertinent metrics, demonstrating improved discernibility between aggregates and mortar, and providing the most favourable balance of sensitivity and precision for aggregate-phase identification.

Kaustav Das, Gaston Rauchs, Jan Sykora +1

Department of Mechanics, Faculty of Civil Engineering, Czech Technical University in Prague, Thákurova 7, 166 29 Prague 6, Czech Republic · Structural Composites Unit, Luxembourg Institute of Science and Technology (LIST), 5 avenue des Hauts-Fourneaux L-4362 Esch-sur-Alzette, Luxembourg

Multi Organ Computed Tomography Segmentation Semantic Segmentation Cone-Beam Computed Tomography High-Order Correlations

Jan 22, 2026eess.IV

Phi-SegNet: Phase-Integrated Supervision for Medical Image Segmentation

Deep learning has substantially advanced medical image segmentation, yet achieving robust generalization across diverse imaging modalities and anatomical structures remains a major challenge. A key contributor to this limitation lies in how existing architectures, ranging from CNNs to Transformers and their hybrids, primarily encode spatial information while overlooking frequency-domain representations that capture rich structural and textural cues. Although recent studies have begun exploring spectral information at the feature level, supervision-level integration of frequency cues-crucial for fine-grained object localization-remains largely untapped. To this end, we propose Phi-SegNet, a CNN-based architecture that incorporates phase-aware information at both architectural and optimization levels. The network integrates Bi-Feature Mask Former (BFMF) modules that blend neighboring encoder features to reduce semantic gaps, and Reverse Fourier Attention (RFA) blocks that refine decoder outputs using phase-regularized features. A dedicated phase-aware loss aligns these features with structural priors, forming a closed feedback loop that emphasizes boundary precision. Evaluated on five public datasets spanning X-ray, US, histopathology, MRI, and colonoscopy, Phi-SegNet consistently achieved state-of-the-art performance, with an average relative improvement of 1.54+/-1.26% in IoU and 0.98+/-0.71% in F1-score over the next best-performing model. In cross-dataset generalization scenarios involving unseen datasets from the known domain, it also exhibits robust and superior performance, highlighting its adaptability and modality-agnostic design. These findings demonstrate the potential of leveraging spectral priors in both feature representation and supervision, paving the way for generalized segmentation frameworks that excel in fine-grained object localization. Our code is available on GitHub.

Shams Nafisa Ali, Taufiq Hasan

Semi-Supervised Medical Image Segmentation Feature Pyramid Semantic Segmentation Medical Imaging Deep Learning Regularization

Nov 9, 2025cs.CV

Label-Efficient 3D Forest Mapping: Self-Supervised and Transfer Learning for Instance Segmentation, Semantic Segmentation, and Species Classification

Detailed structural and species information on individual tree level is increasingly important to support precision forestry, biodiversity conservation, and provide reference data for biomass and carbon mapping. Point clouds from airborne and ground-based laser scanning are currently the most suitable data source to rapidly derive such information at scale. Recent advancements in deep learning improved segmenting and classifying individual trees and identifying semantic tree components. However, deep learning models typically require large amounts of annotated training data which limits further improvement. Producing dense, high-quality annotations for 3D point clouds, especially in complex forests, is labor-intensive and challenging to scale. We explore strategies to reduce dependence on large annotated datasets using self-supervised and transfer learning. Our objective is to improve performance across three tasks: instance segmentation, semantic segmentation, and tree classification using realistic and operational training sets. We observe improvements across all tasks, compared to training from scratch, evaluated with their respective metrics. For instance segmentation, self-supervised learning combined with domain adaptation improves AP50 by 16.98%. For semantic segmentation, self-supervised learning alone improves mIoU by 1.79%. For tree classification, hierarchical transfer learning improves mean Jaccard by 6.07%. To simplify use and encourage uptake, we integrated the tasks into a unified framework, streamlining the process from raw point clouds to tree delineation, structural analysis, and species classification. Pretrained models reduce energy consumption and carbon emissions by ~21%. This open-source contribution aims to accelerate operational extraction of individual tree information from laser scanning point clouds to support forestry, biodiversity, and carbon mapping.

Aldino Rizaldy, Fabian Ewald Fassnacht, Ahmed Jamal Afifi +3

Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Helmholtz Institute Freiberg for Resource Technology (HIF), 09599 Freiberg, Germany. · Institute of Geomatics, BOKU University, 1190 Vienna, Austria · Remote Sensing and Geoinformatics, Freie Universität Berlin, 12249 Berlin, Germany +1

Forests Point Clouds Species Identification Semantic Segmentation 3D Mapping Delineation+1

Oct 30, 2025cs.CV

AD-SAM: Adapting the Segment Anything Model for Semantic Segmentation in Autonomous Driving

This paper presents the Autonomous Driving Segment Anything Model (AD-SAM), a foundation-model adaptation framework for semantic segmentation in autonomous driving. AD-SAM combines a frozen Segment Anything Model (SAM) Vision Transformer (ViT-H) encoder with a trainable ResNet-50 encoder to integrate general visual representations with multi-scale, domain-specific spatial features. Features from the two encoders are integrated through deformable convolution and channel attention, followed by a multi-stage deformable decoder for semantic prediction. Training employs a hybrid objective combining Focal, Dice, Lovász-Softmax, and Surface losses. Experiments on Cityscapes and Berkeley DeepDrive 100K (BDD100K) show that AD-SAM outperforms SAM, Generalized SAM (G-SAM), and DeepLabV3 under a controlled training protocol. AD-SAM achieves 76.27% mIoU on Cityscapes and 64.74% on BDD100K, exceeding DeepLabV3 by 3.45 and 5.00 percentage points, respectively, with larger gains over the SAM-based baselines. Sample-size experiments reveal dataset-dependent behavior. In particular, AD-SAM performs strongly across training sizes on Cityscapes, while its advantage on the more heterogeneous BDD100K becomes pronounced with increased training data. When trained on Cityscapes and directly evaluated on BDD100K, AD-SAM achieves the highest cross-dataset retention (84.88%) among the evaluated models. AD-SAM also converges rapidly, while precomputed frozen SAM embeddings reduce training memory requirements. These findings demonstrate the potential of combining general foundation-model representations with domain-specific multi-scale features for accurate and robust autonomous-driving semantic segmentation.

Het Patel, Mario Camarena, Fatemeh Nazari +3

Department of Computer Science, The University of Texas Rio Grande Valley, Edinburg, TX 78539 USA · Department of Civil Engineering, The University of Texas Rio Grande Valley, Edinburg, TX 78539 USA · Department of Computer Science and Engineering, University of California Riverside, Riverside, CA 92521 USA

Efficient Semantic Segmentation Autonomous Driving Semantic Segmentation Segment Anything Model Recent Vision Foundation Models Vision Transformer

Oct 29, 2025cs.CV

BikeScenes: LiDAR Semantic Segmentation for Bicycles

The vulnerability of cyclists, exacerbated by the rising popularity of faster e-bikes, motivates adapting automotive perception technologies for bicycle safety. We use our multi-sensor SenseBike research platform to study 3D LiDAR semantic segmentation for bicycles. We introduce the novel BikeScenes-lidarseg Dataset, comprising 3021 consecutive LiDAR scans around the university campus of the TU Delft, semantically annotated for 29 dynamic and static classes. As an initial baseline study, we evaluate how a SemanticKITTI pre-trained FRNet model transfers to this bicycle-mounted solid-state LiDAR setting. Fine-tuning on BikeScenes increases mean Intersection-over-Union (mIoU) from 13.8% to 63.6% on our held-out subsequences. These results show the practical value of in-domain data for this platform, while also highlighting the need for larger bicycle datasets. We contribute BikeScenes as a resource for advancing research in cyclist-centric perception.

Denniz Goren, Holger Caesar

Bivariate Bicycle Codes Lidar Autonomous Driving Perception Light Detection And Ranging Data Multi-Modal Sensing Semantic Segmentation

Sep 29, 2025cs.CV

Improved Robustness from Biologically Inspired Sparse Contrast Representations

Deep neural networks surpass humans on many vision benchmarks, yet remain far less robust to distribution shifts such as illumination and weather changes. Existing approaches address this challenge by additional training data, extensive augmentation, architectural modifications, or test-time adaptation. In this work, we explore a complementary direction: inspired by the human retina, we propose a fixed, model-agnostic preprocessing module that extracts signals that are more stable with respect to variations of illumination. Our method combines color remapping with local contrast extraction, producing sparse representations that emphasize structural features. We study its impact on semantic segmentation by training on Cityscapes and evaluating generalization under adverse conditions on Dark Zurich and ACDC. Our results show that the biologically inspired preprocessing preserves in-distribution performance while consistently improving robustness in challenging lighting scenarios, such as nighttime, where annotated training data are scarce. Moreover, the segmentation accuracy remains stable even when the contrast-based representation is sparsified by up to 70%. These gains suggest that rethinking the input representation itself can improve robustness while also opening opportunities for lower-latency, transmission-aware imaging sensors when sparsity can be exploited close to acquisition.

Lorena Stracke, Lia Nimmermann, Shashank Agnihotri +3

Media Systems, University of Siegen, Germany · Data and Web Science Group, University of Mannheim, Germany · Analogue Circuits and Image Sensors, University of Siegen, Germany +1

Semantic Segmentation Hyperspectral Image Contrast

Sep 16, 2025cs.CV

Few to Big: Prototype Expansion Network via Diffusion Learner for Point Cloud Few-shot Semantic Segmentation

Few-shot 3D point cloud semantic segmentation aims to segment novel categories using a minimal number of annotated support samples. However, prototypes derived from the limited non-structural point cloud support set are often misaligned and have a small capacity, hindering effective gen eralization to novel categories. This stems from two core issues: i) the prototype possess limited representational capacity fails to cover the full intra-class diversity of a novel category, and ii) the prototypes suffer from misalignment with the query space due to the inter-set inconsistency between support and query sets. To address these issues, our work focuses on leveraging the few support samples to construct a well-aligned big-capacity prototype. Motivated by the powerful generative capabilities of diffusion models, we re-purpose its pre-trained conditional encoder to provide rich feature components for prototype ex pansion. Subsequently, a push-pull force aligns this expanded prototype towards the query feature space. Under this setup, we introduce the Prototype Expansion Network (PENet), a framework that constructs aligned big-capacity prototypes from two complementary feature sources. Specifically, PENet employs a dual-stream learner architecture: it retains a conventional fully supervised Intrinsic Learner (IL) to distill representative features, while introducing a novel Diffusion Learner (DL) to provide rich generalizable features. The resulting dual prototypes are then processed by a Prototype Assimilation Module (PAM), which adopts a push-pull attention block to align the prototypes with the query space. Furthermore, a Prototype Calibration Mechanism (PCM) regularizes the final big-capacity prototype to prevent semantic drift. Extensive experiments on the S3DIS and ScanNet datasets demonstrate that PENet outperforms state-of-the-art methods across various few-shot settings.

Qianguang Zhao, Dongli Wang, Yan Zhou +2

School of Mathematics and Computational Science, Xiangtan University, Xiangtan 411105, China · School of Automation and Electronics Information, Xiangtan University, Xiangtan 411105, China · School of Automation, Shanghai Jiao Tong University, Shanghai 200232, China

Few-Shot Segmentation Point Clouds Learnable Prototypes Semantic Segmentation Diffusion Models Representational Capacity

Aug 8, 2025cs.CV

SynSeg: Feature Synergy for Multi-Category Contrastive Learning in End-to-End Open-Vocabulary Semantic Segmentation

Semantic segmentation in open-vocabulary scenarios presents significant challenges due to the wide range and granularity of semantic categories. Existing weakly-supervised methods often rely on category-specific supervision and ill-suited feature construction methods for contrastive learning, leading to semantic misalignment and poor performance. In this work, we introduce a novel weakly-supervised approach, SynSeg, to address the challenges. SynSeg performs Multi-Category Contrastive Learning (MCCL) as a stronger training signal which robustly injecting intra- and inter-category knowledge during training. We also propose a new feature reconstruction framework named Feature Synergy Structure (FSS). FSS reconstructs discriminative features for contrastive learning through prior fusion and semantic-activation-map enhancement, effectively avoiding the foreground bias introduced by the visual encoder. Furthermore, SynSeg is a lightweight end-to-end solution capable for real-time inference. In general, SynSeg effectively improves the abilities in semantic localization and discrimination under weak supervision in an efficient manner. Extensive experiments on benchmarks demonstrate that our method outperforms state-of-the-art (SOTA) performance, with mIoU score gains ranging from 0.6% up to 8.9% across all reported benchmarks.

Weichen Zhang, Kebin Liu, Fan Dang +3

Tsinghua University, 100084, Beijing, China · Beijing Jiaotong University, 100044, Beijing, China

Semantic Segmentation Contrastive Learning Feature Pyramid Class Activation Mapping Synergy

Jun 13, 2025cs.CV

OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots

We introduce OV-MAP, a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities. A significant challenge arises when overlapping features from adjacent voxels reduce instance-level precision, as features spill over voxel boundaries, blending neighboring regions together. Our method overcomes this by employing a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds. This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models. We assess the effectiveness of our method through comprehensive experiments on public datasets such as ScanNet200 and Replica, demonstrating superior zero-shot performance, robustness, and adaptability across diverse environments. Additionally, we conducted real-world experiments to demonstrate our method's adaptability and robustness when applied to diverse real-world environments.

Juno Kim, Yesol Park, Hye-Jung Yoon +1

Interdisciplinary Program in AI, Seoul National University · Artificial Intelligence Institute, Seoul National University · Department of Computer Science, Seoul National University

Open-Vocabulary 3D Segmentation 3D Mapping 3D Object Detection Robotic Perception 3D Generation Mobile Robots+3

May 21, 2025cs.CV

From Pixels to Images: A Structural Survey of Deep Learning Paradigms in Remote Sensing Image Semantic Segmentation

Remote sensing images (RSIs) capture both natural and human-induced changes on the Earth's surface. Semantic segmentation (SS) of RSIs enables the fine-grained interpretation of surface features, making it a critical task in RS analysis. With the increasing diversity and volume of RSIs collected by sensors on various platforms, traditional processing methods struggle to maintain efficiency and accuracy. In response, deep learning (DL) has emerged as a transformative approach, enabling substantial advances in remote sensing image semantic segmentation (RSISS). As researchers continue to explore end-to-end SS, DL-based RSISS has undergone a structural evolution from pixel-level and patch-based classification to tile-level and image-level segmentation. However, existing reviews often focus on individual components, such as supervision strategies or fusion stages, and lack a unified operational perspective aligned with segmentation granularity and the training/inference pipeline. This paper provides a comprehensive review by organizing DL-based RSISS into a pixel-patch-tile-image hierarchy, covering early pixel-based methods, prevailing patch-based and tile-based techniques, and emerging image-based approaches. Specifically, the survey analyzes four supervision strategies, eleven feature extraction strategies, and six information fusion strategies, revealing the field's progression from local to global feature extraction, from traditional DL architectures to foundation models, and from unimodal to multimodal segmentation. This review offers a holistic and structured understanding of DL-based RSISS, highlighting representative datasets, comparative insights, and open challenges related to data scale, model efficiency, domain robustness, and multimodal integration. Furthermore, to facilitate reproducible research, curated code collections are provided at: https://github.com/quanweiliu/RSISS.

Quanwei Liu, Tao Huang, Jiaqi Yang +1

College of Science and Engineering, James Cook University, Cairns QLD 4878, Australia · Center for AI and Data Science Innovation, James Cook University, Cairns QLD 4878, Australia · Department of Forest and Wildlife Ecology, University of Wisconsin-Madison, Madison, WI 53706 USA +1

Remote Sensing Image Segmentation Remote Sensing Semantic Segmentation Deep Learning Pixels

Dec 20, 2024cs.CV

SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

Improving the reliability and completeness of colonoscopic inspection is critical for reducing missed lesions and improving colorectal cancer prevention. Reliable scene understanding is essential for navigation, reconstruction, and assessment of inspection completeness. Anatomical structures such as mucosal folds provide stable geometric cues for endoscope localization, while surgical instruments introduce dynamic occlusions that complicate visual interpretation. However, existing gastrointestinal endoscopy datasets largely focus on disease detection or artifact segmentation, leaving a gap in precise annotations of structural landmarks and instruments. We introduce SegCol, a dataset and benchmark for semantic segmentation of colon fold edges and surgical instruments derived from the EndoMapper dataset. SegCol provides manually annotated pixel-level masks for three instrument classes and thin fold-edge structures across temporally consistent image sequences. It forms the basis of the SegCol Challenge, organized as part of the EndoVis Challenge at MICCAI 2024, evaluating both supervised segmentation and annotation-efficient active learning. We further study segmentation metrics, including Dice, ODS/OIS, AP, and CLDice, under structural perturbations and different object geometries, and analyze participating methods, architectural choices, and active learning strategies. Our findings show that metric behavior strongly depends on target structure, highlighting the need for carefully selected evaluation protocols in endoscopic segmentation. Details are available at https://www.synapse.org/Synapse:syn54124209/wiki/626563, and code at https://github.com/surgical-vision/segcol_challenge.

Xinwei Ju, Rema Daher, Razvan Caramalau +7

UCL Hawkes Institute, University College London, London, UK · Department of Computer Science, University College London, London, UK · Medtronic, London, UK +3

Colonoscopy Semantic Segmentation Edge

May 12, 2024eess.IV

Explainable Convolutional Neural Networks for Retinal Fundus Classification and Cutting-Edge Segmentation Models for Retinal Blood Vessels from Fundus Images

Early detection of vision-threatening conditions such as diabetic retinopathy, glaucoma, and age-related macular degeneration depends on retinal fundus image analysis, but manual assessment is slow and expert-dependent. Automated convolutional neural networks classify fundus images accurately yet act as black boxes, and existing retinal vessel segmentation methods lose discriminative power under pathology and seldom exploit attention or transformer backbones. Using the FIVES and DRIVE fundus datasets, we develop a two-pipeline framework that pairs four-class disease classification with attention- and transformer-based vessel segmentation, organised in three stages: (1) FIVES images are augmented by rotation and horizontal and vertical flips and used to fine-tune eight ImageNet-pretrained CNNs: ResNet101, DenseNet169, Xception, InceptionV3, DenseNet121, InceptionResNetV2, ResNet50, and EfficientNetB0. (2) Five gradient-based explanation methods, Grad-CAM, Grad-CAM++, Score-CAM, Faster Score-CAM, and Layer-CAM, are computed on the final convolutional block of each classifier and compared qualitatively across architectures. (3) Ten U-Net variants are benchmarked for vessel segmentation: TransUNet (hybrid CNN--Transformer encoder) and Attention U-Net (gated skip connections), evaluated with ResNet50V2, ResNet101V2, and ResNet152V2 backbones, along with additional Attention U-Net configurations using DenseNet backbones, and the fully transformer-based Swin-UNet. ResNet101 gives the highest classification accuracy: 94.17% (F1 0.942)

>

88.33% for EfficientNetB0. For segmentation, the architecture ranking is consistent on both datasets: Attention U-Net

>

TransUNet

>

Swin-UNet. The strongest configuration is Attention U-Net with a ResNet101V2 backbone: FIVES IoU 0.722, Dice 0.838; DRIVE IoU 0.648, Dice 0.787, lifting DRIVE IoU 60.80

\rightarrow

64.83 over a prior custom U-Net.

Fatema Tuj Johora Faria, Mukaffi Bin Moin, Pronay Debnath +2

aDepartment of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh

Vessel Segmentation Fundus Images Retinal Imaging Semantic Segmentation Convolutional

Date pendingcs.CV

CGSM: Concept-Guided Segmentation Model for Precise Pulmonary Lesion Delineation

Accurate segmentation of pulmonary lesions is essential for effective clinical diagnosis and treatment strategies. Existing segmentation approaches often lack task-specific semantic guidance, as text-based annotations typically offer coarse localization of lesions, leading to inadequate delineation of lesion boundaries and poor performance on small-scale lesions. To address this, we propose CGSM, a Concept-Guided Segmentation Model that integrates LLM-generated and clinically reviewed concepts into the segmentation process. Specifically, we design a Concept-Visual Alignment Module (CVAM) to activate relevant tokens within the concepts that align with visual features, enhancing the interaction between textual and visual information. In addition, we introduce a Concept Modulated Decoder (CM-Decoder), which uses concepts from CVAM as modulation signals to facilitate the adaptive fusion of image and text features, improving the segmentation accuracy. Extensive experiments on two public datasets show that CGSM achieves state-of-the-art performance, with results of 91.59% Dice and 84.49% mIoU on the QaTa-COV19 dataset, demonstrating its effectiveness in pulmonary lesion segmentation.

Changheng Lin, Wenjie Zhang, Yushan Lu +3

Shandong University, China

Lesion Segmentation Semantic Segmentation