Image Segmentation

Ask about this topic

Overview

Image segmentation assigns each pixel or voxel to a meaningful region, object, or structure, enabling tasks such as measuring anatomy, mapping land, and understanding scenes. Methods range from thresholding and variational contours to U-Net-style networks, transformers, and foundation models such as SAM, often guided by text, points, boxes, or reference images. Current work also explores learning from sparse labels, synthetic data, multiple modalities, and video, with benchmarks spanning medical scans, natural images, remote sensing, and microscopy. A central challenge is balancing accurate boundaries and topology with reliable performance on new domains, limited annotations, uncertainty, and practical limits on speed and memory.

Latest papers

CardsList

401 papers24 papers in the last four weeks, up 9% on the four weeks before. 0.1% of all new papers.

Mar 20, 2026

Synergistic Perception and Generative Recomposition: A Multi-Agent Orchestration for Expert-Level Building Inspection

Building facade defect inspection is fundamental to structural health monitoring and sustainable urban maintenance, yet it remains a formidable challenge due to extreme geometric variability, low contrast against complex backgrounds, and the inherent complexity of composite defects (e.g., cracks co-occurring with spalling). Such characteristics lead to severe pixel imbalance and feature ambiguity, which, coupled with the critical scarcity of high-quality pixel-level annotations, hinder the generalization of existing detection and segmentation models. To address gaps, we propose FacadeFixer, a unified multi-agent framework that treats defect perception as a collaborative reasoning task rather than isolated recognition. Specifically,FacadeFixer orchestrates specialized agents for detection and segmentation to handle multi-type defect interference, working in tandem with a generative agent to enable semantic recomposition. This process decouples intricate defects from noisy backgrounds and realistically synthesizes them onto diverse clean textures, generating high-fidelity augmented data with precise expert-level masks. To support this, we introduce a comprehensive multi-task dataset covering six primary facade categories with pixel-level annotations. Extensive experiments demonstrate that FacadeFixer significantly outperforms state-of-the-art (SOTA) baselines. Specifically, it excels in capturing pixel-level structural anomalies and highlights generative synthesis as a robust solution to data scarcity in infrastructure inspection. Our code and dataset will be made publicly available.

Hui Zhong, Yichun Gao, Luyan Liu +4

The Hong Kong University of Science and Technology (Guangzhou), Systems Hub, China · Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things, Guangdong, China · The Hong Kong University of Science and Technology, Hongkong +1

Image SegmentationSynthetic Data AugmentationStructural Health Monitoring
AskPDF
Mar 19, 2026

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.

Anqi Zhang, Xiaokang Ji, Guangyu Gao +3

Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University +1

Image SegmentationSemantic SegmentationMultimodal Large Language Models
AskPDF
Mar 19, 2026

Ca2+ transient detection and segmentation with the Astronomically motivated algorithm for Background Estimation And Transient Segmentation (Astro-BEATS)

Fluorescence-based Ca2+^{2+}-imaging is a powerful tool for studying localized neuronal activity, including miniature Synaptic Calcium Transients, providing real-time insights into synaptic activity. These transients induce only subtle changes in the fluorescence signal, often barely above baseline, which poses a significant challenge for automated synaptic transient detection and segmentation. Detecting astronomical transients similarly requires efficient algorithms that will remain robust over a large field of view with varying noise properties. We leverage techniques used in astronomical transient detection for miniature Synaptic Calcium Transient detection in fluorescence microscopy. We present Astro-BEATS, an automatic miniature Synaptic Calcium Transient segmentation algorithm that incorporates image estimation and source-finding techniques used in astronomy and designed for Ca2+^{2+}-imaging videos. Astro-BEATS outperforms current threshold-based approaches for synaptic Ca2+^{2+} transient detection and segmentation. The produced segmentation masks can be used to train a supervised deep learning algorithm for improved synaptic Ca2+^{2+} transient detection in Ca2+^{2+}-imaging data. The speed of Astro-BEATS and its applicability to previously unseen datasets without re-optimization makes it particularly useful for generating training datasets for deep learning-based approaches.

Bolin Fan, Anthony Bilodeau, Frederic Beaupre +4

Dunlap Institute for Astronomy and Astrophysics, University of Toronto, Toronto, Canada · David A. Dunlap Department for Astronomy and Astrophysics, University of Toronto, Toronto, Canada · CERVO Brain Research Center, Québec, Canada +4

Microscopy Image AnalysisImage SegmentationFluorescence Microscopy
AskPDF
Mar 18, 2026

LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation

Precise localization and delineation of brain tumors using magnetic resonance imaging (MRI) are essential for planning therapy and guiding surgical decisions. To address this, we propose LoGSAM, a parameter-efficient, detection-driven framework that transforms radiologist dictation into text prompts for foundation-model-based localization and segmentation. Radiologist speech is first transcribed and translated using a pretrained Whisper ASR model, followed by negation-aware clinical NLP to extract tumor-specific textual prompts. These prompts guide text-conditioned tumor localization via a LoRA-adapted vision-language detection model, Grounding DINO (GDINO). The predicted bounding boxes are used as prompts for MedSAM to generate pixel-level tumor masks without any additional fine-tuning. On BRISC 2025, LoGSAM attains a Dice score of 80.32%, reaching 98.6% of a fully fine-tuned GDINO + MedSAM baseline while training fewer than 5% of its parameters, indicating a favorable accuracy/parameter trade-off. In addition, we evaluate the full pipeline using German dictations from a board-certified radiologist on unseen MRI scans, achieving 91.7% case-level class-extraction accuracy. These results highlight the feasibility of constructing a modular speech-to-segmentation pipeline from pretrained foundation models with minimal parameter updates.

Mohammad Robaitul Islam Bhuiyan, Sheethal Bhat, Melika Qahqaie +4

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Siemens Healthineers, Erlangen, Germany

Image SegmentationMedical Imaging Foundation ModelsText-Guided Image SegmentationMedical Image SegmentationMedical Image GroundingBrain Tumor Segmentation
AskPDF
Mar 17, 2026

TCATSeg: A Tooth Center-Wise Attention Network for 3D Dental Model Semantic Segmentation

Accurate semantic segmentation of 3D dental models is essential for digital dentistry applications such as orthodontics and dental implants. However, due to complex tooth arrangements and similarities in shape among adjacent teeth, existing methods struggle with accurate segmentation, because they often focus on local geometry while neglecting global contextual information. To address this, we propose TCATSeg, a novel framework that combines local geometric features with global semantic context. We introduce a set of sparse yet physically meaningful superpoints to capture global semantic relationships and enhance segmentation accuracy. Additionally, we present a new dataset of 400 dental models, including pre-orthodontic samples, to evaluate the generalization of our method. Extensive experiments demonstrate that TCATSeg outperforms state-of-the-art approaches.

Qiang He, Wentian Qu, Jiajia Dai +9

Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Department of Computer Science and Technology, Tsinghua University +2

3D Semantic Segmentation3D Point Cloud Segmentation3D Medical Image SegmentationImage Segmentation
AskPDF
Mar 17, 2026

LenghuSky-8: An 8-Year All-Sky Cloud Dataset with Star-Aware Masks and Alt-Az Calibration for Segmentation and Nowcasting

Ground-based time-domain observatories require minute-by-minute, site-scale awareness of cloud cover, yet existing all-sky datasets are short, daylight-biased, or lack astrometric calibration. We present LenghuSky-8, an eight-year (2018-2025) all-sky imaging dataset from a premier astronomical site, comprising 429,620 512×512512 \times 512 frames with 81.2% night-time coverage, star-aware cloud masks, background masks, and per-pixel altitude-azimuth (Alt-Az) calibration. For robust cloud segmentation across day, night, and lunar phases, we train a linear probe on DINOv3 local features and obtain 93.3% ±\pm 1.1% overall accuracy on a balanced, manually labeled set of 1,111 images. Using stellar astrometry, we map each pixel to local alt-az coordinates and measure calibration uncertainties of approximately 0.37 deg at zenith and approximately 1.34 deg at 30 deg altitude, sufficient for integration with telescope schedulers. Beyond segmentation, we introduce a short-horizon nowcasting benchmark over per-pixel three-class logits (sky/cloud/contamination) with four baselines: persistence (copying the last frame), optical flow, ConvLSTM, and VideoGPT. ConvLSTM performs best but yields only limited gains over persistence, underscoring the difficulty of near-term cloud evolution. We release the dataset, calibrations, and an open-source toolkit for loading, evaluation, and scheduler-ready alt-az maps to boost research in segmentation, nowcasting, and autonomous observatory operations.

Yicheng Rui, Xiao-Wei Duan, Licai Deng +9

State Key Laboratory of Dark Matter Physics, Tsung-Dao Lee Institute & School of Physics and Astronomy, Shanghai Jiao Tong University, Shanghai 201210, China · National Astronomical Observatories, Chinese Academy of Sciences, Beijing 100101, China · School of Computer Technology and Application, Qinghai University, Xining 810016, China

Image SegmentationRemote Sensing
AskPDF
Mar 16, 2026

Topology-Preserving Polygon Augmentation for Segmentation in Structured Visual Domains

Geometric data augmentation is widely used in segmentation workflows, but polygon annotations are often assumed to remain valid after transformation. This assumption can fail in structured domains such as architectural floorplan analysis, where a region may contain an interior void encoded as part of a single ordered polygon chain. Cropping or clipping can remove bridge vertices in this chain, causing one semantic region to split into disconnected components. We propose a lightweight topology-preserving augmentation strategy that repairs missing adjacency relations in index space while preserving the original vertex order. The method adds minimal overhead and can be integrated into existing preprocessing workflows. Experiments show that the proposed approach achieves near-perfect Cyclic Adjacency Preservation (CAP) across common geometric transformations and improves annotation consistency in polygon-based segmentation.

Sudip Laudari, Sang Hun Baek

Independent Researcher

Data AugmentationImage SegmentationTopology-Preserving Image Segmentation
AskPDF
Mar 15, 2026

Personalized Cell Segmentation: Benchmark and Framework for Reference-Guided Cell Type Segmentation

Accurate cell segmentation is critical for biological and medical imaging studies. Although recent deep learning models have advanced this task, most methods are limited to generic cell segmentation, lacking the ability to differentiate specific cell types. In this work, we introduce the Personalized Cell Segmentation (PerCS) task, which aims to segment all cells of a specific type given a reference cell. To support this task, we establish a benchmark by reorganizing publicly available datasets, yielding 1,372 images and over 110,000 annotated cells. As a pioneering solution, we propose PerCS-DINO, a framework built on the DINOv2 backbone. By integrating image features and reference embeddings via a cross-attention transformer and contrastive learning, PerCS-DINO effectively segments cells matching the reference. Extensive experiments demonstrate the effectiveness of the proposed PerCS-DINO and highlight the challenges of this new task. We expect PerCS to serve as a useful testbed for advancing research in cell-based applications.

Bisheng Wang, Jaime S. Cardoso, Lin Wu

INESC TEC, Portugal · INESC TEC, Faculdade de Engenharia, Universidade do Porto, Portugal · James Watt School of Engineering, University of Glasgow, United Kingdom

Image Segmentation
AskPDF
Mar 15, 2026

Multi-Period Texture Contrast Enhancement for Low-Contrast Wafer Defect Detection and Segmentation

Wafer defect segmentation is pivotal for semiconductor yield optimization yet remains challenged by the intrinsic conflict between microscale anomalies and highly periodic, overwhelming background textures. Existing deep learning paradigms often falter due to feature dilution during downsampling and the lack of explicit mechanisms to disentangle low-contrast defects from process-induced noise. To transcend these limitations, we propose TexWDS, a texture-aware framework that harmonizes multi-scale feature retention with frequency-domain perturbation modeling. Our methodology incorporates three strategic innovations: (1) A Multi-scale Receptive Field Reweighting strategy is introduced to mitigate aliasing effects and preserve high-frequency details of micro-defects often lost in standard pyramidal architectures. (2) The Multi-scale Unified Semantic Enhancer (MUSE) integrates local appearance with global context encoding, effectively enhancing feature discriminability in low-visibility regions. (3) Crucially, we design a plug-and-play Multi-Periodic Texture Contrast Enhancement (MPTCE) module. By modeling texture disruptions in the frequency domain, MPTCE explicitly decouples non-periodic anomalies from structured backgrounds, boosting contrast for camouflaged defects. Extensive experiments on real-world industrial datasets demonstrate that TexWDS achieves a new state-of-the-art, surpassing the baseline by 8.3% in mAP50-95 and 7.7% in recall, while reducing the false positive rate by approximately 8.6%. These results underscore the framework's robustness in handling complex periodic patterns and its suitability for high-precision manufacturing inspection.

Zihan Zhang

Engineering Technology Research Center of Henan Province for MEMS Manufacturing and Applications, School of Mechanics and Safety Engineering, Zhengzhou University, Zhengzhou 450001, China

Image SegmentationIndustrial Anomaly DetectionSemiconductor ManufacturingMulti-Scale Feature Fusion
AskPDF
Mar 14, 2026

DualSwinFusionSeg: Multimodal Martian Landslide Segmentation via Dual Swin Transformer with Multi-Scale Fusion and UNet++

Automated segmentation of Martian landslides, particularly in tectonically active regions such as Valles Marineris,is important for planetary geology, hazard assessment, and future robotic exploration. However, detecting landslides from planetary imagery is challenging due to the heterogeneous nature of available sensing modalities and the limited number of labeled samples. Each observation combines RGB imagery with geophysical measurements such as digital elevation models, slope maps, thermal inertia, and contextual grayscale imagery, which differ significantly in resolution and statistical properties. To address these challenges, we propose DualSwinFusionSeg, a multimodal segmentation architecture that separates modality-specific feature extraction and performs multi-scale cross-modal fusion. The model employs two parallel Swin Transformer V2 encoders to independently process RGB and auxiliary geophysical inputs, producing hierarchical feature representations. Corresponding features from the two streams are fused at multiple scales and decoded using a UNet++ decoder with dense nested skip connections to preserve fine boundary details. Extensive ablation studies evaluate modality contributions, loss functions, decoder architectures, and fusion strategies. Experiments on the MMLSv2 dataset from the PBVS 2026 Mars-LS Challenge show that modality-specific encoders and simple concatenation-based fusion improve segmentation accuracy under limited training data. The final model achieves 0.867 mIoU and 0.905 F1 on the development benchmark and 0.783 mIoU on the held-out test set, demonstrating strong performance for multimodal planetary surface segmentation.

Shahriar Kabir, Abdullah Muhammed Amimul Ehsan, Istiak Ahmmed Rifti +1

Bangladesh University of Engineering and Technology, Bangladesh · University of California, Riverside, USA

Multimodal Semantic SegmentationImage SegmentationRemote Sensing Image SegmentationMulti-Scale Feature Fusion
AskPDF
Mar 14, 2026

USIS-PGM: Photometric Gaussian Mixtures for Underwater Salient Instance Segmentation

Underwater salient instance segmentation (USIS) is crucial for marine robotic systems, as it enables both underwater salient object detection and instance-level mask prediction for visual scene understanding. Compared with its terrestrial counterpart, USIS is more challenging due to the underwater image degradation. To address this issue, this paper proposes USIS-PGM, a single-stage framework for USIS. Specifically, the encoder enhances boundary cues through a frequency-aware module and performs content-adaptive feature reweighting via a dynamic weighting module. The decoder incorporates a Transformer-based instance activation module to better distinguish salient instances. In addition, USIS-PGM employs multi-scale Gaussian heatmaps generated from ground-truth masks through Photometric Gaussian Mixture (PGM) to supervise intermediate decoder features, thereby improving salient instance localization and producing more structurally coherent mask predictions. Experimental results demonstrate the superiority and practical applicability of the proposed USIS-PGM model.

Lin Hong, Xiangtong Yao, Mürüvvet Bozkurt +2

Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · School of CIT, Chair i6, Technical University of Munich, Munich, Germany · School of CIT, Technical University of Munich, Munich, Germany +1

Image SegmentationInstance SegmentationUnderwater ImagingUnderwater Object Detection
AskPDF
Mar 14, 2026

Efficient Semi-Automated Material Microstructure Analysis Using Deep Learning: A Case Study in Additive Manufacturing

Image segmentation is fundamental to microstructural analysis for defect identification and structure-property correlation, yet remains challenging due to pronounced heterogeneity in materials images arising from varied processing and testing conditions. Conventional image processing techniques often fail to capture such complex features rendering them ineffective for large-scale analysis. Even deep learning approaches struggle to generalize across heterogeneous datasets due to scarcity of high-quality labeled data. Consequently, segmentation workflows often rely on manual expert-driven annotations which are labor intensive and difficult to scale. Using an additive manufacturing (AM) dataset as a case study, we present a semi-automated active learning based segmentation pipeline that integrates a U-Net based convolutional neural network with an interactive user annotation and correction interface and a representative core-set image selection strategy. The active learning workflow iteratively updates the model by incorporating user corrected segmentations into the training pool while the core-set strategy identifies representative images for annotation. Three subset selection strategies, manual selection, uncertainty driven sampling and proposed maximin Latin hypercube sampling from embeddings (SMILE) method were evaluated over six refinement rounds. The SMILE strategy consistently outperformed other approaches, improving the macro F1 score from 0.74 to 0.93 while reducing manual annotation time by about 65 percent. The segmented defect regions were further analyzed using a coupled classification model to categorize defects based on microstructural characteristics and map them to corresponding AM process parameters. The proposed framework reduces labeling effort while maintaining scalability and robustness and is broadly applicable to image based analysis across diverse materials systems.

Sanjeev S. Navaratna, Nikhil Thawari, Gunashekhar Mari +3

Department of Metallurgical and Materials Engineering, Indian Institute of Technology Madras, Chennai 600036, India · Center for Atomistic Modelling and Materials Design, IIT Madras, Chennai 600036, India

Interactive Image SegmentationImage SegmentationActive LearningCoreset Selection
AskPDF
Mar 13, 2026

GP-VM×\timesSMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentation

Medical image segmentation (MIS) is a fundamental component of computer-assisted diagnosis and clinical decision support. Over the past decade, numerous architectures specifically tailored to medical imaging have emerged to address domain-specific challenges such as low contrast, small anatomical structures, and limited annotated data. In parallel, rapid progress in computer vision has produced highly capable general-purpose vision models (GP-VMs) originally designed for natural images. Despite their strong performance on standard vision benchmarks, their effectiveness for MIS remains insufficiently understood. In this controlled empirical study, we examine whether specialized medical segmentation architectures (SMAs) provide systematic advantages over modern GP-VMs for 2D MIS. We compare eleven SMAs and GP-VMs using a unified training and evaluation protocol across three heterogeneous datasets, each covering different 2D imaging modalities, class structures, and data characteristics. Beyond segmentation performance, we employ linear mixed-effects models (LMMs) for statistical assessment and qualitative Grad-CAM visualizations to investigate explainability (XAI)-related model behavior. Our results imply that, for the analyzed settings, GP-VMs achieve performance comparable to that of specialized MIS models. Moreover, XAI analyses indicate that GP-VMs are capable of identifying clinically relevant structures despite the absence of explicit domain-specific architectural priors. These findings suggest that GP-VMs can represent a viable alternative to domain-specific methods for 2D MIS, highlighting the importance of informed model selection. All code and resources are available at GitHub.

Vanessa Borst, Anna Riedmann, Samuel Kounev

Julius-Maximilians-University Würzburg, 97070 Würzburg, Germany

Image SegmentationVision Foundation ModelsMedical Image SegmentationMedical Image Benchmarks
AskPDF
Mar 13, 2026

HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation

Marine Animal Segmentation (MAS) aims at identifying and segmenting marine animals from complex marine environments. Most of previous deep learning-based MAS methods struggle with the long-distance modeling issue. Recently, Segment Anything Model (SAM) has gained popularity in general image segmentation. However, it lacks of perceiving fine-grained details and frequency information. To this end, we propose a novel learning framework, named Hierarchical Frequency Prompted SAM (HFP-SAM) for high-performance MAS. First, we design a Frequency Guided Adapter (FGA) to efficiently inject marine scene information into the frozen SAM backbone through frequency domain prior masks. Additionally, we introduce a Frequency-aware Point Selection (FPS) to generate highlighted regions through frequency analysis. These regions are combined with the coarse predictions of SAM to generate point prompts and integrate into SAM's decoder for fine predictions. Finally, to obtain comprehensive segmentation masks, we introduce a Full-View Mamba (FVM) to efficiently extract spatial and channel contextual information with linear computational complexity. Extensive experiments on four public datasets demonstrate the superior performance of our approach. The source code is publicly available at https://github.com/Drchip61/TIP-HFP-SAM.

Pingping Zhang, Tianyu Yan, Yuhao Wang +7

School of Future Technology, Dalian University of Technology. · Central Hospital of Dalian University of Technology. · Affiliated Zhongshan Hospital of Dalian University. +1

Image SegmentationPromptable Image Segmentation
AskPDF
Mar 9, 2026

Bi-CamoDiffusion: A Boundary-informed Diffusion Approach for Camouflaged Object Detection

Bi-CamoDiffusion is introduced, an evolution of the CamoDiffusion framework for camouflaged object detection. It integrates edge priors into early-stage embeddings via a parameter-free injection process, enhancing boundary sharpness and preventing structural ambiguity. An optimization objective that unifies spatial accuracy, structural constraints, and uncertainty supervision is also proposed, allowing the model to capture of both the object's global context and its intricate boundary transitions. Evaluations across the CAMO, COD10K, and NC4K datasets show that Bi-CamoDiffusion surpasses the baseline, delivering sharper delineation of thin structures and protrusions while also minimizing false positives. The model consistently outperforms existing state-of-the-art methods across all evaluated metrics, including SmS_m, FβwF_β^{w}, EmE_m, and MAEMAE, demonstrating a more precise object-background separation and sharper boundary recovery. Code available at: https://github.com/plsuarez/Bi-CamoDiffusion

Patricia L. Suarez, Leo Thomas Ramos, Angel D. Sappa

ESPOL Polytechnic University · Computer Vision Center · Universitat Aut`onoma de Barcelona

Camouflaged Object DetectionImage SegmentationBoundary-Aware SegmentationDiffusion Models
AskPDF
Mar 8, 2026

RPG-SAM: Reliability-Weighted Prototypes and Geometric Adaptive Threshold Selection for Training-Free One-Shot Polyp Segmentation

Training-free one-shot segmentation offers a scalable alternative to expert annotations where knowledge is often transferred from support images and foundation models. But existing methods often treat all pixels in support images and query response intensities models in a homogeneous way. They ignore the regional heterogeity in support images and response heterogeity in query.To resolve this, we propose RPG-SAM, a framework that systematically tackles these heterogeneity gaps. Specifically, to address regional heterogeneity, we introduce Reliability-Weighted Prototype Mining (RWPM) to prioritize high-fidelity support features while utilizing background anchors as contrastive references for noise suppression. To address response heterogeneity, we develop Geometric Adaptive Selection (GAS) to dynamically recalibrate binarization thresholds by evaluating the morphological consensus of candidates. Finally, an iterative refinement loop method is designed to polishes anatomical boundaries. By accounting for multi-layered information heterogeneity, RPG-SAM achieves a 5.56% mIoU improvement on the Kvasir dataset. Code will be released.

Weikun Lin, Yunhao Bai, Yan Wang

Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University

Image SegmentationPolyp SegmentationMedical Image Segmentation
AskPDF
Mar 7, 2026

StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models

Recent token merging techniques for Vision Transformers (ViTs) provide substantial speedups by reducing the number of tokens processed by self-attention, often without retraining. However, their direct application to the Segment Anything Model (SAM) family is nontrivial: SAM's image encoder mixes windowed and global attention, and its mask decoder relies on dense, prompt-conditioned features for precise boundary prediction. We systematically evaluate representative token-merging methods on SAM and Medical SAM in a strict off-the-shelf setting, and find that existing destination-selection heuristics can erode boundaries and leak prompt information as merge rates increase. We propose StructSAM, a resolution-preserving merge-unmerge framework tailored to SAM. StructSAM computes a lightweight token-energy score from first-order feature gradients, uses grid-based flatness screening to protect boundary and prompt regions, and merges tokens within flat areas toward low-energy destinations with explicit token recovery. We further provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines. Across eight natural and medical benchmarks, StructSAM reduces encoder FLOPs by 25-30% (up to 40%+ with prompt-aware merging) with minor drops in mIoU/Dice, consistently outperforming ToMe, PiToMe, ToMeSD, VidToMe, and ALGM at the same compute.

Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17

University of Stuttgart · DFKI · Max Planck Research School +10

Token MergingImage SegmentationSelf-AttentionPromptable Image SegmentationEfficient ViTsAttention Mechanisms+2
AskPDF
Mar 6, 2026

MultiSolSegment: Multi-channel segmentation of overlapping features in electroluminescence images of photovoltaic cells

Electroluminescence (EL) imaging is widely used to detect defects in photovoltaic (PV) modules, and machine learning methods have been applied to enable large-scale analysis of EL images. However, existing methods cannot assign multiple labels to the same pixel, limiting their ability to capture overlapping degradation features. We present a multi-channel U-Net architecture for pixel-level multi-label segmentation of EL images. The model outputs independent probability maps for cracks, busbars, dark areas, and non-cell regions, enabling accurate co-classification of interacting features such as cracks crossing busbars. The model achieved an accuracy of 98% and has been shown to generalize to unseen datasets. This framework offers a scalable, extensible tool for automated PV module inspection, improving defect quantification and lifetime prediction in large-scale PV systems.

Ojas Sanghi, Norman Jost, Benjamin G. Pierce +3

Sandia National Laboratories · Sandia National Laboratories, 1515 Eubank Blvd SE, Albuquerque, New Mexico, 87123, USA · Case Western Reserve University +3

Image SegmentationFault DetectionU-Net
AskPDF
Mar 5, 2026

Generalizable Multiscale Segmentation of Heterogeneous Map Collections

Historical map collections are highly diverse in style, scale, and geographic focus, often consisting of many single-sheet documents. Yet most work in map recognition focuses on specialist models tailored to homogeneous map series. In contrast, this article aims to develop generalizable semantic segmentation models and ontology. First, we introduce Semap, a new open benchmark dataset comprising 1,439 manually annotated patches designed to reflect the variety of historical map documents. Second, we present a segmentation framework that combines procedural data synthesis with multiscale integration to improve robustness and transferability. This framework achieves state-of-the-art performance on both the HCMSSD and Semap datasets, showing that a diversity-driven approach to map recognition is not only viable but also beneficial. The results indicate that segmentation performance remains largely stable across map collections, scales, geographic regions, and publication contexts. By proposing benchmark datasets and methods for the generic segmentation of historical maps, this work opens the way to integrating the long tail of cartographic archives to historical geographic studies.

Remi Petitpierre

EPFL, Swiss Federal Institute of Technology in Lausanne. Digital Humanities Laboratory (DHLAB), School of Computer and Communication Sciences (IC). CM 1 468, Station 10, 1015 Lausanne, Switzerland. · EPFL. Laboratory of Urban Sociology (LASUR), School of Architecture, Civil and Environmental Engineering (ENAC).

Semantic SegmentationImage SegmentationDomain Generalization
AskPDF
Mar 5, 2026

LAW & ORDER: Adaptive Spatial Weighting for Medical Diffusion and Segmentation

Medical image analysis depends on accurate segmentation and controllable synthesis, but both tasks face severe spatial imbalance: lesions occupy small regions against large backgrounds. We study adaptive spatial weighting as a task-level design principle and instantiate it in two adapters. LAW learns per-pixel loss weights for mask-conditioned diffusion by modulating a ratio prior with a feature-dependent delta map, with normalization, clamping, and Dice regularization for stability. ORDER improves lightweight segmentation by adding selective bidirectional skip attention with stage-wise confidence gating. On held-out diffusion test sets, LAW lowers FID from 158.13±\pm0.15 to 108.43±\pm0.71 on Polyps, from 144.13±\pm0.31 to 89.51±\pm0.96 on KiTS19, and from 139.22±\pm0.38 to 112.58±\pm0.68 on BRISC, while improving held-out mask-recovery Dice from 0.681±\pm0.013 to 0.825±\pm0.003 on Polyps. When the resulting images are added to nnUNet training, downstream Polyps mDice rises from 71.7±\pm0.4 to 74.1±\pm0.8. On the cleaned Polyps segmentation protocol, the reported ORDER configuration reaches 76.3±\pm1.9 mDice and 67.2±\pm2.0 mIoU at 42K parameters and 0.11 GFLOPs, versus 70.3±\pm1.5 mDice and 59.9±\pm1.7 mIoU for matched MK-UNet. On BRISC under the same training recipe, ORDER reaches 77.4±\pm0.8 mDice and 68.1±\pm0.7 mIoU. These results position adaptive spatial weighting as a practical design idea for both medical diffusion and efficient segmentation.

Anugunj Naman, Ayushman Singh, Gaibo Zhang +1

Elmore Family School of Electrical and Computer Engineering Purdue University, West Lafayette, USA · Sesame AI, New York, USA · Department of Computer Science Purdue University, West Lafayette, USA +1

Image SegmentationHealthcareAdaptive Loss WeightingDiffusion-Based Image GenerationMedical Image SegmentationMedical Image Synthesis
AskPDF
Mar 4, 2026

WSI-INR: Implicit Neural Representations for Lesion Segmentation in Whole-Slide Images

Whole-slide images (WSIs) are fundamental for computational pathology, where accurate lesion segmentation is critical for clinical decision making. Existing methods partition WSIs into discrete patches, disrupting spatial continuity and treating multi-resolution views as independent samples, which leads to spatially fragmented segmentation and reduced robustness to resolution variations. To address the issues, we propose WSI-INR, a novel patch-free framework based on Implicit Neural Representations (INRs). WSI-INR models the WSI as a continuous implicit function mapping spatial coordinates directly to tissue semantics features, outputting segmentation results while preserving intrinsic spatial information across the entire slide. In the WSI-INR, we incorporate multi-resolution hash grid encoding to regard different resolution levels as varying sampling densities of the same continuous tissue, achieving a consistent feature representation across resolutions. In addition, by jointly training a shared INR decoder, WSI-INR can capture general priors across different cases. Experimental results showed that WSI-INR maintains robust segmentation performance across resolutions; at Base/4, our resolution-specific optimization improves Dice score by +26.11%, while U-Net and TransUNet decrease by 54.28% and 36.18%, respectively. Crucially, this work enables INRs to segment highly heterogeneous pathological lesions beyond structurally consistent anatomical tissues, offering a fresh perspective for pathological analysis.

Yunheng Wu, Wenqi Huang, Liangyi Wang +4

Graduate School of Informatics, Nagoya University, Nagoya, Japan · Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · School of Computation and Information Technology, Technical University of Munich, Munich, Germany +4

Image SegmentationHistopathology Image SegmentationWhole-Slide Image AnalysisImplicit Neural Representations
AskPDF
Mar 4, 2026

Polyp Segmentation Using Wavelet-Based Cross-Band Integration for Enhanced Boundary Representation

Accurate polyp segmentation is essential for early colorectal cancer detection, yet achieving reliable boundary localization remains challenging due to low mucosal contrast, uneven illumination, and color similarity between polyps and surrounding tissue. Conventional methods relying solely on RGB information often struggle to delineate precise boundaries due to weak contrast and ambiguous structures between polyps and surrounding mucosa. To establish a quantitative foundation for this limitation, we analyzed polyp-background contrast in the wavelet domain, revealing that grayscale representations consistently preserve higher boundary contrast than RGB images across all frequency bands. This finding suggests that boundary cues are more distinctly represented in the grayscale domain than in the color domain. Motivated by this finding, we propose a segmentation model that integrates grayscale and RGB representations through complementary frequency-consistent interaction, enhancing boundary precision while preserving structural coherence. Extensive experiments on four benchmark datasets demonstrate that the proposed approach achieves superior boundary precision and robustness compared to conventional models.

Haesung Oh, Jaesung Lee

Department of Artificial Intelligence, Chung-Ang University, Republic of Korea

Image SegmentationPolyp SegmentationBoundary-Aware Segmentation
AskPDF
Mar 1, 2026

Adaptive Bidirectional Task Interaction for Joint Segmentation and Classification of Breast Ultrasound

Joint lesion segmentation and tissue classification in breast ultrasound are usually trained with a shared encoder, so the two branches stop exchanging information once their decoders separate. That is exactly where boundary detail and semantic evidence are most complementary. The proposed method restores this exchange during decoding and, because its value differs between images, lets the network decide per image how much to keep. A Task Interaction Module (TIM) at each of four decoder levels passes pooled boundary context into the classification representation and modulates decoder channels with class-conditioned priors. An Adaptive Interaction Weighting (AIW) unit then blends interacted and original features with a coefficient computed for each image and level. On BUSI the model reaches 74.19% IoU and 90.60% accuracy, and on BUSI-WHU 86.40% IoU and 95.00% accuracy, ahead of encoder-sharing multi-task, transformer segmentation and decoder-interaction baselines evaluated under the same protocol. The ablation shows that multi-scale context and cross-task exchange are not independent: applied separately they contribute 4.00 points of IoU in total, applied together 6.76. Adding the adaptive blend to task interaction alone raises AUC from 94.41% to 97.31%, indicating that the blend acts primarily on the classification branch. Code: https://github.com/C-loud-Nine/Adaptive-Task-Interaction-BUS.

Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed +2

Khulna University of Engineering and Technology, Bangladesh · University Clermont Auvergne, France

Ultrasound Image SegmentationImage SegmentationMulti-Task LearningMedical Image ClassificationUltrasound ImagingUltrasound Image Classification
AskPDF
Mar 1, 2026

Vision-Language Feature Alignment for Road Anomaly Segmentation

Safe autonomous systems in complex environments require robust road anomaly segmentation to identify unknown obstacles. However, existing approaches often rely on pixel-level statistics to determine whether a region appears anomalous. This reliance leads to high false-positive rates on semantically normal background regions such as sky or vegetation, and poor recall of true Out-of-distribution (OOD) instances, thereby posing safety risks for robotic perception and decision-making. To address these challenges, we propose VL-Anomaly, a vision-language anomaly segmentation framework that incorporates semantic priors from pre-trained Vision-Language Models (VLMs). Specifically, we design a prompt learning-driven alignment module that adapts Mask2Forme's visual features to CLIP text embeddings of known categories, effectively suppressing spurious anomaly responses in background regions. At inference time, we further introduce a multi-source inference strategy that integrates text-guided similarity, CLIP-based image-text similarity and detector confidence, enabling more reliable anomaly prediction by leveraging complementary information sources. Extensive experiments demonstrate that VL-Anomaly achieves state-of-the-art performance on benchmark datasets including RoadAnomaly, SMIYC and Fishyscapes.Code is released on https://github.com/NickHezhuolin/VL-aligner-Road-anomaly-segment.

Zhuolin He, Jiacheng Tang, Jian Pu +1

School of Computer Science, Fudan University, Shanghai 200433, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, 200433, China

Image SegmentationAutonomous Driving PerceptionOOD DetectionVision-Language Alignment
AskPDF
Feb 26, 2026

Zero-Shot and Supervised Bird Image Segmentation Using Foundation Models: A Dual-Pipeline Approach with Grounding DINO1.5, YOLOv11, and SAM2.1

Bird image segmentation remains a challenging task in computer vision due to extreme pose diversity, complex plumage patterns, and variable lighting conditions. This paper presents a dual-pipeline framework for binary bird image segmentation leveraging 2025 foundation models. We introduce two operating modes built upon Segment Anything Model 2.1 (SAM 2.1) as a shared frozen backbone: (1) a zero-shot pipeline using Grounding DINO 1.5 to detect birds via the text prompt "bird" before prompting SAM 2.1 with bounding boxes requiring no labelled bird data; and (2) a supervised pipeline that fine-tunes YOLOv11 on the CUB-200-2011 dataset for high-precision detection, again prompting SAM 2.1 for pixel-level masks. The segmentation model is never retrained for new species or domains. On CUB-200-2011 (11,788 images, 200 species), the supervised pipeline achieves IoU 0.912, Dice 0.954, and F1 0.953 outperforming all prior baselines including SegFormer-B2 (IoU 0.842) by +7.0 percentage points. The zero-shot pipeline achieves IoU 0.831 using only a text prompt, the first such result reported on this benchmark. We demonstrate that prompt-based foundation model pipelines outperform task specific end-to-end trained segmentation networks, while requiring only lightweight detector fine-tuning (~1 hour) for domain adaptation. Complete PyTorch implementation, dataset preparation scripts, and trained weights are publicly available.

Abhinav Munagala

Zero-Shot Referring Image SegmentationImage SegmentationSemantic SegmentationPromptable Image SegmentationVision Foundation Models
AskPDF
Feb 26, 2026

A Boundary-Metric Evaluation Protocol for Whiteboard Stroke Segmentation Under Extreme Imbalance

The binary segmentation of whiteboard strokes is hindered by extreme class imbalance, caused by stroke pixels that constitute only 1.791.79% of the image on average, and in addition, the thin-stroke subset averages 1.141.14% \pm 0.41% in the foreground. Standard region metrics (F1, IoU) can mask thin-stroke failures because the vast majority of the background dominates the score. In contrast, adding boundary-aware metrics and a thin-subset equity analysis changes how loss functions rank and exposes hidden trade-offs. We contribute an evaluation protocol that jointly examines region metrics, boundary metrics (BF1, B-IoU), a core/thin-subset equity analysis, and per-image robustness statistics (median, IQR, worst-case) under seeded, multi-run training with non-parametric significance testing. Five losses -- cross-entropy, focal, Dice, Dice+focal, and Tversky -- are trained three times each on a DeepLabV3-MobileNetV3 model and evaluated on 12 held-out images split into core and thin subsets. Overlap-based losses improve F1 by more than 20 points over cross-entropy (0.6630.663 vs 0.4380.438, p<0.001p < 0.001). In addition, the boundary metrics confirm that the gain extends to the precision of the contour. Adaptive thresholding and Sauvola binarization at native resolution achieve a higher mean F1 (0.7870.787 for Sauvola) but with substantially worse worst-case performance (F1 =0.452= 0.452 vs 0.5650.565 for Tversky), exposing a consistency-accuracy trade-off: classical baselines lead on mean F1 while the learned model delivers higher worst-case reliability. Doubling training resolution further increases F1 by 12.7 points.

Nicholas Korcynski

Rowan University USA

Image SegmentationThin-Structure SegmentationBoundary-Aware Segmentation
AskPDF
Feb 24, 2026

Leveraging GenAI for Segmenting and Labeling Centuries-old Technical Documents

Image segmentation and image recognition are well established computational techniques in the broader discipline of image processing. Segmentation allows to locate areas in an image, while recognition identifies specific objects within an image. These techniques have shown remarkable accuracy with modern images, mainly because the amount of training data is vast. Achieving similar accuracy in digitized images of centuries-old documents is more challenging. This difficulty is due to two main reasons: first, the lack of sufficient training data, and second, because the degree of specialization in a given domain. Despite these limitations, the ability to segment and recognize objects in these collections is important for automating the curation, cataloging, and dissemination of knowledge, making the contents of priceless collections accessible to scholars and the general public. In this paper, we report on our ongoing work in segmenting and labeling images pertaining to shipbuilding treatises from the XVI and XVII centuries, a historical period known as the Age of Exploration. To this end, we leverage SAM2 for image segmentation; Florence2 and ChatGPT for labeling; and a specialized ontology ontoShip and glossary glosShip of nautical architecture for enhancing the labeling process. Preliminary results demonstrate the potential of marrying these technologies for improving curation and retrieval of priceless historical documents. We also discuss the challenges and limitations encountered in this approach and ideas on how to overcome them in the future.

Carlos Monroy, Benjamin Navarro

Department of Mathematics and Computer Science, University of St. Thomas, Houston, Texas

Historical Document ProcessingDocument Image UnderstandingImage Segmentation
AskPDF
Feb 24, 2026

Region of Interest Segmentation and Morphological Analysis for Membranes in Cryo-Electron Tomography

Cryo-electron tomography (cryo-ET) enables high resolution, three-dimensional reconstruction of biological structures, including membranes and membrane proteins. Identification of regions of interest (ROIs) is central to scientific imaging, as it enables isolation and quantitative analysis of specific structural features within complex datasets. In practice, however, ROIs are typically derived indirectly through full structure segmentation followed by post hoc analysis. This limitation is especially apparent for continuous and geometrically complex structures such as membranes, which are segmented as single entities. Here, we developed TomoROIS-SurfORA, a two step framework for direct, shape-agnostic ROI segmentation and morphological surface analysis. TomoROIS performs deep learning-based ROI segmentation and can be trained from scratch using small annotated datasets, enabling practical application across diverse imaging data. SurfORA processes segmented structures as point clouds and surface meshes to extract quantitative morphological features, including inter-membrane distances, curvature, and surface roughness. It supports both closed and open surfaces, with specific considerations for open surfaces, which are common in cryo-ET due to the missing wedge effect. We demonstrate both tools using in vitro reconstituted membrane systems containing deformable vesicles with complex geometries, enabling automatic quantitative analysis of membrane contact sites and remodeling events such as invagination. While demonstrated here on cryo-ET membrane data, the combined approach is applicable to ROI detection and surface analysis in broader scientific imaging contexts.

Xingyi Cheng, Julien Maufront, Aurélie Di Cicco +3

Institut Curie, CNRS UMR168, Université PSL, Sorbonne Université, Paris, France · Inria Center at University of Rennes, SAIRPICO team, Rennes, France · Leiden Institute of Advanced Computer Science (LIACS), Universiteit Leiden, the Netherlands

Image SegmentationElectron MicroscopyCryo-Electron MicroscopySemantic Segmentation
AskPDF
Feb 24, 2026

Onboard-Targeted Segmentation of Straylight in Space Camera Sensors

This study details an artificial intelligence (AI)-based methodology for the semantic segmentation of space camera faults. Specifically, we address the segmentation of straylight effects induced by solar presence around the camera's Field of View (FoV). Anomalous images are sourced from our published dataset. Our approach emphasizes generalization across diverse flare textures, leveraging pre-training on a public dataset (Flare7k++) including flares in various non-space contexts to mitigate the scarcity of realistic space-specific data. A DeepLabV3 model with MobileNetV3 backbone performs the segmentation task. The model design targets deployment in spacecraft resource-constrained hardware. Finally, based on a proposed interface between our model and the onboard navigation pipeline, we develop custom metrics to assess the model's performance in the system-level context.

Riccardo Gallon, Fabian Schiemenz, Alessandra Menicucci +1

Department of Space Systems Engineering, Faculty of Aerospace Engineering, Kluyverweg 1, HS Delft, 2629, Netherlands · Airbus Defence and Space GmbH, Claude-Dornier Straße, Immenstaad am Bodensee, 88090, Germany

Efficient Semantic SegmentationImage SegmentationSemantic Segmentation
AskPDF
Feb 24, 2026

CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objects

Verbal-prompted segmentation is inherently limited by the expressiveness of natural language and struggles with uncommon, instance-specific, or difficult-to-describe objects: scenarios frequently encountered in manufacturing and 3D printing environments. While image exemplars provide an alternative, they primarily encode appearance cues such as color and texture, which are often unrelated to a part's geometric identity. In industrial settings, a single component may be produced in different materials, finishes, or colors, making appearance-based prompting unreliable. In contrast, such objects are typically defined by precise CAD models that capture their canonical geometry. We propose a CAD-prompted segmentation framework built on SAM3 that uses canonical multi-view renderings of a CAD model as prompt input. The rendered views provide geometry-based conditioning independent of surface appearance. The model is trained using synthetic data generated from mesh renderings in simulation under diverse viewpoints and scene contexts. Our approach enables single-stage, CAD-prompted mask prediction, extending promptable segmentation to objects that cannot be robustly described by language or appearance alone.

Zhenran Tang, Rohan Nagabhirava, Changliu Liu

the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA

Image SegmentationInstance SegmentationPromptable Image Segmentation
AskPDF