Image Segmentation

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 44% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 302

Jan 21, 2026cs.CV

Tracing 3D Anatomy in 2D Strokes: A Multi-Stage Projection Driven Approach to Cervical Spine Fracture Identification

Cervical spine fractures require rapid and accurate diagnosis, yet automatic CT interpretation remains challenging as subtle injuries must be assessed across large 3D volumes. We ask whether full 3D vertebra segmentation is necessary for automated fracture recognition, or whether vertebra masks approximated from 2D projections can preserve sufficient diagnostic context. We propose an end-to-end pipeline that localizes the cervical spine, estimates C1-C7 vertebra masks from optimized 2D projections, and uses the resulting vertebra-level volumes for downstream fracture classification. A YOLOv8 detector first localizes spine regions of interest from multi-view variance projections, achieving a 3D mean Intersection over Union of 94.45%. Multi-label vertebra segmentation is then performed with a DenseNet121-Unet on energy-based sagittal and coronal projections, attaining a mean Dice score of 87.86%. The predicted 2D masks are back-projected and fused into approximate 3D masks for each vertebra to extract volumes of interest from the original CT. These volumes are analyzed by an ensemble of 2.5D spatio-sequential CNN-Transformer models, yielding vertebra-level and patient-level F1 scores of 68.15 and 82.26, area under the receiver operating characteristic curve of 91.62 and 90.95, and area under the precision-recall curve of 75.60 and 92.00, respectively. The projection-derived volumes achieved fracture-recognition performance comparable to a full 3D-segmentation baseline, while shifting the vertebra segmentation stage into a lower-dimensional domain. Saliency-based explainability and interobserver variability analysis further examine interpretability and reliability. Overall, the results indicate that projection-based mask approximation is a viable proxy for full 3D vertebra segmentation in cervical fracture recognition.

Fabi Nahian Madhurja, Rusab Sarmun, Muhammad E. H. Chowdhury +3

Department of Computer Science and Engineering, BRAC University, Dhaka 1212, Bangladesh · Department of Electrical and Electronic Engineering, University of Dhaka, Dhaka 1000, Bangladesh · Department of Electrical Engineering, Qatar University, Doha 2713, Qatar +1

Image Segmentation Medical Image Analysis Medical Image Segmentation Computed Tomography

Jan 8, 2026cs.CV

DivAS: Interactive 3D Segmentation by Depth-Weighted Voxel Aggregation

Interactive 3D segmentation of a reconstructed scene should not require a representation-specific optimization loop. We observe that the recipe for lifting 2D foundation-model masks into 3D, namely prompting a few views, refining the resulting masks with rendered depth, and fusing the multi-view evidence into a voxel grid, is shared across scene representations. What remains representation-specific is only the depth signal returned by the renderer and the occupancy prior that gates fusion. We present DivAS (Depth-interactive Voxel Aggregation Segmentation), an optimization-free, training-free framework that realizes this recipe as a single interaction-and-fusion skeleton with lightweight, representation-specific adapters, instantiated on both Gaussian Splatting (GS) and NeRF backbones. On standard forward-facing and unbounded benchmarks, the GS instantiation attains segmentation quality competitive with state-of-the-art optimization-based methods, and the best on LLFF, while being the only one to reach this quality within the consumer-hardware memory envelope at standard resolution. Both instantiations run end-to-end around

2

x faster than feature-field baselines, with a per-update fusion-kernel cost below

70

ms. Because segmentation evidence is gathered from a small, bounded set of anchor views, user effort and computation remain independent of the training-set size. The same skeleton applied to a NeRF backbone matches or exceeds the performance of optimization-based NeRF baselines, confirming that the recipe transfers across fundamentally different 3D representations.

Ayush Pande, Mayank Vatsa

Department of Computer Science and Engineering Indian Institute of Technology Kanpur · Department of Computer Science and Engineering Indian Institute of Technology Jodhpur

Image Segmentation 3D Semantic Segmentation

Jan 6, 2026cs.CV

LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole-Slide Images

Background and Objective: Precise and scalable instance segmentation of cell nuclei is a fundamental prerequisite for computational pathology, yet gigapixel whole-slide images (WSIs) pose significant computational challenges. While patch-based processing is standard during training, existing methods are often limited to small tile sizes during inference due to architectural bottlenecks or reliance on computationally expensive post-processing for instance separation. We introduce a faster, scalable, and end-to-end framework capable of processing large-scale image tiles while accurately modeling biologically realistic overlapping nuclei. Methods: We propose LSP-DETR (Local Star Polygon DEtection TRansformer). The model represents nuclei as star-convex polygons and employs a lightweight transformer with linear complexity, enabling the processing of high-resolution images in a single forward pass. A novel radial distance loss accommodates annotation uncertainty, allowing the segmentation of overlapping nuclei to emerge naturally without explicit overlap labels. Results: LSP-DETR achieves state-of-the-art efficiency, with an inference time of 0.45 s/mm^2, a 3.2x speedup over StarDist, the next-fastest method. On PanNuke, the model achieves competitive accuracy (67.5 bPQ), while yielding an F

_1

-score of 0.964 in polygon overlap when evaluated against consensus annotations from two expert pathologists. Furthermore, it outperforms larger models such as LKCell in generalization robustness, reaching an F1-score of 85.0 on MoNuSeg. Conclusions: LSP-DETR bridges the gap between high-fidelity segmentation and practical clinical requirements by eliminating heuristic post-processing. By providing a scalable, linear-complexity solution that naturally handles overlaps between nuclei, this framework sets a new direction for efficient high-throughput WSI analysis in digital pathology.

Matěj Pekár, Vít Musil, Rudolf Nenutil +2

Masaryk University, Faculty of Informatics, Botanická 68a, Brno, 602 00, Czech Republic · Masaryk Memorial Cancer Institute, Žlutý kopec 7, Brno, 656 53, Czech Republic · BBMRI-ERIC, Neue Stiftingtalstraße 2/B/6, Graz, 8010, Austria

Computational Pathology Image Segmentation Histopathology Image Segmentation Instance Segmentation Detection Transformer Whole-Slide Image Analysis

Jan 2, 2026cs.CV

Learning to Segment Liquids in Real-world Images

Liquids like water, wine and medicine are everywhere. However, limited attention has been given to the task of segmenting liquids, hindering the ability of robots to safely avoid and interact with them. The segmentation of liquids is difficult because liquids come in diverse appearances and shapes; moreover, they can be both transparent or reflective, taking on arbitrary objects and scenes from their background and surroundings. To take on this challenge, we construct a liquid dataset, LQDS, consisting of 5000 real-world images annotated into 14 distinct classes, and design a novel liquid detection model, LQDM, which leverages cross-attention between a dedicated boundary branch and the main segmentation branch to enhance mask predictions. Extensive experiments demonstrate the effectiveness of LQDM on the testing set of LQDS, outperforming state-of-the-art methods to establish a strong baseline for the semantic segmentation of liquids. We believe that LQDS and LQDM will facilitate future research in liquid segmentation and enable practical applications in robotics. Our dataset and code is released at https://lonaslee.github.io/LQDM/.

Jonas Li, Michelle Li, Luke Liu +2

Department of Computer Science and Engineering, University of North Texas, Denton, TX 76203, USA

Image Segmentation Boundary-Aware Segmentation Robotic Perception Semantic Segmentation

Dec 20, 2025cs.CV

Atlas is Your Perfect Context: One-Shot Customization for Generalizable Foundational Medical Image Segmentation

Accurate segmentation of anatomical structures in medical images is essential for diagnosis and treatment planning. While recent interactive segmentation foundation models enhance generalization through large-scale multimodal pretraining, they still depend on precise prompts and can fail in underrepresented clinical contexts (e.g., small organs-at-risk). We present AtlasSegFM, an atlas-guided framework that customizes off-the-shelf foundation models to new clinical contexts with a single annotated example. AtlasSegFM 1) performs atlas-query registration to generate context-aware prompts, 2) refines the segmentation with a frozen foundation model, and 3) applies a lightweight adaptive fusion module to combine atlas priors with foundation-model inputs and predictions. Extensive experiments on six public and in-house datasets across radiotherapy and vascular scenarios show consistent gains, with the largest improvements on small and delicate structures. AtlasSegFM provides a lightweight, deployable solution for one-shot customization of segmentation foundation models in real-world clinical workflows.

Ziyu Zhang, Yi Yu, Simeng Zhu +4

Nanjing University · The Ohio State University · Stanford University

Image Segmentation Medical Imaging Foundation Models Promptable Image Segmentation Medical Image Segmentation

Dec 16, 2025cs.AI

Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning

Recent medical MLLMs have made significant progress in generating step-by-step textual reasoning chains. However, they still struggle with complex clinical tasks that necessitate dynamic and iterative focusing on fine-grained visual regions. To close this gap, we introduce Ophiuchus, a versatile, tool-augmented framework that equips an MLLM to (i) decide when fine-grained visual evidence is needed, (ii) determine where to probe and ground within the medical image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved, multimodal chain of thought for precise segmentation and diagnosis. Ophiuchus moves beyond mere tool-calling by tightly fusing the MLLM's inherent grounding and reasoning capabilities with external tools, enabling more accurate and trustworthy decisions. The core of our method is a three-stage training strategy: cold-start SFT for basic tool selection; self-reflection fine-tuning to strengthen decision revision; and agentic tool reinforcement learning to elicit sophisticated, expert-like diagnostic behaviors. Extensive experiments show that Ophiuchus consistently outperforms both closed-source and open-source SOTA methods across diverse medical benchmarks, including VQA, detection, and reasoning-based segmentation. Our project code is available at https://github.com/SII-zyj/Ophiuchus.

Yankai Jiang, Yujie Zhang, Peng Zhang +5

Zhejiang University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +5

Image Segmentation Tool Use in VLMs Healthcare Medical VQA Medical VLMs Medical Image Segmentation+2

Dec 12, 2025cs.CV

Uncertainty-Aware Domain Adaptation for Vitiligo Segmentation in Clinical Photographs

Accurately quantifying vitiligo extent in routine clinical photographs is crucial for longitudinal monitoring of treatment response. We propose a trustworthy, frequency-aware segmentation framework built on three synergistic pillars: (1) a data-efficient training strategy combining domain-adaptive pre-training on the ISIC 2019 dataset with an ROI-constrained dual-task loss to suppress background noise; (2) an architectural refinement via a ConvNeXt V2-based encoder enhanced with a novel High-Frequency Spectral Gating (HFSG) module and stem-skip connections to capture subtle textures; and (3) a clinical trust mechanism employing K-fold ensemble and Test-Time Augmentation (TTA) to generate pixel-wise uncertainty maps. Extensive validation on an expert-annotated clinical cohort demonstrates superior performance, achieving a Dice score of 85.05% and significantly reducing boundary error (95% Hausdorff Distance improved from 44.79 px to 29.95 px), consistently outperforming strong CNN (ResNet-50 and UNet++) and Transformer (MiT-B5) baselines. Notably, our framework demonstrates high reliability with zero catastrophic failures and provides interpretable entropy maps to identify ambiguous regions for clinician review. Our approach suggests that the proposed framework establishes a robust and reliable standard for automated vitiligo assessment.

Wentao Jiang, Vamsi Varra, Caitlin Perez-Stable +3

Electrical and Computer Engineering, Rice University, Houston, TX, United States · U.S. Dermatology Partners, Houston, TX, United States · School of Medicine, Baylor College of Medicine, Houston, TX, United States

Image Segmentation Domain Adaptation Medical Image Segmentation Convolutional Neural Networks Segmentation Uncertainty Estimation

Dec 5, 2025cs.CV

Ideal Observer for Segmentation of Dead Leaves Images

The visible parts of a scene are determined by occlusion among overlapping surfaces. Here we consider "dead leaves" models, which replicate this by independently sampling objects ("leaves") with position, shape, color, and texture and layering them until the image is covered. Building on prior theory, we present a self-contained framework that rigorously defines the dead leaves model and derives an analytical Bayesian ideal observer for partitioning finite pixel sets. The longest part of the paper spans the derivation of the prior probability, which elevates the observer beyond pixel-similarity methods by incorporating geometric information. These computations are practical only for small pixel sets (up to 9-10 pixels). We emphasize accessibility through step-by-step derivations, extensive visualizations, and examples. We empirically evaluate three tractable observers (prior-only, likelihood-only, and the full ideal observer), plus a random baseline on 108 dead leaves image datasets varying in texture intensity, leaf size, and image size. Likelihood-only performance falls with increasing texture intensity and image size. Prior-only performance falls with decreasing leaf size and increasing maximal image dimension. All model-based observers strongly outperform the random baseline, and the ideal observer consistently outperforms the others by combining both information sources. The model provides a principled upper bound on segmentation performance for limited pixel sets, enabling comparisons with human observers and algorithms.

Swantje Mahncke, Malte Ott, Lars C. Reining +1

Centre for Cognitive Science, Institute of Psychology, TU Darmstadt, Germany · Center for Mind, Brain and Behavior (CMBB), Universities of Marburg, Giessen, and Darmstadt, Germany · Department of Mathematics, TU Darmstadt, Germany

Image Segmentation Occlusion Reasoning Bayesian Inference

Nov 24, 2025cs.CV

MedSAM3: Delving into Segment Anything with Medical Concepts

Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.

Anglin Liu, Xu R. Cao, Yifan Shen +4

The Hong Kong University of Science and Technology (Guangzhou) · University of Illinois Urbana-Champaign · Southeast University +1

Image Segmentation Medical Imaging Foundation Models Text-Guided Image Segmentation Medical Image Segmentation

Sep 29, 2025cs.CV

Improved Robustness from Biologically Inspired Sparse Contrast Representations

Deep neural networks surpass humans on many vision benchmarks, yet remain far less robust to distribution shifts such as illumination and weather changes. Existing approaches address this challenge by additional training data, extensive augmentation, architectural modifications, or test-time adaptation. In this work, we explore a complementary direction: inspired by the human retina, we propose a fixed, model-agnostic preprocessing module that extracts signals that are more stable with respect to variations of illumination. Our method combines color remapping with local contrast extraction, producing sparse representations that emphasize structural features. We study its impact on semantic segmentation by training on Cityscapes and evaluating generalization under adverse conditions on Dark Zurich and ACDC. Our results show that the biologically inspired preprocessing preserves in-distribution performance while consistently improving robustness in challenging lighting scenarios, such as nighttime, where annotated training data are scarce. Moreover, the segmentation accuracy remains stable even when the contrast-based representation is sparsified by up to 70%. These gains suggest that rethinking the input representation itself can improve robustness while also opening opportunities for lower-latency, transmission-aware imaging sensors when sparsity can be exploited close to acquisition.

Lorena Stracke, Lia Nimmermann, Shashank Agnihotri +3

Media Systems, University of Siegen, Germany · Data and Web Science Group, University of Mannheim, Germany · Analogue Circuits and Image Sensors, University of Siegen, Germany +1

Distribution Shift Robustness Image Segmentation Image Corruption Robustness Visual Representation Learning Neural Network Robustness Semantic Segmentation

Sep 26, 2025cs.CV

RefAM: Attention Magnets for Zero-Shot Referral Segmentation

Most existing approaches to referring segmentation achieve strong performance only through fine-tuning or by composing multiple pre-trained models, often at the cost of additional training and architectural modifications. Meanwhile, large-scale generative diffusion models encode rich semantic information, making them attractive as general-purpose feature extractors. In this work, we introduce a new method that directly exploits features (attention scores) from diffusion transformers for downstream tasks, requiring neither architectural modifications nor additional training. To systematically evaluate these features, we extend benchmarks with vision-language grounding tasks spanning both images and videos. Our key insight is that stop words act as attention magnets: they accumulate surplus attention and can be filtered to reduce noise. Moreover, we identify global attention sinks (GAS) emerging in deeper layers and show that they can be safely suppressed or redirected onto auxiliary tokens, leading to sharper and more accurate grounding maps. We further propose an attention redistribution strategy, where appended stop words partition background activations into smaller clusters, yielding sharper and more localized heatmaps. Building on these findings, we develop RefAM, a simple training-free grounding framework that combines cross-attention maps, GAS handling, and redistribution. Across zero-shot referring image and video segmentation benchmarks, our approach achieves strong performance and surpasses prior methods on most datasets, establishing a new state of the art among training-free methods, without task-specific training or architectural modification. For video, we localize the referent from the spatio-temporal features of a video diffusion transformer via a first-frame query that is then propagated; we therefore do not explicitly resolve temporal (which-frame) expressions.

Anna Kukleva, Enis Simsar, Alessio Tonioni +4

Max Planck Institute for Informatics, SIC · ETH Zürich · Google +1

Zero-Shot Learning Image Segmentation Video Object Segmentation Diffusion Transformer Referring Video Object Segmentation Referring Image Segmentation+3

Sep 26, 2025cs.CV

RAU: Reference-based Anatomical Understanding with Vision Language Models

Anatomical understanding, which is the ability to identify, localize, or segment anatomical structures, is critical in medical image analysis; however, its progress is constrained by the scarcity of expert-labeled data. A promising remedy is to leverage an annotated reference image to guide the interpretation of an unlabeled target. Although recent vision-language models (VLMs) exhibit non-trivial visual reasoning, their reference-based understanding and fine-grained localization remain limited. We introduce RAU, a framework for reference-based anatomical understanding with VLMs. We first show that a VLM learns to identify anatomical regions through relative spatial reasoning between reference and target images, trained on a moderately sized dataset. We validate this capability through visual question answering (VQA) and bounding box prediction. Next, we demonstrate that the VLM-derived spatial cues can be seamlessly integrated with the fine-grained segmentation capability of SAM2, enabling localization and pixel-level segmentation of small anatomical regions, such as vessel segments. Across two in-distribution and two out-of-distribution datasets, RAU consistently outperforms a SAM2 fine-tuning baseline using the same memory setup, yielding more accurate segmentations and more reliable localization. More importantly, its generalization ability to unseen modalities makes it scalable to unseen datasets, a property crucial for medical image applications. To the best of our knowledge, RAU is the first to explore the capability of VLMs for reference-based identification, localization, and segmentation of anatomical structures in medical images. Its promising performance highlights the potential of VLM-driven approaches for anatomical understanding in automated clinical workflows.

Yiwei Li, Yikang Liu, Jiaqi Guo +7

United Imaging Intelligence, Boston, MA, USA · School of Computing, University of Georgia, Athens, GA, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA

Image Segmentation Medical VLMs Medical Image Segmentation Medical Image Localization

Sep 26, 2025cs.CV

Johnson-Lindenstrauss Lemma Guided Network for Efficient 3D Medical Segmentation

Lightweight 3D medical image segmentation remains constrained by a fundamental \textit{

efficiency / robustness conflict''}, particularly when processing complex anatomical structures and heterogeneous modalities. In this paper, we study how to redesign the framework based on the characteristics of high-dimensional 3D images, and explore data synergy to overcome the fragile representation of lightweight methods. Our approach, VeloxSeg, begins with a deployable and extensible dual-stream CNN-Transformer architecture composed of Paired Window Attention (PWA) and Johnson-Lindenstrauss lemma-guided convolution (JLC). For each 3D image, we invoke a

glance-and-focus'' principle, where PWA rapidly retrieves multi-scale information, and JLC ensures robust local feature extraction with minimal parameters, significantly enhancing the model's ability to operate with low computational budget. Followed by an extension of the dual-stream architecture that incorporates modal interaction into the multi-scale image-retrieval process, VeloxSeg efficiently models heterogeneous modalities. Finally, Spatially Decoupled Knowledge Transfer (SDKT) via Gram matrices injects the texture prior extracted by a self-supervised network into the segmentation network, yielding stronger representations than baselines at no extra inference cost. Experimental results on multimodal benchmarks show that VeloxSeg achieves a 26% Dice improvement, alongside increasing GPU throughput by 11

\times

, CPU by 48

\times

, and reducing training peak GPU memory usage by

1/20

, inference by

1/24

. Code is available at https://github.com/JinPLu/VeloxSeg.

Jinpeng Lu, Linghan Cai, Yinda Chen +4

University of Science and Technology of China · Dresden University of Technology · Harbin Institute of Technology, Shenzhen

Cross-Modality Medical Image Segmentation Image Segmentation Hybrid CNN-Transformer Architectures 3D Medical Image Segmentation

Sep 5, 2025math.NA

Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation

Deep learning (DL) has transformed neuroimaging by delivering state-of-the-art performance with reduced computation times. Yet, the numerical uncertainty inherent to DL training remains largely underexplored despite its potential to significantly impact the reliability of model outcomes. We show that training the FastSurfer segmentation model introduces substantial numerical uncertainty that exceeds its non-DL counterpart (FreeSurfer 7.3.2) in cortical regions, potentially impacting downstream clinical results. We also characterize this training-time uncertainty using random seed perturbations and demonstrate that seed-induced variability is structurally comparable to numerical variability. We then show that seed variability can be leveraged as a data augmentation technique through ensembling to improve downstream brain age regression performance. These findings position numerical uncertainty during DL training as a substantive factor in neuroimaging reliability, with measurable consequences for downstream tasks, and demonstrate that it can simultaneously be harnessed as a data augmentation technique.

Inés Gonzalez-Pepe, Vinuyan Sivakolunthu, Yohan Chatelain +1

Department of Computer Science Concordia University Montreal, QC H3G 1M8, Canada · Department of Computer Science Concordia University2026 Montreal, QC H3G 1M8, Canada · Krembil Centre for Neuroinformatics Centre for Addiction and Mental Health Toronto, ON M5T 0S8, Canada +1

Image Segmentation Data Augmentation Neuroimaging Neural Network Training Dynamics Segmentation Uncertainty Estimation Brain MRI

Aug 24, 2025cs.CV

Achieving detailed medial temporal lobe segmentation with upsampled isotropic training from implicit neural representation

Imaging biomarkers in magnetic resonance imaging (MRI) are important tools for diagnosing, tracking and treating Alzheimer's disease (AD). Neurofibrillary tau pathology in AD is closely linked to neurodegeneration and generally follows a pattern of spread in the brain, with early stages involving subregions of the medial temporal lobe (MTL). Accurate segmentation of MTL subregions is needed to extract granular biomarkers of AD progression. MTL subregions are often imaged using T2-weighted (T2w) MRI scans that are highly anisotropic due to constraints of MRI physics and image acquisition, making it difficult to reliably model MTL subregions geometrically and extract morphological measures, such as thickness. In this study, we propose a segmentation framework for MTL subregions in isotropic space, in which an implicit neural representation is used to construct the isotropic training atlas from the anisotropic low-resolution T2w data, with T1w MRI as an auxiliary modality to support the INR and segmentation. In an independent test set, the morphological measures extracted using this isotropic model showed stronger effect sizes than those from models trained on anisotropic data in distinguishing participants with mild cognitive impairment (MCI) from cognitively unimpaired individuals. In the test-retest analysis, the morphological measures extracted using the isotropic model showed greater stability than those from the anisotropic segmentation. This study demonstrates improved reliability of MRI-derived MTL subregion biomarkers without additional atlas annotation effort, which may more accurately quantify and track the relationship between AD pathology and brain atrophy for monitoring disease progression.

Yue Li, Pulkit Khandelwal, Rohit Jena +8

University of Pennsylvania, Philadelphia, United States · Siemens Healthineers, Princeton, United States · Lund University, Lund, Sweden

Alzheimer's Disease Image Segmentation Medical Image Segmentation Neuroimaging Brain MRI

May 27, 2025cs.CV

Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets

Manually refining radiological segmentation masks is highly resource-intensive. To determine when this expert commitment is truly justified for the training of segmentation models, we investigate the relationship between label quality and model performance. Expanding beyond models trained directly for inference, we conduct the first study isolating the impact of label quality in pre-training datasets. While high-quality labels remain essential for models proceeding directly to deployment, we find no evidence that strict label quality is crucial for pre-training efficacy. These results question the necessity of exhaustive human-in-the-loop refinement for massive corpora intended for pretraining and suggest that expert effort is more effectively invested in well-curated downstream target datasets.

Alexander Jaus, Zdravko Marinov, Constantin Seibold +4

Image Segmentation Medical Imaging Foundation Models Healthcare Medical Image Segmentation Learning with Noisy Labels Medical Image Representation Learning

May 26, 2025cs.CV

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

We present \textbf{LlamaSeg}, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the \textbf{SA-OVRS} dataset, which contains \textbf{2M} segmentation masks annotated with over \textbf{5,800} open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance (

d_{\mathrm{AHD}}

) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at https://github.com/GML-FMGroup/llamaseg.

Jiru Deng, Tengjin Weng, Tianyu Yang +3

Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2

Image Segmentation Text-Guided Image Segmentation Promptable Image Segmentation Open-Vocabulary Semantic Segmentation

May 17, 2025eess.IV

Bridging the Inter-Domain Gap through Low-Level Features for Cross-Modal Medical Image Segmentation

This paper addresses cross-modal medical image segmentation, focusing on MRI-CT transfer in a source-only domain generalization setting. During training, only source-modality samples are available, while unlabeled target-modality images are used for testing. We propose LowBridge, which builds on the observation that cross-modal images share similar low-level features (e.g. edges) as they depict the same types of anatomical structures. Specifically, we first train a generative model to recover the source images from their edge features, followed by training a segmentation model on the generated source images, separately. At test time, edge features from the target images are input to the pretrained generative model to generate source-style target domain images, which are then segmented using the pretrained segmentation network. Experiments on various public datasets demonstrate that LowBridge achieves state-of-the-art performance, outperforming ten existing approaches. Ablation studies further show that LowBridge is compatible with different types of generative and segmentation models, suggesting its generalizability and potential to benefit from future advances in these models. The code will be available at https://github.com/JoshuaLPF/LowBridge.

Pengfei Lyu, Pak-Hei Yeung, Jing Xia +5

Northeastern University, Shenyang, China · Nanyang Technological University, Singapore

Cross-Modal Learning Cross-Modality Medical Image Segmentation Image Segmentation Domain Generalization Cross-Domain Transfer Learning

May 6, 2025cs.CV

Synergistic Vision-Language Reinforcement Enables Scalable On-Demand Analysis across Diverse Clinical Tasks

Accurate delineation of tumors and surrounding organs-at-risk is essential for radiotherapy, surgery and treatment response assessment, yet remains time-consuming and expertise-intensive. Existing artificial intelligence systems often require manual spatial prompts or task-specific retraining, while generic class labels provide limited semantic grounding for heterogeneous disease targets. Here we present SyRe, a promptable segmentation foundation model based on Synergistic vision-language Reinforcement. SyRe strengthens bidirectional interaction between visual and linguistic representations to improve semantically grounded spatial understanding. To support large-scale training, we introduce the Color Region Description strategy and construct SyReData, comprising 20 million image-mask-description triplets across 9 modalities and 229 segmentation tasks. Training with diversified prompt forms further enables open-ended prompting, invalid-prompt rejection and flexible switching between single- and multi-target analysis. SyRe achieves accurate text-prompted segmentation across diverse clinical scenarios, with particularly strong performance on disease-related targets. Across 28 unseen external datasets, including 20 cancer types and multinational in-house cohorts, SyRe generalizes robustly under real-world distribution shifts. SyRe-generated masks also preserve clinically relevant quantitative information in pathology and yield radiomics features that stratify survival and improve prognostic modeling across five retrospective CT and MRI tumor cohorts. Finally, clinician-in-the-loop refinement enables efficient case-level correction when greater precision is required. These results establish SyRe as a generalizable foundation for scalable quantitative oncology and clinician-guided segmentation refinement.

Haonan Wang, Jiaji Mao, Lehan Wang +11

Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · Department of Radiology, Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University, Guangzhou 510120, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR

Image Segmentation Medical Imaging Foundation Models Text-Guided Image Segmentation Medical VLMs Medical Image Segmentation Oncology+1

Apr 2, 2025cs.CV

Semi-Supervised Biomedical Image Segmentation via Diffusion Models and Teacher-Student Co-Training

Supervised deep learning achieves strong performance in biomedical image segmentation but relies on costly pixel-wise annotations, motivating semi-supervised approaches that exploit unlabeled data. We introduce a diffusion-based teacher--student framework in which segmentation predictions are used to condition image denoising, encouraging the production of more informative pseudo-labels. The teacher is first pretrained through an unsupervised reconstruction task using diffusion-style corruption, timestep conditioning, and denoising. Starting from a corrupted empty mask, the model predicts an intermediate segmentation that conditions image denoising, encouraging the predicted mask to capture structural information useful for recovering the original image. The resulting teacher is then co-trained with a student using supervised segmentation on labeled samples and cross pseudo-supervision on unlabeled data. We further introduce a multi-round extension during co-training, in which the teacher generates multiple stochastic image reconstructions and corresponding segmentation predictions, providing additional reconstruction and alignment signals to improve its pseudo-labels. We evaluate the proposed framework on three public 2D biomedical segmentation datasets and a 3D left atrial segmentation benchmark. Across several labeling regimes, our method achieves competitive or superior performance compared with state-of-the-art semi-supervised approaches, with the largest gains observed under severe label scarcity.

Luca Ciampi, Gabriele Lagani, Giuseppe Amato +1

ISTI-CNR, Pisa, Italy

Image Segmentation Semi-Supervised Medical Image Segmentation Medical Image Segmentation Pseudo-Labeling Teacher-Student Learning Diffusion Models

Dec 20, 2024cs.CV

SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

Improving the reliability and completeness of colonoscopic inspection is critical for reducing missed lesions and improving colorectal cancer prevention. Reliable scene understanding is essential for navigation, reconstruction, and assessment of inspection completeness. Anatomical structures such as mucosal folds provide stable geometric cues for endoscope localization, while surgical instruments introduce dynamic occlusions that complicate visual interpretation. However, existing gastrointestinal endoscopy datasets largely focus on disease detection or artifact segmentation, leaving a gap in precise annotations of structural landmarks and instruments. We introduce SegCol, a dataset and benchmark for semantic segmentation of colon fold edges and surgical instruments derived from the EndoMapper dataset. SegCol provides manually annotated pixel-level masks for three instrument classes and thin fold-edge structures across temporally consistent image sequences. It forms the basis of the SegCol Challenge, organized as part of the EndoVis Challenge at MICCAI 2024, evaluating both supervised segmentation and annotation-efficient active learning. We further study segmentation metrics, including Dice, ODS/OIS, AP, and CLDice, under structural perturbations and different object geometries, and analyze participating methods, architectural choices, and active learning strategies. Our findings show that metric behavior strongly depends on target structure, highlighting the need for carefully selected evaluation protocols in endoscopic segmentation. Details are available at https://www.synapse.org/Synapse:syn54124209/wiki/626563, and code at https://github.com/surgical-vision/segcol_challenge.

Xinwei Ju, Rema Daher, Razvan Caramalau +7

UCL Hawkes Institute, University College London, London, UK · Department of Computer Science, University College London, London, UK · Medtronic, London, UK +3

Endoscopy Image Segmentation Active Learning Medical Image Segmentation Semantic Segmentation Medical Image Benchmarks

Dec 17, 2024cs.CV

Open-World Panoptic Segmentation

Robots need to be able to understand their surroundings in order to operate safely and robustly, and to interact with the surrounding environment. Robots deployed in unconstrained real-world scenarios must additionally be able to deal with novel situations and objects that have never been seen before. In this article, we tackle the problem of open-world panoptic segmentation, i.e., the task of discovering new semantic categories and new object instances at test time, while enforcing consistency among the categories that we incrementally discover. We present Con2MAV, a general method for open-world panoptic segmentation. Experiments across a wide range of datasets, from road scenes to underwater environments, highlight its compelling capabilities in open-world segmentation and its competitive performance on known classes. We will open-source the implementation of our approach upon acceptance. In addition, we propose PANIC (Panoptic ANomalies In Context), a benchmark for evaluating open-world segmentation tasks in autonomous driving scenarios. This dataset, recorded with a multi-modal sensor suite mounted on a car, and then manually annotated, provides high-quality, pixel-wise annotations of anomalous objects at both semantic and instance level. PANIC contains 800 images, more than 50 unknown classes, i.e., classes that do not appear in the training set, and over 4,000 object instances, providing a comprehensive benchmark for evaluating open-world segmentation methods in autonomous driving scenarios. We provide competitions for multiple open-world segmentation tasks on a hidden test set. Our dataset and competitions are available at https://www.ipb.uni-bonn.de/data/panic.

Matteo Sodano, Federico Magistri, Jens Behley +1

Center for Robotics, University of Bonn, Germany, and the Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

Image Segmentation Panoptic Segmentation Autonomous Driving Perception

Dec 2, 2024eess.AS

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.

Hugo Malard, Michel Olvera, Stephane Lathuiliere +1

LTCI, Télécom Paris, Institut Polytechnique de Paris

Image Segmentation Audio-Visual Representation Learning

Nov 4, 2024cs.CV

Segment Anything for Dendrites from Electron Microscopy

Segmentation of cellular structures in electron microscopy (EM) images is fundamental to analyzing the morphology of neurons and glial cells in the healthy and diseased brain tissue. Current neuronal segmentation applications are based on convolutional neural networks (CNNs) and do not effectively capture global relationships within images. Here, we present DendriteSAM, a vision foundation model based on Segment Anything, for interactive and automatic segmentation of dendrites in EM images. The model is trained on high-resolution EM data from healthy rat hippocampus and is tested on diseased rat and human data. Our evaluation results demonstrate better mask quality compared to the original and other fine-tuned models, leveraging the features learned during training. This study introduces the first implementation of vision foundation models in dendrite segmentation, paving the path for computer-assisted diagnosis of neuronal anomalies.

Zewen Zhuo, Ilya Belevich, Ville Leinonen +4

A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland · Electron Microscopy Unit Institute of Biotechnology University of Helsinki Helsinki, Finland · Department of Medicine Faculty of Health Sciences University of Eastern Finland Kuopio, Finland

Image Segmentation Vision Foundation Models Promptable Image Segmentation Microscopy Image Analysis

May 12, 2024eess.IV

Explainable Convolutional Neural Networks for Retinal Fundus Classification and Cutting-Edge Segmentation Models for Retinal Blood Vessels from Fundus Images

Early detection of vision-threatening conditions such as diabetic retinopathy, glaucoma, and age-related macular degeneration depends on retinal fundus image analysis, but manual assessment is slow and expert-dependent. Automated convolutional neural networks classify fundus images accurately yet act as black boxes, and existing retinal vessel segmentation methods lose discriminative power under pathology and seldom exploit attention or transformer backbones. Using the FIVES and DRIVE fundus datasets, we develop a two-pipeline framework that pairs four-class disease classification with attention- and transformer-based vessel segmentation, organised in three stages: (1) FIVES images are augmented by rotation and horizontal and vertical flips and used to fine-tune eight ImageNet-pretrained CNNs: ResNet101, DenseNet169, Xception, InceptionV3, DenseNet121, InceptionResNetV2, ResNet50, and EfficientNetB0. (2) Five gradient-based explanation methods, Grad-CAM, Grad-CAM++, Score-CAM, Faster Score-CAM, and Layer-CAM, are computed on the final convolutional block of each classifier and compared qualitatively across architectures. (3) Ten U-Net variants are benchmarked for vessel segmentation: TransUNet (hybrid CNN--Transformer encoder) and Attention U-Net (gated skip connections), evaluated with ResNet50V2, ResNet101V2, and ResNet152V2 backbones, along with additional Attention U-Net configurations using DenseNet backbones, and the fully transformer-based Swin-UNet. ResNet101 gives the highest classification accuracy: 94.17% (F1 0.942)

>

88.33% for EfficientNetB0. For segmentation, the architecture ranking is consistent on both datasets: Attention U-Net

>

TransUNet

>

Swin-UNet. The strongest configuration is Attention U-Net with a ResNet101V2 backbone: FIVES IoU 0.722, Dice 0.838; DRIVE IoU 0.648, Dice 0.787, lifting DRIVE IoU 60.80

\rightarrow

64.83 over a prior custom U-Net.

Fatema Tuj Johora Faria, Mukaffi Bin Moin, Pronay Debnath +2

aDepartment of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh

Image Segmentation Retinal Vessel Segmentation Hybrid CNN-Transformer Architectures U-Net Medical Image Classification Medical Image Segmentation+3

Jan 10, 2024cs.CV

A locally statistical active contour model for SAR image segmentation can be solved by denoising algorithms

In this paper, we propose a novel locally statistical variational active contour model based on I-divergence-TV denoising model, which hybrides geodesic active contour (GAC) model with active contours without edges (ACWE) model, and can be used to segment images corrupted by multiplicative gamma noise. By adding a diffusion term into the level set evolution (LSE) equation of the proposed model, we construct a reaction-diffusion (RD) equation, which can gradually regularize the level set function (LSF) to be piecewise constant in each segment domain and gain the stable solution. We further transform the proposed model into a general ROF model by adding a proximity term ,and it can be solved by a fast denoising algorithm proposed by Jia-Zhao or soved by BM3D and NLM denoising algorithm, which also provide a unified solution framework for formally generalized-ROF-like subproblems arising in multivariate splitting algorithms.

Guangming Liu

Yantai Science and Technology Association

Image Segmentation Remote Sensing Image Segmentation

Dec 29, 2023cs.CV

The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography

The Circle of Willis (CoW) is an important network of arteries connecting major circulations of the brain. Its vascular architecture is believed to influence the risk, severity, and outcome of serious neurovascular diseases. However, characterizing the highly variable CoW anatomy remains a manual and time-consuming expert task. The CoW is commonly imaged by two non-invasive angiographic imaging modalities, magnetic resonance angiography (MRA) and computed tomography angiography (CTA), yet few datasets with annotated CoW anatomy exist, and there have been no established benchmarks for comparing CoW segmentation algorithms. We organized the TopCoW benchmark challenge alongside the release of an annotated CoW dataset with 125 paired MRA and CTA scans from the same patients. Voxel-level annotations for 13 vessel components were created using virtual reality technology and verified by clinical experts. Participants submitted algorithms for CoW segmentation and variant classification, which we evaluated on internal and external test sets comprising 226 scans from over five centers. The benchmark includes voxel-level segmentation, CoW component detection, CoW variant classification, and two clinical application tasks. We received submissions from over 250 participants across six continents. Top-performing teams achieved over 90% Dice scores for CoW segmentation, over 80% F1 scores for detecting key vessel components, and over 70% balanced accuracy in CoW variant classification across nearly all test sets. The best algorithms also supported clinically relevant downstream tasks by accurately classifying fetal-type posterior cerebral arteries and localizing aneurysms in relation to CoW anatomy. This benchmark demonstrated the utility of CoW segmentation algorithms for some downstream clinical applications with explainability.

Kaiyuan Yang, Fabio Musio, Yihui Ma +112

Image Segmentation Medical Image Segmentation Medical Image Benchmarks

Dec 19, 2023cs.CV

Active contours driven by local and global intensity fitting energy with application to SAR image segmentation and its fast solvers

In this paper, we propose a novel variational active contour model based on Aubert-Aujol (AA) denoising model, which hybrides geodesic active contour (GAC) model with active contours without edges (ACWE) model and can be used to segment images corrupted by multiplicative gamma noise. We transform the proposed model into classic ROF model by adding a proximity term.[26] was submitted on 29-Aug-2013, and our early edition was ever submitted to TGRS on 12-Jun-2012, Venkatakrishnan et al.[27] proposed their PnP algorithm on 29-May-2013, so Venkatakrishnan and we proposed the PnP algorithm almost simultaneously. Inspired by a fast denosing algorithm proposed by Jia-Zhao recently, we propose two fast fixed point algorithms to solve SAR image segmentation question.

Guangming Liu

Yantai Science and Technology Association

Image Segmentation Remote Sensing Image Segmentation

Dec 9, 2023cs.CV

SAR image segmentation algorithms based on I-divergence-TV model

In this paper, we propose a novel variational active contour model based on I-divergence-TV model to segment Synthetic aperture radar (SAR) images with multiplicative gamma noise, which hybrides edge-based model with region-based model. The proposed model can efficiently stop the contours at weak or blurred edges, and can automatically detect the exterior and interior boundaries of images. We further transform the proposed model into a general ROF model by adding a proximity term ,and it can be solved by a fast denoising algorithm proposed by Jia-Zhao or soved by BM3D and NLM denoising algorithm, which also provide a unified solution framework for formally generalized-ROF-like subproblems arising in multivariate splitting algorithms[25]. [25] was submitted on 29-Aug-2013, and our early edition was ever submitted to TGRS on 12-Jun-2012, Venkatakrishnan et al. [26] proposed their PnP algorithm on 29-May-2013, so Venkatakrishnan and we proposed the PnP algorithm almost simultaneously.

Guangming Liu

Yantai Science and Technology Association

Image Segmentation Remote Sensing Image Segmentation Synthetic Aperture Radar

Dec 8, 2023cs.CV

A global optimization SAR image segmentation model can be easily transformed to a general ROF denoising model

In this paper, we propose a novel locally statistical active contour model (LACM) based on Aubert-Aujol (AA) denoising model and variational level set method, which can be used for SAR images segmentation with intensity inhomogeneity. Then we transform the proposed model into a global optimization model by using convex relaxation technique. Firstly, we apply the Split Bregman technique to transform the global optimization model into two alternating optimization processes of Shrink operator and Laplace operator, which is called SB_LACM model. Moreover, we propose two fast models to solve the global optimization model , which are more efficient than the SB_LACM model. The first model is: we add the proximal function to transform the global optimization model to a general ROF model[29], which can be solved by a fast denoising algorithm proposed by R.-Q.Jia, and H.Zhao; [29] was submitted on 29-Aug-2013, and our early edition was ever submitted to TGRS on 12-Jun-2012, Venkatakrishnan et al. [30] proposed their PnP algorithm on 29-May-2013, so Venkatakrishnan and we proposed the PnP algorithm almost simultaneously. Thus we obtain a fast segmentation algorithm with global optimization solver that does not involve partial differential equations or difference equation, and only need simple difference computation. The second model is: we use a different splitting approach than one model to transform the global optimization model into a differentiable term and a general ROF model term, which can be solved by the same technique as the first model.

Guangming Liu

Yantai Science and Technology Association

Image Segmentation Remote Sensing Image Segmentation