ReSeg: Retraining-free segmentation of abdominal organs via generalizable semantic representation learning

Derong Yu; Guoyan Zheng

cs.CVOct 21, 2026

ReSeg: Retraining-free segmentation of abdominal organs via generalizable semantic representation learning

Authors: Derong Yu, Guoyan Zheng

Organizations: Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No.800 Dongchuan Road, Shanghai, 200240, China

Abstract

Abdominal organ segmentation is essential in modern medical applications. While existing deep learning approaches demonstrate promising performance, their heavy reliance on large-scale image data raises concerns over generalization capability and privacy risks. This motivates the development of generalizable segmentation training paradigms based on domain-randomized image synthesis, facilitating generalization across unseen real image domains. However, existing paradigms either suffer from limited segmentation performance when applied to abdominal organs, or require dataset-specific finetuning. To address these limitations, we propose ReSeg, a novel two-stage retraining-free framework for domain-agnostic zero-shot abdominal organ segmentation. At the first stage, our organ-agnostic synthetic data-driven pretraining approach exploits semantically and visually diverse synthetic images generated by a dedicated data engine to optimize a Generalizable Semantic Representation Network (GSRNet). The network is thereby capable of learning semantic features robust against variations in anatomical structure, imaging contrast, resolution and spatial context, which provide universal representation capabilities that serve as a foundation for segmentation learning. Subsequently, in the second stage, we continue to train an Organ-specific Encoder-decoder Segmentation Network (OESNet), whose encoder is initialized with the pretrained weights. This training process is facilitated by another dedicated data engine, which generates diversified visual images while strictly retaining the inherent spatial layout of abdominal organs. Comprehensive evaluation on seven abdominal organ segmentation datasets across three different modalities including CT, MRI, and PET demonstrates that ReSeg consistently outperforms existing state-of-the-art (SOTA) methods while eliminating the need for real training images.

Figures & tables

Figure 1: Illustration of the proposed two-stage retraining-free framework and the zero-shot segmentation performance. (a) We utilize organ shape templates to randomly sample images from a wide range of distributions and train the model on the synthetic images through two learning stages. Stage I learns organ-agnostic, generalizable semantic features that are robust to imaging variations, while stage II leverages these semantic priors to learn abdominal-organ-specific semantic features for image segmentation. In the inference stage, our model can segment real images from multiple domains without retraining or finetuning. (b) ReSeg achieves superior performance compared with existing state-of-the-art (SOTA) methods on seven abdominal segmentation datasets across three different modalities including CT, MRI, and PET. In particular, SynthSeg ( Billot et al., 2023 ) is trained only on synthetic data while ZeroShotSAM ( Shaharabany and Wolf, 2024 ) leverages Segment Anything Model (SAM) ( Kirillov et al., 2023 ) to perform minimally-guided zero-shot segmentation using foreground prompt points as the input. The solid line in the radar plot depicts the average Dice Similarity Coefficient (DSC) on each dataset.

Figure 2: A schematic overview of ReSeg. We leverage segmentation masks from the TotalSegmentator CT dataset ( Wasserthal et al., 2023 ) , excluding real images entirely. Two data engines, referred to as GenRep and GenSeg, are introduced to construct a Randomized Paired Image (RPI) database and an Anatomical Multi-view Image (AMI) database, respectively. ReSeg is a two-stage learning framework that is trained on these two databases. In particular, in stage I, the RPI database is used to train a Generalizable Semantic Representation Network (GSRNet), which learns organ-agnostic, generalizable semantic features with an Appearance-aware Semantic (AS) decoder and a Context-aware Semantic (CS) decoder. In stage II, an Organ-specific Encoder-decoder Segmentation Network (OESNet) is initialized with the pretrained encoder weights and then trained on the AMI database. OESNet is guided by semantic priors calculated by the well-trained GSRNet to enhance domain-invariant anatomical feature extraction within a Semantics-guided Segmentation (SS) decoder. We reduce the dimension of the embeddings to 3 using PCA and visualize them as RGB images. Embed.: embedding.

Figure 3: Illustration of the Generalizable Appearance (GA) model. Given a label map $\mathcal{L}$ with $L$ labels, GMMs are used to randomly sample intensities. For all label $l\in\{1,2,\cdots,L\}$ , a lower bound $a_{\mu}$ for the mean parameters are first drawn from a uniform distribution $U(a_{\mu}^{L},a_{\mu}^{H})$ , where $a_{\mu}^{L}$ and $a_{\mu}^{H}$ are two fixed hyperparameters controlling respectively the lowest and the highest value for this dynamic lower bound. Then, for each label $l$ in $\mathcal{L}$ , the intensity $I_{l}$ is sampled from a Gaussian distribution $\mathcal{N}(\mu_{l},\sigma_{l}^{2})$ , where $\mu_{l}\sim U(a_{\mu},b_{\mu})$ , $\sigma_{l}\sim U(a_{\sigma},b_{\sigma})$ , and $b_{\mu}$ , $a_{\sigma}$ , and $b_{\sigma}$ are fixed hyperparameters. After GMMs, Perlin noise ( Perlin, 1985 ) is applied, followed by random offline and online augmentations.

Figure 4: Illustration of the two data engines for ReSeg. Both data engines use segmentation masks sampled from the TotalSegmentator CT dataset ( Wasserthal et al., 2023 ) . The GenRep data engine randomly samples organ shape templates from all patients to create randomized label maps with paired synthetic images, forming a Randomized Paired Image (RPI) database. In contrast, the GenSeg data engine extracts segmentations of all organs from one randomly drawn patient to synthesize pseudo-images while preserving the inherent spatial layout of abdominal organs, constituting an Anatomical Multi-view Image (AMI) database.

Figure 5: Framework for the organ-agnostic semantic representation learning stage. During training, a synthetic image pair is randomly sampled from the RPI database to train a Generalizable Semantic Representation Network (GSRNet). This network consists of a shared CNN encoder with two distinct decoders: an Appearance-aware Semantic (AS) decoder optimized via a Dual-view Prototypical Contrastive (DPC) loss, and a Context-aware Semantic (CS) decoder trained by a Multi-patch Context-aware Contrastive (MCC) loss. During inference, an input image is fed into GSRNet to generate the AS embedding and the CS embedding from the AS decoder and the CS encoder, respectively. These embeddings are then concatenated to produce the generalizable semantic embedding. We reduce the dimension of the embeddings to 3 using PCA and visualize them as RGB images. Embed.: embedding.

Figure 6: Framework for the abdominal-organ-specific segmentation learning stage. During training, a synthetic image with its GT segmentation label is randomly sampled from the AMI database and then randomly cropped to get an image patch with its GT segmentation label patch. The synthetic image patch is fed into an Organ-specific Encoder-decoder Segmentation Network (OESNet), which employs the pretrained weights to initialize the encoder, followed by a Semantics-guided Segmentation (SS) decoder. In the inference stage, OESNet can segment on real images from multiple domains without finetuning. We reduce the dimension of the embeddings to 3 using PCA and visualize them as RGB images. Embed.: embedding. CE: cross entropy.

Figure 7: Architecture of the SS decoder in OESNet. The SS decoder is integrated with hierarchical SFE modules. Each SFE module takes the segmentation feature of the current layer $f_{d}^{in}$ and the generalizable semantic embedding $f_{s}$ from the frozen GSRNet as inputs. Within the SFE module, $f_{s}$ is first aligned with $f_{d}^{in}$ and then generates attention map to enhance the segmentation feature. After that, the feature undergoes multi-kernel processing and finally the output feature $f_{d}^{out}$ is obtained.

Figure 8: Visual results for the comparison with the SOTA methods on both BTCV and AMOS-CT datasets. 3D examples are shown in 2D slices along the transverse plane. Left to right: input images, GT segmentation labels, and segmentation results of different methods including SynthSeg ( Billot et al., 2023 ) , ZeroShotSAM ( Shaharabany and Wolf, 2024 ) , DSPNet ( Tang et al., 2025 ) , UniverSeg ( Butoi et al., 2023 ) , VoCo ( Wu et al., 2024 ) , Anatomix ( Dey et al., 2025 ) , and the proposed ReSeg. Colors correspond to the organs as follows: red - the liver, green - the spleen, blue - the right kidney, yellow - the left kidney.

Figure 9: Visual results for the comparison with the SOTA methods on both L2RAb-MR and AMOS-MR datasets. 3D examples are shown in 2D slices along the transverse plane. Left to right: input images, GT segmentation labels, and segmentation results of different methods including SynthSeg ( Billot et al., 2023 ) , ZeroShotSAM ( Shaharabany and Wolf, 2024 ) , DSPNet ( Tang et al., 2025 ) , UniverSeg ( Butoi et al., 2023 ) , VoCo ( Wu et al., 2024 ) , Anatomix ( Dey et al., 2025 ) , and the proposed ReSeg. Colors correspond to the organs as follows: red - the liver, green - the spleen, blue - the right kidney, yellow - the left kidney.

Figure 10: Box plots of the DSC scores across the BTCV, the AMOS-CT, the L2RAb-MR, and the AMOS-MR datasets, demonstrating the performance of ReSeg versus SOTA methods. Statistical significances from Wilcoxon signed-rank test ( Woolson, 2007 ) are annotated above the box plots.

Figure 11: Visual comparison results for abdominal segmentation on PET images. 3D examples from the AutoPET-Organ dataset are shown in 2D slices along the frontal plane. Left to right: input images, GT segmentation labels, and segmentation results of ZeroShotSAM ( Shaharabany and Wolf, 2024 ) and the proposed ReSeg. Colors correspond to the organs as follows: red - the liver, green - the spleen, blue - the right kidney, yellow - the left kidney.

Figure 12: Visual comparison results for cardiac segmentation. 3D examples from the MMWHS-CT and the MMWHS-MR datasets are shown in 2D slices along the frontal plane. Left to right: input images, GT segmentation labels, and segmentation results of SynthSeg ( Billot et al., 2023 ) and the proposed ReSeg. Colors correspond to the organs as follows: red - the myocadium, green - the left atrium, blue - the right atrium, yellow - the left ventricle, cyan - the right ventricle, purple - the ascending aorta, beige - the pulmonary artery.

Zero-Shot Medical Image Segmentation

Zero-shot medical image segmentation uses pretrained models to delineate anatomy and disease without task-specific mask training.

See all 18 papers

Optimal Transport Metric Learning for Feature Alignment in Partially Supervised Segmentation
Sep 15, 2026Dakini Mallam Garba, Salim Abdou DaouraMulti-Organ Segmentation Medical Image Segmentation
ML Collective
ThreshGuide: Class-Aware Labeled-Guided Thresholding for Semi-Supervised 3D Abdominal Multi-Organ Segmentation
Sep 14, 2026Hongyu Liu, Yinlong Wang, Lusha Li +1Semi-Supervised Medical Image Segmentation Medical Image Segmentation
School of Intelligent Science and Technology, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
Frequency Adapter with SAM for Generalized Medical Image Segmentation
May 11, 2026Phuoc-Nguyen Bui, Van-Nguyen Pham, Duc-Tai Le +2Frequency-Domain Feature Learning Domain Generalization
Sungkyunkwan University, Korea

Sep 15, 2026J/K move · Enter open · S save

Optimal Transport Metric Learning for Feature Alignment in Partially Supervised Segmentation

Dakini Mallam Garba, Salim Abdou Daoura

ML Collective

Multi-Organ Segmentation Medical Image Segmentation Metric Learning

Open paper AskPDF arXiv

Multi-organ segmentation is often challenged by partially annotated datasets and domain shifts across different imaging sources. To address these limitations, we propose a two-stage learning framework that efficiently leverages partial supervision. In the first stage, the model learns from available annotations to produce accurate segmentations of annotated organs, establishing robust feature representations. In the second stage, we introduce learnable organ prototypes and a Sinkhorn-triplet loss to enforce organ-wise feature consistency across datasets. This encourages latent embeddings of the same organ to remain close, while increasing separation between different organs, even when annotations are missing. Our approach achieves performance comparable to state-of-the-art methods on the BTCV dataset, while remaining computationally efficient. By explicitly aligning feature distributions rather than relying solely on pseudo-labels, the framework effectively mitigates domain shift, making it particularly suitable for medical image segmentation tasks with limited annotation resources.

Abstract

Zero-Shot Medical Image Segmentation

Explore similar work

Optimal Transport Metric Learning for Feature Alignment in Partially Supervised Segmentation

ThreshGuide: Class-Aware Labeled-Guided Thresholding for Semi-Supervised 3D Abdominal Multi-Organ Segmentation

Frequency Adapter with SAM for Generalized Medical Image Segmentation