ReSeg: Retraining-free segmentation of abdominal organs via generalizable semantic representation learning
Authors: Derong Yu, Guoyan Zheng
Organizations: Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No.800 Dongchuan Road, Shanghai, 200240, China
Abdominal organ segmentation is essential in modern medical applications. While existing deep learning approaches demonstrate promising performance, their heavy reliance on large-scale image data raises concerns over generalization capability and privacy risks. This motivates the development of generalizable segmentation training paradigms based on domain-randomized image synthesis, facilitating generalization across unseen real image domains. However, existing paradigms either suffer from limited segmentation performance when applied to abdominal organs, or require dataset-specific finetuning. To address these limitations, we propose ReSeg, a novel two-stage retraining-free framework for domain-agnostic zero-shot abdominal organ segmentation. At the first stage, our organ-agnostic synthetic data-driven pretraining approach exploits semantically and visually diverse synthetic images generated by a dedicated data engine to optimize a Generalizable Semantic Representation Network (GSRNet). The network is thereby capable of learning semantic features robust against variations in anatomical structure, imaging contrast, resolution and spatial context, which provide universal representation capabilities that serve as a foundation for segmentation learning. Subsequently, in the second stage, we continue to train an Organ-specific Encoder-decoder Segmentation Network (OESNet), whose encoder is initialized with the pretrained weights. This training process is facilitated by another dedicated data engine, which generates diversified visual images while strictly retaining the inherent spatial layout of abdominal organs. Comprehensive evaluation on seven abdominal organ segmentation datasets across three different modalities including CT, MRI, and PET demonstrates that ReSeg consistently outperforms existing state-of-the-art (SOTA) methods while eliminating the need for real training images.
Figures & tables
Figure 1: Illustration of the proposed two-stage retraining-free framework and the zero-shot segmentation performance. (a) We utilize organ shape templates to randomly sample images from a wide range of distributions and train the model on the synthetic images through two learning stages. Stage I learns organ-agnostic, generalizable semantic features that are robust to imaging variations, while stage II leverages these semantic priors to learn abdominal-organ-specific semantic features for image segmentation. In the inference stage, our model can segment real images from multiple domains without retraining or finetuning. (b) ReSeg achieves superior performance compared with existing state-of-the-art (SOTA) methods on seven abdominal segmentation datasets across three different modalities including CT, MRI, and PET. In particular, SynthSeg ( Billot et al., 2023 ) is trained only on synthetic data while ZeroShotSAM ( Shaharabany and Wolf, 2024 ) leverages Segment Anything Model (SAM) ( Kirillov et al., 2023 ) to perform minimally-guided zero-shot segmentation using foreground prompt points as the input. The solid line in the radar plot depicts the average Dice Similarity Coefficient (DSC) on each dataset.
Figure 2: A schematic overview of ReSeg. We leverage segmentation masks from the TotalSegmentator CT dataset ( Wasserthal et al., 2023 ) , excluding real images entirely. Two data engines, referred to as GenRep and GenSeg, are introduced to construct a Randomized Paired Image (RPI) database and an Anatomical Multi-view Image (AMI) database, respectively. ReSeg is a two-stage learning framework that is trained on these two databases. In particular, in stage I, the RPI database is used to train a Generalizable Semantic Representation Network (GSRNet), which learns organ-agnostic, generalizable semantic features with an Appearance-aware Semantic (AS) decoder and a Context-aware Semantic (CS) decoder. In stage II, an Organ-specific Encoder-decoder Segmentation Network (OESNet) is initialized with the pretrained encoder weights and then trained on the AMI database. OESNet is guided by semantic priors calculated by the well-trained GSRNet to enhance domain-invariant anatomical feature extraction within a Semantics-guided Segmentation (SS) decoder. We reduce the dimension of the embeddings to 3 using PCA and visualize them as RGB images. Embed.: embedding.
Figure 3: Illustration of the Generalizable Appearance (GA) model. Given a label map L with L labels, GMMs are used to randomly sample intensities. For all label l∈{1,2,⋯,L} , a lower bound aμ for the mean parameters are first drawn from a uniform distribution U(aμL,aμH) , where aμL and aμH are two fixed hyperparameters controlling respectively the lowest and the highest value for this dynamic lower bound. Then, for each label l in L , the intensity Il is sampled from a Gaussian distribution N(μl,σl2) , where μl∼U(aμ,bμ) , σl∼U(aσ,bσ) , and bμ , aσ , and bσ are fixed hyperparameters. After GMMs, Perlin noise ( Perlin, 1985 ) is applied, followed by random offline and online augmentations.
Figure 4: Illustration of the two data engines for ReSeg. Both data engines use segmentation masks sampled from the TotalSegmentator CT dataset ( Wasserthal et al., 2023 ) . The GenRep data engine randomly samples organ shape templates from all patients to create randomized label maps with paired synthetic images, forming a Randomized Paired Image (RPI) database. In contrast, the GenSeg data engine extracts segmentations of all organs from one randomly drawn patient to synthesize pseudo-images while preserving the inherent spatial layout of abdominal organs, constituting an Anatomical Multi-view Image (AMI) database.
Figure 5: Framework for the organ-agnostic semantic representation learning stage. During training, a synthetic image pair is randomly sampled from the RPI database to train a Generalizable Semantic Representation Network (GSRNet). This network consists of a shared CNN encoder with two distinct decoders: an Appearance-aware Semantic (AS) decoder optimized via a Dual-view Prototypical Contrastive (DPC) loss, and a Context-aware Semantic (CS) decoder trained by a Multi-patch Context-aware Contrastive (MCC) loss. During inference, an input image is fed into GSRNet to generate the AS embedding and the CS embedding from the AS decoder and the CS encoder, respectively. These embeddings are then concatenated to produce the generalizable semantic embedding. We reduce the dimension of the embeddings to 3 using PCA and visualize them as RGB images. Embed.: embedding.
Figure 6: Framework for the abdominal-organ-specific segmentation learning stage. During training, a synthetic image with its GT segmentation label is randomly sampled from the AMI database and then randomly cropped to get an image patch with its GT segmentation label patch. The synthetic image patch is fed into an Organ-specific Encoder-decoder Segmentation Network (OESNet), which employs the pretrained weights to initialize the encoder, followed by a Semantics-guided Segmentation (SS) decoder. In the inference stage, OESNet can segment on real images from multiple domains without finetuning. We reduce the dimension of the embeddings to 3 using PCA and visualize them as RGB images. Embed.: embedding. CE: cross entropy.
Figure 7: Architecture of the SS decoder in OESNet. The SS decoder is integrated with hierarchical SFE modules. Each SFE module takes the segmentation feature of the current layer fdin and the generalizable semantic embedding fs from the frozen GSRNet as inputs. Within the SFE module, fs is first aligned with fdin and then generates attention map to enhance the segmentation feature. After that, the feature undergoes multi-kernel processing and finally the output feature fdout is obtained.
Figure 8: Visual results for the comparison with the SOTA methods on both BTCV and AMOS-CT datasets. 3D examples are shown in 2D slices along the transverse plane. Left to right: input images, GT segmentation labels, and segmentation results of different methods including SynthSeg ( Billot et al., 2023 ) , ZeroShotSAM ( Shaharabany and Wolf, 2024 ) , DSPNet ( Tang et al., 2025 ) , UniverSeg ( Butoi et al., 2023 ) , VoCo ( Wu et al., 2024 ) , Anatomix ( Dey et al., 2025 ) , and the proposed ReSeg. Colors correspond to the organs as follows: red - the liver, green - the spleen, blue - the right kidney, yellow - the left kidney.
Figure 9: Visual results for the comparison with the SOTA methods on both L2RAb-MR and AMOS-MR datasets. 3D examples are shown in 2D slices along the transverse plane. Left to right: input images, GT segmentation labels, and segmentation results of different methods including SynthSeg ( Billot et al., 2023 ) , ZeroShotSAM ( Shaharabany and Wolf, 2024 ) , DSPNet ( Tang et al., 2025 ) , UniverSeg ( Butoi et al., 2023 ) , VoCo ( Wu et al., 2024 ) , Anatomix ( Dey et al., 2025 ) , and the proposed ReSeg. Colors correspond to the organs as follows: red - the liver, green - the spleen, blue - the right kidney, yellow - the left kidney.
Figure 10: Box plots of the DSC scores across the BTCV, the AMOS-CT, the L2RAb-MR, and the AMOS-MR datasets, demonstrating the performance of ReSeg versus SOTA methods. Statistical significances from Wilcoxon signed-rank test ( Woolson, 2007 ) are annotated above the box plots.
Figure 11: Visual comparison results for abdominal segmentation on PET images. 3D examples from the AutoPET-Organ dataset are shown in 2D slices along the frontal plane. Left to right: input images, GT segmentation labels, and segmentation results of ZeroShotSAM ( Shaharabany and Wolf, 2024 ) and the proposed ReSeg. Colors correspond to the organs as follows: red - the liver, green - the spleen, blue - the right kidney, yellow - the left kidney.
Figure 12: Visual comparison results for cardiac segmentation. 3D examples from the MMWHS-CT and the MMWHS-MR datasets are shown in 2D slices along the frontal plane. Left to right: input images, GT segmentation labels, and segmentation results of SynthSeg ( Billot et al., 2023 ) and the proposed ReSeg. Colors correspond to the organs as follows: red - the myocadium, green - the left atrium, blue - the right atrium, yellow - the left ventricle, cyan - the right ventricle, purple - the ascending aorta, beige - the pulmonary artery.
Multi-organ segmentation is often challenged by partially annotated datasets and domain shifts across different imaging sources. To address these limitations, we propose a two-stage learning framework that efficiently leverages partial supervision. In the first stage, the model learns from available annotations to produce accurate segmentations of annotated organs, establishing robust feature representations. In the second stage, we introduce learnable organ prototypes and a Sinkhorn-triplet loss to enforce organ-wise feature consistency across datasets. This encourages latent embeddings of the same organ to remain close, while increasing separation between different organs, even when annotations are missing. Our approach achieves performance comparable to state-of-the-art methods on the BTCV dataset, while remaining computationally efficient. By explicitly aligning feature distributions rather than relying solely on pseudo-labels, the framework effectively mitigates domain shift, making it particularly suitable for medical image segmentation tasks with limited annotation resources.
Pseudo-labeling is a strong paradigm for semi-supervised medical image segmentation, yet its effectiveness is highly sensitive to confidence thresholding. In abdominal multi-organ segmentation, a fixed global threshold is particularly suboptimal because organ classes differ substantially in size, appearance, and learning difficulty. In this work, we propose ThreshGuide, a class-aware threshold adaptation framework that uses labeled data to guide pseudo-label selection on unlabeled data. Built upon a standard teacher-student architecture, the teacher model evaluates labeled samples during training to estimate class-aware threshold targets by maximizing an error-aware F\b{eta} criterion that balances precision and coverage. These targets are then smoothed with an exponential moving average (EMA) and used to filter unlabeled voxels in a class-dependent manner. Experiments on FLARE2022 and AMOS2022 show that ThreshGuide performs competitively overall, yielding clear improvements specifically on hard-to-learn organs.
Hongyu Liu, Yinlong Wang, Lusha Li +1
School of Intelligent Science and Technology, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
Medical image segmentation is a critical task in computer-aided diagnosis and treatment planning. However, deep learning models often struggle to generalize across datasets due to domain shifts arising from variations in imaging protocols, scanner types, and patient populations. Traditional domain generalization (DG) methods utilize causal feature learning, adversarial consistency, and style augmentation to improve segmentation robustness. While effective, these approaches rely on explicit feature alignment, adversarial objectives, or handcrafted augmentations, which may not fully exploit the capabilities of foundation models. Recently, the Segment Anything Model (SAM) has demonstrated strong generalization capabilities in segmentation tasks. SAM-based DG methods attempt to improve medical image segmentation. However, these approaches primarily operate in the spatial domain and overlook frequency-based discrepancies that significantly affect model robustness. In this work, we propose Frequency-based Domain Generalization with SAM (FSAM), a novel framework that integrates Low-Rank Adaptation (LoRA) for efficient fine-tuning and a frequency adapter to incorporate frequency-domain representations for single-source domain generalization. FSAM enhances SAM's segmentation robustness by extracting domain-invariant high-frequency features, mitigating frequency-related domain shifts. Experimental results on fundus and prostate datasets demonstrate that FSAM outperforms existing traditional DG and SAM-based DG approaches in domain generalization. Codes and pre-trained models will be made available on GitHub.