Biomedical image segmentation must keep up with new anatomical structures, subjects, imaging modalities, and acquisition protocols, yet dense expert annotations are scarce and models trained on one cohort often struggle to transfer. In-context segmentation (ICS) offers a way out: a new structure is specified by a few annotated exemplars at inference time. However, reliable correspondence remains challenging in biomedical images, where adjacent structures often share similar appearances, and variations across subjects and imaging conditions further obscure their distinctions. We introduce CAST, a training-free framework built on a frozen DINOv3 backbone to calibrate fine-grained correspondences. Its stability test improves feature reliability at both global and patch-specific levels, while its coherence test assesses whether candidate correspondences are supported by consistent local structural context. Together, these tests enable more reliable correspondence across diverse anatomical structures and imaging conditions. Across eight biomedical benchmarks spanning 2D and 3D segmentation, cross-subject, and cross-modality transfer, CAST outperforms the strongest overall baseline by 12.34 percentage points in Dice and ranks first across all six volumetric settings. The versatility of CAST further opens up opportunities for challenging biomedical applications, including 2D-to-3D segmentation and atlas label transfer.
Figures & tables
Figure 2: Comparison of DINOv3 feature similarity between natural and biomedical images. Similarity maps are obtained by computing cosine similarities between query patch features and annotated target features from the source image. Natural images exhibit clear similarity contrast between foreground and background, whereas biomedical images show similar responses across neighboring anatomical structures. Green contours indicate source annotations.
Figure 3: Overview of CAST. Given an annotated source image, its mask, and an unlabeled query image, CAST calibrates and refines source–query patch correspondences through two complementary tests. The stability test characterizes feature variations induced by geometric transformations, providing global debiasing before candidate retrieval and patch-specific metrics for subsequent reranking. The coherence test evaluates retrieved candidates through neighborhood feature contrast and target–non-target layout consistency. Within the correspondence refinement framework, globally debiased features guide class-conditional candidate retrieval, after which source-specific metrics and neighborhood evidence jointly rerank the candidates. The top- Ntop candidates from each class are then aggregated through voting to produce the final segmentation.
Method
2D segmentation
3D segmentation
Avg.
Chest [-1pt] X-ray
REFUGE [-1pt] Fundus
GlaS [-1pt] Histology
DMBM
CTPelvic1K [-1pt] CT
LPBA40 [-1pt] MRI
CHAOS
AMOS22
Fluorescence
CT / MRI
CT / MRI
Same
Cross
C → C
M → C
M → M
C → M
Segmentation-trained or test-time adapted
SegGPT [ 35 ]
95.40
82.63
71.99
72.82
67.93
39.11
27.32
79.70
50.31
32.34
24.76
60.05
UniverSeg [ 1 ]
91.16
72.36
58.17
75.50
71.93
39.23
24.86
76.23
61.33
30.49
20.16
56.70
Table 1: Comparison of in-context segmentation methods on eight biomedical benchmarks (Dice, %, ↑ ). All tasks use five annotated source exemplars ( K=5 ). Same/Cross denote within-/cross-marker transfer on DMBM; C/M denote CT/MRI, with arrows indicating transfer directions. 2D Dice is computed per target image, while 3D Dice is computed over the entire target volume. Avg. is the equal-weight mean across datasets. Gray marks UniverSeg results on datasets included in its pretraining (CHAOS and AMOS22), and their contribution to Avg. Best and second-best results are shown in bold and underlined , respectively.
Figure 4: Qualitative comparison of CAST with SegGPT, INSID3, and FROST under the one-shot ( K=1 ) setting. Left: cross-subject segmentation on REFUGE, CTPelvic1K, and LPBA40. Right: transfer across imaging appearance on DMBM (cross-marker), CHAOS and AMOS22 (cross-modality). Green overlays indicate reference annotations and ground truth, while purple overlays denote predicted masks. White boxes highlight the zoomed-in regions in REFUGE.
Figure 5: Effect of the source annotation budget on CHAOS MRI → CT. Top: Whole-volume Dice of INSID3, FROST, and CAST with K∈{1,3,5,10} annotated source slices. Bottom: Representative 3D liver segmentations at K=1 and K=10 , with ground truth (GT) for reference.
Method
Setup
Inference
Total
Avg. per slice
INSID3
0.29
74.19
74.48
0.394
FROST
3.55
61.99
65.54
0.347
CAST
7.01
27.75
34.76
0.184
Table 2: Runtime comparison on CHAOS MRI → CT at K=5 (seconds). Setup is performed once per annotated source set. Total is the sum of setup and inference time per volume. Avg. per slice includes amortized setup time.
Stability
Coherence
LPBA40
CHAOS
Direct label transfer (1-NN)
37.72
77.53
–
–
41.09
79.10
✓
–
42.54
79.44
–
✓
42.58
79.67
CAST
✓
✓
44.88
81.51
Table 3: Ablation of CAST (whole-volume Dice, %, ↑ ). ✓ and – indicate enabled and disabled tests. All configurations use K=5 . CHAOS reports the mean across three MRI → CT settings.
Method
Setting
SegGPT
GF-SAM
INSID3
FROST
CAST (Ours)
PSD95
50.38
27.77
37.70
59.73
66.92
SAP102
50.05
27.43
44.89
63.57
70.04
Mean
50.22
27.60
41.30
61.65
68.48
Table 4: Atlas label transfer to DMBM using five annotated source slices (Dice, %, ↑ ). Mean denotes the average of the PSD95 and SAP102 results.
Figure 6: Qualitative atlas label transfer results in DMBM for the striatum (top) and ventral posterolateral (VPL) nucleus of the thalamus (bottom).
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: https://github.com/MohammedOussamaBEN/GazeRefine.git
Mohammed Oussama Benyahia, Marouane Tliba, Mohamed Amine Kerkouri +10
Université Sorbonne Paris Nord · Northwestern University · VSB-Technical University of Ostrava
In-context learning (ICL) enables medical image segmentation models to adapt to new anatomical structures from limited examples, reducing the clinical annotation burden. However, standard ICL methods typically rely on dense, global cross-attention, which scales poorly with image resolution. While recent approaches have introduced localized attention mechanisms, they often lack explicit supervision on the selection process, leading to redundant computation in non-informative regions. We propose PatchICL, a hierarchical framework that combines selective image patching with multi-level supervision. Our approach learns to actively identify and attend only to the most informative anatomical regions. Compared to UniverSeg, a strong global-attention baseline, PatchICL achieves competitive in-domain CT segmentation accuracy while reducing compute by 44% at 512×512 resolution. On 35 out-of-domain datasets spanning diverse imaging modalities, PatchICL outperforms the baseline on 6 of 13 modality categories, with particular strength on modalities dominated by localized pathology such as OCT and dermoscopy. Training and evaluation code are available at https://github.com/tidiane-camaret/ic_segmentation
T. Camaret Ndir, Marco Reisert, Robin T. Schirrmeister
Division of Medical Physics, Department of Radiology, Medical Center – University of Freiburg, Freiburg, Germany
We present MaskGen, a theoretically grounded and deliberately simple approach for domain generalization in 3D biomedical image segmentation. Modern segmentation models degrade sharply under shifts in modality, disease severity, clinical sites, and more, limiting their reliable adoption. Existing generalization methods address this using extreme augmentations, hand-engineered domain statistics mixing, or architectural redesigns that add significant implementation overhead while yielding inconsistent performance across biomedical settings. MaskGen instead presents a principled learning strategy with marginal overhead that utilizes both source-domain image intensities and domain-stable foundation model representations to train robust segmentation models. As a result, MaskGen achieves strong gains in both fully supervised and few-shot segmentation across broad clinical shifts in biomedical studies. Unlike prior approaches, MaskGen is architecture- and loss-agnostic, compatible with standard augmentation pipelines, easy to implement, and tackles arbitrary anatomical regions. Its implementation is freely available at https://github.com/sebodiaz/MaskGen.