Biomedical image segmentation must keep up with new anatomical structures, subjects, imaging modalities, and acquisition protocols, yet dense expert annotations are scarce and models trained on one cohort often struggle to transfer. In-context segmentation (ICS) offers a way out: a new structure is specified by a few annotated exemplars at inference time. However, reliable correspondence remains challenging in biomedical images, where adjacent structures often share similar appearances, and variations across subjects and imaging conditions further obscure their distinctions. We introduce CAST, a training-free framework built on a frozen DINOv3 backbone to calibrate fine-grained correspondences. Its stability test improves feature reliability at both global and patch-specific levels, while its coherence test assesses whether candidate correspondences are supported by consistent local structural context. Together, these tests enable more reliable correspondence across diverse anatomical structures and imaging conditions. Across eight biomedical benchmarks spanning 2D and 3D segmentation, cross-subject, and cross-modality transfer, CAST outperforms the strongest overall baseline by 12.34 percentage points in Dice and ranks first across all six volumetric settings. The versatility of CAST further opens up opportunities for challenging biomedical applications, including 2D-to-3D segmentation and atlas label transfer.
Figures & tables
Figure 2: Comparison of DINOv3 feature similarity between natural and biomedical images. Similarity maps are obtained by computing cosine similarities between query patch features and annotated target features from the source image. Natural images exhibit clear similarity contrast between foreground and background, whereas biomedical images show similar responses across neighboring anatomical structures. Green contours indicate source annotations.
Figure 3: Overview of CAST. Given an annotated source image, its mask, and an unlabeled query image, CAST calibrates and refines source–query patch correspondences through two complementary tests. The stability test characterizes feature variations induced by geometric transformations, providing global debiasing before candidate retrieval and patch-specific metrics for subsequent reranking. The coherence test evaluates retrieved candidates through neighborhood feature contrast and target–non-target layout consistency. Within the correspondence refinement framework, globally debiased features guide class-conditional candidate retrieval, after which source-specific metrics and neighborhood evidence jointly rerank the candidates. The top- Ntop candidates from each class are then aggregated through voting to produce the final segmentation.
Method
2D segmentation
3D segmentation
Avg.
Chest [-1pt] X-ray
REFUGE [-1pt] Fundus
GlaS [-1pt] Histology
DMBM
CTPelvic1K [-1pt] CT
LPBA40 [-1pt] MRI
CHAOS
AMOS22
Fluorescence
CT / MRI
CT / MRI
Same
Cross
C → C
M → C
M → M
C → M
Segmentation-trained or test-time adapted
SegGPT [ 35 ]
95.40
82.63
71.99
72.82
67.93
39.11
27.32
79.70
50.31
32.34
24.76
60.05
UniverSeg [ 1 ]
91.16
72.36
58.17
75.50
71.93
39.23
24.86
76.23
61.33
30.49
20.16
56.70
Table 1: Comparison of in-context segmentation methods on eight biomedical benchmarks (Dice, %, ↑ ). All tasks use five annotated source exemplars ( K=5 ). Same/Cross denote within-/cross-marker transfer on DMBM; C/M denote CT/MRI, with arrows indicating transfer directions. 2D Dice is computed per target image, while 3D Dice is computed over the entire target volume. Avg. is the equal-weight mean across datasets. Gray marks UniverSeg results on datasets included in its pretraining (CHAOS and AMOS22), and their contribution to Avg. Best and second-best results are shown in bold and underlined , respectively.
Figure 4: Qualitative comparison of CAST with SegGPT, INSID3, and FROST under the one-shot ( K=1 ) setting. Left: cross-subject segmentation on REFUGE, CTPelvic1K, and LPBA40. Right: transfer across imaging appearance on DMBM (cross-marker), CHAOS and AMOS22 (cross-modality). Green overlays indicate reference annotations and ground truth, while purple overlays denote predicted masks. White boxes highlight the zoomed-in regions in REFUGE.
Figure 5: Effect of the source annotation budget on CHAOS MRI → CT. Top: Whole-volume Dice of INSID3, FROST, and CAST with K∈{1,3,5,10} annotated source slices. Bottom: Representative 3D liver segmentations at K=1 and K=10 , with ground truth (GT) for reference.
Method
Setup
Inference
Total
Avg. per slice
INSID3
0.29
74.19
74.48
0.394
FROST
3.55
61.99
65.54
0.347
CAST
7.01
27.75
34.76
0.184
Table 2: Runtime comparison on CHAOS MRI → CT at K=5 (seconds). Setup is performed once per annotated source set. Total is the sum of setup and inference time per volume. Avg. per slice includes amortized setup time.
Stability
Coherence
LPBA40
CHAOS
Direct label transfer (1-NN)
37.72
77.53
–
–
41.09
79.10
✓
–
42.54
79.44
–
✓
42.58
79.67
CAST
✓
✓
44.88
81.51
Table 3: Ablation of CAST (whole-volume Dice, %, ↑ ). ✓ and – indicate enabled and disabled tests. All configurations use K=5 . CHAOS reports the mean across three MRI → CT settings.
Method
Setting
SegGPT
GF-SAM
INSID3
FROST
CAST (Ours)
PSD95
50.38
27.77
37.70
59.73
66.92
SAP102
50.05
27.43
44.89
63.57
70.04
Mean
50.22
27.60
41.30
61.65
68.48
Table 4: Atlas label transfer to DMBM using five annotated source slices (Dice, %, ↑ ). Mean denotes the average of the PSD95 and SAP102 results.
Figure 6: Qualitative atlas label transfer results in DMBM for the striatum (top) and ventral posterolateral (VPL) nucleus of the thalamus (bottom).
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.