In-context learning (ICL) enables medical image segmentation models to adapt to new anatomical structures from limited examples, reducing the clinical annotation burden. However, standard ICL methods typically rely on dense, global cross-attention, which scales poorly with image resolution. While recent approaches have introduced localized attention mechanisms, they often lack explicit supervision on the selection process, leading to redundant computation in non-informative regions. We propose PatchICL, a hierarchical framework that combines selective image patching with multi-level supervision. Our approach learns to actively identify and attend only to the most informative anatomical regions. Compared to UniverSeg, a strong global-attention baseline, PatchICL achieves competitive in-domain CT segmentation accuracy while reducing compute by 44% at 512×512 resolution. On 35 out-of-domain datasets spanning diverse imaging modalities, PatchICL outperforms the baseline on 6 of 13 modality categories, with particular strength on modalities dominated by localized pathology such as OCT and dermoscopy. Training and evaluation code are available at https://github.com/tidiane-camaret/ic_segmentation
Figures & tables
Figure 1: Coarse-to-fine patch selection across resolution levels on subject 0013 of the TotalSegmentator CT dataset. Columns show the target image with patch boxes, ground-truth overlay, and prediction overlay. Rows correspond to cascade levels at increasing resolution.
Figure 2: Overview of the PatchICL architecture. Image patches are selectively sampled on the target and context images, and mask predictions are aggregated.
Figure 3: Patch-level attention mechanism. Target and context patches are processed jointly through transformer layers with 2D RoPE, type embeddings, and bidirectional attention.
quadriceps femoris m (L/R), sartorius m (L/R), thigh med./post. comp. m (L/R)
Table 1: Anatomical structure categories for TotalSegmentator evaluation. Left/right pairs abbreviated (L/R). Unmarked structures appear in both CT and MRI; c = CT only, m = MRI only.
Figure 4: Dice score and computational cost (GFLOPs) as a function of input image resolution on TotalSegmentator CT. PatchICL achieves competitive accuracy with significantly reduced compute at higher resolutions.
Category
N
PatchICL
UniverSeg
Bones (Limbs/Shldr./Pelvis)
1,992
0.572 ± 0.296
0.523 ± 0.331
Bones (Spine)
780
0.464 ± 0.271
0.552 ± 0.281
Organs (Thorax/Head/Spine)
1,077
0.568 ± 0.328
0.584 ± 0.316
Organs (Abd./Pelvis)
8,498
0.461 ± 0.314
0.470 ± 0.335
Muscles
4,494
0.444 ± 0.328
0.454 ± 0.332
Bones (Ribs/Sternum)
13,620
0.390 ± 0.308
0.321 ± 0.334
Table 2: Dice scores on TotalSegmentator CT validation cases and held-out classes (mean ± std). N denotes the number of evaluated slices. Best per category in bold .
Figure 5: Qualitative comparison on TotalSegmentator CT held-out classes. Columns: context pair, target with yellow ground truth, blue PatchICL prediction, orange UniverSeg prediction. Dashed yellow contours show GT on prediction columns. Dice scores inset. Top rows: PatchICL wins (iliopsoas, liver); bottom rows: UniverSeg wins (lung lobe, autochthon).
Figure 6: Additional coarse-to-fine patch selection examples on TotalSegmentator CT. Columns show the target image with patch boxes, ground-truth overlay, and prediction overlay. Rows correspond to cascade levels at increasing resolution.
Category
N
PatchICL
UniverSeg
Organs (Thorax/Head/Spine)
0 936
0.550 ± 0.291
0.547 ± 0.294
Muscles (Trunk)
1 396
0.447 ± 0.304
0.488 ± 0.305
Organs (Abd./Pelvis)
2 072
0.436 ± 0.298
0.459 ± 0.307
Bones (Spine)
0 441
0.405 ± 0.234
0.500 ± 0.236
Bones (Limbs/Pelvis)
0 520
0.417 ± 0.340
0.423 ± 0.350
Vessels
0 831
0.381 ± 0.312
0.398 ± 0.326
Table 3: Dice scores on TotalSegmentator MRI (mean ± std). N denotes the number of evaluated slices. The overall row averages across all individual classes; category rows are shown for interpretability but contain varying numbers of classes. Best per category in bold .
Figure 7: Cross-modality comparison on TotalSegmentator MRI (both methods trained on CT only). Columns: context pair, target with yellow ground truth, blue PatchICL prediction, orange UniverSeg prediction. Dice scores inset.
Modality
N
PatchICL
UniverSeg
X-Ray
0,0 11
0.727 ± 0.045
0.896 ± 0.025
Dermoscopy
0, 209
0.643 ± 0.204
0.578 ± 0.190
Chest X-Ray
2 581
0.610 ± 0.174
0.684 ± 0.216
Microscopy
0, 263
0.539 ± 0.252
0.495 ± 0.285
Ultrasound
1 209
0.484 ± 0.241
0.553 ± 0.261
MRI
0, 147
0.433 ± 0.270
0.565 ± 0.211
Table 4: Dice scores on MedSegBench by imaging modality (mean ± std), using 3 context pairs. N denotes the number of evaluated samples. Best per modality in bold .
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Department of Computer Science, University of Virginia, Virginia, USA · Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Kahlert School of Computing, University of Utah, Utah, USA +1
Harbin Institute of Technology at Shenzhen, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · University Hospital Tübingen, Tübingen, Germany +2