When to Unpair: Regulating Pairing Dependence in Medical Visual In-Context Learning

Cheng Wan; Chenjun Li; Qingyu Zhao

cs.CVOct 7, 2026

When to Unpair: Regulating Pairing Dependence in Medical Visual In-Context Learning

Authors: Cheng Wan, Chenjun Li, Qingyu Zhao

Organizations: Cornell University · Weill Cornell Medicine

Abstract

Visual in-context learning (ICL), well suited to label-scarce medical imaging, uses support image-label pairs to demonstrate input-output mappings, while the labels collectively indicate the requested task. We diagnose dependence on individual pairings with a test-time derangement that reassigns every support label to another support image while preserving the query, support images, and label multiset. The resulting pairing gap, defined as shuffled-minus-matched performance, shows that all four released models depend on the pairing, to widely varying degrees. Further analysis of a paired-trained model reveals support-associated spurious regions and lesion-size biases even with real, unaltered supports, alongside sensitivity to mis-registered support labels. To regulate this dependence, we introduce a late unpairing curriculum (LUC), which starts with matched training and then applies random unpairing, replacing each support label with that of another support in the same episode. LUC nearly closes the pairing gap on two backbones while maintaining or improving matched-support performance across all evaluated task types, with gains extending to held-out tasks and cross-dataset episodes. It also mitigates these failure modes. On BraTS whole-tumor segmentation, matched-support DSC rises from 0.733 to 0.857 while the gap shrinks from -0.184 to -0.008. In a released model, brief fine-tuning with random unpairing reduces the gap. A reversed curriculum that places the same number of unpairing epochs at the start of training leaves a large gap. This shows that pairing dependence is shaped by the order of training and not only by the amount of unpaired training.

Figures & tables

Figure 1: Which label sits on which support image changes the prediction. (a) Grey: every support image keeps its own label; red: a derangement , which moves each label onto another image and leaves the label multiset unchanged. (b) One BraTS validation case at $K{=}4$ with matched supports and three derangements; beside each row are the support set that model trains on and its schedule $p(e)$ . Green outlines the ground truth; numbers are DSC. (c) Pairing gap and matched-support DSC over the BraTS whole-tumor evaluation.

Figure 2: The two operations on the support set. (a) Matched supports. (b) The test-time derangement: the same four labels, none on its own image. (c) Random unpairing in training: every support takes a copy of another support’s label, so a label can repeat, which never happens in (b).

		Whole tumor		Edema
Backbone	Fusion	matched	gap	matched	gap
2D
UniverSeg ( Butoi et al., 2023 )	averaging	0.731	$-0.41$	0.508	$-0.36$
SegGPT ( Wang et al., 2023b )	feature ensemble	0.713	$-0.63$	0.510	$-0.51$
3D
Neuroverse3D ( Hu et al., 2025 )	mean-over- $K$	0.846	$-0.11$	0.593	$-0.35$

Table 1: Pairing gap in four released models on two BraTS tasks. Each model is run with its own protocol using released weights (inference only); we report DSC on matched supports and the pairing gap, defined as the difference between shuffled and matched DSC.

Figure 3: The influence of support labels on query predictions. Green outline: ground truth label; filled: model’s prediction. The first column shows one of the $K{=}4$ support images, with its label in yellow. Each row shows a prediction failure, with the right-hand panel reporting statistics of the artifact over $n{=}60$ validation subjects. (a) Based on real, unaltered supports, the paired model predicts additional regions that do not overlap the ground truth (white dashed circles). (b) Based on real, unaltered supports, volumes of the tumor predicted by the paired model correlate with tumor size in the support. LUC 0.5 removes this spurious correlation. (c) Shifting every support label by $8$ voxels along each of two axes (white arrow in the first column) causes the paired model’s prediction to shift in the same direction; in the prediction panels, the dashed outline is the prediction with the original supports and the filled region the prediction with the shifted supports.

Figure 4: Matched-support and shuffled-support DSC on BraTS whole-tumor segmentation. Filled marker: DSC evaluated on matched supports; open marker: DSC evaluated with the shuffled support labels (test-time derangement); the line between them is the pairing gap. LUC combines a near-zero pairing gap with high matched-support performance. Reversing the schedule with EUC retains a larger gap, while unpairing in every epoch (the unpaired model) lowers matched-support performance.

	paired ( $p{=}0$ )		LUC 0.5		LUC 0.2
Task (metric)	matched	gap	matched	gap	matched	gap
Segmentation (DSC $\uparrow$ )	$0.733$	$-0.184$	$0.857$	$-0.008$	$0.878$	$-0.003$
Translation (PSNR $\uparrow$ )	$22.41$	$-1.22$	$22.56$	$-0.03$	$22.70$	$-0.08$
Anatomical seg. (DSC $\uparrow$ )	$0.704$	$-0.035$	$0.806$	$-0.001$	$0.791$	$-0.007$
Inpainting (PSNR $\uparrow$ )	$17.52$	$-2.27$	$17.76$	$+0.05$	$17.88$	$-0.02$
Bias corr. (PSNR $\uparrow$ )	$26.59$	$-4.20$	$27.45$	$-0.24$	$26.34$	$-0.18$

Table 2: Matched-support score and pairing gap on five task types. The models differ only in the unpairing schedule

p(e)

. Green: an LUC cell with a near-zero gap and a matched-support score at least equal to that of the paired model; red: the paired model’s gap. Per-task detail is in Appendix D .

Figure 5: Performance on tasks outside training. Matched-support score on held-out tasks and on cross-dataset episodes, in which the supports and the query come from different datasets. Held-out segmentation tasks use new combinations of anatomical structures that are segmented individually during training, while the bias-field-correction task in (c) uses a coarser

3^{3}

bias-field grid than in training. Both LUC models score significantly above the paired model on every task shown (subject-level Wilcoxon,

p{<}0.05

); Table 10 in Appendix G lists all such tasks. Datasets in parentheses: B: BraTS, A: ABCD, D: ADNI, O: OASIS-3, I: IXI; T1: the three T1 cohorts.

Figure 6: Pairing gap during training, and how fast it closes after the switch. Validation gap logged at every epoch. (a) Whole-tumor gap, five-epoch moving average; dotted lines mark the schedule switches. (b) Gap on each of five tasks (BraTS whole tumor, ABCD, ADNI, OASIS-3, T2

\to

FLAIR), divided by its mean over the ten epochs before the switch, for LUC 0.5 ; three-epoch moving average, thin lines for individual tasks and the thick line for their median. Figure 12 in Appendix E shows that LUC 0.2 behaves the same way.

Figure 7: Fine-tuning the released Neuroverse3D of Table 1 , with random unpairing or with matched supports. Pairing gap on edema (a) and on whole tumor (b), and matched-support DSC (c), against fine-tuning steps. Step 0 is the released model; every evaluation uses the same queries, supports and derangements as Table 1 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Ingredient	Value (fixed across the family)
Architecture	Neuroverse3D, channels $[32,64,128,256,512]$ , $\sim$ 71M params
Data mix	BraTS 2021 $+$ three T1 datasets (ABCD, ADNI, OASIS-3)
Tasks	segmentation, translation, anatomical seg., inpainting, bias correction
Optimizer	Adam, lr $10^{-4}$ , no weight decay
Learning-rate schedule	held fixed across the family
Loss	$50\times\mathcal{L}_{\text{smooth-}L_{3}\text{-}L_{1}}$

Appendix

Table 3: Training recipe for the controlled Neuroverse3D family. All six models share these settings and differ only in the unpairing schedule

p(e)

model	$p(e)$	$x$	mirror
paired	$0$ for all $e$	0	none
unpaired	$1$ for all $e$	1	none
EUC 0.5	$1$ for $e{<}75$ , else $0$	0.5	LUC 0.5
LUC 0.5	$0$ for $e{<}75$ , else $1$	0.5	EUC 0.5
EUC 0.2	$1$ for $e{<}30$ , else $0$	0.2	LUC 0.2
LUC 0.2	$0$ for $e{<}120$ , else $1$	0.2	EUC 0.2

Appendix

Table 4: The six models. All share the recipe of Table 3 and differ only in

p(e)

;

x

is the fraction of epochs with unpairing, and “mirror” names the time-reversed partner.

model	matched	cross-patient	zero	noise	shuffled
paired	0.733	0.661	0.655	0.461	0.549
unpaired	0.611	0.713	0.714	0.573	0.719
LUC 0.5	0.857	0.832	0.826	0.710	0.848
LUC 0.2	0.878	0.878	0.879	0.808	0.875
EUC 0.5	0.731	0.669	0.631	0.541	0.640
EUC 0.2	0.731	0.646	0.650	0.534	0.601

Appendix

Table 5: Label-perturbation tests on BraTS segmentation (subject-level mean DSC,

n{=}30

). Shuffling changes only the assignment; cross-patient, zeroed and noise labels also change the label content. For the paired model, shuffled supports score below the zeroed and the cross-patient condition; the LUC models move little between matched and shuffled supports and hold the highest matched-support DSC.

Figure 8: Predictions with real supports and with a mask painted into the supports. Each row shows a support pair, the query, and the paired and LUC 0.5 predictions. Top: real supports; the query has a single tumor (green). Middle: the injected-mask test, with a fixed synthetic label (red, white dashed ring) painted into every support where the query has no tumor. Bottom: injected-mask response across the models, grouped by schedule.

model	Seg DSC	Transl PSNR	AnatSeg DSC	Inp PSNR	Bias PSNR
paired	0.733 / $-0.184$	22.406 / $-1.216$	0.704 / $-0.035$	17.517 / $-2.267$	26.588 / $-4.201$
unpaired	0.611 / $+0.109$	22.146 / $+0.042$	0.685 / $+0.005$	17.327 / $-0.011$	25.958 / $+0.159$
EUC 0.5	0.731 / $-0.091$	22.389 / $-1.099$	0.708 / $-0.024$	17.339 / $-0.880$	25.661 / $-3.234$
LUC 0.5	0.857 / $-0.008$	22.557 / $-0.027$	0.806 / $-0.001$	17.756 / $+0.052$	27.447 / $-0.242$
EUC 0.2	0.731 / $-0.130$	22.486 / $-1.340$	0.694 / $-0.025$	17.651 / $-1.872$	26.516 / $-3.759$
LUC 0.2	0.878 / $-0.003$	22.698 / $-0.081$	0.791 / $-0.007$	17.878 / $-0.021$	26.338 / $-0.182$

Appendix

Table 6: In-distribution results (matched / gap) for the six models trained under the recipe. Matched

=

the score with matched supports, in the metric named in the column header; gap

=

shuffled

-

matched, so a more negative gap means more pairing dependence. Green: the LUC models, where the gap is near zero and the matched-support score is at or above the paired model. Red: the paired model’s gap. All rows share the one recipe and differ only in

p(e)

	Paired ( $p{=}0$ )		LUC 0.5		LUC 0.2		EUC 0.5		EUC 0.2
Cohort	matched	gap	matched	gap	matched	gap	matched	gap	matched	gap
ABCD (children)	$0.859$	$-0.074$	$0.868$	$+0.002$	$0.871$	$-0.006$	$0.848$	$-0.049$	$0.857$	$-0.048$
ADNI (elderly)	$0.466$	$-0.030$	$0.737$	$-0.004$	$0.702$	$-0.016$	$0.504$	$-0.024$	$0.438$	$-0.026$
OASIS-3 (mixed)	$0.785$	$0.000$	$0.813$	$0.000$	$0.800$	$0.000$	$0.774$	$0.000$	$0.788$	$0.000$

Appendix

Table 7: Pairing gap by cohort (anatomical segmentation,

K{=}4

). Matched

=

matched-support DSC; gap

=

shuffled-support minus matched-support DSC. Green: matched-support DSC under LUC at or above the paired model. Red: the paired model’s gap on ABCD. On all three cohorts the LUC models have a gap within

0.016

of zero and matched-support DSC above paired; EUC 0.5 and EUC 0.2 (which end on matched supports) have more negative gaps than LUC 0.5 and LUC 0.2 where the paired model has one.

Figure 9: Qualitative comparison across tasks and models ( $K{=}4$ ). Six models (paired $\to$ EUC 0.5 $\to$ EUC 0.2 $\to$ unpaired $\to$ LUC 0.2 $\to$ LUC 0.5 ); green $=$ ground-truth contour, red $=$ predicted mask; the corner badge is DSC (segmentation) or PSNR in dB (synthesis/restoration; inpainting scored in the filled box only). Rows are labeled task (dataset). Top, shuffled supports: two tasks; the paired model degrades (partial or displaced mask, lost detail) while LUC 0.5 and LUC 0.2 hold, EUC 0.5 and EUC 0.2 degrade with the paired model, and the unpaired model sits in between. Bottom, matched supports: the trained task suite across all four datasets. Accuracy is comparable throughout: LUC 0.5 matches the paired model on translation and restoration, is cleaner on the two segmentation tasks (whole-tumor over-segmentation; atrophic ADNI ventricle), and fills the inpainting box slightly better. The bottom-row cases were selected to show the pattern; population numbers are in Tables 6 and 7 .

backbone	training	matched DSC	pairing gap (DSC)
Neuroverse3D ( Hu et al., 2025 ) (3D)	paired	0.733	$-0.184$
Neuroverse3D ( Hu et al., 2025 ) (3D)	$+$ LUC 0.5	0.857	$-0.008$
UniverSeg ( Butoi et al., 2023 ) (2D)	paired	0.590	$-0.222$
UniverSeg ( Butoi et al., 2023 ) (2D)	$+$ LUC 0.5	0.602	$\approx\!0.000$

Appendix

Table 8: Pairing gap with and without LUC, in a 3D and a 2D backbone. Each backbone is trained with its own recipe; its two rows differ only in the unpairing schedule

p(e)

. Neuroverse3D: BraTS whole tumor; UniverSeg: BraTS multi-region, mean over WT/TC/ET/edema.

Figure 10: Score under replaced, mis-registered, and eroded support labels. Left: BraTS whole-tumor DSC as

0

4

of the

K{=}4

support labels are replaced by another patient’s label. Right: DSC drop from the matched-support score under a support mis-registration and a partial erosion of the support masks. Neuroverse3D,

n{=}60

Figure 11: LUC versus EUC at

x{=}0.5

and

x{=}0.2

. Injected-mask response, DSC lost under the derangement and support sensitivity; the two models of a pair unpair for the same number of epochs, the LUC model ends on unpaired supports and its EUC mirror on matched supports, and the dashed line is the paired model.

Figure 12: Pairing gap of LUC 0.2 after its switch at epoch

120

. The gap on each of five tasks, divided by its mean over the ten epochs before the switch; thin lines for tasks and the thick line for their median.

model	correct	deranged	rolled16
paired	0.754	0.585	0.685
unpaired	0.630	0.741	0.710
EUC 0.5	0.755	0.667	0.686
EUC 0.2	0.751	0.650	0.699
LUC 0.5	0.860	0.848	0.852
LUC 0.2	0.880	0.879	0.880

Appendix

Table 9: Three support-label conditions (BraTS whole-tumor DSC,

n{=}40

). The paired model and both EUC models lose DSC at deranged ; LUC 0.5 and LUC 0.2 are flat across all three support-label conditions.

task	data	$n$	input	paired ( $p{=}0$ )		LUC 0.5		LUC 0.2
				matched	gap	matched	gap	matched	gap
Segmentation (DSC)
Hippocampus †	T1 cohorts	60	–	0.608	$-$ 0.011	0.706	$+$ 0.004	0.707	$-$ 0.017
Caudate	T1 cohorts	60	–	0.632	$-$ 0.014	0.720	$-$ 0.004	0.716	$-$ 0.014
Thalamus †	T1 cohorts	60	–	0.758	$-$ 0.001	0.853	0.000	0.832	$-$ 0.002
Edema from FLAIR †	BraTS	40	–	0.627	$-$ 0.088	0.695	$-$ 0.003	0.711	$-$ 0.004

Appendix

Table 10: Tasks outside training. Matched-support score (DSC; PSNR in dB, inside the filled boxes for inpainting) and pairing gap; “input” is the PSNR of the unprocessed input, and an arrow in the data column marks a cross-dataset episode. Both LUC models score significantly above the paired model on every unmarked task (subject-level Wilcoxon,

p{<}0.05

); on tasks marked ∘ , one LUC model does and the other does not differ significantly from it. † Shown in Figure 5 .

Explore similar work

Aug 5, 2026cs.CV

Context Matters: Support Set Selection and Failure Detection for In-Context Medical Image Segmentation

In-context learning (ICL) adapts medical image segmentation models to unseen structures and modalities without retraining by conditioning on a task-specific support set of image-mask exemplars. Because this support set is the model's only task-specific signal, its composition directly influences segmentation performance. In this work, we investigate the support set as a controllable determinant of ICL reliability. First, we compare random sampling against similarity-based selection, where exemplars are retrieved based on their visual similarity to the query image. Second, we train a transformer-based classifier to predict, from the query and support images alone, whether a segmentation will fall below a specified Intersection-over-Union (IoU) threshold. Using MultiverSeg with DINOv3 embeddings across four benchmarks and three imaging modalities, we show that similarity-based selection consistently matches or outperforms random sampling, with the largest gains at the smallest support set sizes. Furthermore, our classifier predicts segmentation failure above chance on all four benchmarks. Ultimately, these results demonstrate that the reliability of in-context segmentation can be both improved via informed support selection and anticipated before use, providing practical mechanisms for safer clinical deployment.

Youssef Gehad, Emmanuel Zerefa, Krish Kabra +1

Rice University

Semi-Supervised Medical Image Segmentation Medical Imaging In-Context Learning Dinov3

Ask PDF

Sep 21, 2026cs.AI

Representation-guided in-context learning for medical image interpretation with multimodal large language models

Medical image interpretation is central to diagnosis and care, yet adapting general-purpose multimodal large language models (MLLMs) often requires resource-intensive domain-specific fine-tuning. Here we introduce representation-guided in-context learning (RG-ICL), a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates. Across eight datasets spanning histopathology, radiology and retinal fundoscopy, RG-ICL improved classification (mean gain 20 percentage points) and visual question answering (VQA) (mean gain 13 percentage points) over no-context and conventional ICL, approaching or exceeding training-based comparators. Which cases were retrieved mattered more than how many: 6 query-aligned cases outperformed up to 32 randomly selected ones, whereas fixed or random cases often reduced accuracy below baseline. For VQA, aligning reference cases with both image content and question intent produced further gains. These findings indicate that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

Minda Zhao, Fangyu Hu, Yan Luo +8

Harvard AI and Robotics Lab, Schepens Eye Research Institute of Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA · Department of Ophthalmology, Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA · Department of Ophthalmology, National Taiwan University Hospital, Taipei City, Taiwan +6

Medical Vision-Language Models Multimodal Clinical Data Medical Images Multimodal Large Language Models In-Context Learning In-Context+1

Ask PDF

Sep 14, 2026cs.CV

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.

Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5

University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University +2

Multimodal Learning In-Context Learning Modality Gap Twins Task Framing

Ask PDF