Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models

Jingxuan Kang; Qianying Yue; Che Liu; Chen Qin

cs.CVSep 27, 2026

Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models

Authors: Jingxuan Kang, Qianying Yue, Che Liu, Chen Qin

Organizations: Imperial College London · The Chinese University of Hong Kong

Abstract

Pretrained biomedical vision-language models achieve strong zero-shot performance in biomedical image classification. However, downstream biomedical classification often depends on subtle visual differences between classes that may not be fully captured by pretrained representations. Few-shot adaptation addresses this mismatch by optimizing a task-specific predictor on a small labeled support set. Because the selected examples capture only part of the visual variation within the target classes, the adapted predictions can depend strongly on their composition. We propose Prompt-Anchored Residual Adaptation (PARA), which retains the frozen prompt prediction as a support-invariant semantic anchor and incorporates a visual prediction learned from the support set through an anchor-relative residual. The residual step is computed in a closed form from frozen support embeddings using anchor discrepancy and support agreement. Support-set dependence also limits evaluation: comparisons are fair within a shared draw but remain conditional on its composition. To obtain more reliable comparisons, we introduce a repeated-support protocol that separates support-selection variation from optimization randomness and reports both average and worst-20% performance. PARA achieves state-of-the-art performance in both few-shot classification and base-to-novel generalization.

Figures & tables

Figure 1: Lowest and highest Accuracy across 20 support draws for three prompt learning methods on COVID-19.

Figure 2: Overview of PARA. During training, the support set is used to optimize the LoRA parameters. At inference, all support examples form the adapted class prototypes, and a query image is scored by both the frozen prompt branch and the adapted visual branch. Their difference is scaled by $\eta(\mathcal{S})$ and added to the anchor prediction, where $\eta(\mathcal{S})$ aggregates class-wise anchor discrepancy and support agreement computed from frozen support embeddings. Panel (a) details prototype construction during training and inference, while panel (b) illustrates how the two geometry quantities jointly determine the residual step.

Zero-shot Method
	Accuracy						Macro-F1
Method	$K=1$	$K=2$	$K=4$	$K=8$	$K=16$	Avg.	$K=1$	$K=2$	$K=4$	$K=8$	$K=16$	Avg.
BiomedCLIP	54.35						44.55
CLIP-Based Adaptation Methods
Tip-Adapter	55.12 $\pm$ 0.60	55.99 $\pm$ 0.81	57.16 $\pm$ 1.10	59.22 $\pm$ 1.45	61.91 $\pm$ 1.42	57.88 $\pm$ 1.08	45.46 $\pm$ 0.53	46.24 $\pm$ 0.78	47.63 $\pm$ 0.89	49.89 $\pm$ 1.25	52.91 $\pm$ 1.52	48.43 $\pm$ 1.00
CLIP-LoRA	54.82 $\pm$ 1.18	54.82 $\pm$ 1.09	55.23 $\pm$ 0.91	56.15 $\pm$ 1.15	59.71 $\pm$ 1.40	56.15 $\pm$ 1.15	46.53 $\pm$ 0.92	47.07 $\pm$ 0.86	47.77 $\pm$ 0.76	50.46 $\pm$ 0.88	55.33 $\pm$ 1.20	49.43 $\pm$ 0.92

Table 1: Comparison with state-of-the-art methods on 11 biomedical datasets (%). Entries report repeated-support means

\pm

average within-dataset support-draw standard deviations. BiomedCLIP is support-invariant; best means are bold.

Zero-shot Method
	Shots per class ( $K$ )
Method	1	2	4	8	16	Avg.
BiomedCLIP	54.35
CLIP-Based Adaptation Methods
Tip-Adapter	54.27	54.86	55.59	57.13	59.71	56.31
CLIP-LoRA	53.06	53.31	53.92	54.50	57.67	54.49

Table 2: Performance over the worst 20% of support draws. Accuracy CVaR 20 (%) is averaged across 11 biomedical datasets. BiomedCLIP is invariant to support selection. The best adaptation result in each column is shown in bold.

	Accuracy						Macro-F1
	Mean $\pm$ SD			CVaR 20			Mean $\pm$ SD			CVaR 20
Method	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
BiomedCLIP	55.87	79.91	65.76	55.87	79.91	65.76	49.09	68.77	57.29	49.09	68.77	57.29
CLIP-LoRA	78.00 $\pm$ 2.23	72.56 $\pm$ 4.47	75.18 $\pm$ 2.76	74.62	66.40	70.27	76.41 $\pm$ 2.19	66.43 $\pm$ 3.53	71.07 $\pm$ 2.47	73.09	61.48	66.78
CoOp	72.54 $\pm$ 2.75	68.59 $\pm$ 5.07	70.51 $\pm$ 3.47	68.38	62.10	65.09	70.94 $\pm$ 2.58	56.00 $\pm$ 4.40	62.59 $\pm$ 3.31	66.98	49.75	57.09
CoCoOp	71.20 $\pm$ 2.94	68.37 $\pm$ 5.56	69.76 $\pm$ 3.69	66.82	60.52	63.51	69.54 $\pm$ 2.72	54.48 $\pm$ 5.03	61.10 $\pm$ 3.63	65.42	47.47	55.02

Table 3: Base-to-novel generalization over 20 support selections on 10 datasets (%). Adaptation uses 16-shot base-class support. SD is averaged across datasets. For both mean and CVaR 20 , HM is computed from the displayed Base and Novel values; HM SD is computed from the corresponding draw-wise harmonic means.

Figure 3: Accuracy performance profiles for PARA and the five baselines used in paired significance tests across 11 datasets. A curve gives the fraction of datasets on which a method is within a factor

\tau

of the lowest error among the methods shown ( Dolan and Moré 2002 ) .

Figure 4: The base–novel trade-off. Base and novel Accuracy (%) averaged over 10 datasets and 20 support selections.

Variant	Mean Acc.	CVaR 20	SD
w/o Prototype Anchoring	66.66	61.70	3.27
w/o Prompt-Anchored Residual	64.98	57.75	4.84
w/o Support Geometry	64.78	60.17	3.17
PARA (Full)	66.95	62.41	3.06

Table 4: Component ablation. SD denotes the standard deviation across support draws.

Figure 5: Paired Wilcoxon tests. Filled markers indicate Holm-adjusted

p<0.05

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Classes	Total	Support pool	Test
BTMRI	Magnetic Resonance Imaging	4	5,712	1,714	3,998
BUSI	Ultrasound	3	780	234	546
CHMNIST	Histopathology	8	5,000	1,504	3,496
COVID-19	X-Ray	4	21,165	6,351	14,814
CTKidney	Computerized Tomography	4	12,446	3,734	8,712
DermaMNIST	Dermoscopy	7	10,014	3,006	7,008

Appendix

Table A1: Dataset overview and the fixed split at the image level, stratified by class.

Figure A1: Few-shot Accuracy for each dataset. The Average panel is the mean across the 11 datasets at each shot level. Each adaptation point is the mean over 20 support draws after averaging optimizer seeds within each draw. BiomedCLIP denotes the zero-shot anchor at

K=0

, which is independent of the sampled support set.

Figure A2: Few-shot Macro-F1 for each dataset under the same protocol as Figure A1 . The Average panel is the mean across the 11 datasets at each shot level.

Accuracy
	Mean			CVaR 20
Baseline	$\Delta$	Wins	$p_{\rm H}$	$\Delta$	Wins	$p_{\rm H}$
Tip-Adapter	9.07	9/11	0.0244	6.09	9/11	0.0645
CLIP-LoRA	10.80	11/11	0.0117	7.91	10/11	0.0195
TaskRes	4.44	10/11	0.0244	6.65	10/11	0.0137
CLAP	6.57	11/11	0.0117	9.91	11/11	0.0117

Appendix

Table A2: Paired comparisons between PARA and all 12 adaptation baselines across 11 datasets.

\Delta

is the mean paired difference in percentage points, Wins counts positive differences, and

p_{\rm H}

is adjusted with Holm’s method over the 12 baselines separately for each summary. (a) Accuracy.

Macro-F1
	Mean			CVaR 20
Baseline	$\Delta$	Wins	$p_{\rm H}$	$\Delta$	Wins	$p_{\rm H}$
Tip-Adapter	12.91	11/11	0.0117	10.17	11/11	0.0117
CLIP-LoRA	11.90	11/11	0.0117	9.09	11/11	0.0117
TaskRes	2.37	9/11	0.0391	3.76	9/11	0.0244
CLAP	4.34	10/11	0.0156	6.73	11/11	0.0117

Appendix

Table A3: Paired comparisons between PARA and all 12 adaptation baselines (continued). (b) Macro-F1.

Accuracy: Mean
Dataset	BiomedCLIP	Tip-Adapter	CLIP-LoRA	TaskRes	CLAP	LP++	SS-Text+	TAMP	PARA
BTMRI	62.53	65.79	61.95	74.49	72.50	74.19	71.36	73.54	76.93
BUSI	54.95	56.30	57.98	59.89	59.57	60.03	56.70	59.81	63.31
CHMNIST	30.43	41.86	44.95	67.88	66.34	68.69	71.63	67.55	66.96
COVID-19	66.46	71.40	62.89	70.16	66.51	69.95	67.40	69.45	75.50
CTKidney	57.45	60.47	65.38	72.01	70.13	71.92	59.55	69.11	75.22

Appendix

Table A4: Few-shot results for each dataset across all methods in the main comparison (%). Each value is averaged over the five shot levels. Mean and CVaR 20 are shown in separate blocks. Bold marks the best result across all methods. (a) Frozen and CLIP-based adaptation methods.

Accuracy: Mean
Dataset	CoOp	CoCoOp	ProGrad	BiomedCoOp	vMFCoOp	PARA
BTMRI	72.93	68.06	72.92	74.24	74.66	76.93
BUSI	58.16	56.36	59.51	59.10	59.10	63.31
CHMNIST	64.30	58.07	66.77	68.69	66.53	66.96
COVID-19	69.96	63.84	65.20	73.78	70.97	75.50
CTKidney	68.03	64.32	67.76	69.24	68.80	75.22

Appendix

Table A5: Few-shot results for each dataset (continued). (b) Prompt-learning methods; PARA is repeated for comparison.

Explore similar work

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
May 15, 2026Huanyang Tong, Kai Liu, Fangjun Kuang +1Vision-Language Model Adaptation Multimodal Clinical Data
Wenzhou University · Wenzhou Business College
Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification
Apr 27, 2026Xiaoliu Luo, Minxue Xiao, Ting Xie +5Vision-Language Model Adaptation Medical Image Classification
Chongqing University of Technology · Hebei University of Technology · Nanyang Technological University +3
TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models
Jul 10, 2026Tianyou Jiang, Ziyu ZhouVision-Language Model Adaptation Medical Vision-Language Models
University of Bern, Bern, Switzerland · Shanghai Jiao Tong University, Shanghai, China

May 15, 2026, cs.CVJ/K move · Enter open · S save

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation

Huanyang Tong, Kai Liu, Fangjun Kuang, Huiling Chen

Wenzhou University · Wenzhou Business College

Vision-Language Model Adaptation Multimodal Clinical Data Cross-Modal Fusion

Open paper AskPDF arXiv

Biomedical Vision--Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis but face a critical bottleneck: \textit{fragility to prompt variations}.Existing adaptation frameworks typically optimize visual and textual prompts as independent streams, relying on ideal ``Golden Prompts''. In clinical reality, where descriptions are often noisy and heterogeneous, this modality isolation leads to unstable cross-modal alignment. To address this, we propose BiomedAP, a vision-informed dual-anchor framework with gated cross-modal fusion.BiomedAP enforces synergistic alignment through two mechanisms: (1) Gated Cross-Modal Fusion, which enables layer-wise interaction between modalities, acting as a dynamic noise regulator to suppress irrelevant textual cues; and (2) a Dual-Anchor Constraint that regularizes learnable prompts toward stable semantic centroids derived from both expert templates (High Anchors) and few-shot visual prototypes (Low Anchors). Extensive experiments across 11 benchmarks demonstrate that BiomedAP consistently surpasses baselines, achieving competitive few-shot accuracy and markedly enhanced robustness under prompt perturbations. Our code is available at: https://github.com/tongdiedie/BiomedAP. Keywords: Vision-Language Models; Prompt Learning; Parameter-Efficient Fine-Tuning; Few-shot Learning

Abstract

Explore similar work

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation

Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models