ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models

M. Arda Aydın; Melih B. Yilmaz; Aykut Koç; Tolga Çukur

cs.CVMar 17, 2026

ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models

Authors: M. Arda Aydın, Melih B. Yilmaz, Aykut Koç, Tolga Çukur

Organizations: Bilkent University · National Magnetic Resonance Research Center (UMRAM)

Abstract

The success of CLIP-like vision-language models (VLMs) on natural images has inspired medical counterparts, yet existing approaches largely fall into two extremes: specialist models trained on single-domain data, which capture domain-specific details but generalize poorly, and generalist medical VLMs trained on multi-domain data, which retain broad semantics but dilute fine-grained diagnostic cues. Bridging this specialization-generalization trade-off remains challenging. To address this problem, we propose ACE-LoRA, a parameter-efficient adaptation framework for generalist medical VLMs that maintains robust zero-shot generalization. ACE-LoRA integrates Low-Rank Adaptation (LoRA) modules into frozen image-text encoders and introduces an Attention-based Context Enhancement Hypergraph Neural Network (ACE-HGNN) module that captures higher-order contextual interactions beyond pairwise similarity to enrich global representations with localized diagnostic cues, addressing a key limitation of prior Parameter-Efficient Fine-Tuning (PEFT) methods that overlook fine-grained details. To further enhance cross-modal alignment, we formulate a label-guided InfoNCE loss to effectively suppress false negatives between semantically related image-text pairs. Despite adding only 0.95M trainable parameters, ACE-LoRA consistently outperforms state-of-the-art medical VLMs and PEFT baselines across zero-shot classification, segmentation, and detection benchmarks spanning multiple domains. Our code is available at https://github.com/icon-lab/ACE-LoRA.

Figures & tables

Figure 1: Overview of ACE-LoRA. ACE-LoRA integrates low-rank adaptation modules into self-attention blocks of image and text encoders in a generalist medical VLM, and introduces ACE-HGNN, a hypergraph-based module that models high-order topological dependencies between local (e.g., image patches or report snippets) and global embeddings. For clarity, ACE-HGNN is described using image embeddings, though the same procedure is applied to text embeddings.

Figure 2: False negatives in contrastive learning. The CLIP loss treats all non-matching pairs as negatives, which can falsely push apart semantically similar samples, whereas our formulation avoids separating pairs that share the same disease label.

Method	CheXpert 5 $\times$ 200		RSNA		SIIM
Method	ACC (%)	AUC (%)	ACC (%)	AUC (%)	ACC (%)	AUC (%)
BiomedCLIP [ 62 ]	35.50	67.46	74.34	81.14	61.40	64.31
CoOp [ 69 ]	40.90	69.32	75.07	79.92	65.69	69.87
CoCoOp [ 68 ]	40.10	69.74	75.02	79.95	67.24	71.21
CLIP-Adapter [ 11 ]	42.70	77.55	68.47	75.14	60.38	61.26
TaskRes [ 58 ]	36.50	68.17	74.77	81.36	63.53	67.14

Table 1: Comparison of zero-shot classification performance across different fine-tuning methods. Full FT represents full fine-tuning, in which all parameters of the pre-trained model are updated. All methods are trained using Label-guided InfoNCE loss to ensure a fair comparison. The best results are highlighted in bold , and the second-best are underlined .

Specialist Medical VLMs
Method	CheXpert 5 $\times$ 200		RSNA		SIIM
Method	ACC (%)	AUC (%)	ACC (%)	AUC (%)	ACC (%)	AUC (%)
ConVIRT [ 64 ]	45.10	75.82	42.64	42.13	46.54	49.09
GLoRIA [ 18 ]	24.80	71.60	68.42	76.73	48.67	50.13
MGCA [ 49 ]	21.40	52.62	63.75	68.21	58.52	62.55
BioVIL [ 2 ]	48.40	76.99	56.66	62.23	53.32	59.99

Table 2: Zero-shot classification performance on the CheXpert 5

\times

200, RSNA, and SIIM datasets. Competing methods are categorized into specialist and generalist medical VLMs.

Table 5

Components			Datasets
LoRA	ACE-HGNN	Label	CheXpert	RSNA	SIIM
✗	✗	✗	35.50	74.34	61.40
✓	✗	✗	45.60	67.07	68.29
✓	✓	✗	49.20	74.64	70.56
✓	✗	✓	45.80	72.85	72.50
✓	✓	✓	49.80	79.54	73.35

Table 5: Impact of each component in ACE-LoRA on classification accuracy. “Label” denotes the Label-guided InfoNCE loss. When this loss is not applied (✗), the CLIP loss is used during training.

Table 7

Method	CheXpert	RSNA	SIIM
BMC-CLIP	24.70	53.83	48.67
+ ACE-LoRA	50.50	67.35	70.70
$\Updelta$	+25.80	+13.52	+22.03

Table 8: Performance of ACE-LoRA when deployed on an alternative generalist medical VLM (BMC-CLIP).

\Updelta

denotes the performance boost in zero-shot accuracy that ACE-LoRA provides when adopted for BMC-CLIP.

Figure 3: Comparison of cross-modal similarity maps on the RSNA Dataset. Similarity maps show the correspondence between image regions and the text query “Pneumonia” , with red boxes indicating ground-truth (GT) abnormal regions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Dataset	Classes	Total Size	Train/Val/Test (%)
Classification	CheXpert 5 $\times$ 200 [ 21 ]	5	1000	-/-/100
Classification	RSNA [ 43 ]	2	29,684	-/–/15
Classification	SIIM [ 61 ]	2	12,047	–/–/30
Classification	NIH ChestX-ray [ 50 ]	14	112,120	–/5/-
Classification	LC25000 (Lung) [ 3 ]	3	15,000	–/–/100
Classification	LC25000 (Colon) [ 3 ]	2	10,000	–/–/100

Appendix

Table A.1: Summary of dataset specifications and statistics. “Classes” and “Total Size” refer to the number of categories and total images, respectively.

Dataset	Classes
CheXpert 5 $\times$ 200 [ 21 ]	"Atelectasis", "Cardiomegaly", "Consolidation", "Edema", "Pleural Effusion"
RSNA [ 43 ]	"No Finding", "Pneumonia"
SIIM [ 61 ]	"No Finding", "Pneumothorax"
NIH ChestX-ray [ 50 ]	"Atelectasis", "Cardiomegaly", "Effusion", "Infiltration", "Mass", ’"Nodule", "Pneumonia", "Pneumothorax", "Consolidation", "Edema", "Emphysema", "Fibrosis", "Pleural Thickening", "Hernia"
LC25000 (Lung) [ 3 ]	"Lung adenocarcinoma", "Benign lung", "Lung squamous cell carcinoma"
LC25000 (Colon) [ 3 ]	"Colon adenocarcinoma", "Benign colonic tissue"

Appendix

Table A.2: Class names for each dataset in zero-shot image classification.

Figure A.1 : Average zero-shot accuracy across three CXR benchmarks vs. number of trainable parameters (log-scale). The bubble size denotes the computational cost in GFLOPs during the forward pass.

IE	TE	CheXpert	RSNA	SIIM
✗	✗	45.80	72.85	72.50
✓	✗	49.00	75.84	73.60
✗	✓	50.40	76.92	72.80
✓	✓	49.80	79.54	73.35

Appendix

Table A.3 : Effects of ACE-HGNN module integration on the image encoder (IE) and the text encoder (TE). Classification accuracy is reported in all datasets.

Method	CheXpert	RSNA	SIIM
BiomedCLIP [ 62 ]	35.50	74.34	61.40
TransCLIP [ 60 ]	37.50	73.54	65.36

Appendix

Table A.4 : Analysis of transductive inference. Classification accuracy is reported in all datasets.

Figure A.2 : Zero-shot accuracy across three datasets for varying

k

values. We find that selecting

k=5

yields optimal performance.

Figure A.3 : Zero-shot accuracy across three datasets for different

r

values. Setting the rank

r=4

delivers the best performance.

Figure A.4 : Zero-shot accuracy as a function of pre-training data scale (log-scale). Dashed lines indicate the zero-shot performance of BiomedCLIP without domain adaptation.

Figure A.5 : Qualitative comparison of different image encoder backbones. We compare our method with PRIOR [ 5 ] and BiomedCLIP [ 62 ] on the semantic segmentation and object detection tasks. GT denotes the ground truth.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding
Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal Learning VLM Adaptation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence +1
TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models
Jul 10, 2026Tianyou Jiang, Ziyu ZhouFew-Shot Domain Adaptation Logit Adjustment
University of Bern, Bern, Switzerland · Shanghai Jiao Tong University, Shanghai, China
Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning
Sep 13, 2026Zhanghao Chen, Yuanyuan Li, Zhenyu Lu +3VLM Adaptation Medical VLMs
School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · School of Biological Science & Medical Engineering, Southeast University, Nanjing 211189, China

Aug 6, 2025J/K move · Enter open · S save

NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding

Zelin Peng, Yichen Zhao, Yu Huang, Piao Yang+4

MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence+1

Cross-Modal Learning VLM Adaptation Cross-Modal Alignment Medical VLMs Cross-Modal Attention Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Computer-aided medical image analysis is crucial for disease diagnosis and treatment planning. While vision-language models (VLMs) such as CLIP exhibit strong generalization ability, their direct application to medical imaging remains hindered by a substantial domain gap. Existing methods for bridging this gap, including prompt learning and unidirectional modality interaction, typically introduce domain knowledge into only one modality. However, such approaches fail to fully exploit CLIP's inherent dual-modality structure and overlook the synergistic effect of bidirectional cross-modal interaction, resulting in persistent modality misalignment. In this paper, we propose NEARL (iNteracted quEry Adaptation with oRthogonaL Regularization), a novel parameter-efficient VLM framework for bidirectional cross-modal interaction. NEARL consists of two key components: (1) the Unified Synergy Embedding Transformer (USEformer), which dynamically generates compact cross-modal queries to facilitate interaction; and (2) the Orthogonal Cross-Attention Adapter (OCA), which decouples new knowledge into truly novel and incremental components through orthogonal regularization. This design reduces interference from incremental components, enabling more focused learning of novel information and improving modality interaction in VLMs. Notably, NEARL introduces only 1.46M learnable parameters. Extensive experiments on three medical imaging modalities demonstrate state-of-the-art performance (e.g., a 2.3% relative improvement on the pneumonia dataset), along with fast inference and low memory overhead, highlighting its effectiveness for real-world medical vision-language understanding.

Abstract

VLM Adaptation

Explore similar work

NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning