Enhancing RAG-based Medical VQA with Gated Multimodal Representations

Daniela Jimenez-Pano; Hugo Jair Escalante; Delia Irazú Hernández-Farías

cs.CVOct 16, 2026

Enhancing RAG-based Medical VQA with Gated Multimodal Representations

Authors: Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-Farías

Organizations: LabTL - INAOE, Puebla, Mexico · Institute for AAII, UTEP, El Paso, TX, USA · Department of Computer Science, UTEP, El Paso, TX, USA

Abstract

Medical Vision-Language Models (Med-LVLMs) have shown strong performance in Medical Visual Question Answering (MVQA), yet they may still produce factually inconsistent responses. Retrieval-Augmented Generation (RAG) can mitigate this issue by incorporating external clinical evidence; however, retrieval is often guided by a single modality, potentially overlooking the joint semantics of medical images and clinical questions. We introduce a multimodal RAG framework that learns a joint query representation through a Gated Multimodal Unit, enabling evidence retrieval to leverage information from both modalities while keeping the underlying Med-LVLM frozen. We evaluate the proposed approach on benchmark datasets. Results show that the multimodal query representation consistently outperforms a reference Med-LVLM and remains competitive with state-of-the-art methods. Our findings demonstrate the potential of multimodal query representations for more effective evidence retrieval in Medical VQA.

Figures & tables

Figure 1: Overview of the proposed framework for Medical VQA.

Dataset	Source data	Train (+/-)	Test
MIMIC-CXR [ 5 ]	377,110 X-rays / 227,835 studies	962 / 6,532	3,468
IU-XRAY [ 2 ]	7,470 X-rays / 3,955 reports	1,868 / 6,022	2,569
Harvard-FairVLMed [ 8 ]	10,000 glaucoma samples	851 / 6,457	4,266

Table 1: Datasets and binary VQA splits used in this work.

	IU-Xray			MIMIC-CXR			FairVLMed
Method	Acc	F1	AUC	Acc	F1	AUC	Acc	F1	AUC
GMU with dual encoder
Baseline	62.79	55.78	70.92	73.79	81.86	67.12	72.11	81.80	70.90
F-R	66.33	59.33	74.64	74.94	81.97	69.66	72.39	82.24	68.08
F-Q	61.74	55.78	71.00	76.73	82.48	73.05	78.79	86.55	78.34
F-I	59.28	54.84	70.03	76.01	82.51	71.23	87.58	92.90	70.25

Table 2: Results on closed-ended Medical VQA. F-R, F-Q and F-I denote the GMU configurations. Results marked with

*

correspond to the retrieval-only configurations of RULE [ 16 ] and MMed-RAG [ 15 ] , i.e., retrieved contexts added to the Med-LVLM prompt without preference fine-tuning, taken from the original papers. Best results per column are in bold .

		IU-Xray			MIMIC-CXR			FairVLMed
Model	$k$	Acc	F1	AUC	Acc	F1	AUC	Acc	F1	AUC
Baseline	1	62.39	56.44	82.74	80.36	85.75	86.85	89.06	93.82	57.48
	3	75.16	64.79	85.45	82.75	86.60	88.16	90.18	94.54	68.33
	5	76.06	65.31	85.56	82.61	86.41	88.32	90.72	94.80	71.26
F-R	1	62.08	56.28	82.64	80.19	85.64	86.74	89.45	94.02	89.23
	3	74.81	64.31	85.51	82.49	86.37	87.95	85.93	91.55	89.20

Table 3: Ablation study for the number of retrieved contexts

k

Gated Multimodal Fusion

See all 14 papers

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
Apr 30, 2026Xupeng Chen, Binbin Shi, Chenqian Le +5Multimodal RAG Retrieval-Augmented Generation
New York University, New York, USA · Tsinghua University, Beijing, China · Independent Researcher +1
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction
Mar 5, 2026Jiashu Yang, Chi Zhang, Abudukelimu Wuerkaixi +5Multimodal RAG Multimodal QA
Dalian University of Technology · Meituan LongCat Interaction Team · Tsinghua University

Apr 30, 2026J/K move · Enter open · S save

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang+4

New York University, New York, USA · Tsinghua University, Beijing, China · Independent Researcher+1

Multimodal RAG Retrieval-Augmented Generation Iterative RAG Healthcare Medical VLMs Medical QA+1

Open paper AskPDF arXiv

Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual content (tables, figures, structured layouts) of original document pages. We propose MED-VRAG, an iterative multimodal RAG framework that retrieves and reasons over PMC document page images instead of OCR'd text. The system pairs ColQwen2.5 patch-level page embeddings with a sharded MapReduce LLM filter, scaling to ~350K pages while keeping Stage-1 retrieval under 30 ms via an offline coarse-to-fine index (C=8 centroids per page, ANN over centroids, exact two-way scoring on the top-R shortlist). A vision-language model (VLM) then iteratively refines its query and accumulates evidence in a memory bank across up to 3 reasoning rounds, with a single iteration costing ~15.9 s and the full three-round pipeline ~47.8 s on 4xA100. Across four medical QA benchmarks (MedQA, MedMCQA, PubMedQA, MMLU-Med), MEDVRAG reaches 78.6% average accuracy. Under controlled comparison with the same Qwen2.5-VL-32B backbone, retrieval contributes a +5.8 point gain over the no-retrieval baseline; we also note a +1.8 point edge over MedRAG + GPT-4 (76.8%), with the caveat that this is a cross-paper rather than head-to-head comparison. Ablations isolate +1.0 from page-image vs text-chunk retrieval, +1.5 from iteration, and +1.0 from the memory bank.

Abstract

Gated Multimodal Fusion

Explore similar work

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction