MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA

Haowen Gu; Gensheng Pei; Zeren Sun; Mingwu Ren; Xiangbo Shu; Yazhou Yao; Fumin Shen

cs.CVAug 27, 2026

MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA

Authors: Haowen Gu, Gensheng Pei, Zeren Sun, Mingwu Ren, Xiangbo Shu, Yazhou Yao, Fumin Shen

Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery · Department of Electrical and Computer Engineering, Sungkyunkwan University · School of Computer Science and Engineering, University of Electronic Science and Technology of China

Abstract

Medical Visual Question Answering (Med-VQA) holds significant promise for clinical decision support, yet faces challenges due to limited annotated data and the high computational demands of existing large vision-language models. We propose MedFG-VQA, a lightweight framework that leverages a memory bank to augment DCT-based low-frequency features and employs graph-enhanced cross-attention for effective visual-textual alignment. Specifically, our approach features two key components: Frequency-Memory Fusion (FMF), which enhances low-frequency features by retrieving from a learnable memory bank built on DCT decomposition, and Graph-Aware Cross-Attention (GACA), which aligns visual-textual features via cross-attention and refines them through graph-convolutional aggregation. To address data scarcity, we construct SynMed-VQA, a large-scale synthetic dataset comprising over 2 million question-answer pairs across 9 imaging modalities and 10 major organs, generated with GPT-4o. Extensive experiments on SynMed-VQA and three other standard biomedical VQA benchmarks demonstrate that MedFG-VQA achieves competitive or superior performance compared to much larger models while maintaining significantly lower computational costs, highlighting its efficiency and potential for clinical deployment.

Figures & tables

Figure 1: Illustration of VLMs evaluated on the SynMedVQA dataset. Among the 6 VLMs, MedFG-VQA achieves the highest overall score. Qualitative VQA comparison between two models, showcasing the effectiveness of MedFG-VQA.

Figure 2: The overall architecture of MedFG-VQA. The model consists of a vision encoder, a FreqMemoryFusion(FMF) module, a Graph-Aware Cross-Attention(GACA) module and a LLM.

Dataset	Images (train $\|$ val $\|$ test)	Questions (train $\|$ val $\|$ test)
CHMNIST [ 18 ]	2496 $\|$ 1000 $\|$ 1504	12480 $\|$ 5000 $\|$ 7520
RETINA [ 20 ]	2108 $\|$ 841 $\|$ 1268	10540 $\|$ 4205 $\|$ 6340
BTMRI [ 33 ]	2854 $\|$ 1141 $\|$ 1717	14270 $\|$ 5705 $\|$ 8585
DermaMNIST [ 40 ]	7006 $\|$ 1003 $\|$ 2005	35030 $\|$ 5015 $\|$ 10025
BUSI [ 3 ]	389 $\|$ 155 $\|$ 236	1945 $\|$ 775 $\|$ 1180
OCTMNIST [ 19 ]	97476 $\|$ 10832 $\|$ 1000	487380 $\|$ 54160 $\|$ 5000

Table 1: Overview of datasets statistics, including image and question counts for training, validation, and test splits. Open and Close questions share the same quantities.

Figure 3: The construction pipeline of SynMedVQA. The pipeline includes prompt template, samples of Q&A pair generated by GPT-4o and the quality control process.

Figure 4: Sunburst visualization of dataset category proportions, where the inner circle corresponds to datasets and the outer circle to category ratios within each dataset.

Method	Params	OCTMNIST	CHMNIST	LungColon	CTKidney	COVID-19	BTMRI	KneeXray	RETINA	Kvasir	DermaMNIST	BUSI	Average
InternVL3.5 [ 42 ]	1B	0.538	0.4569	0.4802	0.4714	0.4943	0.4261	0.5147	0.4355	0.4588	0.5662	0.433	0.4846
MiniCPM-V 4.0 [ 44 ]	4B	0.5374	0.4964	0.4987	0.52	0.503	0.4425	0.4838	0.4276	0.4503	0.5288	0.5033	0.4964
Qwen3-VL [ 43 ]	4B	0.5872	0.5334	0.5473	0.5547	0.5601	0.4869	0.5647	0.4877	0.5083	0.6076	0.5559	0.5492
Gemma3 [ 39 ]	4B	0.5208	0.4537	0.4474	0.4927	0.4751	0.4162	0.4739	0.4202	0.4142	0.4387	0.4483	0.4590
LLaVA-Med [ 24 ]	7B	0.4834	0.4961	0.5395	0.3991	0.3165	0.4096	0.5272	0.4719	0.3703	0.5788	0.4720	0.4834
Ours	795M	0.7056	0.5372	0.6493	0.6239	0.6733	0.6155	0.6609	0.5372	0.5440	0.7225	0.6284	0.6441

Table 2: Comparison of model performance on SynMedVQA. All baseline models use their publicly available pretrained weights.

Model	Params			SLAKE [ 27 ]		VQA-RAD [ 23 ]		PathVQA [ 15 ]
Model	Vision	LLM	Total	Closed	Open	Closed	Open	Closed	Open
InternVL3.5 [ 42 ]	0.3B	0.8B	1.1B	0.6459	0.8156	0.6140	0.6180	0.5694	0.3187
MiniCPM-V 4.0 [ 44 ]	0.4B	3.0B	4.1B	0.4246	0.8315	0.7353	0.7978	0.6376	0.4513
Qwen3-VL [ 43 ]	0.4B	4.0B	4.4B	0.5120	0.9006	0.7868	0.8764	0.6679	0.5249
Gemma3 [ 39 ]	0.4B	0.6B	3.2B	0.5287	0.7337	0.6507	0.8090	0.8410	0.4608
LLaVA-Med [ 24 ]	0.3B	7.2B	7.5B	0.6567	0.1232	0.6434	0.0730	0.7308	0.2828

Table 3: Comparative results on three public medical VQA benchmarks. The best and second-best results are indicated in bold and underline , respectively.

Table 8

Panel of Figure 5 — Figure 5: Qualitative results against (a) Gemma3-4B , (b) Qwen3-VL-4B, and (c) LLaVA-Med v1.5. Responses are abridged for brevity.

Table 5: Ablation studies on different frequency-domain transformation strategies and module against standard cross-attention.

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

InViC: Intent-aware Visual Cues for Medical Visual Question Answering
Mar 17, 2026Zhisong Wang, Ziyang Chen, Zanting Ye +3Medical VQA Medical VLMs
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University
Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering
Aug 8, 2026Yusra Tariq, Rakesh Chandra JoshiMedical VQA Multimodal Fusion
Department of Artificial Intelligence, Amity University, Noida, India
KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering
Apr 1, 2026Xianyao Zheng, Hong Yu, Hui Cui +7Medical VQA Medical VLMs
School of Software, Northwestern Polytechnical University, Shaanxi, China · Tianjin Central Hospital of Gynecology Obstetrics, Tianjin, China · Department of Computer Science and Information Technology, La Trobe University, Melbourne, Australia +6

Mar 17, 2026J/K move · Enter open · S save

InViC: Intent-aware Visual Cues for Medical Visual Question Answering

Zhisong Wang, Ziyang Chen, Zanting Ye, Hongze Zhu+2

National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University

Medical VQA Medical VLMs Visual Token Compression

Open paper AskPDF arXiv

Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.

Dataset	Images (train $\|$ val $\|$ test)	Questions (train $\|$ val $\|$ test)
CHMNIST [ 18 ]	2496 $\|$ 1000 $\|$ 1504	12480 $\|$ 5000 $\|$ 7520
RETINA [ 20 ]	2108 $\|$ 841 $\|$ 1268	10540 $\|$ 4205 $\|$ 6340
BTMRI [ 33 ]	2854 $\|$ 1141 $\|$ 1717	14270 $\|$ 5705 $\|$ 8585
DermaMNIST [ 40 ]	7006 $\|$ 1003 $\|$ 2005	35030 $\|$ 5015 $\|$ 10025
BUSI [ 3 ]	389 $\|$ 155 $\|$ 236	1945 $\|$ 775 $\|$ 1180
OCTMNIST [ 19 ]	97476 $\|$ 10832 $\|$ 1000	487380 $\|$ 54160 $\|$ 5000

Abstract

Medical VQA

Explore similar work

InViC: Intent-aware Visual Cues for Medical Visual Question Answering

Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering

KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering