Organizations: National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University
Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.
Figures & tables
Figure 1: Overview of InViC. Left: CTE module with learnable cue slots initialized from a pooled question representation, followed by cross-attention to question tokens, cross-attention to visual tokens, and adapter-style calibration. Middle: Two-stage training strategy where visual tokens are visible only to cue tokens in Stage I and fully visible in Stage II. Right: Sequence construction and LLM training with cross-entropy loss.
Figure 2: Qualitative comparison of Lingshu-7B and Qwen3-VL-4B on the SLAKE dataset. We visualize model predictions after zero-shot inference, LoRA fine-tuning, and integration of the InViC architecture. Blue highlights indicate correct predictions, while red indicates incorrect predictions.
Method
VQA-Med 2019
SLAKE
VQA-RAD
Accuracy
Recall
BLEU
Accuracy
Recall
BLEU
Accuracy
Recall
BLEU
Closed-source MLLMs
GPT-5.2
0.672
0.412
0.370
0.778
0.503
0.453
0.705
0.640
0.559
GPT-4.1
0.664
0.413
0.326
0.709
0.387
0.350
0.641
0.610
0.554
Medical-focused backbones
Hulu-Med-4B [ 5 ]
0.600
0.349
0.341
0.658
0.474
0.471
0.687
0.630
0.629
Table 1: Main results on three Med-VQA benchmarks (VQA-Med 2019, SLAKE, and VQA-RAD). We report Accuracy, Recall, and BLEU for each method. In the LoRA fine-tuning and InViC integration block, we highlight InViC results in bold.
Learnable tokens
Cue tokens
Cue token count
Metric
Stage II
Stage I + II
Stage II
Stage I + II
K=4
K=8
K=16
K=32
Accuracy
0.609
0.799
0.770
0.849
0.785
0.832
0.849
0.800
Recall
0.448
0.745
0.699
0.832
0.733
0.809
0.832
0.740
BLEU
0.449
0.747
0.701
0.835
0.735
0.811
0.835
0.740
Table 2: Ablation results on SLAKE with Qwen3-VL-4B. Left: comparison between learnable tokens and cue tokens under different training schedules ( K=16 ). Right: effect of the cue token count K .
Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.
Suyang Xi, Songtao Hu, Yuxiang Lai +4
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries. Recent approaches typically connect off-the-shelf vision encoders with large language models (LLMs) through lightweight mapping networks to reduce computational cost. However, these methods often overlook the importance of handling noise and small irrelevant changes in visual representations. To address these challenges, we propose a noise-aware Med-VQA framework that incorporates a denoising autoencoder before visual embeddings are mapped into the input space of an LLM. The denoising autoencoder is pretrained to reconstruct clean visual embeddings from corrupted inputs, encouraging the model to learn robust visual representations that are less sensitive to noise. The resulting embeddings are then projected into the language model embedding space using a multi-layer perceptron (MLP), forming visual prefix tokens that provide image information to the LLM. To enable efficient adaptation without full retraining, we employ parameter-efficient fine-tuning using low-rank adaptation (LoRA). The proposed method is evaluated on the SLAKE and PathVQA benchmarks. Experimental results show improved robustness to noisy input embeddings while maintaining competitive clean performance across multiple evaluation criteria. These findings suggest that learning more robust visual representations can enhance Med-VQA performance and robustness.
I Putu Adi Pratama, Bahadorreza Ofoghi, Atul Sajjanhar +1
Deakin University, 221 Burwood Highway, Burwood, Victoria 3125, Australia
Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with MedProb, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.