MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Authors: Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan, Yaqi Liu, Shansong Wang, Xiaofeng Yang
Organizations: Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.
Figures & tables
Figure 1: Comparison between conventional text-space reasoning and Medical Latent Visual Reasoning. In conventional text-space reasoning, the model autoregressively generates intermediate textual reasoning tokens before producing the final answer. In Medical Latent Visual Reasoning, the last hidden states are directly reused as input embeddings to form latent reasoning steps, providing additional hidden-space computation before answer generation.
Figure 2: Construction and composition of the large-scale medical grounding corpus. a , Overview of the corpus construction pipeline. We collect 30 public medical imaging datasets spanning eight modalities. For the 29 non-CXR source datasets, expert spatial annotations are converted into bounding boxes and diagnostic labels. The labels are enriched with clinically relevant context retrieved from PrimeKG and passed to MedGemma-1.5 for text-only QA generation, while the raw image and bounding box coordinates are withheld from the generation model. The quality of the automatically generated QA–box triplets is estimated through a stratified expert audit of 1,200 samples. The GEMeX chest X-ray corpus is incorporated directly and unified with the automatically generated data, resulting in 1,223,332 QA–box triplets. b , Composition and representative examples of the resulting corpus. The left panel shows the number and proportion of QA–box triplets across the eight imaging modalities; the middle panel summarizes the number of source datasets per modality and the overall corpus statistics; and the right panel presents representative triplets from ultrasound, CT, and MRI. In total, the corpus contains 505,554 automatically generated triplets from 29 non-CXR datasets together with 717,778 CXR pairs from GEMeX.
Dataset
Modality
# QA-Box
Diagnostic Categories
Computed Tomography (CT)
LiTS
CT
120,321
Liver; Liver tumor
MSD-Task7
CT
99,708
Pancreas; Pancreas tumor
MSD-Task8
CT
49,727
Hepatic tumor
MSD-Task10
CT
17,257
Colon cancer
MSD-Task6
CT
7,477
Lung tumours
Table 1: Summary of the 29 source datasets (excluding GEMeX for CXR) used to construct the medical grounding corpus. Each dataset name is hyperlinked to its public source. We report the imaging modality, number of generated QA–box triplets, and annotated diagnostic categories. The grand total includes 717,778 GEMeX CXR pairs incorporated directly without automated QA generation (Section 3.3 ).
Figure 3: Overview of the proposed MedLVR framework for medical VQA. The top panel shows the autoregressive multimodal reasoning pipeline, in which visual, textual, latent, and special embeddings are jointly processed by the multimodal language model. Rather than generating long textual reasoning chains, MedLVR reuses hidden states between <MedLVR_Start> and <MedLVR_End> as latent reasoning steps before producing the final answer. The bottom panel illustrates the two-stage training strategy: Stage 1 uses ROI-supervised latent evidence formation to shape intermediate latent states, while Stage 2 applies reinforcement learning with the VLPO objective to directly optimize latent reasoning and answer generation.
Category
Parameter
Stage 1
Stage 2
Category
Parameter
Stage 1
Stage 2
Optimization
Learning rate
1×10−5
2×10−8
VLPO
Group size G
–
4
Weight decay
0.1
0.1
Exploration scale σ
–
0.05
Warmup ratio
0.03
0.03
Clipping range ϵ
–
0.2
LR schedule
Cosine
Cosine
KL coefficient β
–
0.04
Training steps
2,500
One epoch
Latent weight λlat
–
0.1
Latent reasoning
Latent budget K
8
8
Decoding
Temperature (training)
–
0.2
Table 2: Hyperparameter settings for Stage 1 and Stage 2 training.
Figure 4: Mechanistic validation of latent visual reasoning. a , Stage-wise ROI-specific evidence gap for B3, B4, and B5. Latent steps are grouped into Early (1–3), Middle (4–8), Late (9–11), and Final (12) stages; Overall averages all 12 steps. b , Sample-level distribution of the overall ROI-specific evidence gap on the 200-case held-out mechanism set. Boxes denote the interquartile range, horizontal lines denote the median, circles denote the mean, and whiskers denote mean ± s.d. Percentages below the axis indicate the fraction of samples with a positive gap. Adjusted p values are reported for pairwise comparisons. c , Stage-wise ROI perturbation sensitivity for B3 and B5. d , ROI-specific accuracy drop after masking the clinically relevant region. Points denote the estimated drop and horizontal intervals denote 95% confidence intervals. e , Average accuracy across the five external medical VQA benchmarks for matched B1–B5 controls. f , Blank-image gap on MMMU (Health & Medicine), VQA-RAD, and PMC-VQA for B1, B3, and B7.
Variant
SLAKE
VQA-RAD
PMC-VQA
MMMU-HM
MedXpertQA
B1: Same-Data Qwen
63.2
60.7
49.6
48.1
22.7
B3: Latent SFT, no ROI
64.9
62.5
51.3
52.5
23.2
B5: MedLVR + ROI Formation
65.7
64.7
52.8
54.7
23.8
B6: Text-only Policy Optimization
66.0
65.1
53.1
55.3
24.0
B7: Full MedLVR
66.4
65.9
53.6
56.6
24.3
Table 5: Contribution of the main training components. Performance of different reasoning and optimization variants across five medical VQA benchmarks.
Figure 6: Visual evidence allocation during reasoning. Spatial activation maps compare the base model and MedLVR across decoder layers, while the layer-wise statistics quantify the relative allocation of attention to visual tokens.
Method
Avg. Tokens
Total Time
Overall Accuracy
MedGemma-1.5
102.1
1300.2s
9.1%
Chiron-o1
16.6
147.4s
44.3%
Qwen2.5-VL-7B
12.5
137.9s
35.9%
MedLVR
7.8
158.0s
45.2%
Table 6: Inference efficiency and accuracy. Avg. Tokens denotes the average number of generated output tokens per sample, and Total Time denotes total inference time on the 485-sample evaluation set.
Figure 7: Sensitivity to the latent reasoning budget. a , Mean accuracy across eight imaging modalities under K∈{2,4,8,14,16} . Performance remains stable across moderate latent budgets, and increasing K beyond 8 does not provide a consistent improvement. The shaded region denotes ±1 standard deviation across modalities. b , Accuracy deficit relative to the modality-specific optimum for each latent budget. At the fixed default setting of K=8 , the maximum deficit is 0.27 points and the mean deficit is 0.09 points, showing that a single shared budget retains near-optimal performance across modalities without modality-specific tuning.
Latent reasoning enables reasoning over continuous hidden states rather than explicit tokens, avoiding the language bottleneck and inference overhead of chain-of-thought for medical VQA. However, existing methods suffer from modality collapse, insufficient visual supervision, and train-inference mismatch. Moreover, their opaque latent states offer no interpretability, which is critical in clinical applications. We propose VITAL, a latent-space reasoning framework for medical MLLMs with visual-semantic dual supervision: an auxiliary text decoder reconstructs reasoning chains from latent states, while a visual projector regresses ROI features from a frozen, independent medical vision encoder. Both modules are discarded at inference with zero overhead, yet can be re-attached post-hoc for dual interpretability, providing textual and visual explanations of the reasoning process without sacrificing efficiency. We construct a 61K dataset spanning 9 imaging modalities, exceeding prior medical visual latent reasoning datasets by an order of magnitude. Experiments on 7 benchmarks show that VITAL consistently and substantially outperforms the backbone, all latent reasoning baselines, and medical MLLMs trained on far larger data, achieving state-of-the-art results competitive with trillion-parameter proprietary models.
Qiaoru Li, Shaotian Liang, Jintao Chen +4
Zhejiang University · Tencent · Shanghai AI Laboratory +2
Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.
Zhisong Wang, Ziyang Chen, Zanting Ye +3
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University
Medical Vision-Language Models (VLMs) hold immense promise for complex clinical tasks, but their reasoning capabilities are often constrained by text-only paradigms that fail to ground inferences in visual evidence. This limitation not only curtails performance on tasks requiring fine-grained visual analysis but also introduces risks of visual hallucination in safety-critical applications. Thus, we introduce MedVR, a novel reinforcement learning framework that enables annotation-free visual reasoning for medical VLMs. Its core innovation lies in two synergistic mechanisms: Entropy-guided Visual Regrounding (EVR) uses model uncertainty to direct exploration, while Consensus-based Credit Assignment (CCA) distills pseudo-supervision from rollout agreement. Without any human annotations for intermediate steps, MedVR achieves state-of-the-art performance on diverse public medical VQA benchmarks, significantly outperforming existing models. By learning to reason directly with visual evidence, MedVR promotes the robustness and transparency essential for accelerating the clinical deployment of medical AI.
Zheng Jiang, Heng Guo, Chengyu Fang +4
Tsinghua University · DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China +2