MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Authors: Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan, Yaqi Liu, Shansong Wang, Xiaofeng Yang
Organizations: Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.
Figures & tables
Figure 1: Comparison between conventional text-space reasoning and Medical Latent Visual Reasoning. In conventional text-space reasoning, the model autoregressively generates intermediate textual reasoning tokens before producing the final answer. In Medical Latent Visual Reasoning, the last hidden states are directly reused as input embeddings to form latent reasoning steps, providing additional hidden-space computation before answer generation.
Figure 2: Construction and composition of the large-scale medical grounding corpus. a , Overview of the corpus construction pipeline. We collect 30 public medical imaging datasets spanning eight modalities. For the 29 non-CXR source datasets, expert spatial annotations are converted into bounding boxes and diagnostic labels. The labels are enriched with clinically relevant context retrieved from PrimeKG and passed to MedGemma-1.5 for text-only QA generation, while the raw image and bounding box coordinates are withheld from the generation model. The quality of the automatically generated QA–box triplets is estimated through a stratified expert audit of 1,200 samples. The GEMeX chest X-ray corpus is incorporated directly and unified with the automatically generated data, resulting in 1,223,332 QA–box triplets. b , Composition and representative examples of the resulting corpus. The left panel shows the number and proportion of QA–box triplets across the eight imaging modalities; the middle panel summarizes the number of source datasets per modality and the overall corpus statistics; and the right panel presents representative triplets from ultrasound, CT, and MRI. In total, the corpus contains 505,554 automatically generated triplets from 29 non-CXR datasets together with 717,778 CXR pairs from GEMeX.
Dataset
Modality
# QA-Box
Diagnostic Categories
Computed Tomography (CT)
LiTS
CT
120,321
Liver; Liver tumor
MSD-Task7
CT
99,708
Pancreas; Pancreas tumor
MSD-Task8
CT
49,727
Hepatic tumor
MSD-Task10
CT
17,257
Colon cancer
MSD-Task6
CT
7,477
Lung tumours
Table 1: Summary of the 29 source datasets (excluding GEMeX for CXR) used to construct the medical grounding corpus. Each dataset name is hyperlinked to its public source. We report the imaging modality, number of generated QA–box triplets, and annotated diagnostic categories. The grand total includes 717,778 GEMeX CXR pairs incorporated directly without automated QA generation (Section 3.3 ).
Figure 3: Overview of the proposed MedLVR framework for medical VQA. The top panel shows the autoregressive multimodal reasoning pipeline, in which visual, textual, latent, and special embeddings are jointly processed by the multimodal language model. Rather than generating long textual reasoning chains, MedLVR reuses hidden states between <MedLVR_Start> and <MedLVR_End> as latent reasoning steps before producing the final answer. The bottom panel illustrates the two-stage training strategy: Stage 1 uses ROI-supervised latent evidence formation to shape intermediate latent states, while Stage 2 applies reinforcement learning with the VLPO objective to directly optimize latent reasoning and answer generation.
Category
Parameter
Stage 1
Stage 2
Category
Parameter
Stage 1
Stage 2
Optimization
Learning rate
1×10−5
2×10−8
VLPO
Group size G
–
4
Weight decay
0.1
0.1
Exploration scale σ
–
0.05
Warmup ratio
0.03
0.03
Clipping range ϵ
–
0.2
LR schedule
Cosine
Cosine
KL coefficient β
–
0.04
Training steps
2,500
One epoch
Latent weight λlat
–
0.1
Latent reasoning
Latent budget K
8
8
Decoding
Temperature (training)
–
0.2
Table 2: Hyperparameter settings for Stage 1 and Stage 2 training.
Figure 4: Mechanistic validation of latent visual reasoning. a , Stage-wise ROI-specific evidence gap for B3, B4, and B5. Latent steps are grouped into Early (1–3), Middle (4–8), Late (9–11), and Final (12) stages; Overall averages all 12 steps. b , Sample-level distribution of the overall ROI-specific evidence gap on the 200-case held-out mechanism set. Boxes denote the interquartile range, horizontal lines denote the median, circles denote the mean, and whiskers denote mean ± s.d. Percentages below the axis indicate the fraction of samples with a positive gap. Adjusted p values are reported for pairwise comparisons. c , Stage-wise ROI perturbation sensitivity for B3 and B5. d , ROI-specific accuracy drop after masking the clinically relevant region. Points denote the estimated drop and horizontal intervals denote 95% confidence intervals. e , Average accuracy across the five external medical VQA benchmarks for matched B1–B5 controls. f , Blank-image gap on MMMU (Health & Medicine), VQA-RAD, and PMC-VQA for B1, B3, and B7.
Variant
SLAKE
VQA-RAD
PMC-VQA
MMMU-HM
MedXpertQA
B1: Same-Data Qwen
63.2
60.7
49.6
48.1
22.7
B3: Latent SFT, no ROI
64.9
62.5
51.3
52.5
23.2
B5: MedLVR + ROI Formation
65.7
64.7
52.8
54.7
23.8
B6: Text-only Policy Optimization
66.0
65.1
53.1
55.3
24.0
B7: Full MedLVR
66.4
65.9
53.6
56.6
24.3
Table 5: Contribution of the main training components. Performance of different reasoning and optimization variants across five medical VQA benchmarks.
Figure 6: Visual evidence allocation during reasoning. Spatial activation maps compare the base model and MedLVR across decoder layers, while the layer-wise statistics quantify the relative allocation of attention to visual tokens.
Method
Avg. Tokens
Total Time
Overall Accuracy
MedGemma-1.5
102.1
1300.2s
9.1%
Chiron-o1
16.6
147.4s
44.3%
Qwen2.5-VL-7B
12.5
137.9s
35.9%
MedLVR
7.8
158.0s
45.2%
Table 6: Inference efficiency and accuracy. Avg. Tokens denotes the average number of generated output tokens per sample, and Total Time denotes total inference time on the 485-sample evaluation set.
Figure 7: Sensitivity to the latent reasoning budget. a , Mean accuracy across eight imaging modalities under K∈{2,4,8,14,16} . Performance remains stable across moderate latent budgets, and increasing K beyond 8 does not provide a consistent improvement. The shaded region denotes ±1 standard deviation across modalities. b , Accuracy deficit relative to the modality-specific optimum for each latent budget. At the fixed default setting of K=8 , the maximum deficit is 0.27 points and the mean deficit is 0.09 points, showing that a single shared budget retains near-optimal performance across modalities without modality-specific tuning.
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University