Improving Medical VQA through Trajectory-Aware Process Supervision

Halil Ibrahim Gulluk; Olivier Gevaert

cs.LGApr 10, 2026

Improving Medical VQA through Trajectory-Aware Process Supervision

Authors: Halil Ibrahim Gulluk, Olivier Gevaert

Organizations: Electrical Engineering, Stanford University · Stanford Center For Biomedical Informatics Research

Abstract

Reasoning capabilities are crucial for reliable medical visual question answering (VQA); however, existing datasets rarely include reasoning explanations. We address this by generating reasoning trajectories for six medical VQA benchmarks using the COMCTS algorithm with open-source vision-language models, with an LLM serving as the verification judge. Building on these generated datasets, we propose a two-stage training framework: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO) with a novel process-based reward. While standard approaches rely solely on exact-match rewards for final answers, we introduce a trajectory-aware reward that measures the similarity between generated and ground-truth reasoning processes. Specifically, we embed reasoning steps using sentence transformers and compute the Dynamic Time Warping (DTW) distance between the resulting vector sequences. Experiments across six benchmarks demonstrate that combining the DTW-based process reward with exact-match reward consistently outperforms SFT-only training, raising mean accuracy from 0.598 to 0.689, mean BERTScore from 0.845 to 0.881, and mean ROUGE-L from 0.665 to 0.748. Our results highlight the importance of process supervision in training reasoning-capable medical VLMs. We make our code and generated reasoning datasets publicly available at https://anonymous.4open.science/r/MICCAI-R1-MED-VQA-code-B14B/

Figures & tables

Panel of Figure 2 — Figure 2: Example of a reasoning trajectory from the generated dataset $\mathcal{D}_{\text{reas}}$ . Given a chest X-ray and a question about heart size, the model produces a chain of reasoning steps ( $r_{1}$ to $r_{9}$ ) that analyze the image before arriving at the final answer ( $r_{10}$ ).

Panel of Figure 3 — Figure 3: Dynamic Time Warping (DTW) aligns two reasoning trajectories represented as sequences of sentence embeddings. DTW allows many-to-one matching, enabling flexible alignment when the generated reasoning $\mathcal{S}_{g}$ has more or fewer steps than the ground truth $\mathcal{S}_{gt}$ .

Symbol	Description
$q$	Input prompt (image and question)
$o_{i}$	The $i$ -th sampled output from the policy
$G$	Group size (number of sampled outputs per prompt)
$\pi_{\theta}$	Current policy parameterized by $\theta$
$\pi_{\theta_{\text{old}}}$	Policy from the previous iteration
$\pi_{\text{ref}}$	Reference policy (e.g., SFT model)

Table 1: Notation used in the GRPO formulation.

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.514\pm 0.012$	$0.643\pm 0.013$	$0.664\pm 0.008$	$\mathbf{0.710\pm 0.009}$
PMC-VQA	$0.545\pm 0.009$	$0.558\pm 0.015$	$0.613\pm 0.012$	$\mathbf{0.641\pm 0.006}$
RAD-VQA	$0.578\pm 0.011$	$0.546\pm 0.014$	$0.665\pm 0.008$	$\mathbf{0.676\pm 0.012}$
SLAKE-VQA	$0.761\pm 0.009$	$0.716\pm 0.007$	$0.771\pm 0.009$	$\mathbf{0.817\pm 0.006}$
VQA-MED	$0.681\pm 0.010$	$0.688\pm 0.007$	$0.717\pm 0.007$	$\mathbf{0.733\pm 0.007}$
OmniMed-VQA	$0.510\pm 0.015$	$0.410\pm 0.014$	$0.522\pm 0.016$	$\mathbf{0.557\pm 0.016}$

Table 2: Accuracy

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.515\pm 0.013$	$0.644\pm 0.013$	$0.665\pm 0.007$	$\mathbf{0.713\pm 0.010}$
PMC-VQA	$0.663\pm 0.008$	$0.666\pm 0.011$	$0.712\pm 0.007$	$\mathbf{0.738\pm 0.005}$
RAD-VQA	$0.597\pm 0.015$	$0.571\pm 0.015$	$0.681\pm 0.006$	$\mathbf{0.695\pm 0.011}$
SLAKE-VQA	$0.786\pm 0.007$	$0.745\pm 0.007$	$0.791\pm 0.007$	$\mathbf{0.841\pm 0.006}$
VQA-MED	$0.705\pm 0.009$	$0.713\pm 0.006$	$0.740\pm 0.006$	$\mathbf{0.758\pm 0.006}$
OmniMed-VQA	$0.552\pm 0.007$	$0.459\pm 0.016$	$0.560\pm 0.017$	$\mathbf{0.581\pm 0.011}$

Table 3: Sentence BLEU-1

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.092\pm 0.002$	$0.115\pm 0.002$	$0.119\pm 0.001$	$\mathbf{0.127\pm 0.002}$
PMC-VQA	$0.334\pm 0.006$	$0.329\pm 0.009$	$0.358\pm 0.005$	$\mathbf{0.370\pm 0.005}$
RAD-VQA	$0.112\pm 0.003$	$0.108\pm 0.004$	$0.128\pm 0.002$	$\mathbf{0.131\pm 0.004}$
SLAKE-VQA	$0.147\pm 0.001$	$0.142\pm 0.002$	$0.154\pm 0.002$	$\mathbf{0.165\pm 0.003}$
VQA-MED	$0.249\pm 0.004$	$0.243\pm 0.004$	$0.255\pm 0.003$	$\mathbf{0.259\pm 0.004}$
OmniMed-VQA	$0.157\pm 0.003$	$0.120\pm 0.008$	$0.160\pm 0.007$	$\mathbf{0.168\pm 0.007}$

Table 4: Sentence BLEU-4

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.810\pm 0.003$	$0.847\pm 0.004$	$0.852\pm 0.002$	$\mathbf{0.864\pm 0.004}$
PMC-VQA	$0.812\pm 0.005$	$0.810\pm 0.007$	$0.834\pm 0.004$	$\mathbf{0.846\pm 0.003}$
RAD-VQA	$0.826\pm 0.005$	$0.824\pm 0.009$	$0.861\pm 0.003$	$\mathbf{0.864\pm 0.004}$
SLAKE-VQA	$0.909\pm 0.002$	$0.900\pm 0.002$	$0.915\pm 0.004$	$\mathbf{0.929\pm 0.003}$
VQA-MED	$0.852\pm 0.005$	$0.852\pm 0.003$	$0.864\pm 0.002$	$\mathbf{0.874\pm 0.003}$
OmniMed-VQA	$0.798\pm 0.004$	$0.755\pm 0.006$	$0.795\pm 0.010$	$\mathbf{0.803\pm 0.005}$

Table 5: COMET

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.776\pm 0.007$	$0.833\pm 0.007$	$0.847\pm 0.002$	$\mathbf{0.861\pm 0.005}$
PMC-VQA	$0.869\pm 0.004$	$0.868\pm 0.006$	$0.889\pm 0.003$	$\mathbf{0.899\pm 0.003}$
RAD-VQA	$0.813\pm 0.007$	$0.799\pm 0.006$	$0.858\pm 0.006$	$\mathbf{0.863\pm 0.005}$
SLAKE-VQA	$0.911\pm 0.002$	$0.894\pm 0.004$	$0.915\pm 0.004$	$\mathbf{0.935\pm 0.003}$
VQA-MED	$0.867\pm 0.004$	$0.871\pm 0.003$	$0.885\pm 0.003$	$\mathbf{0.891\pm 0.005}$
OmniMed-VQA	$0.833\pm 0.004$	$0.784\pm 0.005$	$0.824\pm 0.010$	$\mathbf{0.835\pm 0.005}$

Table 6: BERTScore

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.261\pm 0.006$	$0.325\pm 0.007$	$0.335\pm 0.003$	$\mathbf{0.359\pm 0.005}$
PMC-VQA	$0.579\pm 0.008$	$0.581\pm 0.010$	$0.621\pm 0.008$	$\mathbf{0.643\pm 0.005}$
RAD-VQA	$0.316\pm 0.009$	$0.304\pm 0.008$	$0.360\pm 0.005$	$\mathbf{0.367\pm 0.009}$
SLAKE-VQA	$0.428\pm 0.004$	$0.407\pm 0.006$	$0.435\pm 0.006$	$\mathbf{0.464\pm 0.006}$
VQA-MED	$0.466\pm 0.006$	$0.466\pm 0.005$	$0.483\pm 0.003$	$\mathbf{0.493\pm 0.004}$
OmniMed-VQA	$0.432\pm 0.006$	$0.346\pm 0.010$	$0.415\pm 0.016$	$\mathbf{0.443\pm 0.011}$

Table 7: METEOR

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.521\pm 0.013$	$0.647\pm 0.014$	$0.669\pm 0.007$	$\mathbf{0.716\pm 0.011}$
PMC-VQA	$0.682\pm 0.008$	$0.684\pm 0.011$	$0.729\pm 0.007$	$\mathbf{0.752\pm 0.006}$
RAD-VQA	$0.629\pm 0.017$	$0.611\pm 0.017$	$0.722\pm 0.009$	$\mathbf{0.734\pm 0.007}$
SLAKE-VQA	$0.805\pm 0.007$	$0.761\pm 0.006$	$0.806\pm 0.010$	$\mathbf{0.851\pm 0.008}$
VQA-MED	$0.705\pm 0.009$	$0.716\pm 0.005$	$0.744\pm 0.006$	$\mathbf{0.760\pm 0.006}$
OmniMed-VQA	$0.647\pm 0.010$	$0.570\pm 0.015$	$0.646\pm 0.021$	$\mathbf{0.674\pm 0.011}$

Table 8: ROUGE-L F1

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.858\pm 0.003$	$\mathbf{0.861\pm 0.005}$	$0.850\pm 0.004$
PMC-VQA	$0.897\pm 0.003$	$\mathbf{0.899\pm 0.003}$	$0.894\pm 0.002$
RAD-VQA	$0.859\pm 0.006$	$\mathbf{0.863\pm 0.005}$	$0.854\pm 0.007$
SLAKE-VQA	$0.913\pm 0.006$	$\mathbf{0.935\pm 0.003}$	$0.928\pm 0.002$
VQA-MED	$\mathbf{0.895\pm 0.004}$	$0.891\pm 0.005$	$0.892\pm 0.004$
OmniMed-VQA	$0.832\pm 0.007$	$0.835\pm 0.005$	$\mathbf{0.838\pm 0.006}$

Table 9: BERTScore – NW-Based Reward Comparison

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: COMCTS generated sample from VQA-RAD. Question : What are the hyperdense lesions noted at the edges of the aorta? Gt-answer (in $\mathcal{D}_{VQA}$ ) : calcified atherosclerosis Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a cross-sectional CT scan of the abdomen. It shows the aorta, kidneys, and surrounding structures. There are hyperdense lesions noted at the edges of the aorta. The CT scan image shows a cross-section of the abdomen. The aorta is visible in the center of the image. There are hyperdense lesions noted at the edges of the aorta. The final answer is: The hyperdense lesions noted at the edges of the aorta are likely calcifications or atherosclerotic plaques.

Figure 5: COMCTS generated sample from VQA-RAD. Question : Do you see cardiomegaly? Gt-answer (in $\mathcal{D}_{VQA}$ ) : no Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a chest X-ray showing the thoracic cavity with the heart, lungs, and ribs visible. The heart appears to be of normal size, and there are no obvious signs of cardiomegaly. Cardiomegaly is the medical term for an enlarged heart. In chest X-rays, cardiomegaly is typically identified by the heart’s size relative to the lungs and other structures. The heart’s size in this X-ray appears to be within normal limits. The final answer is: No, there is no cardiomegaly

Figure 6: COMCTS generated sample from Slake-VQA. Question : Does the picture contain heart? Gt-answer (in $\mathcal{D}_{VQA}$ ) : No Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a grayscale axial CT scan of the abdomen. It shows the liver on the left, the stomach and intestines in the center, and the spine in the lower center. The kidneys are visible on either side of the spine. There is fat and muscle tissue surrounding the organs. The image is a cross-sectional view, and structures that would be anterior or posterior to the plane of the image are not fully visible. The question asks about the presence of the heart in the image. A CT scan of the abdomen focuses on the abdominal organs. The heart is located in the chest, not the abdomen. Therefore, it would not be expected to be visible in an abdominal CT scan. The final answer is: No.

Figure 7: COMCTS generated sample from Slake-VQA. Question : Does the picture contain kidney? Gt-answer (in $\mathcal{D}_{VQA}$ ) : No Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a grayscale axial CT scan of the upper abdomen. It shows the liver, gallbladder, portions of the stomach and duodenum, and the spine. The liver takes up a large portion of the left side of the image, appearing as a relatively homogenous density. The gallbladder is visible adjacent to the liver. A portion of the stomach and duodenum can be seen on the right side. The vertebral body is also visible in the inferior aspect of the image. There are some artifacts in the upper and right portions of the image. The question asks about the presence of kidneys in the image. To answer this, one needs to identify the anatomical location of kidneys and whether they are visible within the field of view of the CT scan. Typically, kidneys are found in the retroperitoneal space along the posterior abdominal wall, slightly inferior to the liver and stomach. The final answer is: No.

Figure 8: COMCTS generated sample from PMC-VQA. Question : What is the condition shown in the CT scan? Choices: A) :Pancreatic cancer B) :Right upper lobe herniation C) :Broken rib D) :Appendicitis Gt-answer (in $\mathcal{D}_{VQA}$ ) : Right upper lobe herniation Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a CT scan of the chest. The scan shows the lungs and surrounding structures. There is a noticeable protrusion or displacement of lung tissue into the chest cavity, indicated by an arrow. The lower part of the right lung appears to be displaced downwards and possibly compressed. The question asks to identify the condition displayed in the CT scan. The image shows a clear displacement of lung tissue. We must analyze the provided options and correlate them with the observed anatomical changes in the scan. Options like appendicitis or pancreatic cancer are unlikely to present with such a finding in a chest CT scan. Broken rib can’t fully explain the lung displacement. Herniation is the most plausible option given the observed lung displacement. The final answer is: B) :Right upper lobe herniation.

Figure 9: COMCTS generated sample from PMC-VQA. Question : Which bone is affected according to the radiograph? Choices: A) :Femur B) :Humerus C) :Tibia D) :Fibula Gt-answer (in $\mathcal{D}_{VQA}$ ) : Humerus Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a black and white radiograph (X-ray) of a human upper limb. The image shows a clear view of the shoulder joint and part of the upper arm. There appears to be a fracture involving the proximal end of the humerus, with significant comminution (multiple fragments). The bone appears fragmented and displaced. The radiograph is labeled with "R" on the right side and "A" on the lower left. The question asks to identify the affected bone in the radiograph. To answer this correctly, one needs to identify anatomical structures visible in the X-ray and match them with the provided options. The radiograph clearly shows the shoulder joint and the upper arm bone. The final answer is: Humerus.

Figure 10: COMCTS generated sample from PMC-VQA. Question : What is the location of the mass observed in the CT scan? Choices: A) : Hepatic lobe B) : Pulmonary artery C) : Renal artery D) : Renal Gt-answer (in $\mathcal{D}_{VQA}$ ) : Renal Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a grayscale axial CT scan of the abdomen. It displays the liver, stomach, spleen, kidneys, and surrounding structures. A large, well-defined, low-density mass is visible in the right-side abdomen, adjacent to the right kidney. The mass appears to be significantly larger than the kidney itself. There are also air-filled bowel loops visible within the abdomen. The vertebral column is visible centrally. The question asks for the location of a mass identified in a CT scan. We need to identify the mass in the image and determine which anatomical structure it is associated with. The choices provide different anatomical locations, and we must select the one that best corresponds to the observed mass. The final answer is: D) Renal.

Figure 11: COMCTS generated sample from PMC-VQA. Question : Is liver present? Gt-answer (in $\mathcal{D}_{VQA}$ ) : Yes Reasoning-answer generated by COMCTS (in $\mathcal{D}_{reas}$ ): The image is a microscopic view of tissue stained with hematoxylin and eosin (H&E). Numerous dark purple, small, round structures (likely nuclei) are visible scattered throughout a pinkish-purple background. There are several cells with visible nuclei, and a larger, pale area potentially representing a vessel or space. The overall appearance suggests an inflammatory process with numerous immune cells. There is a granular texture and darker staining areas scattered throughout the field. The question asks if liver tissue is present. Analyzing the histological features is crucial. Identification of hepatocytes (liver cells) and the liver’s typical lobular architecture would suggest the presence of liver tissue. The presence of inflammation within a structural framework resembling liver tissue is a strong indicator. The final answer is: Yes.

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.515\pm 0.013$	$0.644\pm 0.013$	$0.665\pm 0.007$	$\mathbf{0.713\pm 0.010}$
PMC-VQA	$0.663\pm 0.008$	$0.666\pm 0.011$	$0.712\pm 0.007$	$\mathbf{0.738\pm 0.005}$
RAD-VQA	$0.597\pm 0.015$	$0.571\pm 0.015$	$0.681\pm 0.006$	$\mathbf{0.695\pm 0.011}$
SLAKE-VQA	$0.786\pm 0.007$	$0.745\pm 0.007$	$0.791\pm 0.007$	$\mathbf{0.841\pm 0.006}$
VQA-MED	$0.705\pm 0.009$	$0.713\pm 0.006$	$0.740\pm 0.006$	$\mathbf{0.758\pm 0.006}$
OmniMed-VQA	$0.552\pm 0.007$	$0.459\pm 0.016$	$0.560\pm 0.017$	$\mathbf{0.581\pm 0.011}$

Appendix

Table 10: Sentence BLEU-1

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.164\pm 0.004$	$0.204\pm 0.004$	$0.211\pm 0.002$	$\mathbf{0.226\pm 0.003}$
PMC-VQA	$0.530\pm 0.008$	$0.532\pm 0.011$	$0.573\pm 0.008$	$\mathbf{0.594\pm 0.006}$
RAD-VQA	$0.208\pm 0.005$	$0.197\pm 0.006$	$0.240\pm 0.005$	$\mathbf{0.242\pm 0.009}$
SLAKE-VQA	$0.275\pm 0.003$	$0.262\pm 0.005$	$0.288\pm 0.005$	$\mathbf{0.315\pm 0.006}$
VQA-MED	$0.378\pm 0.007$	$0.375\pm 0.007$	$0.388\pm 0.003$	$\mathbf{0.394\pm 0.004}$
OmniMed-VQA	$0.294\pm 0.004$	$0.219\pm 0.015$	$0.288\pm 0.011$	$\mathbf{0.304\pm 0.010}$

Appendix

Table 11: Sentence BLEU-2

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.111\pm 0.003$	$0.139\pm 0.003$	$0.144\pm 0.001$	$\mathbf{0.154\pm 0.002}$
PMC-VQA	$0.401\pm 0.008$	$0.400\pm 0.010$	$0.432\pm 0.006$	$\mathbf{0.448\pm 0.005}$
RAD-VQA	$0.138\pm 0.003$	$0.132\pm 0.005$	$0.158\pm 0.004$	$\mathbf{0.161\pm 0.006}$
SLAKE-VQA	$0.181\pm 0.001$	$0.174\pm 0.003$	$0.191\pm 0.005$	$\mathbf{0.207\pm 0.005}$
VQA-MED	$0.314\pm 0.006$	$0.308\pm 0.007$	$0.321\pm 0.003$	$\mathbf{0.326\pm 0.004}$
OmniMed-VQA	$0.213\pm 0.004$	$0.156\pm 0.010$	$0.213\pm 0.009$	$\mathbf{0.227\pm 0.008}$

Appendix

Table 12: Sentence BLEU-3

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.404\pm 0.013$	$0.498\pm 0.014$	$0.517\pm 0.006$	$\mathbf{0.544\pm 0.014}$
PMC-VQA	$0.688\pm 0.009$	$0.686\pm 0.014$	$0.723\pm 0.011$	$\mathbf{0.737\pm 0.008}$
RAD-VQA	$0.499\pm 0.017$	$0.479\pm 0.021$	$0.560\pm 0.011$	$\mathbf{0.571\pm 0.019}$
SLAKE-VQA	$0.696\pm 0.005$	$0.679\pm 0.021$	$0.731\pm 0.015$	$\mathbf{0.782\pm 0.018}$
VQA-MED	$0.677\pm 0.009$	$0.666\pm 0.008$	$0.687\pm 0.021$	$\mathbf{0.707\pm 0.006}$
OmniMed-VQA	$0.487\pm 0.011$	$0.416\pm 0.024$	$0.536\pm 0.020$	$\mathbf{0.549\pm 0.027}$

Appendix

Table 13: Corpus BLEU-1

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.017\pm 0.010$	$0.016\pm 0.009$	$\mathbf{0.023\pm 0.013}$	$0.013\pm 0.000$
PMC-VQA	$0.617\pm 0.010$	$0.618\pm 0.017$	$0.656\pm 0.012$	$\mathbf{0.669\pm 0.010}$
RAD-VQA	$0.111\pm 0.019$	$0.112\pm 0.020$	$\mathbf{0.136\pm 0.020}$	$0.136\pm 0.035$
SLAKE-VQA	$0.165\pm 0.011$	$0.194\pm 0.023$	$0.235\pm 0.021$	$\mathbf{0.270\pm 0.018}$
VQA-MED	$0.490\pm 0.009$	$0.471\pm 0.011$	$0.487\pm 0.019$	$\mathbf{0.500\pm 0.007}$
OmniMed-VQA	$0.326\pm 0.010$	$0.253\pm 0.027$	$0.367\pm 0.021$	$\mathbf{0.379\pm 0.025}$

Appendix

Table 14: Corpus BLEU-2

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.004\pm 0.002$	$0.004\pm 0.001$	$\mathbf{0.005\pm 0.002}$	$0.004\pm 0.000$
PMC-VQA	$0.549\pm 0.012$	$0.550\pm 0.019$	$0.589\pm 0.012$	$\mathbf{0.597\pm 0.011}$
RAD-VQA	$0.024\pm 0.011$	$0.035\pm 0.019$	$0.036\pm 0.019$	$\mathbf{0.042\pm 0.024}$
SLAKE-VQA	$0.036\pm 0.015$	$0.077\pm 0.024$	$0.104\pm 0.017$	$\mathbf{0.111\pm 0.020}$
VQA-MED	$0.387\pm 0.008$	$0.366\pm 0.011$	$0.382\pm 0.018$	$\mathbf{0.391\pm 0.009}$
OmniMed-VQA	$0.227\pm 0.015$	$0.168\pm 0.024$	$0.266\pm 0.022$	$\mathbf{0.277\pm 0.024}$

Appendix

Table 15: Corpus BLEU-3

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.002\pm 0.001$	$0.002\pm 0.001$	$\mathbf{0.003\pm 0.001}$	$0.002\pm 0.000$
PMC-VQA	$0.493\pm 0.013$	$0.494\pm 0.021$	$0.533\pm 0.012$	$\mathbf{0.536\pm 0.013}$
RAD-VQA	$0.009\pm 0.003$	$0.016\pm 0.012$	$0.014\pm 0.010$	$\mathbf{0.017\pm 0.015}$
SLAKE-VQA	$0.013\pm 0.007$	$0.040\pm 0.016$	$\mathbf{0.051\pm 0.014}$	$0.051\pm 0.015$
VQA-MED	$0.273\pm 0.007$	$0.253\pm 0.010$	$0.266\pm 0.015$	$\mathbf{0.274\pm 0.010}$
OmniMed-VQA	$0.140\pm 0.021$	$0.106\pm 0.021$	$0.180\pm 0.024$	$\mathbf{0.188\pm 0.025}$

Appendix

Table 16: Corpus BLEU-4

Dataset	SFT	$\mathcal{R}_{form}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$
Path-VQA	$0.022\pm 0.011$	$0.080\pm 0.023$	$0.088\pm 0.018$	$\mathbf{0.136\pm 0.017}$
PMC-VQA	$0.545\pm 0.009$	$0.558\pm 0.015$	$0.613\pm 0.012$	$\mathbf{0.641\pm 0.006}$
RAD-VQA	$0.248\pm 0.029$	$0.243\pm 0.030$	$0.334\pm 0.025$	$\mathbf{0.352\pm 0.025}$
SLAKE-VQA	$0.704\pm 0.013$	$0.658\pm 0.009$	$0.720\pm 0.016$	$\mathbf{0.772\pm 0.013}$
VQA-MED	$0.671\pm 0.010$	$0.667\pm 0.008$	$0.698\pm 0.008$	$\mathbf{0.714\pm 0.009}$
OmniMed-VQA	$\mathbf{0.541\pm 0.014}$	$0.387\pm 0.019$	$0.496\pm 0.018$	$0.532\pm 0.017$

Appendix

Table 17: Open Accuracy

Panel of Figure 12 — Figure 12: Needleman-Wunsch (NW) alignment for reasoning trajectories. Unlike DTW, NW enforces one-to-one matching and explicitly models gaps with affine penalties: gap opening ( $g_{o}$ ) and gap extension ( $g_{e}$ ). This allows handling cases where generated reasoning has missing or extra steps compared to ground truth.

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.696\pm 0.004$	$\mathbf{0.710\pm 0.009}$	$0.674\pm 0.010$
PMC-VQA	$0.630\pm 0.007$	$\mathbf{0.641\pm 0.006}$	$0.632\pm 0.007$
RAD-VQA	$0.666\pm 0.012$	$\mathbf{0.676\pm 0.012}$	$0.667\pm 0.016$
SLAKE-VQA	$0.767\pm 0.018$	$\mathbf{0.817\pm 0.006}$	$0.798\pm 0.005$
VQA-MED	$\mathbf{0.740\pm 0.009}$	$0.733\pm 0.007$	$0.734\pm 0.006$
OmniMed-VQA	$0.532\pm 0.009$	$\mathbf{0.557\pm 0.016}$	$0.554\pm 0.013$

Appendix

Table 18: Accuracy

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.134\pm 0.022$	$\mathbf{0.136\pm 0.017}$	$0.091\pm 0.017$
PMC-VQA	$0.630\pm 0.007$	$\mathbf{0.641\pm 0.006}$	$0.632\pm 0.007$
RAD-VQA	$\mathbf{0.352\pm 0.024}$	$\mathbf{0.352\pm 0.025}$	$0.331\pm 0.037$
SLAKE-VQA	$0.699\pm 0.027$	$\mathbf{0.772\pm 0.013}$	$0.741\pm 0.012$
VQA-MED	$\mathbf{0.723\pm 0.008}$	$0.714\pm 0.009$	$0.713\pm 0.007$
OmniMed-VQA	$0.509\pm 0.013$	$\mathbf{0.532\pm 0.017}$	$0.530\pm 0.013$

Appendix

Table 19: Open Accuracy

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.698\pm 0.005$	$\mathbf{0.713\pm 0.010}$	$0.676\pm 0.011$
PMC-VQA	$0.730\pm 0.005$	$\mathbf{0.738\pm 0.005}$	$0.725\pm 0.004$
RAD-VQA	$0.685\pm 0.010$	$\mathbf{0.695\pm 0.011}$	$0.686\pm 0.016$
SLAKE-VQA	$0.794\pm 0.016$	$\mathbf{0.841\pm 0.006}$	$0.819\pm 0.004$
VQA-MED	$\mathbf{0.764\pm 0.010}$	$0.758\pm 0.006$	$0.758\pm 0.005$
OmniMed-VQA	$0.571\pm 0.009$	$0.581\pm 0.011$	$\mathbf{0.593\pm 0.013}$

Appendix

Table 20: Sentence BLEU-1

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.222\pm 0.002$	$\mathbf{0.226\pm 0.003}$	$0.214\pm 0.004$
PMC-VQA	$0.588\pm 0.006$	$\mathbf{0.594\pm 0.006}$	$0.587\pm 0.005$
RAD-VQA	$0.239\pm 0.008$	$\mathbf{0.242\pm 0.009}$	$0.241\pm 0.009$
SLAKE-VQA	$0.282\pm 0.006$	$\mathbf{0.315\pm 0.006}$	$0.300\pm 0.006$
VQA-MED	$\mathbf{0.398\pm 0.004}$	$0.394\pm 0.004$	$0.395\pm 0.006$
OmniMed-VQA	$0.300\pm 0.007$	$0.304\pm 0.010$	$\mathbf{0.313\pm 0.012}$

Appendix

Table 21: Sentence BLEU-2

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.151\pm 0.001$	$\mathbf{0.154\pm 0.002}$	$0.146\pm 0.003$
PMC-VQA	$0.439\pm 0.004$	$\mathbf{0.448\pm 0.005}$	$0.439\pm 0.003$
RAD-VQA	$0.158\pm 0.004$	$\mathbf{0.161\pm 0.006}$	$0.159\pm 0.004$
SLAKE-VQA	$0.188\pm 0.004$	$\mathbf{0.207\pm 0.005}$	$0.197\pm 0.004$
VQA-MED	$\mathbf{0.329\pm 0.003}$	$0.326\pm 0.004$	$0.327\pm 0.006$
OmniMed-VQA	$0.220\pm 0.005$	$0.227\pm 0.008$	$\mathbf{0.231\pm 0.008}$

Appendix

Table 22: Sentence BLEU-3

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.125\pm 0.001$	$\mathbf{0.127\pm 0.002}$	$0.120\pm 0.002$
PMC-VQA	$0.359\pm 0.004$	$\mathbf{0.370\pm 0.005}$	$0.360\pm 0.003$
RAD-VQA	$0.129\pm 0.003$	$\mathbf{0.131\pm 0.004}$	$0.129\pm 0.003$
SLAKE-VQA	$0.153\pm 0.003$	$\mathbf{0.165\pm 0.003}$	$0.159\pm 0.003$
VQA-MED	$\mathbf{0.261\pm 0.002}$	$0.259\pm 0.004$	$0.259\pm 0.004$
OmniMed-VQA	$0.163\pm 0.004$	$0.168\pm 0.007$	$\mathbf{0.172\pm 0.005}$

Appendix

Table 23: Sentence BLEU-4

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.541\pm 0.006$	$\mathbf{0.544\pm 0.014}$	$0.522\pm 0.010$
PMC-VQA	$0.728\pm 0.005$	$\mathbf{0.737\pm 0.008}$	$0.728\pm 0.008$
RAD-VQA	$0.565\pm 0.015$	$0.571\pm 0.019$	$\mathbf{0.574\pm 0.016}$
SLAKE-VQA	$0.721\pm 0.018$	$\mathbf{0.782\pm 0.018}$	$0.748\pm 0.015$
VQA-MED	$\mathbf{0.710\pm 0.007}$	$0.707\pm 0.006$	$0.706\pm 0.010$
OmniMed-VQA	$0.544\pm 0.022$	$0.549\pm 0.027$	$\mathbf{0.553\pm 0.054}$

Appendix

Table 24: Corpus BLEU-1

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$\mathbf{0.027\pm 0.014}$	$0.013\pm 0.000$	$0.016\pm 0.009$
PMC-VQA	$0.656\pm 0.007$	$\mathbf{0.669\pm 0.010}$	$0.659\pm 0.011$
RAD-VQA	$0.135\pm 0.022$	$0.136\pm 0.035$	$\mathbf{0.141\pm 0.024}$
SLAKE-VQA	$0.207\pm 0.013$	$\mathbf{0.270\pm 0.018}$	$0.236\pm 0.022$
VQA-MED	$\mathbf{0.503\pm 0.006}$	$0.500\pm 0.007$	$0.500\pm 0.011$
OmniMed-VQA	$0.375\pm 0.021$	$0.379\pm 0.025$	$\mathbf{0.386\pm 0.045}$

Appendix

Table 25: Corpus BLEU-2

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$\mathbf{0.006\pm 0.002}$	$0.004\pm 0.000$	$0.004\pm 0.001$
PMC-VQA	$0.582\pm 0.009$	$\mathbf{0.597\pm 0.011}$	$0.588\pm 0.012$
RAD-VQA	$0.039\pm 0.018$	$\mathbf{0.042\pm 0.024}$	$0.041\pm 0.019$
SLAKE-VQA	$0.083\pm 0.010$	$\mathbf{0.111\pm 0.020}$	$0.093\pm 0.023$
VQA-MED	$\mathbf{0.394\pm 0.005}$	$0.391\pm 0.009$	$0.392\pm 0.009$
OmniMed-VQA	$0.269\pm 0.020$	$0.277\pm 0.024$	$\mathbf{0.284\pm 0.035}$

Appendix

Table 26: Corpus BLEU-3

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$\mathbf{0.003\pm 0.001}$	$0.002\pm 0.000$	$0.002\pm 0.001$
PMC-VQA	$0.520\pm 0.011$	$\mathbf{0.536\pm 0.013}$	$0.527\pm 0.013$
RAD-VQA	$0.016\pm 0.012$	$\mathbf{0.017\pm 0.015}$	$0.016\pm 0.012$
SLAKE-VQA	$0.040\pm 0.008$	$\mathbf{0.051\pm 0.015}$	$0.046\pm 0.017$
VQA-MED	$\mathbf{0.274\pm 0.006}$	$0.274\pm 0.010$	$0.274\pm 0.006$
OmniMed-VQA	$0.178\pm 0.021$	$0.188\pm 0.025$	$\mathbf{0.198\pm 0.027}$

Appendix

Table 27: Corpus BLEU-4

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.351\pm 0.002$	$\mathbf{0.359\pm 0.005}$	$0.340\pm 0.005$
PMC-VQA	$0.639\pm 0.004$	$\mathbf{0.643\pm 0.005}$	$0.636\pm 0.005$
RAD-VQA	$0.363\pm 0.008$	$\mathbf{0.367\pm 0.009}$	$0.365\pm 0.010$
SLAKE-VQA	$0.436\pm 0.010$	$\mathbf{0.464\pm 0.006}$	$0.452\pm 0.007$
VQA-MED	$\mathbf{0.497\pm 0.005}$	$0.493\pm 0.004$	$0.493\pm 0.005$
OmniMed-VQA	$0.426\pm 0.013$	$0.443\pm 0.011$	$\mathbf{0.444\pm 0.012}$

Appendix

Table 28: METEOR

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.861\pm 0.002$	$\mathbf{0.864\pm 0.004}$	$0.852\pm 0.004$
PMC-VQA	$0.845\pm 0.004$	$\mathbf{0.846\pm 0.003}$	$0.841\pm 0.002$
RAD-VQA	$0.862\pm 0.005$	$\mathbf{0.864\pm 0.004}$	$0.859\pm 0.007$
SLAKE-VQA	$0.913\pm 0.004$	$\mathbf{0.929\pm 0.003}$	$0.920\pm 0.004$
VQA-MED	$\mathbf{0.877\pm 0.003}$	$0.874\pm 0.003$	$0.874\pm 0.004$
OmniMed-VQA	$0.801\pm 0.007$	$0.803\pm 0.005$	$\mathbf{0.804\pm 0.008}$

Appendix

Table 29: COMET

Dataset	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NWW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{DTW}$	$\mathcal{R}_{form}+\mathcal{R}_{exact}+\mathcal{R}_{NW}+\mathcal{R}_{DTW}$
Path-VQA	$0.700\pm 0.004$	$\mathbf{0.716\pm 0.011}$	$0.679\pm 0.011$
PMC-VQA	$0.748\pm 0.005$	$\mathbf{0.752\pm 0.006}$	$0.743\pm 0.005$
RAD-VQA	$0.726\pm 0.013$	$\mathbf{0.734\pm 0.007}$	$0.725\pm 0.019$
SLAKE-VQA	$0.811\pm 0.017$	$\mathbf{0.851\pm 0.008}$	$0.833\pm 0.007$
VQA-MED	$\mathbf{0.768\pm 0.010}$	$0.760\pm 0.006$	$0.761\pm 0.006$
OmniMed-VQA	$0.657\pm 0.016$	$0.674\pm 0.011$	$\mathbf{0.674\pm 0.013}$

Appendix

Table 30: ROUGE-L F1

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering
Mar 14, 2026Lin Fan, Yafei Ou, Zhipeng Deng +8Medical VQA Visual CoT
Southwest Jiaotong University · RIKEN · Hokkaido University +4
OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
Jun 10, 2026Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6Medical VQA Medical VLMs
York University · Vector Institute · University of British Columbia +5

Apr 10, 2026J/K move · Enter open · S save

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan+3

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Open paper AskPDF arXiv

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.

Abstract

Medical VQA

Explore similar work

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models