From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training

Wang Jingxin

cs.CVSep 25, 2026

From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training

Authors: Wang Jingxin

Organizations: Institute of Neuroscience, CAS

Abstract

Medical vision-language model (VLM) post-training is commonly evaluated through answer accuracy. We examine how changes in accuracy and training objectives relate to image-conditioned decisions in a controlled Qwen2.5-VL-3B study on PMC-VQA. We compare supervised fine-tuning (SFT) with low-rank adaptation (LoRA) restricted to the language model, expanded multimodal adaptation scopes, standard answer-only Group Relative Policy Optimization (GRPO), and a counterfactual evidence objective. On 2,000 clean-test questions, language model LoRA SFT changes correct-image accuracy by +1.10 percentage points (95% paired bootstrap CI:-0.85 to +3.05), while visual-benefit events decrease by 2.40 points and image sensitivity decreases by 5.60 points. Paired records reveal 155 acquired and 203 lost visual-benefit events. Broader adaptation yields lower correct-image accuracy than language-model LoRA SFT. Standard GRPO produces mixed-reward groups and parameter updates, with an uncertain clean test accuracy change. A generation audit reveals that canonical option scores can follow a different token path from generated answers. With scores taken along the greedy generation path, the evidence target improves on the training set; its gains over standard GRPO remain inconsistent on validation data at matched training doses. Sample-level analyses trace how evidence scores, decision margins, and generated answers change during post-training. This empirical and measurement audit identifies gaps between optimization activity, target acquisition, and useful held-out visual behavior.

Figures & tables

Figure 1: Evaluation under Correct, NoImage, and Shuffled image conditions. The three-bit patterns record answer correctness in C/N/S order. Visual-benefit and visual-harm events decompose the Correct-minus-Shuffled accuracy difference; the full patterns also describe success under the NoImage condition.

Model	Correct	NoImage	Shuffled	C-S	VBR	VHR	VDS/100	LSR/111
M0	51.70	36.45	31.25	20.45	29.05	8.60	17.55	16.95
M1-L	52.80	36.20	34.55	18.25	26.65	8.40	17.60	20.15
M1-ML	51.25	36.40	34.15	17.10	25.35	8.25	16.20	20.45
M1-VML	51.15	36.85	33.85	17.30	25.00	7.70	16.50	20.70
M2-500	52.60	36.70	34.50	18.10	26.60	8.50	17.35	20.25

Table 1: Performance on the frozen PMC-VQA clean-test cohort of 2,000 QA. Values are percentages, except C-S, which is in percentage points. VDS and LSR correspond to patterns 100 and 111. Paired confidence intervals for model differences are reported in the text.

Figure 2: SFT performance under the frozen clean-test protocol. Paired-bootstrap intervals show decreases in VBR and IS and an increase in pattern111, while the task and C-S changes remain uncertain. NoImage accuracy is nearly unchanged.

Figure 3: Paired pattern transitions and bidirectional event flows from M0 to M1-L. Matrix entries are QA counts and sum to 2,000; color uses log(1 + count). The 155 VBR gains and 203 losses recover the aggregate -2.40 -point change.

Panel of Figure 4 — Figure 4: Examples from three paired-transition categories. Labels follow the dataset annotations. The first two questions share a source image but concern different panels and use different replacement images. The third shows a gain in correct-image-only success.

Figure 5: Adaptation-scope comparison using VALID-selected checkpoints on a common clean-test cohort. The three configurations vary which components are trained: language attention adapters, the multimodal merger, and vision adapters. Their effects reflect the combined changes in trainable capacity, optimization, and regularization.

Figure 6: Standard GRPO training diagnostics and clean-test performance. The training signal is the fraction of prompt groups with nonzero reward variance. A separate checkpoint audit verifies adapter updates. The clean-test comparison shows a small accuracy change with a paired confidence interval spanning zero.

Figure 7: Generation-path measurement and decision margins. Panels A-B show the natural token branch and trace-consistency checks. Panels C-D compare M4-500 with M2-500 on two VALID examples: evidence and decision margins can move in different directions or share a change in the correct-image gold score. The evidence-up/decision-down example is drawn from three cases meeting both strict criteria.

Figure 8: Training-target acquisition and held-out transfer at matched doses. Panel A reports TRAIN score gains relative to M1-L. Panels B-D compare M4 with standard GRPO at equal prompt counts. The diagnostic cohorts contain 256 TRAIN pattern100 questions, 188 VALID pattern100 questions, and 112 VALID pattern110 questions; behavioral comparisons use the diagnostic VALID cohort of 645. Score effects are in raw branch-logit units and behavioral effects are in percentage points.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Three-bit pattern distributions across the SFT parameter scopes. Rates are computed over 2,000 questions from 1,440 images. Figure 3 complements these aggregate distributions with paired question-level transitions.

Figure S2: Examples of answer changes under expanded adaptation and standard GRPO: loss of the correct panel answer, loss of the Correct-image foot-scar answer, and a Correct-image gain on an abdominal-CT question for which NoImage and Shuffled were already correct. Each case is the lowest-index member of its specified selection pool.

Panel of Figure 12 — Figure S3: (Continued.) In the abdominal-CT case, GRPO brings the Correct-image answer into agreement with the already correct NoImage and Shuffled answers.

Figure S4: Answer-token acquisition and stopping behavior in the SLAKE native-answer experiment. Separate scores for semantic answer tokens and EOS reveal improved answer-token likelihood alongside deteriorating termination behavior.

Experiment	Evaluation cohort	Measurement	Result
SLAKE base model (R0)	VALID: 1,053 QA	Native-answer exact match	C 55.56%, N 30.58%, S 32.67%
SLAKE native-answer SFT (R1)	TRAIN generation; 5,971 teacher-forced scores per model	Answer-token likelihood and termination	Answer likelihood improves while repetitive generation and termination failure emerge
VQA-Med-2019 base model (R0)	VALID: 2,000 QA / 500 images	Native-answer exact match	C 13.90%, N 5.45%, S 10.35%

Appendix

Table A1: Native-answer experiments on SLAKE and VQA-Med-2019 ( Ben Abacha et al., 2019 ) . VQA-Med-2019 baseline C-N is +8.45 points (CI [7.00, 9.95]) and C-S is +3.55 points ([2.05, 5.05]). These results use each dataset’s native-answer protocol. The SLAKE shuffled mapping contains 2 self-maps among 96 source-image groups, reducing the number of effective image replacements.

Experimental question	Finding	Supporting analysis
How much does the base model benefit from the correct image?	M0 C-S is +20.45 points	PMC clean-test image interventions
How does SFT change image-conditioned success?	Substantial question-level gains and losses coexist with uncertain aggregate task gain	Paired transitions and confidence intervals, Figures 2-4
Does expanding trainable scope improve utility?	Broader adaptation lowers correct-image accuracy and has mixed effects on visual metrics	SFT scope comparison, Figure 5
Do GRPO signals translate into selected-endpoint gains?	Selected clean-test accuracy changes by -0.20 points despite active optimization	Signal audit and clean-test evaluation, Figure 6
Does the canonical score follow the generated answer?	Generation-branch traces expose scoring-path discrepancies and tie-handling differences	Multimodal generation audit, Figure 7
Does the aligned evidence objective acquire its training target?	The measured evidence target improves on TRAIN	Training-cohort diagnostic, Figure 8A

Appendix

Table A2: Experimental questions, principal findings, and supporting analyses.

Explore similar work

When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
Mar 1, 2026Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh +4Medical VLMs RL for VLMs
University of Toronto, Canada · Vector Institute, Canada · KITE Research Institute, University Health Network, Canada +1
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM Evaluation Counterfactual Evaluation
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1
Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
Sep 21, 2026Santi Ram Tiwari, Nihal Naik, Devbrat Pandey +1Vision-Language Grounding VLM Reasoning
KGraph AI Solutions Pvt. Ltd. Bangalore, India – 560016

Mar 1, 2026J/K move · Enter open · S save

When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains

Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh, Natasha Sharan+3

University of Toronto, Canada · Vector Institute, Canada · KITE Research Institute, University Health Network, Canada+1

Medical VLMs RL for VLMs Medical VQA Supervised Fine-Tuning

Open paper AskPDF arXiv

Reinforcement learning (RL) is increasingly used to post-train medical Vision-Language Models (VLMs), yet it remains unclear whether RL improves medical visual reasoning or mainly sharpens behaviors already induced by supervised fine-tuning (SFT). We present a controlled study that disentangles these effects along three axes: vision, SFT, and RL. Using MedMNIST as a multi-modality testbed, we probe visual perception by benchmarking VLM vision towers against vision-only baselines, quantify reasoning support and sampling efficiency via Accuracy@1 versus Pass@K, and evaluate when RL closes the support gap and how gains transfer across modalities. We find that RL is most effective when the model already has non-trivial support (high Pass@K): it primarily sharpens the output distribution, improving Acc@1 and sampling efficiency, while SFT expands support and makes RL effective. Based on these findings, we propose a boundary-aware recipe and instantiate it by RL post-training an OctoMed-initialized model on a small, balanced subset of PMC multiple-choice VQA, achieving strong average performance across six medical VQA benchmarks.

Abstract

Explore similar work

When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning