GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

Wanqi Dong; Wanyue Dong; Jingze Ge; Mehul Motani

cs.CVOct 25, 2026

GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

Authors: Wanqi Dong, Wanyue Dong, Jingze Ge, Mehul Motani

Organizations: National University of Singapore · Shandong Provincial Daizhuang Hospital

Abstract

Adapting general-purpose vision-language models (VLMs) to medical visual question answering (VQA) is valuable for helping clinicians read images and make diagnoses, but annotated data are scarce and provide only short answers, without reasoning annotations. Supervised fine-tuning (SFT) fits these labels but overfits to them and amplifies their biases, eroding out-of-domain (OOD) generalization. Reinforcement learning (RL) better preserves generalization but never uses the gold answer as a training target: GRPO-style algorithms learn from the contrast between correct and incorrect answers within a group of sampled responses, so all-correct or all-wrong groups give no correctness signal. Existing SFT-RL hybrids either apply SFT and RL in separate stages or switch between them per question, or need reasoning traces from experts or stronger models, which medical VQA data lack. We propose GRASP, which combines RL and SFT in every update using only short answers: RL lets the model explore its own reasoning, SFT teaches the gold answer after that reasoning to supply the direct supervision RL lacks, and their balance adapts to how well the model answers each question. Tuning only about 1% of the parameters with LoRA, GRASP outperforms all compared training objectives both in domain and on OOD open-ended questions across Qwen3-VL-8B, InternVL3-8B and Gemma 3-4B. On an OOD benchmark spanning eight imaging modalities, GRASP-trained InternVL3-8B and Qwen3-VL-8B beat all 10 compared general and medical VLMs in open-ended accuracy, including models trained on up to 2,600 times more data or with over 250 times more trainable parameters. We release MedTerse, a dataset of 12.9K short-answer questions, and our code at https://github.com/sea-comet/GRASP-MedVQA.

Figures & tables

Figure 1: Grasp : simultaneous RL and SFT training on the same rollouts. For each question with only a short gold answer, the policy samples $G{=}8$ reason-then-answer rollouts whose group accuracy is $p$ . Top (RL): group-relative advantages plus an adaptive extreme-group correction $c(p)$ for all-wrong and all-correct groups drive the GSPO-EGC loss. Bottom (SFT): each rollout’s own reasoning is kept as a loss-free prefix, followed by the teacher-forced gold answer, weighted by $\lambda(p)$ . Policy: Qwen3-VL-8B with frozen base weights; both losses update only the LoRA adapters in the ViT and the LLM, and no reasoning annotations are used.

Group	RL signal	$c(p)$	$\lambda(p)$
all-wrong	none	$-\kappa\mu_{t}s_{t}/G$	$\lambda_{\max}$
mixed	right vs. wrong	$0$	$\lambda_{\max}(1{-}p)^{\gamma}$
all-correct	none	$+\kappa(1{-}\mu_{t})s_{t}/G$	$0$

Table 1: Division of labor between the two branches. RL signal: whether

A^{\text{rel}}

separates right from wrong.

			In domain (local test splits, 6,600 questions)							OOD (OmniMedVQA)
										open-ended
Objective	Type	Ep.	SLAKE	VQA-RAD	PathVQA	Val	Open	Y/N	All	CT	X-ray	MRI	Open	Y/N bal	All
Zero-shot	–	–	43.4	52.5	59.6	36.6	15.2	68.5	51.9	1.0	24.2	30.1	15.9	63.6	25.6
SFT	single	2	71.3	54.2	66.8	47.0	32.9	76.0	62.5	19.8	34.5	31.6	22.9	47.1	21.2
GSPO	single	2	59.8	55.7	66.8	44.7	26.3	75.8	60.4	18.7	37.8	31.4	22.8	60.8	28.0
GSPO-EGC	single	2	64.3	56.4	67.8	47.3	29.5	77.1	62.2	18.6	42.9	31.6	24.2	61.7	29.3

Table 2: Training objectives on Qwen3-VL-8B (EM, %). All rows share the backbone, data, LoRA, rewards and evaluation protocol. Open/Y/N/All: pooled results; CT-MRI: OOD open-ended subsets; Y/N bal : balanced OOD yes/no accuracy. Best in bold; seed variation in App. Table 6 .

Figure 2: Data. (a) MedTerse: 12,893 short-answer training questions from four public sources. (b) 11,575 OOD test questions over eight OmniMedVQA modalities. Inner rings: share of yes/no questions.

			In domain				OOD (OmniMedVQA)				Kvasir		VQA-Med
			Open		All		Open		All		All		All
Model	Tuned	Data	EM	LLM	EM	LLM	EM	LLM	EM	LLM	EM	LLM	EM	LLM
Qwen3-VL-8B	–	–	15.2	25.4	51.9	55.1	15.9	26.9	25.6	30.9	35.6	42.6	4.8	5.1
InternVL3-8B	–	–	16.6	27.4	49.7	53.3	19.2	30.1	22.2	31.5	38.7	40.7	5.1	5.7
MedGemma-4B	4.3B	33.6M	20.9	25.1	53.5	54.8	17.2	24.1	25.7	31.6	20.7	21.3	6.0	6.2
MedGemma-27B	${\approx}$ 27B	– ‡	26.0	33.2	55.3	57.5	15.3	21.2	20.9	25.9	33.4	34.3	8.2	8.8

Table 3: Comparison with general and medical VLMs , using one shared prompt (own-prompt results: App. Table 7 ). EM: exact match; LLM: accuracy when an LLM judges semantic equivalence to the gold answer (App. F ); Tuned: trainable parameters; Data: post-training examples. Grasp -Qwen/-InternVL: Qwen3-VL-8B/InternVL3-8B, 2 epochs. Best in bold. † Inferred from released weights; ‡ not reported separately.

	Components						In domain		Out of domain
Variant	Simult.	$\lambda(p)$	Own $z$	EGC	$r_{\text{syn}}$	$r_{\text{img}}$	Open	All	Open	All
Grasp (2 ep.)	✓	cont.	✓	adapt.	✓	✓	40.0	71.9	25.5	28.6
answer only (no own $z$ )	✓	cont.	✗	adapt.	✓	✗	35.9 ( $-$ 4.1)	69.7 ( $-$ 2.2)	23.1 ( $-$ 2.4)	24.0 ( $-$ 4.6)
exclusive switch (HPT)	✗	switch	✓	fixed	✓	✓	36.2 ( $-$ 3.8)	68.7 ( $-$ 3.2)	24.2 ( $-$ 1.3)	29.8 (+1.2)
constant $\lambda{=}0.2$	✓	const.	✓	fixed	✓	✓	36.6 ( $-$ 3.4)	68.6 ( $-$ 3.3)	24.6 ( $-$ 0.9)	27.0 ( $-$ 1.6)
w/o EGC	✓	cont.	✓	✗	✓	✓	37.3 ( $-$ 2.7)	69.2 ( $-$ 2.7)	24.5 ( $-$ 1.0)	27.6 ( $-$ 1.0)

Table 4: Ablations (EM, %; Qwen3-VL-8B, two epochs); parentheses give the difference to Grasp . Simult.: RL on every group and concurrent SFT wherever

p<1

; Own

z

: answer after the rollout’s own reasoning;

\lambda(p)

: continuous, constant or switch; EGC: adaptive (both extremes or

p{=}0

only), fixed (constant shift) or none. Seed s.d. of Grasp : 0.3 (in-domain) and 0.2 (OOD) open-ended points.

Figure 3: Training objectives on three backbones (EM, all questions; labels: gain of the simultaneous hybrid over zero-shot). OOD open-ended EM on InternVL3-8B: SFT 22.5, GSPO-EGC 25.0, SFT

\rightarrow

GSPO-EGC 24.9, Grasp 27.3.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone / adapters	Qwen3-VL-8B-Instruct; BF16 LoRA $r{=}32$ , $\alpha{=}32$ on attention and MLP layers; 102.7M parameters
Sequence / generation	4,096 / 256 tokens
Group / batch	8 rollouts; 12 questions per step
Sampling	temperature 1.15; fresh rollouts each step
Optimizer	8-bit AdamW
Learning rate	$2{\times}10^{-5}$ , cosine, 10% warm-up
Epochs / steps	2 / 2,148

Appendix

Table 5: Hyperparameters of the main run. Our implementation builds on TRL ( von Werra et al., 2020 ) and Unsloth ( Han et al., 2023 ) .

Run	L-all	L-open	O-open
Grasp ( $n{=}3$ )	71.35 $\,\pm\,$ 0.56	39.72 $\,\pm\,$ 0.27	25.34 $\,\pm\,$ 0.24
SFT ( $n{=}3$ )	62.89 $\,\pm\,$ 0.31	33.35 $\,\pm\,$ 0.46	22.91 $\,\pm\,$ 0.06
GSPO ( $n{=}3$ )	59.80 $\,\pm\,$ 0.67	25.36 $\,\pm\,$ 1.04	22.70 $\,\pm\,$ 0.40

Appendix

Table 6: Seed variation (EM, %; mean

\pm

standard deviation over three training seeds). L: in domain; O: OmniMedVQA; open: open-ended questions.

			In domain				OmniMedVQA				Kvasir
			Open		All		Open		All		All
Model	Tuned	Data	EM	LLM	EM	LLM	EM	LLM	EM	LLM	EM	LLM
Qwen3-VL-8B	–	–	26.8	33.1	55.7	57.6	20.4	28.5	22.3	29.2	44.9	45.7
InternVL3-8B	–	–	32.0	35.5	56.6	57.7	21.3	28.1	25.3	31.1	40.3	41.0
MedGemma-4B	4.3B	33.6M	26.8	35.6	54.0	56.8	15.8	26.9	23.2	32.6	19.1	19.2
MedGemma-27B	${\approx}$ 27B	–	26.5	34.8	53.9	56.5	18.2	25.2	17.2	23.2	27.3	30.9

Appendix

Table 7: Comparison with VLMs under each model’s own prompt. Direct-answer models are asked for a single word or phrase, while reasoning models and Grasp use their own reasoning format. Columns and Grasp rows as in Table 3 ;

{>}

: lower bound, as HuatuoGPT-V-34B also trains on medical text of unreported size.

Figure 4: OOD open-ended accuracy on the radiology modalities (OmniMedVQA, own prompt): (a) exact match and (b) LLM-judged. Grasp : Grasp -Qwen.

Figure 5: A curriculum without a schedule (main run, 60-step average). (a) Rollout groups by number of correct answers;

\bar{\lambda}

: supervision weight averaged over all groups in each epoch. (b) Training-adaptive extreme-group correction.

Figure 6: Sensitivity to

\lambda_{\max}

\gamma

and

\kappa

(open-ended EM, two epochs). Each non-default point is a single run that changes only the indicated parameter. The default point is the three-seed mean, and the shaded bands show

\pm 1

standard deviation.

Setting	L-all	L-open	O-open
default ( $\lambda_{\max}{=}0.4,\gamma{=}1,\kappa{=}1$ )	71.4	39.7	25.3
$\lambda_{\max}{=}0.2$	68.7	35.9	25.0
$\lambda_{\max}{=}0.6$	69.8	38.7	25.6
$\gamma{=}0.5$	69.4	37.5	25.0
$\gamma{=}2$	68.4	37.1	24.6
$\kappa{=}0.5$	69.0	37.3	24.6

Appendix

Table 8: Hyperparameter sweep (EM, %; two epochs). Each non-default row changes one value; the default row is the three-seed mean.

Panel of Figure 7 — Figure 7: Qualitative examples with verbatim greedy outputs. Marks follow exact match. Each example was selected because Grasp answers correctly and its reasoning identifies evidence relevant to the answer.

Sampling stratum	$n$	Consensus pairs	Judge agrees
Judge same	70	67 (95.7%)	66/67 (98.5%)
Near different	60	46 (76.7%)	36/46 (78.3%)
Random different	70	66 (94.3%)	66/66 (100.0%)
All	200	179 (89.5%)	168/179 (93.9%)

Appendix

Table 9: Blinded two-physician audit. A pair counts as a consensus pair if both physicians gave it the same label other than unsure . The strata were sampled on purpose, so the pooled agreement rate is not an estimate for the full population of pairs.

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
Mar 1, 2026Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh +4Medical VLMs RL for VLMs
University of Toronto, Canada · Vector Institute, Canada · KITE Research Institute, University Health Network, Canada +1
Improving Medical VQA through Trajectory-Aware Process Supervision
Apr 10, 2026Halil Ibrahim Gulluk, Olivier GevaertMedical VQA Medical VLMs
Electrical Engineering, Stanford University · Stanford Center For Biomedical Informatics Research
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Mar 1, 2026J/K move · Enter open · S save

When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains

Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh, Natasha Sharan+3

University of Toronto, Canada · Vector Institute, Canada · KITE Research Institute, University Health Network, Canada+1

Medical VLMs RL for VLMs Medical VQA Supervised Fine-Tuning

Open paper AskPDF arXiv

Reinforcement learning (RL) is increasingly used to post-train medical Vision-Language Models (VLMs), yet it remains unclear whether RL improves medical visual reasoning or mainly sharpens behaviors already induced by supervised fine-tuning (SFT). We present a controlled study that disentangles these effects along three axes: vision, SFT, and RL. Using MedMNIST as a multi-modality testbed, we probe visual perception by benchmarking VLM vision towers against vision-only baselines, quantify reasoning support and sampling efficiency via Accuracy@1 versus Pass@K, and evaluate when RL closes the support gap and how gains transfer across modalities. We find that RL is most effective when the model already has non-trivial support (high Pass@K): it primarily sharpens the output distribution, improving Acc@1 and sampling efficiency, while SFT expands support and makes RL effective. Based on these findings, we propose a boundary-aware recipe and instantiate it by RL post-training an OctoMed-initialized model on a small, balanced subset of PMC multiple-choice VQA, achieving strong average performance across six medical VQA benchmarks.

Abstract

Medical VQA

Explore similar work

When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains

Improving Medical VQA through Trajectory-Aware Process Supervision

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering