Recent advances in medical large language models have explored Test-Time Reinforcement Learning (TTRL) to enhance reasoning. However, standard TTRL often relies on majority voting (MV) as a heuristic supervision signal, which can be unreliable in complex medical scenarios where the most frequent reasoning path is not necessarily the clinically correct one. In this work, we propose a novel and unified training paradigm that integrates medical process reward models with TTRL to bridge the gap between test-time scaling (TTS) and parametric model optimization. Specifically, we advance the TTRL framework by replacing the conventional MV with a fine-grained, expert-aligned supervision paradigm using Med-RPM. This integration ensures that reinforcement learning is guided by medical correctness rather than mere consensus, effectively distilling search-based intelligence into the model's parametric memory. Extensive evaluations on four different benchmarks have demonstrated that our developed method consistently and significantly outperforms current TTRL and standalone PRM selection. Our findings establish that transitioning from stochastic heuristics to structured, step-wise rewards is essential for developing reliable and scalable medical AI systems
Figures & tables
Figure 1: Overview of the MAPLE framework. Given a test question, the policy model generates M candidate reasoning chains via multi-sample generation. Each chain is evaluated by a PRM, which assigns step-level scores si,j to every intermediate reasoning step. The scored candidates are aggregated through a Self-Consistency with Reward Model reranking (SC+RM) mechanism to produce a pseudo label y^ . Per-sampel rewards R(yi,y^) are then computed by the candidate and pseudo label, and the resulting reward signals are used to update the policy model online via policy optimization.
Model
Size
MedQA
MedMCQA
DDXPlus
MMLU-Med
Llama3.1 (pass@1)
8B
62.79
57.44
66.41
78.33
Llama3.1 (MV)
8B
68.25
60.00
74.00
83.33
Gemma 2 (pass@1)
9B
52.90
50.97
63.80
79.63
Gemma 2 (MV)
9B
54.76
52.00
69.00
79.63
R1-Distill-Llama(pass@1)
8B
47.10
42.63
49.22
70.25
R1-Distill-Llama (MV)
8B
61.90
44.00
80.00
78.70
Table 1: Performance comparison on medical QA benchmarks. Results are reported on MedQA (USMLE-style), MedMCQA, DDXPlus, and MMLU-Med.
Figure 2: Performance comparison across four medical QA benchmarks. Built on Llama3.1 (8B) as backbone, MAPLE consistently outperforms its base model and surpasses QwQ (32B) on DDXPlus and MMLU-Med despite being 4× smaller in model size. Italic green values indicate MAPLE’s absolute accuracy gain over the Llama3.1 (MV) backbone.
Model
Size
MedQA
MedMCQA
DDXPlus
MMLU-Med
TTRL
8B
69.84
62.00
73.00
85.71
TTRL-Med S3 (MV)
8B
69.05
59.87
74.99
80.95
TTRL-Med S3 (BoM)
8B
69.05
61.00
78.00
82.41
TTRL-Med S3 (SC+RM)
8B
69.05
60.00
75.00
82.41
MAPLE (MV)
8B
68.25
63.00
78.00
90.48
MAPLE (BoM)
8B
73.02
66.00
83.00
85.19
Table 2: Ablation study on model components across four benchmarks. TTRL: base policy optimized without any PRM guidance; MAPLE-Med S3 : TTRL augmented with Med S3 as the process reward model.
Figure 3: Test-time scaling curves on MedMCQA under : MV, BoM, and SC+RM. MAPLE (red) consistently outperforms the Llama3.1-8B backbone (blue) across different rollout budgets M , with the shaded region highlighting the performance gap.
Medical reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing PRM benchmarks largely target mathematics rather than patient-specific clinical errors. We introduce MedPRMBench, a step-level medical reasoning benchmark with 14 clinical error types, including five new and two clinically re-scoped types, and rule-defined severity grades. Built from seven medical QA sources via structured error construction and review by 86 physicians from Grade-A tertiary hospitals, it provides 6{,}500 evaluation questions, 13{,}000 reasoning chains, and 113{,}910 step labels, plus a leakage-clean 6{,}870-question training split. The best of 20 prompted critics reaches only 75.4% PRMScore, and critics differ sharply in error-type profiles and operating points. A Qwen3-8B PRM trained on the split reaches 86.6% PRMScore and 89.0% strict first-error localization; in matched ablations, physician review raises error recall by 6.55 points and cuts clean-chain false alarms from 5.6% to 2.1%. The training signal replicates across model families; without further training, the PRM transfers to ProcessBench and PRMBench, exceeding six of nine math-trained PRMs on each.
Test-time reinforcement learning adapts a model on its own unlabeled test set using majority-vote pseudo-labels and has shown strong results in mathematics. We show that this recipe collapses on medical multiple-choice QA: accuracy stagnates while output diversity rapidly declines. Through a controlled experiment that keeps the questions, model, and optimizer fixed while changing only the answer space, we trace this failure to answer-space structure rather than domain difficulty. In small answer spaces, incorrect rollouts often collide on the same wrong pseudo-label and reinforce it; in large answer spaces, they disperse and receive little reward. This diagnosis motivates PROSE, Process Reward Guided Self-Training, which rewards reasoning quality instead of answer agreement. PROSE scores each reasoning step with a medical process reward model, assigns the trajectory reward as the minimum score across steps, and enforces answer-format constraints. Without labels, PROSE substantially improves a general Llama model, surpassing purpose-built medical models and matching much larger systems. Because the process signal is internalized into the policy, the adapted model requires no reward model at inference and transfers its gains to unseen datasets. We further show that the minimum aggregation is essential: mean aggregation can be exploited, saturating the proxy reward while degrading accuracy.
Kailong Fan, Anqi Pu, Yichen Wu +7
1Harvard Medical School/MGH · 3Harvard University · 2Zhejiang University
We introduce MediX-R1, an open-ended Reinforcement Learning (RL) framework for medical multimodal large language models (MLLMs) that enables clinically grounded, free-form answers beyond multiple-choice formats. MediX-R1 fine-tunes a baseline vision-language backbone with Group Based RL and a composite reward tailored for medical reasoning: an LLM-based accuracy reward that judges semantic correctness with a strict YES/NO decision, a medical embedding-based semantic reward to capture paraphrases and terminology variants, and lightweight format and modality rewards that enforce interpretable reasoning and modality recognition. This multi-signal design provides stable, informative feedback for open-ended outputs where traditional verifiable or MCQ-only rewards fall short. To measure progress, we propose a unified evaluation framework for both text-only and image+text tasks that uses a Reference-based LLM-as-judge in place of brittle string-overlap metrics, capturing semantic correctness, reasoning, and contextual alignment. Despite using only ∼51K instruction examples, MediX-R1 achieves excellent results across standard medical LLM (text-only) and VLM (image + text) benchmarks, outperforming strong open-source baselines and delivering particularly large gains on open-ended clinical tasks. Our results demonstrate that open-ended RL with comprehensive reward signals and LLM-based evaluation is a practical path toward reliable medical reasoning in multimodal models. Our trained models, curated datasets and source code are available at https://medix.cvmbzuai.com
Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed +5
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College