Large language models demonstrate strong medical question-answering capabilities, yet reasoning performance remains uneven across languages. Process reward models (PRMs) provide step-level feedback for improving reasoning, but a fixed PRM may become increasingly misaligned with the trajectory distribution of an evolving policy during reinforcement learning, thereby reducing the reliability of its supervision. We propose a policy--reward alternating training framework that jointly adapts a medical PRM and optimizes a multilingual reasoning policy. Following supervised initialization, the current policy generates multilingual trajectories to update the PRM, which then provides process-level rewards for the next policy update. Medical step supervision maintains the PRM's medical foundation, while process, outcome, and language-related rewards guide policy learning. We develop multilingual medical PRMs (PAIR-PRM) and reasoning models (PAIR-MED) at 8B and 9B scales across 12 languages. On the medical subsets of Global-MMLU and MMLU-ProX, our models improve average accuracy by 3.01--7.47% over their corresponding Qwen backbones. On CUREMED-BENCH, PAIR-MED-9B improves the joint language consistency and logical accuracy metric by 12.88% and 7.45% over baselines for in-domain and out-of-domain languages. Further analyses examine PRM adaptation, multilingual reasoning quality, and the contributions of individual training components.
Figures & tables
Figure 1: Traditional Frozen-PRM Training vs. Alternating Policy–PRM Training in PAIR-Med .
Figure 2: Overview of PAIR-Med: Multilingual Policy and PRM Initialization (Stage One), followed by Alternating PRM Adaptation and Policy Optimization (Stage Two).
Global-MMLU
MMLU-ProX
Model
High
Medium
Low
Avg.
High
Medium
Low
Avg.
Qwen3-8B Yang et al. (2025a)
61.95
62.27
32.19
52.11
43.81
39.32
18.81
34.35
Apollo2-7B Zheng et al. (2025b)
62.38
51.70
31.67
49.47
38.86
28.57
11.75
27.25
BioMistral-7B Labrak et al. (2024)
41.19
39.38
28.27
36.43
18.54
10.24
13.32
14.72
CURE-Med-7B Onyame et al. (2026)
62.62
54.32
33.18
50.73
33.82
27.17
14.96
25.87
HuatuoGPT-3-8B Chen et al. (2026a)
70.23
63.19
34.05
56.41
46.81
37.84
19.39
35.43
Table 1: Accuracy (%) on Global-MMLU-medical and MMLU-ProX-health. High, Medium, and Low denote language resource groups. Within each model group, the highest and second-highest values per column are bolded and underlined, respectively. PAIR-Med accuracy is averaged over N=10 samples, while w/ PAIR-Prm selects the highest-scoring response among N=10 candidates. Per-language results are provided in the Appendix. B .
In-Domain
Out-of-Domain
Model
es
fr
ja
ko
th
sw
yo
Avg.
vi
hi
tr
am
bn
ha
Avg.
Qwen3-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
HuatuoGPT-3-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen3.5-9B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CURE-Med-7B
11.43
8.57
4.76
4.00
2.04
0.00
0.00
4.40
6.67
0.75
1.79
0.00
0.00
0.00
1.53
PAIR-Med -8B
22.86
25.71
15.24
13.33
9.18
2.86
0.00
12.74
14.29
6.72
8.04
0.00
0.71
0.00
4.96
Table 2: Zero-shot prompting results on CUREMED-BENCH: LC&LA. Reasoning baselines use thinking mode.
Figure 3: Accuracy changes relative to each base model across high-, medium-, and low-resource languages.
Figure 4: Stage-One and Stage-Two PRM best-of- N answer selection on Global-MMLU. The vertical zero line denotes self-consistency accuracy.
Table 5: Generation and optimization settings for the alternating-training runs. Each PRM and policy update uses one epoch.
SFT Cold Start
Multilingual PRM Initialization
Backbone
Training hours
GPU-hours
Training hours
GPU-hours
Qwen3-8B
0.73
5.86
13.39
107.10
Qwen3.5-9B
2.38
19.02
45.88
367.07
Appendix
Table 6: Stage-One training cost on eight GPUs.
Backbone
Alternating Training
Qwen3-8B
7.52
Qwen3.5-9B
25.22
Appendix
Table 7: GPU-hours for PRM and policy updates over all rounds of Stage Two training.
High
Medium
Low
Model
en
es
fr
zh
ja
ko
id
th
sw
wo
yo
zu
Avg.
Qwen3-8B
68.04
64.78
60.79
62.65
53.48
54.46
65.54
66.80
37.54
34.60
25.90
30.71
52.11
Apollo2-7B
69.90
57.81
57.34
66.84
60.01
52.62
54.41
48.07
38.67
29.32
29.36
29.31
49.47
BioMistral-7B
55.08
43.78
45.91
37.67
23.52
31.56
37.24
49.33
30.36
29.79
28.84
24.09
36.43
CURE-Med-7B
69.23
60.53
61.46
64.25
57.61
53.28
57.94
51.73
39.13
30.72
32.82
30.04
50.73
HuatuoGPT-3-8B
75.81
70.09
69.30
70.23
65.71
57.34
68.62
63.60
38.40
35.53
30.96
31.30
56.41
Appendix
Table 8: Accuracy (%) for each language on Global-MMLU. Within each of the two model groups, the highest value in each column is bold, and the second highest is underlined.
High
Medium
Low
Model
en
es
fr
zh
ja
ko
id
th
sw
wo
yo
zu
Avg.
Qwen3-8B
48.61
45.71
42.50
41.92
40.32
33.53
48.18
36.24
21.68
19.35
16.45
17.75
34.35
Apollo2-7B
47.89
37.41
37.99
38.86
32.17
32.89
31.44
21.39
13.10
12.22
11.93
9.75
27.25
BioMistral-7B
25.90
22.12
22.27
13.39
9.02
8.15
15.43
7.13
13.10
14.41
12.22
13.53
14.72
CURE-Med-7B
40.17
31.44
32.02
37.55
27.94
26.49
31.44
23.58
14.56
16.30
14.84
14.12
25.87
HuatuoGPT-3-8B
56.33
48.18
48.18
42.06
39.30
33.92
45.12
34.49
20.81
18.05
18.77
19.94
35.43
Appendix
Table 9: Accuracy (%) for each language on MMLU-ProX. Within each of the two model groups, the highest value in each column is bold, and the second highest is underlined.
In-Domain
Out-of-Domain
Model
es
fr
ja
ko
th
sw
yo
Avg.
vi
hi
tr
am
bn
ha
Avg.
Logical Accuracy
CURE-Med-7B
80.95
72.53
67.35
50.27
62.14
42.83
35.74
58.83
67.71
47.16
32.61
23.96
51.48
49.94
45.48
Qwen3-8B
72.98
74.15
73.79
64.43
59.79
25.65
5.30
53.73
66.05
67.97
61.16
26.14
56.60
6.87
47.47
HuatuoGPT-3-8B
35.37
34.27
31.50
28.17
31.03
14.56
5.60
25.79
34.65
36.38
20.80
15.91
30.97
5.24
23.99
PAIR-Med -8B
67.33
71.47
64.81
50.43
53.45
22.30
15.11
49.27
61.89
37.73
41.19
5.47
27.87
3.20
29.56
Appendix
Table 10: Detailed CUREMED-BENCH results (%) under zero-shot prompting.
High-resource
Medium-resource
Low-resource
Training method
en
es
fr
zh
ja
ko
id
th
sw
wo
yo
zu
Avg.
Qwen3-8B
Base model
68.04
64.78
60.79
62.65
53.48
54.46
65.54
66.80
37.54
34.60
25.90
30.71
52.11
Stage-one SFT
75.34
66.71
66.91
68.82
57.47
56.40
53.26
56.06
31.68
23.54
23.47
29.15
50.73
Native SFT
72.82
66.82
66.71
68.49
61.42
57.32
65.75
60.87
39.35
40.24
34.86
35.87
55.88
Native GRPO
74.68
70.23
69.10
71.09
64.65
60.46
69.69
63.60
36.74
27.78
10.89
23.36
53.52
Appendix
Table 11: Accuracy (%) for each training method and language on Global-MMLU. Bold marks the highest value within each backbone block; ties share the same marking.
High-resource
Medium-resource
Low-resource
Training method
en
es
fr
zh
ja
ko
id
th
sw
wo
yo
zu
Avg.
Qwen3-8B
Base model
48.61
45.71
42.50
41.92
40.32
33.53
48.18
36.24
21.68
19.35
16.45
17.75
34.35
Stage-one SFT
57.73
44.91
44.89
46.03
38.05
34.11
34.59
32.69
17.94
10.36
11.11
12.37
32.07
Native SFT
54.10
45.40
45.74
47.52
39.57
34.64
43.25
34.78
20.61
21.92
16.89
18.05
35.21
Native GRPO
56.62
47.88
47.16
49.78
40.32
37.26
47.30
36.53
17.75
13.10
4.30
10.77
34.06
Appendix
Table 12: Accuracy (%) for each training method and language on MMLU-ProX. Avg. is the unweighted mean over all 12 languages. Bold marks the highest value within each backbone block; ties share the same marking.
Inference strategy
High
Medium
Low
Avg.
Native GRPO-8B
Policy alone
69.95
64.58
24.69
53.52
Self-consistency
73.54
68.33
38.45
60.54
Stage-One PRM
75.55
71.20
39.48
62.44
Stage-Two PRM
75.27
71.49
41.08
62.93
PAIR-Med -8B
Appendix
Table 13: Different inference accuracy (%) on Global-MMLU by resource group. Bold marks the highest value among inference strategies for the same policy.
PAIR-Med -8B
PAIR-Med -9B
Language
GPT
PRM-8B
ρ
GPT
PRM-9B
ρ
en
0.855
0.742
0.244
0.871
0.736
0.462
es
0.727
0.695
0.433
0.795
0.735
0.411
fr
0.707
0.668
0.511
0.777
0.684
0.485
zh
0.780
0.718
0.412
0.824
0.716
0.340
ja
0.669
0.703
0.334
0.738
0.723
0.315
Appendix
Table 14: Reasoning quality and PAIR-Prm agreement for each language.
High-resource
Medium-
Low
Inference strategy
en
es
fr
zh
ja
ko
id
th
sw
wo
yo
zu
Avg.
Native GRPO-8B
Policy alone
74.68
70.23
69.10
71.09
64.65
60.46
69.69
63.60
36.74
27.78
10.89
23.36
53.52
Self-consistency
78.47
73.55
73.22
73.88
68.57
63.98
73.09
67.93
42.52
38.41
34.08
38.79
60.54
Stage-One PRM
79.66
74.42
75.38
77.60
70.69
67.71
75.70
70.20
45.58
39.61
33.55
39.18
62.44
Stage-Two PRM
79.26
74.42
74.55
77.47
70.63
68.11
76.03
70.33
46.77
42.35
35.02
40.18
62.93
Appendix
Table 15: Different inference accuracy (%) for each language on Global-MMLU using 8B policies.
High
Medium
Low
Inference strategy
en
es
fr
zh
ja
ko
id
th
sw
wo
yo
zu
Avg.
Native GRPO-9B
Policy alone
79.33
77.40
76.27
75.28
70.56
69.90
76.30
61.86
58.27
49.36
50.09
53.00
66.47
Self-consistency
82.25
78.73
78.67
78.00
73.88
71.82
78.70
67.06
62.65
55.24
55.68
58.74
70.12
Stage-One PRM
83.12
79.73
80.73
79.47
76.27
74.48
79.71
70.93
65.78
57.98
57.07
60.21
72.12
Stage-Two PRM
82.45
80.53
80.53
78.87
76.41
74.61
79.77
70.47
66.51
59.58
55.61
62.21
72.30
Appendix
Table 16: Differnt inference accuracy (%) for each language on Global-MMLU using 9B policies.
Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.
Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
Medical reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing PRM benchmarks largely target mathematics rather than patient-specific clinical errors. We introduce MedPRMBench, a step-level medical reasoning benchmark with 14 clinical error types, including five new and two clinically re-scoped types, and rule-defined severity grades. Built from seven medical QA sources via structured error construction and review by 86 physicians from Grade-A tertiary hospitals, it provides 6{,}500 evaluation questions, 13{,}000 reasoning chains, and 113{,}910 step labels, plus a leakage-clean 6{,}870-question training split. The best of 20 prompted critics reaches only 75.4% PRMScore, and critics differ sharply in error-type profiles and operating points. A Qwen3-8B PRM trained on the split reaches 86.6% PRMScore and 89.0% strict first-error localization; in matched ablations, physician review raises error recall by 6.55 points and cuts clean-chain false alarms from 5.6% to 2.1%. The training signal replicates across model families; without further training, the PRM transfers to ProcessBench and PRMBench, exceeding six of nine math-trained PRMs on each.
Recent advances in medical large language models have explored Test-Time Reinforcement Learning (TTRL) to enhance reasoning. However, standard TTRL often relies on majority voting (MV) as a heuristic supervision signal, which can be unreliable in complex medical scenarios where the most frequent reasoning path is not necessarily the clinically correct one. In this work, we propose a novel and unified training paradigm that integrates medical process reward models with TTRL to bridge the gap between test-time scaling (TTS) and parametric model optimization. Specifically, we advance the TTRL framework by replacing the conventional MV with a fine-grained, expert-aligned supervision paradigm using Med-RPM. This integration ensures that reinforcement learning is guided by medical correctness rather than mere consensus, effectively distilling search-based intelligence into the model's parametric memory. Extensive evaluations on four different benchmarks have demonstrated that our developed method consistently and significantly outperforms current TTRL and standalone PRM selection. Our findings establish that transitioning from stochastic heuristics to structured, step-wise rewards is essential for developing reliable and scalable medical AI systems
Kailong Fan, Anqi Pu, Yichen Wu +7
Zhejiang University · Harvard Medical School · Harvard University