Reinforcement learning (RL) is a promising way to improve the clinical reasoning abilities of large language models (LLMs), but current pipelines often rely on expert-labeled questions or costly synthetic-data generation. Meanwhile, the medical literature offers a wealth of high-quality unlabeled documents, including clinical case reports that trace the full clinical course from initial presentation to final treatment decisions. We study whether document-grounded self-play can convert these reports into effective RL training signals. We identify two challenges that limit existing self-play methods in this setting: knowledge-point fixation}, where the proposer repeatedly asks about a narrow set of salient clues in a source report, and difficulty-quality mismatch, where appropriately difficult questions may still be poorly formed or shallow, limiting their value for reasoner training. To address these challenges, we propose CasePlay, a self-play framework with two key components: (a) a Knowledge-Conditioned Proposer, which anchors question generation to case-specific knowledge points so that questions draw on a broader range of evidence from the source report; and (b) a Rubric-Judge Reasoner, which augments answer-accuracy feedback with rubric-based quality feedback on clue integration, reasoning depth, and option quality, guiding the proposer toward reasoning-intensive questions that better support reasoner training. Extensive experiments across seven medically related benchmarks demonstrate that CasePlay outperforms existing self-play baselines and improves the SFT warm-start model by 3.1 points on average, highlighted by a 6.3-point gain on LiveClin. This approach offers an effective path toward continually improving medical reasoning ability through self-play RL.
Figures & tables
Figure 1: Overview of CasePlay.
Figure 2: (a) Cumulative source-document coverage of generated questions over training. (b) MDS [ 26 ] projection of generated-question embeddings. (c) Rubric-based quality-score distribution among intermediate-difficulty questions with reasoner accuracy of 0.5.
Method
Avg. Acc.
Resp. Len.
Warm-start Model
63.1
3794.5
+ RL (Random)
63.5
3025.4
+ RL (Filtered)
64.3
4932.8
Table 1: Impact of question quality.
Method
Expert Medical Reasoning
Clin. Case
Science QA
MedXpert
MB-OP4
MB-OP5
MedQA
LiveClin
MMLU.
GPQA
Avg.
Base Models
Qwen3-8B-Base [ 28 ]
12.3
41.2
35.3
50.8
48.6
56.4
42.2
41.0
+ SFT (Warm-Start)
22.7
72.9
68.1
83.6
66.0
75.3
53.4
63.1
Medical Reasoning Models
HuatuoGPT-o1-8B [ 2 ]
15.4
52.6
45.2
72.6
55.4
63.1
50.4
50.7
Table 2: Main results on seven medically related benchmarks.
Figure 3: (a) Different knowledge points from the same report yield substantially different difficulty. (b) Knowledge-conditioned generation produces a more balanced distribution of question types.
Figure 4: Training curve of rubric score.
Rubric Dimension
Pearson Corr. ↑
Clue Integration
0.783
Reasoning Depth
0.752
Option Quality
0.627
Average Score
0.705
Table 3: Agreement with DeepSeek-V3.2.
Figure 5: Sensitivity to (a) the number of knowledge points M and (b) the rubric weight λ .
Method
Expert Medical Reasoning
Clin. Case
Science QA
MedXpert
MB-OP4
MB-OP5
MedQA
LiveClin
MMLU.
GPQA
Avg.
Qwen3-4B-Base
SFT (Warm-Start)
16.8
65.6
58.4
72.4
57.3
66.5
43.5
54.4
R-Zero [ 14 ]
16.3
65.5
59.4
73.8
59.4
68.4
44.2
55.3
SPICE [ 17 ]
17.6
66.4
60.9
77.6
61.7
70.7
45.5
57.2
DARC [ 18 ]
18.4
67.1
60.8
78.4
62.6
71.4
47.8
58.1
Table 4: Performance of CasePlay across different base models.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Case study of knowledge-point extraction from a clinical case report.
Figure 7: Examples of knowledge-conditioned question generation from the same clinical case report.
SFT Warm-Start
Epochs
5
Batch size
128
Max response len.
8,192
Learning rate
1×10−5
LR schedule
Cosine
LR warmup ratio
0.1
Self-Play RL
Optimization
Rollout
Others
Learning rate
1×10−6
Proposer samples Np
4
Invalid penalty rformat
-0.1
Train batch
128
Answer samples Nacc
4
Learnability weight
1.0
Appendix
Table 5: Key hyperparameters.
Method
Expert Medical Reasoning
Clin. Case
Science QA
MedXpert
MB-OP4
MB-OP5
MedQA
LiveClin
MMLU.
GPQA
Avg.
R-Zero
19.7 ± 2.3
72.9 ± 0.5
68.6 ± 2.5
83.5 ± 0.4
66.0 ± 1.5
73.3 ± 0.4
52.9 ± 2.6
62.4
95% CI
[16.8, 21.6]
[71.8, 74.2]
[65.1, 71.4]
[82.7, 84.1]
[64.8, 69.1]
[72.6, 74.1]
[50.0, 56.5]
—
SPICE
21.8 ± 2.2
74.2 ± 1.0
70.0 ± 2.0
83.2 ± 0.4
67.1 ± 1.2
75.6 ± 0.2
54.2 ± 1.6
63.7
95% CI
[17.1, 24.3]
[72.8, 75.2]
[66.3, 73.1]
[82.4, 83.9]
[65.1, 69.9]
[74.9, 75.9]
[51.8, 57.1]
—
DARC
23.5 ± 1.0
75.7 ± 1.0
69.8 ± 1.4
84.8 ± 0.5
70.5 ± 0.7
75.3 ± 0.1
55.2 ± 1.3
65.0
Appendix
Table 6: Full results for the main self-play comparisons.
Method
Expert Medical Reasoning
Clin. Case
Science QA
MedXpert
MB-OP4
MB-OP5
MedQA
LiveClin
MMLU.
GPQA
Avg.
Static Synthetic Data + GRPO
22.6
75.0
69.5
84.1
66.9
75.6
55.3
64.1
CasePlay
25.1
76.9
70.1
85.4
72.3
75.9
57.6
66.2
Appendix
Table 7: Comparison with GRPO on a fixed synthetic question pool.
Method
Expert Medical Reasoning
Clin. Case
Science QA
MedXpert
MB-OP4
MB-OP5
MedQA
LiveClin
MMLU.
GPQA
Avg.
Fleming-R1-7B
19.1
57.8
54.1
75.1
60.2
69.3
48.7
54.9
+ CasePlay
20.2
61.6
57.9
76.5
65.3
69.6
50.2
57.3
Appendix
Table 8: Continuing medical RL post-training with CasePlay.
Figure 8: Training curves of rubric scores.
Strategy
DCR ↑
Avg. Acc. ↑
Random
0.40
65.7
Acc.-priority
0.32
64.1
Round-robin
0.42
66.2
Appendix
Table 9: Impact of knowledge-point selection strategies.
Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the convergent, evidence-driven task of inferring a patient's condition from clinical data to produce a diagnosis, and clinical healthcare reasoning: the broader, navigational judgment required to communicate, plan, and adapt across multi-turn clinical interactions where a single correct answer may not exist. Recent benchmarks such as HealthBench and MedXpertQA reveal persistent weaknesses in both areas, exposing failures in complex diagnostic scenarios and limitations in contextual, patient-centered dialogue. We introduce a sequential training framework that targets these facets using synthetic data and rubric-based reinforcement learning. First, we improve diagnostic reasoning using MedBullets-derived questions with rule- and rubric-guided Reinforcement Learning (RL). We then shift to clinical reasoning by generating 5.3k synthetic multi-turn scenarios, each paired with multi-dimensional rubrics to comprehensively assess the response. This approach yields over 10% improvement on MedXpertQA, and our 30B model achieves 50.1% accuracy on HealthBench-Hard, surpassing proprietary baselines including GPT-5 (thinking). Our results show that targeted synthetic datasets and rubric-based training can systematically improve both diagnostic and interactive clinical reasoning in medical LLMs.
While large language models hold promise for complex medical applications, their development is hindered by the scarcity of high-quality reasoning data. To address this issue, existing approaches typically distill chain-of-thought reasoning traces from large proprietary models via supervised fine-tuning, then conduct reinforcement learning (RL). These methods exhibit limited improvement on underrepresented domains like rare diseases while incurring substantial costs from generating complex reasoning chains. To efficiently enhance medical reasoning, we propose MedSSR, a Medical Knowledge-enhanced data Synthesis and Semi-supervised Reinforcement learning framework. Our framework first employs rare disease knowledge to synthesize distribution-controllable reasoning questions. We then utilize the policy model itself to generate high-quality pseudo-labels. This enables a two-stage, intrinsic-to-extrinsic training paradigm: self-supervised RL on the pseudo-labeled synthetic data, followed by supervised RL on the human-annotated real data. MedSSR scales model training efficiently without relying on costly trace distillation. Extensive experiments on Qwen and Llama demonstrate that our method outperforms existing methods across ten medical benchmarks, achieving up to +5.93% gain on rare-disease tasks. Our code is available at https://github.com/tdlhl/MedSSR.
Haolin Li, Shuyang Jiang, Ruipeng Zhang +3
College of Computer Science and Artificial Intelligence, Fudan University · Shanghai AI Laboratory · Department of Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine +3
Large language models (LLMs) have shown promise in medical question answering and clinical reasoning, yet their improvement remains constrained by static parametric knowledge and costly expert supervision. Self-evolving agents offer a promising alternative by enabling models to improve through iterative task generation and problem-solving. However, most existing self-evolving methods are designed for easily verifiable domains such as mathematics and coding, where solutions can be checked by exact answers or executable programs. Medical reasoning is fundamentally different: it is open-ended, knowledge-intensive, and often only partially verifiable. We present MedZERO, a self-evolving framework for open-ended medical reasoning. MedZERO couples an Examiner that generates frontier medical question-option pairs with a Reasoner that solves them through evidence-grounded multi-turn reasoning with external knowledge tools. To support reliable, continual improvement, MedZERO adopts controlled knowledge accumulation, which maintains temporary exploratory knowledge and curated persistent knowledge in reasoning. We evaluate MedZERO on five public medical reasoning benchmarks using 4B- and 8B-scale base models under open-ended evaluation. At both model scales, MedZERO achieves the highest average accuracy among the evaluated methods, with gains of up to 13.7 percentage points over the next-best self-evolving baseline. Code available at https://github.com/IcecreamArtist/MedZERO.
Xilin Dang, Weilin Ruan, Xue Yang +4
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Hematology, Zhongshan Hospital, Fudan University · School of Future Technology, South China University of Technology +1