CasePlay: Self-Play Reinforcement Learning from Case Reports for Medical Reasoning

Hao Wang; Zihan Wang; Kai Wu; Jian Yao; Yiwen Ye; Yongcan Luo; Dapeng Wu; Jiale Chen

cs.LGOct 18, 2026

CasePlay: Self-Play Reinforcement Learning from Case Reports for Medical Reasoning

Authors: Hao Wang, Zihan Wang, Kai Wu, Jian Yao, Yiwen Ye, Yongcan Luo, Dapeng Wu, Jiale Chen,

Organizations: City University of Hong Kong · Northeastern University · Tongji University · The Hong Kong Polytechnic University ·

Abstract

Reinforcement learning (RL) is a promising way to improve the clinical reasoning abilities of large language models (LLMs), but current pipelines often rely on expert-labeled questions or costly synthetic-data generation. Meanwhile, the medical literature offers a wealth of high-quality unlabeled documents, including clinical case reports that trace the full clinical course from initial presentation to final treatment decisions. We study whether document-grounded self-play can convert these reports into effective RL training signals. We identify two challenges that limit existing self-play methods in this setting: knowledge-point fixation}, where the proposer repeatedly asks about a narrow set of salient clues in a source report, and difficulty-quality mismatch, where appropriately difficult questions may still be poorly formed or shallow, limiting their value for reasoner training. To address these challenges, we propose CasePlay, a self-play framework with two key components: (a) a Knowledge-Conditioned Proposer, which anchors question generation to case-specific knowledge points so that questions draw on a broader range of evidence from the source report; and (b) a Rubric-Judge Reasoner, which augments answer-accuracy feedback with rubric-based quality feedback on clue integration, reasoning depth, and option quality, guiding the proposer toward reasoning-intensive questions that better support reasoner training. Extensive experiments across seven medically related benchmarks demonstrate that CasePlay outperforms existing self-play baselines and improves the SFT warm-start model by 3.1 points on average, highlighted by a 6.3-point gain on LiveClin. This approach offers an effective path toward continually improving medical reasoning ability through self-play RL.

Figures & tables

Panel of Figure 2 — Figure 2: (a) Cumulative source-document coverage of generated questions over training. (b) MDS [ 26 ] projection of generated-question embeddings. (c) Rubric-based quality-score distribution among intermediate-difficulty questions with reasoner accuracy of 0.5.

Method	Avg. Acc.	Resp. Len.
Warm-start Model	63.1	3794.5
+ RL (Random)	63.5	3025.4
+ RL (Filtered)	64.3	4932.8

Table 1: Impact of question quality.

Base Models
Method	Expert Medical Reasoning				Clin. Case	Science QA
	MedXpert	MB-OP4	MB-OP5	MedQA	LiveClin	MMLU.	GPQA	Avg.
Qwen3-8B-Base [ 28 ]	12.3	41.2	35.3	50.8	48.6	56.4	42.2	41.0
+ SFT (Warm-Start)	22.7	72.9	68.1	83.6	66.0	75.3	53.4	63.1
Medical Reasoning Models
HuatuoGPT-o1-8B [ 2 ]	15.4	52.6	45.2	72.6	55.4	63.1	50.4	50.7

Table 2: Main results on seven medically related benchmarks.

Panel of Figure 3 — Figure 3: (a) Different knowledge points from the same report yield substantially different difficulty. (b) Knowledge-conditioned generation produces a more balanced distribution of question types.

Figure 4: Training curve of rubric score.

Rubric Dimension	Pearson Corr. $\uparrow$
Clue Integration	0.783
Reasoning Depth	0.752
Option Quality	0.627
Average Score	0.705

Table 3: Agreement with DeepSeek-V3.2.

Panel of Figure 5 — Figure 5: Sensitivity to (a) the number of knowledge points $M$ and (b) the rubric weight $\lambda$ .

	MedXpert	MB-OP4	MB-OP5	MedQA	LiveClin	MMLU.	GPQA	Avg.
Method	Expert Medical Reasoning				Clin. Case	Science QA
Qwen3-4B-Base
SFT (Warm-Start)	16.8	65.6	58.4	72.4	57.3	66.5	43.5	54.4
R-Zero [ 14 ]	16.3	65.5	59.4	73.8	59.4	68.4	44.2	55.3
SPICE [ 17 ]	17.6	66.4	60.9	77.6	61.7	70.7	45.5	57.2
DARC [ 18 ]	18.4	67.1	60.8	78.4	62.6	71.4	47.8	58.1

Table 4: Performance of CasePlay across different base models.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Case study of knowledge-point extraction from a clinical case report.

Panel of Figure 7 — Figure 7: Examples of knowledge-conditioned question generation from the same clinical case report.

SFT Warm-Start
Epochs	5	Batch size	128	Max response len.	8,192
Learning rate	$1{\times}10^{-5}$	LR schedule	Cosine	LR warmup ratio	0.1
Self-Play RL
Optimization		Rollout		Others
Learning rate	$1{\times}10^{-6}$	Proposer samples $N_{p}$	4	Invalid penalty $r^{\mathrm{format}}$	-0.1
Train batch	128	Answer samples $N_{\mathrm{acc}}$	4	Learnability weight	1.0

Appendix

Table 5: Key hyperparameters.

Method	Expert Medical Reasoning				Clin. Case	Science QA
Method	MedXpert	MB-OP4	MB-OP5	MedQA	LiveClin	MMLU.	GPQA	Avg.
R-Zero	19.7 $\pm$ 2.3	72.9 $\pm$ 0.5	68.6 $\pm$ 2.5	83.5 $\pm$ 0.4	66.0 $\pm$ 1.5	73.3 $\pm$ 0.4	52.9 $\pm$ 2.6	62.4
95% CI	[16.8, 21.6]	[71.8, 74.2]	[65.1, 71.4]	[82.7, 84.1]	[64.8, 69.1]	[72.6, 74.1]	[50.0, 56.5]	—
SPICE	21.8 $\pm$ 2.2	74.2 $\pm$ 1.0	70.0 $\pm$ 2.0	83.2 $\pm$ 0.4	67.1 $\pm$ 1.2	75.6 $\pm$ 0.2	54.2 $\pm$ 1.6	63.7
95% CI	[17.1, 24.3]	[72.8, 75.2]	[66.3, 73.1]	[82.4, 83.9]	[65.1, 69.9]	[74.9, 75.9]	[51.8, 57.1]	—
DARC	23.5 $\pm$ 1.0	75.7 $\pm$ 1.0	69.8 $\pm$ 1.4	84.8 $\pm$ 0.5	70.5 $\pm$ 0.7	75.3 $\pm$ 0.1	55.2 $\pm$ 1.3	65.0

Appendix

Table 6: Full results for the main self-play comparisons.

Method	MedXpert	MB-OP4	MB-OP5	MedQA	LiveClin	MMLU.	GPQA	Avg.
Method	Expert Medical Reasoning				Clin. Case	Science QA
Static Synthetic Data + GRPO	22.6	75.0	69.5	84.1	66.9	75.6	55.3	64.1
CasePlay	25.1	76.9	70.1	85.4	72.3	75.9	57.6	66.2

Appendix

Table 7: Comparison with GRPO on a fixed synthetic question pool.

Method	MedXpert	MB-OP4	MB-OP5	MedQA	LiveClin	MMLU.	GPQA	Avg.
Method	Expert Medical Reasoning				Clin. Case	Science QA
Fleming-R1-7B	19.1	57.8	54.1	75.1	60.2	69.3	48.7	54.9
$+$ CasePlay	20.2	61.6	57.9	76.5	65.3	69.6	50.2	57.3

Appendix

Table 8: Continuing medical RL post-training with CasePlay.

Panel of Figure 8 — Figure 8: Training curves of rubric scores.

Strategy	DCR $\uparrow$	Avg. Acc. $\uparrow$
Random	0.40	65.7
Acc.-priority	0.32	64.1
Round-robin	0.42	66.2

Appendix

Table 9: Impact of knowledge-point selection strategies.

Self-Play RL

Self-play reinforcement learning trains agents through repeated interaction with copies, opponents, or co-evolving task generators.

See all 83 papers

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
Sep 21, 2026Kalash Shah, Kunal Singh, Snehan J +1RL for Language Model Reasoning Rubric-Based RL
Fractal AI Research
Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach
Apr 13, 2026Haolin Li, Shuyang Jiang, Ruipeng Zhang +3Medical LLMs RL for Language Model Reasoning
College of Computer Science and Artificial Intelligence, Fudan University · Shanghai AI Laboratory · Department of Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine +3
MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation
Oct 6, 2026Xilin Dang, Weilin Ruan, Xue Yang +4Self-Evolving Agents Medical LLMs
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Hematology, Zhongshan Hospital, Fudan University · School of Future Technology, South China University of Technology +1

Sep 21, 2026J/K move · Enter open · S save

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards

Kalash Shah, Kunal Singh, Snehan J, Shreyas Singh

Fractal AI Research

RL for Language Model Reasoning Rubric-Based RL Clinical Reasoning Medical Reasoning Medical LLMs

Open paper AskPDF arXiv

Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the convergent, evidence-driven task of inferring a patient's condition from clinical data to produce a diagnosis, and clinical healthcare reasoning: the broader, navigational judgment required to communicate, plan, and adapt across multi-turn clinical interactions where a single correct answer may not exist. Recent benchmarks such as HealthBench and MedXpertQA reveal persistent weaknesses in both areas, exposing failures in complex diagnostic scenarios and limitations in contextual, patient-centered dialogue. We introduce a sequential training framework that targets these facets using synthetic data and rubric-based reinforcement learning. First, we improve diagnostic reasoning using MedBullets-derived questions with rule- and rubric-guided Reinforcement Learning (RL). We then shift to clinical reasoning by generating 5.3k synthetic multi-turn scenarios, each paired with multi-dimensional rubrics to comprehensively assess the response. This approach yields over 10% improvement on MedXpertQA, and our 30B model achieves 50.1% accuracy on HealthBench-Hard, surpassing proprietary baselines including GPT-5 (thinking). Our results show that targeted synthetic datasets and rubric-based training can systematically improve both diagnostic and interactive clinical reasoning in medical LLMs.

Abstract

Self-Play RL

Explore similar work

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards

Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach

MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation