Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach
Authors: Haolin Li, Shuyang Jiang, Ruipeng Zhang, Jiangchao Yao, Ya Zhang, Yanfeng Wang
Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Shanghai AI Laboratory · Department of Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine · CMIC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University · Institute of Artificial Intelligence for Medicine, Shanghai Jiao Tong University School of Medicine
While large language models hold promise for complex medical applications, their development is hindered by the scarcity of high-quality reasoning data. To address this issue, existing approaches typically distill chain-of-thought reasoning traces from large proprietary models via supervised fine-tuning, then conduct reinforcement learning (RL). These methods exhibit limited improvement on underrepresented domains like rare diseases while incurring substantial costs from generating complex reasoning chains. To efficiently enhance medical reasoning, we propose MedSSR, a Medical Knowledge-enhanced data Synthesis and Semi-supervised Reinforcement learning framework. Our framework first employs rare disease knowledge to synthesize distribution-controllable reasoning questions. We then utilize the policy model itself to generate high-quality pseudo-labels. This enables a two-stage, intrinsic-to-extrinsic training paradigm: self-supervised RL on the pseudo-labeled synthetic data, followed by supervised RL on the human-annotated real data. MedSSR scales model training efficiently without relying on costly trace distillation. Extensive experiments on Qwen and Llama demonstrate that our method outperforms existing methods across ten medical benchmarks, achieving up to +5.93% gain on rare-disease tasks. Our code is available at https://github.com/tdlhl/MedSSR.
Figures & tables
Figure 1: Distribution of question types in existing medical datasets. Only 22% are reasoning-intensive, and just 3% among them concern rare diseases.
Figure 2: Left : Comparison of existing methods for medical reasoning. “SS RL” is short for self-supervised RL. “Avg Token/Sample” denotes the average number of tokens consumed from an API model to generate one sample. Right : Performance improvement comparison based on Llama. Prior methods show less improvement in rare diseases than common tasks, while ours significantly breaks the rare disease improvement upper bound of +3% .
Figure 3: Overview of the proposed method. We first synthesize questions with tunable distribution, then train the policy model via a semi-supervised reinforcement learning mechanism. Rare disease knowledge injection is triggered when the random value ρ is less than the threshold α .
Model
Symptoms
Causes
Diagnosis
Others
RDs
Treatment
Affects
Avg
Large Language Models
Qwen3-8B
68.63
72.01
57.95
66.44
70.39
52.18
68.28
65.13
R1-Distill-Llama-8B
53.58
63.04
45.49
60.19
55.52
46.57
63.28
55.38
Medical Large Language Models
UltraMedical-3.1-8B
73.53
73.01
57.28
65.28
77.47
50.83
73.79
67.31
HuatuoGPT-o1-8B
74.90
78.47
61.64
70.10
79.53
52.89
78.68
70.89
Table 1: Performance on rare diseases. “RDs” is short for Related Disorders. The best results are in Bold . ↑ denotes the relative improvement of our method compared to the fully-supervised baseline. Ours is highlighted in gray.
Model
BioASQ
MedMCQA
MedQA
MedXpertqa
MMLU
PubMedQA
NEJM
Lancet
Medbullets
Avg
Large Language Models
Qwen3-8B
75.32
62.14
66.22
15.91
83.20
48.00
61.48
63.77
48.86
58.32
R1-Distill-Llama-8B
69.82
46.42
46.60
6.28
65.24
47.30
42.08
48.36
31.66
44.86
Medical Large Language Models
UltraMedical-3.1-8B
81.42
60.75
64.61
13.46
75.16
58.08
64.84
60.14
51.06
58.84
HuatuoGPT-o1-8B
80.97
64.77
77.79
16.68
79.57
58.35
65.92
62.68
54.30
62.34
Table 2: Performance on general medical tasks. All results are the average of four runs.
Stage
BioASQ
MedMCQA
MedQA
MedXpertqa
MMLU
PubMedQA
NEJM
Lancet
Medbullets
Avg
One-stage
82.16
67.10
78.06
17.77
86.04
54.70
69.49
68.32
62.26
65.10
Reverse-stage-I
81.91
64.25
74.98
17.40
83.88
54.15
68.74
67.66
55.11
63.12
Reverse-stage-II
82.83
67.48
78.67
17.27
85.56
53.30
69.36
69.66
60.55
64.96
Two-stage-I
82.51
65.44
75.61
17.09
86.25
55.05
66.91
68.87
55.92
63.74
Two-stage-II
86.00
67.79
81.14
20.66
86.64
54.70
71.97
70.94
63.47
67.03
Table 3: Ablation on training strategy. “One-stage” denotes mixed training without separating synthetic/real data.
Figure 4: Average performance on rare disease and general datasets across different rare disease ratios.
Figure 5: Impact of rare disease knowledge injection. The upper/lower part denotes rare/general tasks.
Figure 6: Rewards (left) and performance (right) curves of different labeling strategies across training steps.
Group
Incorrect
Harmful
Plausibility
Senior
1/200
1/200
4.86/5
Junior
0/200
0/200
4.74/5
Overall
0.50%
0.50%
4.80/5
Table 4: Physician evaluation of synthetic samples.
Figure 7: Comparison of our question-only synthesis and long reasoning chain synthesis (SFT).
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Group
# Sample
# Incorrect
# Harmful
# Rating
# SD
Senior
200
1
1
4.86
0.42
1
50
1
1
4.92
-
2
50
0
0
4.78
-
3
48
0
0
4.78
-
4
52
0
0
4.95
-
Junior
200
0
0
4.74
0.54
Appendix
Table 5: Physician evaluation results of synthetic data. “SD” denotes the standard deviation of the plausibility rating within the group.
Number
Incorrect
Harmful
Plausibility
200
0.50%
0.50%
4.80/5
300
1.00%
0.67%
4.88/5
500
4/500 (0.80%)
3/500 (0.60%)
4.85/5
Appendix
Table 6: Expanded Physician evaluation.
Dataset
# All
# Train
# Test
# Rare
BioASQ
1482
700
782
144
MedQA
11451
10178
1273
154
MedMCQA
184022
182822
1200
45
PubMedQA
1000
500
500
63
HeadQA
296
296
-
-
MedXpertqa
2450
-
2450
301
Appendix
Table 7: The statistics of the datasets.
Param
Qwen
Llama
Data Configuration
Global batch size
128
128
PPO mini batch size
128
128
PPO micro batch size
8
8
Max prompt length
1024
1024
Max response length
4096
4096
Appendix
Table 8: Detailed configuration for RL training.
Model
BioASQ
MedMCQA
MedQA
MedXpertqa
MMLU
PubMedQA
NEJM
Lancet
Medbullets
RareDis
Large Language Models
Qwen3-8B
1.90
0.23
1.33
0.36
1.08
1.19
0.68
0.54
1.31
0.41
R1-Distill-Llama-8B
2.08
0.88
1.31
0.64
1.61
1.63
1.59
0.88
2.51
0.40
Medical Large Language Models
UltraMedical-3.1-8B
1.13
0.97
1.15
0.70
1.31
1.15
0.35
1.64
1.06
0.64
HuatuoGPT-o1-8B
1.42
0.21
0.18
0.58
1.00
0.92
1.05
1.12
0.62
0.31
Appendix
Table 9: Detailed standard deviations across all models and benchmarks.
Figure 8: Rewards and performance curves of offline labeling with more than 1K steps. We report the average performance on test sets with error bars.
Ratio
Symptoms
Causes
Diagnosis
Others
RDs
Treatment
Affects
Avg
0%
73.73
77.76
63.07
70.37
71.22
55.84
76.38
69.77
13%
76.57
77.49
64.89
72.69
74.51
55.57
76.03
71.11
25%
79.71
80.89
68.19
72.22
76.64
57.51
78.62
73.40
33%
78.14
78.49
68.33
75.93
79.77
59.60
80.52
74.40
50%
79.12
78.62
65.84
72.45
78.78
58.63
78.45
73.13
Appendix
Table 10: Performance on RareDis-Sub across different rare disease ratios. “RDs” is short for Related Disorders.
Ratio
BioASQ
MedMCQA
MedQA
MedXpertqa
MMLU
PubMedQA
NEJM
Lancet
Medbullets
Avg
0%
83.47
65.89
75.69
16.40
84.53
55.20
67.50
67.59
56.09
63.60
13%
84.78
68.73
82.54
21.27
87.21
56.20
70.56
69.78
64.12
67.24
25%
86.00
67.79
81.14
20.66
86.64
54.70
71.97
70.94
63.47
67.03
33%
83.38
68.56
81.38
20.97
86.13
53.55
71.39
67.72
64.86
66.44
50%
84.14
67.50
80.20
19.42
86.36
56.15
69.32
66.99
64.45
66.06
Appendix
Table 11: Performance on general medical datasets across different rare disease ratios.
Data Scale
Method
BioASQ
MedMCQA
MedQA
MedXpertqa
MMLU
PubMedQA
NEJM
Lancet
Medbullets
Avg
×1
Fully-supervised
81.91
64.25
74.98
17.40
83.88
54.15
68.74
67.66
55.11
63.12
Self-supervised
82.51
65.44
75.61
17.09
86.25
55.05
66.91
68.87
55.92
63.74
×2
Fully-supervised
81.40
66.79
77.36
16.97
84.78
55.05
69.03
68.14
58.60
64.24
MedSSR
86.00
67.79
81.14
20.66
86.64
54.70
71.97
70.94
63.47
67.03
Appendix
Table 12: Performance across nine datasets under similar training cost. We define “×1 scale” as the 43K data volume used in prior experiments. The results of both “×1 scale” and “×2 scale” experiments are reported for comparison.
Model
Symptoms
Causes
Diagnosis
Others
RDs
Treatment
Affects
Avg
Qwen3-1.7B-Base
37.35
46.65
34.03
40.05
37.17
33.77
35.86
37.84
+ MedSSR
49.71
55.43
44.81
48.84
49.84
40.35
47.24
48.03
↑
12.36
8.78
10.78
8.79
12.67
6.58
11.38
10.19
Qwen3-4B-Base
60.29
66.80
48.99
58.80
53.62
45.73
63.97
56.89
+ MedSSR
70.69
75.14
58.69
68.29
70.23
53.76
72.41
67.03
↑
10.40
8.34
9.70
9.49
16.61
8.03
8.44
10.14
Appendix
Table 13: Performance on RareDis-Sub across different LLM scales. “RDs” is short for Related Disorders.
Model
BioASQ
MedMCQA
MedQA
MedXpertqa
MMLU
PubMedQA
NEJM
Lancet
Medbullets
Avg
Qwen3-1.7B-Base
56.10
41.36
39.20
11.58
53.81
45.35
37.94
32.16
30.20
38.63
+ MedSSR
71.99
48.17
46.86
12.54
61.85
49.05
48.84
46.66
38.48
47.16
↑
15.89
6.81
7.66
0.96
8.04
3.70
10.90
14.50
8.28
8.53
Qwen3-4B-Base
76.53
52.44
55.58
11.87
74.03
50.20
54.39
52.49
41.56
52.12
+ MedSSR
82.61
63.86
72.25
16.54
81.96
54.05
65.30
64.14
57.47
62.02
↑
6.08
11.42
16.67
4.67
7.93
3.85
10.91
11.65
15.91
9.90
Appendix
Table 14: Performance on general medical datasets across different LLM scales.
Case Study w/ rare disease
Seed Question 1
An army recruit, smoker and 6 months into training staed complaining of pain at postero medial aspect of both legs. There was acute point tenderness and the pain was aggravated on physical activity. The most likely diagnosis is A. Bearger’s disease B. Gout C. Lumbar canal stenosis D. Stress fracture
Seed Question 2
A patient presenting with haemoptysis and renal failure with antibasement membrane antibodies has - A. Good pasture’s B. Wegener’s C. Churg Strauss D. Henoch-schonlein purpura
Worster-Drought syndrome is a form of congenital suprabulbar paresis that occurs in some children with cerebral palsy. It is caused by inadequate development of the corticobulbar tracts and causes problems with the mouth and tongue including impaired swallowing. A similar syndrome in adults is called anterior opercular syndrome. A 1986 study of a family in which multiple members had Worster-Drought syndrome suggested it might be hereditary. A 2000 review of cases classified Worster-Drought Syndrome as a form of cerebral palsy, caused by early damage to the brain, but identified no obvious causes during gestation or birth and found some families with a history of the condition. The syndrome was named after Cecil Charles Worster-Drought , the doctor who described it in 1956.
Synthetic Question
A 5-year-old child presents with significant speech delay, difficulty swallowing since infancy, and drooling. Neurological examination reveals impaired movement of the tongue and soft palate, with no limb weakness. The child’s mother reports similar symptoms in her own childhood. MRI shows underdevelopment of the corticobulbar tracts. Which of the following is the most likely diagnosis? A. Moebius syndrome B. Spinal muscular atrophy (bulbar form) C. Worster-Drought syndrome D. Anterior opercular syndrome (acquired suprabulbar palsy)
Majority Answer Num: 5/8
To determine the most likely diagnosis for this 5-year-old child, let’s go through the information step-by-step. ### Step-by-Step Analysis: 1. Clinical Presentation: - Significant speech delay - Difficulty swallowing since infancy - Drooling - Impaired movement of the tongue and soft palate - No limb weakness. 2. Family History: - Similar symptoms in the mother2̆019s childhood. 3. Neurological Examination: - Impaired movement of the tongue and soft palate - No limb weakness. 4. Imaging Findings: - MRI shows underdevelopment of the corticobulbar tracts. ### Differential Diagnosis: 1. Moebius Syndrome (A): - Characterized by congenital facial paralysis (including inability to move the eyes laterally and the mouth) and sometimes other cranial nerve abnormalities. - Often associated with feeding difficulties, facial asymmetry, and small eyes. - Does not typically involve limb weakness or underdevelopment of the corticobulbar tracts. 2. Spinal Muscular Atrophy (Bulbar Form) (B): - Typically presents with muscle weakness and wasting, particularly in the bulbar muscles (speech, swallowing, and respiratory muscles). - Often has a progressive course. - Usually involves limb weakness due to lower motor neuron involvement. 3. ** Worster-Drought Syndrome (C)**: - Also known as congenital central hypoventilation syndrome. - Characterized by underdevelopment of the brainstem, particularly the pontine respiratory centers. - May present with respiratory issues, but not typically with speech delay or swallowing difficulties unless it affects the cranial nerves. - Usually involves other neurological symptoms like poor sucking and feeding. 4. Anterior Opercular Syndrome (D): - Also known as acquired suprabulbar palsy. - Caused by lesions affecting the corticobulbar tracts, leading to weakness or paralysis of the muscles supplied by the lower motor neurons (e.g., facial, tongue, and pharyngeal muscles). - Typically presents in adults, not children. - Associated with spasticity and hyperreflexia due to upper motor neuron involvement. ### Final Considerations: - The combination of significant speech delay, difficulty swallowing since infancy, drooling, impaired movement of the tongue and soft palate, and underdevelopment of the corticobulbar tracts suggests a congenital condition affecting the brainstem and cranial nerves. - The presence of similar symptoms in the mother2̆019s childhood suggests a familial pattern, which is characteristic of Worster-Drought syndrome or Moebius syndrome. Given the specific combination of symptoms and imaging findings, ** Worster-Drought syndrome ** (also known as congenital central hypoventilation syndrome with associated craniofacial anomalies) is the most likely diagnosis. This condition is characterized by underdevelopment of the brainstem, which would explain the underdevelopment of the corticobulbar tracts and the associated cranial nerve abnormalities. ### Final Answer: The final answer is C.
Appendix
Table 15: Case study with rare disease knowledge injection in data synthesis.
Case Study w/o rare disease
Seed Question 1
A 35wks POG, USG parameters more or less corresponding to POG. Doppler showing absent end diastolic flow Management: A. Monitor till it becomes reverse diastolic flow or 37 weeks B. Monitor till 37 weeks C. Give steroids and wait for 48 hours D. Plan for immediate termination
Seed Question 2
A drug ’X’ belongs to the anticholinergic drug group. It is primarily used in pre anesthetic medication and also during surgery. Which of the following can be ’X’ ? A. Glycopyrrolate B. Pipenzolate methyl bromide C. Isopropamide D. Dicyclomine
Synthetic Question
A 32-year-old woman at 33 weeks gestation presents with reduced fetal movements. Ultrasound reveals an estimated fetal weight below the 10th percentile for gestational age. Umbilical artery Doppler shows an elevated systolic/diastolic ratio but preserved end-diastolic flow. Non-stress test is non-reactive. What is the most appropriate next step in management? A. Repeat non-stress test in 24 hours and continue monitoring B. Administer corticosteroids and plan delivery after 48 hours C. Immediate delivery D. Start magnesium sulfate and continue pregnancy until 37 weeks
Majority Answer Num: 6/8
# Step-by-step reasoning: 1. Clinical Presentation: - The patient is a 32-year-old woman at 33 weeks gestation. - She presents with reduced fetal movements, which is concerning for fetal compromise. - Ultrasound reveals an estimated fetal weight below the 10th percentile for gestational age, indicating fetal growth restriction (FGR). - Umbilical artery Doppler shows an elevated systolic/diastolic ratio but preserved end-diastolic flow, suggesting chronic placental insufficiency but not acute compromise. - The non-stress test is non-reactive, which is abnormal and indicates reduced fetal reactivity. 2. Interpretation of Findings: - Reduced fetal movements and estimated fetal weight below the 10th percentile are strong indicators of fetal growth restriction (FGR). - Elevated systolic/diastolic ratio on umbilical artery Doppler suggests chronic placental insufficiency. - Preserved end-diastolic flow indicates that there is still some placental function, but it is compromised. - A non-reactive non-stress test is abnormal and suggests reduced fetal reactivity, which is concerning for fetal distress. 3. Management of FGR with Abnormal Non-Stress Test: - In cases of FGR with abnormal non-stress tests (e.g., non-reactive), the next step is to assess for acute fetal compromise. - If there is evidence of acute fetal distress (e.g., abnormal biophysical profile, abnormal Doppler findings), immediate delivery may be indicated. - In the absence of acute distress, the decision to deliver depends on the gestational age, fetal condition, and maternal factors. 4. Gestational Age and Decision Making: - At 33 weeks gestation, the fetus is considered preterm. - If there is no evidence of acute fetal distress, delivery is generally not indicated unless there is clear evidence of worsening fetal condition. - Corticosteroids may be administered to promote fetal lung maturity if delivery is likely within 7 days. 5. Most Appropriate Next Step: - Given the reduced fetal movements, FGR, and abnormal non-stress test, the next step should be to assess for acute fetal distress. - If there is no evidence of acute distress, corticosteroids should be administered to promote fetal lung maturity, and the patient should be monitored closely for further fetal deterioration. - Delivery is typically not indicated at 33 weeks unless there is clear evidence of acute fetal distress or worsening fetal condition. 6. Final Decision: - Administer corticosteroids to promote fetal lung maturity. - Plan delivery after 48 hours if there is no improvement in fetal condition or if there is evidence of acute fetal distress. # Final Answer: The final answer is B.
Minority Answer Num: 2/8
To determine the most appropriate next step in the management of a 32-year-old woman at 33 weeks gestation with reduced fetal movements, let’s break this down step-by-step. ### Step-by-Step Reasoning: 1. Clinical Presentation: - The patient is at 33 weeks gestation. - She presents with reduced fetal movements. - Ultrasound reveals an estimated fetal weight below the 10th percentile for gestational age (indicating intrauterine growth restriction, IUGR). - Umbilical artery Doppler shows an elevated systolic/diastolic ratio but preserved end-diastolic flow (suggesting chronic placental insufficiency). - Non-stress test is non-reactive (indicating poor fetal well-being). 2. Interpretation of Findings: - Reduced fetal movements + IUGR + abnormal Doppler + non-reactive NST suggest fetal compromise. - Elevated systolic/diastolic ratio indicates chronic placental insufficiency, which is often associated with poor outcomes if not addressed promptly. 3. Management Considerations: - Intrauterine growth restriction (IUGR) with abnormal Doppler findings and non-reactive NST is concerning for fetal compromise. - Immediate intervention may be necessary to prevent adverse outcomes such as stillbirth or neonatal morbidity. 4. Options Analysis: - A: Repeat non-stress test in 24 hours and continue monitoring - This is not appropriate given the presence of non-reactive NST and other concerning findings. - B: Administer corticosteroids and plan delivery after 48 hours - Corticosteroids can be given to promote fetal lung maturity, but immediate delivery might be warranted based on the findings. - C: Immediate delivery - Given the combination of reduced fetal movements, IUGR, abnormal Doppler, and non-reactive NST, immediate delivery is often recommended to prevent further fetal compromise. - D: Start magnesium sulfate and continue pregnancy until 37 weeks - Magnesium sulfate is typically used for neuroprotection in preterm labor, not for managing IUGR with compromised fetal status. 5. Final Decision: - The presence of non-reactive NST and IUGR with abnormal Doppler findings suggests significant fetal compromise. - Immediate delivery is generally recommended in such cases to prevent adverse outcomes. ### Final Answer: Given the clinical presentation and findings, the most appropriate next step in management is immediate delivery. Therefore, the final answer is C.
Appendix
Table 16: Case study without rare disease knowledge injection in data synthesis.
Failure Case
Question
A 55-year-old woman presents with progressive weakness in her right hand and difficulty manipulating small objects over the past 3 months. She also reports mild pain radiating from her neck to her right shoulder. On examination, there is atrophy of the right thenar muscles and decreased sensation over the lateral aspect of the right hand. What is the most likely diagnosis?
Options
A. Carpal tunnel syndrome, B. Cervical radiculopathy (C6-C7), C. Ulnar nerve entrapment at the elbow, D. Thoracic outlet syndrome.
Ground Truth
A. Carpal tunnel syndrome
Voting Results
"A": 0, "B": 8, "C": 0, "D": 0
Response Example
To determine the most likely diagnosis for this patient, let’s break it down step-by-step: ### Step-by-Step Reasoning: 1. Clinical Presentation: - Age: 55 years old - Symptoms: Progressive weakness in the right hand, difficulty manipulating small objects, mild pain radiating from the neck to the right shoulder - Physical Exam: Atrophy of the right thenar muscles, decreased sensation over the lateral aspect of the right hand 2. Key Findings: - Atrophy of the thenar muscles: This is characteristic of median nerve involvement. - Decreased sensation over the lateral aspect of the right hand: The lateral aspect of the hand is innervated by the radial nerve, but the thenar muscles are primarily innervated by the median nerve. - Radiating pain from the neck to the shoulder: This suggests a cervical spine issue rather than a peripheral nerve problem. 3. Differential Diagnosis: - A: Carpal Tunnel Syndrome: Typically presents with numbness and tingling in the thumb, index, middle, and half of the ring finger, but usually not with thenar muscle atrophy or neck pain . - B: Cervical Radiculopathy (C6-C7): Commonly presents with weakness in the wrist extensors (extensor carpi radialis brevis, extensor carpi radialis longus, and extensor carpi ulnaris), which can cause difficulty with small object manipulation. It also often involves pain radiating from the neck to the shoulder, and sensory changes in the thumb, index, and middle fingers (C6) or the middle and ring fingers (C7). Thenar muscle atrophy is more commonly associated with C7 radiculopathy. - C: Ulnar Nerve Entrapment at the Elbow: Presents with weakness in the hypothenar muscles, adductor pollicis, and interossei, along with decreased sensation in the little finger and half of the ring finger. There is no significant atrophy of the thenar muscles. - D: Thoracic Outlet Syndrome: Can present with symptoms similar to those of cervical radiculopathy, including pain, numbness, and weakness in the upper extremity, but typically involves both arms and may be associated with vascular signs like cyanosis or pallor of the hand. 4. Specificity of Symptoms: - Thenar muscle atrophy is strongly suggestive of median nerve involvement, which is often due to compression at the wrist (carpal tunnel syndrome) or cervical spine issues (e.g., C6 or C7 radiculopathy). - Radiating pain from the neck to the shoulder points towards a cervical spine origin. - Lateral aspect of the hand is not typically affected by median nerve issues but could be related to radial nerve issues, which are less likely given the presence of thenar muscle atrophy. 5. Conclusion: - The combination of thenar muscle atrophy, decreased sensation over the lateral aspect of the hand, and radiating pain from the neck to the shoulder is most consistent with C6-C7 cervical radiculopathy. - The lateral aspect of the hand is innervated by the radial nerve, which can be affected by C6 radiculopathy, but the presence of thenar muscle atrophy points more strongly to C7 involvement. ### Final Answer: Based on the clinical presentation and physical exam findings, the most likely diagnosis is B: Cervical radiculopathy (C6-C7). The final answer is B"
Appendix
Table 17: Failure case analysis of majority-voted labels. The incorrect parts are highlighted in red .
Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the convergent, evidence-driven task of inferring a patient's condition from clinical data to produce a diagnosis, and clinical healthcare reasoning: the broader, navigational judgment required to communicate, plan, and adapt across multi-turn clinical interactions where a single correct answer may not exist. Recent benchmarks such as HealthBench and MedXpertQA reveal persistent weaknesses in both areas, exposing failures in complex diagnostic scenarios and limitations in contextual, patient-centered dialogue. We introduce a sequential training framework that targets these facets using synthetic data and rubric-based reinforcement learning. First, we improve diagnostic reasoning using MedBullets-derived questions with rule- and rubric-guided Reinforcement Learning (RL). We then shift to clinical reasoning by generating 5.3k synthetic multi-turn scenarios, each paired with multi-dimensional rubrics to comprehensively assess the response. This approach yields over 10% improvement on MedXpertQA, and our 30B model achieves 50.1% accuracy on HealthBench-Hard, surpassing proprietary baselines including GPT-5 (thinking). Our results show that targeted synthetic datasets and rubric-based training can systematically improve both diagnostic and interactive clinical reasoning in medical LLMs.
Medical question answering requires advanced reasoning that integrates domain knowledge with logical inference. However, existing large language models (LLMs) often generate reasoning chains that lack factual accuracy and clinical reliability. We propose Ranked Preference Reinforcement Optimization (RPRO), a novel framework that combines reinforcement learning with preference-driven reasoning refinement to enhance clinical chain-of-thought (CoT) performance. RPRO distinguishes itself from prior approaches by employing task-adaptive reasoning templates and a probabilistic evaluation mechanism that aligns model outputs with established clinical workflows, while automatically identifying and correcting low-quality reasoning chains. Unlike traditional pairwise preference methods, RPRO introduces a groupwise ranking optimization based on the Bradley--Terry model and incorporates KL-divergence regularization for stable training. Experiments on PubMedQA, MedQA-USMLE, and a real-world clinical dataset from Far Eastern Memorial Hospital (FEMH) demonstrate consistent improvements over strong baselines. Remarkably, our 2B-parameter model outperforms much larger 7B--20B models, including medical-specialized variants. These findings demonstrate that combining preference optimization with quality-driven refinement provides a scalable and clinically grounded approach to building more reliable medical LLMs.
Chia-Hsuan Hsu, Jun-En Ding, Hsin-Ling Hsu +5
Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei City, Taiwan · Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ, USA · Far Eastern Memorial Hospital, New Taipei City, Taiwan +4
We introduce MediX-R1, an open-ended Reinforcement Learning (RL) framework for medical multimodal large language models (MLLMs) that enables clinically grounded, free-form answers beyond multiple-choice formats. MediX-R1 fine-tunes a baseline vision-language backbone with Group Based RL and a composite reward tailored for medical reasoning: an LLM-based accuracy reward that judges semantic correctness with a strict YES/NO decision, a medical embedding-based semantic reward to capture paraphrases and terminology variants, and lightweight format and modality rewards that enforce interpretable reasoning and modality recognition. This multi-signal design provides stable, informative feedback for open-ended outputs where traditional verifiable or MCQ-only rewards fall short. To measure progress, we propose a unified evaluation framework for both text-only and image+text tasks that uses a Reference-based LLM-as-judge in place of brittle string-overlap metrics, capturing semantic correctness, reasoning, and contextual alignment. Despite using only ∼51K instruction examples, MediX-R1 achieves excellent results across standard medical LLM (text-only) and VLM (image + text) benchmarks, outperforming strong open-source baselines and delivering particularly large gains on open-ended clinical tasks. Our results demonstrate that open-ended RL with comprehensive reward signals and LLM-based evaluation is a practical path toward reliable medical reasoning in multimodal models. Our trained models, curated datasets and source code are available at https://medix.cvmbzuai.com
Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed +5
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College