RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning

Chia-Hsuan Hsu; Jun-En Ding; Hsin-Ling Hsu; Chih-Ho Hsu; Li-Hung Yao; Chun-Chieh Liao; Feng Liu; Fang-Ming Hung

cs.CLAug 31, 2025

RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning

Authors: Chia-Hsuan Hsu, Jun-En Ding, Hsin-Ling Hsu, Chih-Ho Hsu, Li-Hung Yao, Chun-Chieh Liao, Feng Liu, Fang-Ming Hung

Organizations: Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei City, Taiwan · Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ, USA · Far Eastern Memorial Hospital, New Taipei City, Taiwan · Department of Management Information Systems, National Chengchi University, Taipei City, Taiwan ·

Abstract

Medical question answering requires advanced reasoning that integrates domain knowledge with logical inference. However, existing large language models (LLMs) often generate reasoning chains that lack factual accuracy and clinical reliability. We propose Ranked Preference Reinforcement Optimization (RPRO), a novel framework that combines reinforcement learning with preference-driven reasoning refinement to enhance clinical chain-of-thought (CoT) performance. RPRO distinguishes itself from prior approaches by employing task-adaptive reasoning templates and a probabilistic evaluation mechanism that aligns model outputs with established clinical workflows, while automatically identifying and correcting low-quality reasoning chains. Unlike traditional pairwise preference methods, RPRO introduces a groupwise ranking optimization based on the Bradley--Terry model and incorporates KL-divergence regularization for stable training. Experiments on PubMedQA, MedQA-USMLE, and a real-world clinical dataset from Far Eastern Memorial Hospital (FEMH) demonstrate consistent improvements over strong baselines. Remarkably, our 2B-parameter model outperforms much larger 7B--20B models, including medical-specialized variants. These findings demonstrate that combining preference optimization with quality-driven refinement provides a scalable and clinically grounded approach to building more reliable medical LLMs.

Figures & tables

Fig. 1 : Overview of the proposed pipeline. The framework distinguishes between medical QA and diagnosis tasks, applies chain-of-thought (CoT) reasoning with multi-dimensional quality assessment and probabilistic refinement, and further improves through ranking-guided preference optimization with dataset construction.

Models	PubMedQA (Accuracy / Macro F1)			MedQA-USMLE (Accuracy / Macro F1)
Models	0-shot	1-shot	5-shot	0-shot	1-shot	5-shot
BioMedLM	35.10 / 20.45	42.23 / 25.75	48.65 / 30.30	22.68 / 13.73	26.07 / 15.92	31.18 / 20.36
BioMistral 7B	36.89 / 27.98	42.76 / 32.41	45.53 / 36.28	19.89 / 7.47	31.53 / 13.41	37.57 / 16.10
Falcon 7B	55.83 / 37.72	57.14 / 39.08	57.92 / 40.63	17.23 / 8.14	24.87 / 16.21	24.89 / 13.94
Gemma 7B	58.08 / 48.02	58.67 / 49.51	58.12 / 50.84	20.37 / 10.18	43.12 / 24.15	51.18 / 32.35
MedAlpaca 7B	44.83 / 42.41	46.72 / 43.94	48.51 / 45.34	24.01 / 16.37	41.27 / 31.46	47.82 / 36.38

TABLE I : Comparison of different models on PubMedQA and MedQA-USMLE under 0-shot, 1-shot, and 5-shot settings. Best results are in bold . Results for our method (Gemma 2B, RGPO) are averaged over 5 runs with different random seeds. The standard deviations (Accuracy / Macro-F1) are as follows: PubMedQA: 0-shot (

\pm

1.41 /

\pm

1.32), 1-shot (

\pm

1.36 /

\pm

1.28), 5-shot (

\pm

1.29 /

\pm

1.35); MedQA-USMLE: 0-shot (

\pm

1.52 /

\pm

1.47), 1-shot (

\pm

1.24 /

\pm

1.39), 5-shot (

\pm

1.33 /

\pm

1.41).

Models	0-shot (BERTScore-F1 / Cosine)	1-shot (BERTScore-F1 / Cosine)	5-shot (BERTScore-F1 / Cosine)
BioMedLM	0.798 / 0.342	0.812 / 0.351	0.825 / 0.353
BioMistral 7B	0.830 / 0.398	0.842 / 0.326	0.885 / 0.405
Falcon 7B	0.856 / 0.342	0.861 / 0.351	0.872 / 0.368
Gemma 7B	0.871 / 0.367	0.874 / 0.381	0.868 / 0.373
MedAlpaca 7B	0.848 / 0.334	0.852 / 0.343	0.861 / 0.357
Mistral 7B	0.859 / 0.345	0.865 / 0.359	0.871 / 0.366

TABLE II : Performance comparison on the FEMH dataset using semantic similarity metrics. Results for our method are averaged over 5 random seeds. The standard deviations (BERTScore-F1 / Cosine Similarity) are: 0-shot (

\pm

0.016 /

\pm

0.012), 1-shot (

\pm

0.010 /

\pm

0.015), 5-shot (

\pm

0.014 /

\pm

0.014).

Method	0-shot			1-shot			5-shot
Method	Accuracy	Recall@3	Recall@5	Accuracy	Recall@3	Recall@5	Accuracy	Recall@3	Recall@5
BioMedLM	52.18	55.27	58.63	53.46	56.83	60.17	54.76	58.31	61.84
BioMistral 7B	53.12	56.04	59.18	54.29	57.36	59.71	55.64	58.73	61.92
Falcon 7B	54.37	57.68	60.74	55.81	58.94	62.13	57.06	60.43	63.82
Gemma 7B	56.94	60.31	63.47	58.19	62.44	65.88	59.63	64.12	67.41
MedAlpaca 7B	53.82	57.03	60.11	55.21	58.52	61.83	56.63	60.11	63.72

TABLE III : Performance comparison on ICD-10 diagnostic coding using our FEMH dataset. Model-generated diagnoses are mapped to standardized ICD-10 codes for evaluation. Accuracy measures exact code-level agreement, while Recall@k measures whether the correct code appears in the top-k predictions. Results are shown for 0-shot, 1-shot, and 5-shot in-context learning settings.

Baseline Model	PubMedQA			MedQA-USMLE
Baseline Model	0-shot	1-shot	5-shot	0-shot	1-shot	5-shot
BioMedLM
BioMistral 7B
Falcon 7B
Gemma 7B	ns	ns			ns	ns
MedAlpaca 7B					**

TABLE IV : Statistical significance of performance improvements over baselines on PubMedQA and MedQA-USMLE across different few-shot settings. Significance levels are determined using McNemar’s test for accuracy comparisons.

Method	PubMedQA		MedQA-USMLE
	ACC (%)	Macro F1 (%)	ACC (%)	Macro F1 (%)
SFT	55.59	38.15	36.05	23.45
SFT + Refinement	57.43	40.53	37.08	23.72
SFT + Listwise Ranking	56.09	37.41	35.18	23.47
DPO + Refinement	59.90	42.72	38.26	24.35
Our Method: RGPO (Full Method)	60.23	51.78	42.38	30.51

TABLE V : Ablation study evaluating the contributions of probabilistic refinement and listwise preference ranking within the RGPO framework on PubMedQA and MedQA-USMLE under zero-shot settings. Results compare baseline SFT, SFT with refinement only, SFT with listwise ranking only, and the full RGPO method. Best results are in bold .

Data	Method	0-shot	1-shot	5-shot
Clean	RGPO w/o Filtering	0.867 / 0.401	0.872 / 0.489	0.878 / 0.507
Clean	RGPO w/ Filtering	0.879 / 0.423	0.884 / 0.507	0.891 / 0.528
Noisy (w/o Clean)	RGPO w/o Filtering	0.851 / 0.372	0.858 / 0.451	0.864 / 0.468
Noisy (w/o Clean)	RGPO w/ Filtering	0.868 / 0.398	0.873 / 0.486	0.879 / 0.502

TABLE VI : Ablation study on the FEMH dataset under clean and noisy (unprocessed) data conditions. We evaluate the effect of probabilistic refinement filtering under different prompting settings (0-shot, 1-shot, and 5-shot). “Noisy” denotes raw clinical records without text cleaning (e.g., line breaks and formatting artifacts). The acceptance threshold

\theta

is fixed at 0.6. Results are reported using semantic similarity metrics (BERTScore-F1 / Cosine Similarity). Best results within each data condition are shown in bold .

Case 1: Myocarditis (post-Moderna) Input (Symptoms/Objective): S: 1101201 Enzyme decrease

\rightarrow

possible improved myocarditis… 110/11/26 2nd Moderna

\rightarrow

11/27 fever

\rightarrow

11/29 mild dyspnea, chest tightness (worse when lying) … LMD CPK WNL, TnT 1559 → 210. EKG: mild pericarditis (r/o). CXR: normal heart size. O: BP, HR stable; clear BS; soft abdomen; no edema. Baseline (SFT) Reasoning: Step 1. The patient presents with mild dyspnea and chest tightness that worsens when lying down, suggesting cardiopulmonary involvement. Step 2. Laboratory findings show elevated troponin levels , indicating myocardial injury . Step 3. The combination of symptoms and biomarker elevation is consistent with myocarditis . Step 4. Given the clinical presentation, the most likely diagnosis is acute or post-infectious myocarditis . Diagnosis: Post-infectious myocarditis . Model Reasoning (RGPO): Step 1. Identify the key symptoms: The patient presents with fever, dyspnea, chest tightness, and mild discomfort. Step 2. Consider the laboratory results: The patient has elevated troponin levels, but CK-MB is normal . This suggests a possible myocarditis . Step 3. Integrate the clinical findings: The patient’s symptoms and laboratory results suggest a possible myocarditis. The EKG is normal, and the CXR is normal . Step 4. Consider differential diagnoses: The differential diagnoses include myocarditis, pericarditis, and other causes of chest pain and dyspnea . Diagnosis: Myocarditis . Prediction & Answer: Baseline Prediction: Post-infectious myocarditis. Our Prediction: Myocarditis. True Diagnosis: r/o mild myocarditis.

Case 2: IOL Dislocation (OD) Input (Symptoms/Objective): S: Stinging sensation (2021/11), high IOP (

>

50), blurred vision OD. Lost f/u 2 years. O: IOP: 53→17 (treated). Cornea clear. PCIOL subluxation inferiorly. OS: NS 1+–2+. Fundus: C/D 0.5–0.6 (OD). OCT RNFL OD 32. Baseline (SFT) Reasoning: Step 1. The patient presents with a stinging sensation in the eye, markedly elevated intraocular pressure, and blurred vision , indicating an ocular pathology affecting visual acuity. Step 2. The symptoms show a temporal progression, with stinging sensation noted in late 2021 and blurred vision reported since early 2019. Step 3. Relevant ocular history includes intraocular lens subluxation , corneal abrasion, and a prior cerebrovascular accident, suggesting chronic postoperative or structural ocular changes. Step 4. Integrating the symptoms, temporal course, and ocular history, the findings are interpreted as most consistent with posterior capsular opacification . Diagnosis: Posterior capsular opacification (PCO) . Model Reasoning (RGPO): Step 1. Potential causes: corneal opacity, cataract, glaucoma, IOL subluxation . Step 2. History: high IOP, decreased vision, recent IOL subluxation . Step 3. Comprehensive exam to assess cornea/lens/retina. Step 4. Conclusion: likely IOL subluxation related to prior surgery/high IOP . Diagnosis: IOL subluxation . Prediction & Answer: Baseline Prediction: Posterior capsular opacification (PCO). Our Prediction: IOL subluxation. True Diagnosis: IOL dislocation (OD).

TABLE VII : Qualitative comparison of baseline (SFT) and RGPO reasoning on representative cases from the FEMH dataset.

Case	Question (summary)	SFT reasoning (baseline)	RGPO reasoning
Case 1	Post-vaccine myocarditis	Primarily relies on symptom–biomarker association to support myocarditis, with limited exploration of alternative diagnoses or explicit articulation of residual diagnostic uncertainty.	Integrates symptoms, laboratory trends, and imaging findings while explicitly considering differential diagnoses and articulating diagnostic uncertainty consistent with clinical practice.
Case 2	IOL dislocation	Attributes symptoms to posterior capsular opacification (PCO) despite documented intraocular lens subluxation and markedly elevated intraocular pressure, underweighting relevant structural findings.	Prioritizes objective structural abnormalities, systematically evaluates plausible causes, and appropriately links symptoms to intraocular lens subluxation associated with prior surgery and high IOP.

TABLE VIII : Qualitative comparison of reasoning generated by standard SFT and RGPO on representative clinical cases, illustrating common reasoning limitations and improved clinical alignment.

Fig. 2 : Performance comparison on PubMedQA, MedQA-USMLE, and FEMH dataset under different $\beta$ values. PubMedQA/MedQA report Accuracy (solid blue) and Macro F1 (dashed orange), while FEMH dataset reports BERTScore-F1 (solid blue) and Cosine Similarity (dashed orange).

Fig. 3 : Performance comparison on PubMedQA, MedQA-USMLE, and FEMH dataset under different rollout numbers ( $K$ ). PubMedQA/MedQA report Accuracy and Macro F1; FEMH dataset reports BERTScore-F1 and Cosine Similarity. Each prompt generates $K$ candidate CoTs. For $K<4$ , all candidates are used ( $M=K$ ); for $K\geq 4$ , the top $M=4$ candidates are selected for training.

Fig. 4 : Performance across different acceptance thresholds on PubMedQA, MedQA-USMLE, and FEMH dataset. PubMedQA/MedQA report Accuracy (solid blue) and Macro F1 (dashed purple), while FEMH dataset reports BERTScore-F1 and Cosine Similarity.

Fig. 5 : Training loss curves on MedQA-USMLE, PubMedQA, and FEMH dataset. The plots show KL divergence, pairwise loss, ranking loss, and total loss across training steps for each dataset.

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 396 papers

Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering
Jun 13, 2026Zaifu Zhan, Shuang Zhou, Rui ZhangMulti-Agent LLM Systems LLM Answer Verification
Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States
To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering
Feb 23, 2026Zaifu Zhan, Min Zeng, Shuang Zhou +6CoT Reasoning Efficient Language Model Reasoning
Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States · Department of Computer Science and Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States +1
Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
Sep 21, 2026Kalash Shah, Kunal Singh, Snehan J +1RL for Language Model Reasoning Rubric-Based RL
Fractal AI Research

Jun 13, 2026J/K move · Enter open · S save

Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

Zaifu Zhan, Shuang Zhou, Rui Zhang

Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States

Multi-Agent LLM Systems LLM Answer Verification Medical QA Multi-Agent Reasoning Clinical Language Model Evaluation

Open paper AskPDF arXiv

Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA). Method: We designed a multi-agent peer-reviewed reasoning method in which multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, then act as peer reviewers to evaluate each other's reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chain-of-thought-based majority voting. Results: Peer-reviewed reasoning consistently outperformed both baselines. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). The method also scaled effectively with more participating models, while peer assessments reliably distinguished high- from low-quality reasoning chains. Conclusion: The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA. By emphasizing reasoning quality rather than answer agreement alone, this approach improves accuracy, interpretability, and robustness, offering a promising direction for trustworthy biomedical AI systems.

Abstract

Medical LLMs

Explore similar work

Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards