ThReadMed-QA: A Multi-Turn Medical Dialogue Benchmark from Real Patient Questions

Monica Munnangi; Saiph Savage

cs.CLMar 11, 2026

ThReadMed-QA: A Multi-Turn Medical Dialogue Benchmark from Real Patient Questions

Authors: Monica Munnangi, Saiph Savage

Organizations: Khoury College of Computer Sciences Northeastern University Boston

Abstract

Medical question-answering benchmarks predominantly evaluate single-turn exchanges, failing to capture the iterative, clarification-seeking nature of real patient consultations. We introduce ThReadMed-QA, a benchmark of 2,437 fully-answered patient-physician conversation threads extracted from r/AskDocs, comprising 8,204 question-answer pairs across up to 9 turns. Unlike prior work relying on simulated dialogues, adversarial prompts, or exam-style questions, ThReadMed-QA captures authentic patient follow-up questions and verified physician responses, reflecting how patients naturally seek medical information online. We evaluate five state-of-the-art LLMs -- GPT-5, GPT-4o, Claude Haiku, Gemini 2.5 Flash, and Llama 3.3 70B -- on a stratified test split of 238 conversations (948 QA pairs) using a calibrated LLM-as-a-judge rubric grounded in physician ground truth. Even the strongest model, GPT-5, achieves only 41.2% fully-correct responses. All five models degrade significantly from turn 0 to turn 2 (p < 0.001), with wrong-answer rates roughly tripling by the third turn. We identify a fundamental tension between single-turn capability and multi-turn reliability: models with the strongest initial performance (GPT-5: 75.2; Claude Haiku: 72.3 out of 100) exhibit the steepest declines by turn 2 (dropping 16.2 and 25.0 points respectively), while weaker models plateau or marginally improve. We introduce two metrics to quantify multi-turn failure modes: Conversational Consistency Score (CCS) and Error Propagation Rate (EPR). CCS reveals that nearly one in three Claude Haiku conversations swings between a fully correct and a completely wrong response within the same thread. EPR shows that a single wrong turn raises the probability of a subsequent wrong turn by 1.9-6.1x across all models.

Figures & tables

Figure 1: We compare our dataset with the existing medical QA datasets. MedQA [ 9 ] , PubMedQA [ 10 ] , and MedMCQA [ 11 ] are derived from medical examinations and are multiple-choice questions. MedQA-Followup [ 5 ] and MeDiaQA [ 12 ] are adversarial follow-up questions with multiple choice options. MedRedQA [ 13 ] , MedicationQA [ 14 ] and HealthSearchQA [ 1 ] have patient-authored questions but in single-turn. Our dataset ThReadMed-QA is the first to have patient-authored questions and followups, along with physician responses.

Figure 2: Dataset construction and filtering process. We selected posts from r/AskDocs from Jan 2015 to Jun 2023, totaling approximately 1.8 million posts. After preprocessing and keeping the thread with the original poster (OP) and physician, we have the final set of 2,437 conversation threads.

Scale
Metric	Full Dataset	Fully Answered Dataset
Total conversation threads	9,741	2,437
Total number of questions	37,191	8,204
Total number of answered questions	21,595 (58.1%)	8,204 (100%)
Conversational depth
Follow-ups per thread (min / max)	2 / 32	2 / 9

Table 1: Comparison of the full multi-turn patient-physician QA dataset and the fully answered subset. Both datasets are derived from r/AskDocs threads with at least one verified medical professional respondent. The full dataset includes all threads with three or more QA turns (9,741 threads; 37,191 questions, 58.1% answered). The fully answered subset retains only threads where every question has a physician response (2,437 threads; 8,204 questions). Token counts use tiktoken

cl100k\_base

(GPT-4/5).

Model	Mean [95% CI]	Correct (%)	Wrong (%)	Partially correct (%)	Avg. Words
Physician	—	—	—	—	62
GPT-5	65.0 [62.9, 67.0]	41.2	11.3	47.5	386
Claude Haiku	54.4 [52.3, 56.6]	28.1	19.2	52.7	227
GPT-4o	53.4 [51.7, 55.2]	18.6	11.7	69.7	264
Gemini 2.5 Flash	45.7 [43.8, 47.6]	12.9	21.5	65.6	330
Llama 3.3 70B	42.2 [40.7, 43.9]	6.2	21.7	72.0	323

Table 2: Overall performance (948 QA pairs, 238 conversations). Bootstrap 95% CIs in brackets. Mean score is on a 0–100 scale (0 = wrong, 50 = partially correct, 100 = correct). All models produce responses 3.7–6.2

\times

longer than those of physicians.

Model	T0 ( $n$ =238)	T1 ( $n$ =238)	T2 ( $n$ =238)	T3–5 ( $n$ =210)	T6+† ( $n$ =24)
GPT-5	75.2 [71.4, 79.0]	64.3 [60.1, 68.5]	59.0 [54.6, 63.2]	61.4 [56.9, 66.0]	60.4 [47.9, 70.8]
Claude Haiku	72.3 [68.5, 75.8]	51.9 [47.7, 56.1]	47.3 [43.1, 51.5]	46.4 [41.9, 51.2]	43.8 [31.2, 56.2]
GPT-4o	58.2 [55.1, 61.3]	51.8 [48.2, 55.3]	50.2 [46.5, 53.9]	52.6 [48.8, 56.3]	50.0 [41.7, 58.3]
Gemini 2.5 Flash	51.3 [48.1, 54.6]	44.7 [41.2, 48.5]	42.6 [38.9, 46.4]	43.6 [39.3, 47.9]	47.9 [37.5, 58.3]
Llama 3.3 70B	47.1 [44.5, 49.6]	40.5 [37.4, 43.7]	40.1 [36.8, 43.5]	41.0 [37.1, 44.8]	43.8 [35.4, 52.1]

Table 3: Mean judge score (0–100 scale) with bootstrap 95% CIs by turn group. †: turns 6+ are indicative only (

n=24

). See Appendix A , table 7 for full significance tests.

Figure 3: Mean judge score by conversation turn (turns 0-5, n = 238, 238, 238, 109, 61, 40 respectively). Shaded bands show 95% confidence intervals. All models degrade from turn 0 to turn 2; the degradation is statistically significant across all models (p < 0.001, Mann-Whitney U).

Model	CCS [95% CI]	Floor	Ceiling	Volatile (%)	Degraded (%)
GPT-5	51.5 [47.7, 55.3]	39.1	87.6	16.8	57.1
Claude Haiku	45.2 [41.0, 49.4]	27.3	82.1	27.7	63.9
GPT-4o	60.5 [56.7, 64.3]	34.2	73.7	9.7	37.8
Gemini 2.5 Flash	57.1 [53.2, 61.1]	25.0	67.9	13.0	39.9
Llama 3.3 70B	63.9 [60.1, 67.4]	22.9	59.0	8.0	38.7

Table 4: Conversational Consistency Score (0–100 scale) with bootstrap 95% CIs (238 conversations,

\geq

3 turns). Floor / Ceiling : mean per-conversation worst- and best-case scores (0–100 scale). Volatile : conversation contains both a fully correct and a completely wrong response. Degraded : later-turn mean

>

10 points below turn-0 score.

Model	EPR [95% CI]	After Correct	Amplification
GPT-5	38.6% [27.1, 50.0]	6.3%	6.1 $\times$
Claude Haiku	40.7% [31.9, 50.4]	15.6%	2.6 $\times$
GPT-4o	35.1% [24.7, 46.8]	8.3%	4.2 $\times$
Gemini 2.5 Flash	44.1% [36.0, 52.2]	13.3%	3.3 $\times$
Llama 3.3 70B	40.9% [32.6, 49.2]	22.0%	1.9 $\times$

Table 5: Error Propagation Rate with bootstrap 95% CIs. After Correct : wrong rate given prior correct turn. Amplification : EPR / after-correct wrong rate.

Figure 4: Cascade failure rates: percentage of wrong responses at turn t + 1 conditioned on the score at turn t. All models show severe cascade effects after a wrong turn, with failure rates up to 6.1× higher than after a correct turn.

Model	T0	Baseline T2	Oracle T2	Q-difficulty †	Context effect
GPT-5	75.2	59.0	50.0	156% of drop	$-9.0$ 0 ( $p<0.001$ )
Claude Haiku	72.3	47.3	50.8	86% of drop	$+3.6$ 0 ( $p=0.024$ )
GPT-4o	58.2	50.2	45.4	160% of drop	$-4.8$ 0 ( $p=0.003$ )
Gemini 2.5 Flash	51.3	42.6	45.6	66% of drop	$+2.9^{\ddagger}$ 0 ( $p=0.36$ )
Llama 3.3 70B	47.1	40.1	35.7	164% of drop	$-4.4$ 0 ( $p=0.003$ )

Table 6: Oracle-physician ablation at turn 2. Q-difficulty : share of total degradation explained by question difficulty alone (baseline T0

-

oracle T2). Context effect : oracle T2

-

baseline T2; negative values mean the oracle condition is worse than baseline.

p

-values test baseline vs. oracle across turns 1–5 (Mann-Whitney U). † Values

>

100% indicate the oracle condition scores below baseline T2, implying the model’s own prior responses serve as net-positive scaffolding; question difficulty thus more than fully explains the total degradation. ‡ Gemini 2.5 Flash oracle vs. baseline difference is not significant (

p=0.36

); no directional claim is made for this model.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	T0 ( $n$ =238)	T1 ( $n$ =238)	T2 ( $n$ =238)	T3–5 ( $n$ =210)	T6+† ( $n$ =24)
GPT-5	75.2 [71.4, 79.0]	64.3 [60.1, 68.5] ∗∗∗	59.0 [54.8, 63.2] ∗∗∗	61.4 [56.9, 66.0] ∗∗∗	60.4 [50.0, 72.9] ∗∗
Claude Haiku	72.3 [68.5, 75.8]	51.9 [47.7, 56.1] ∗∗∗	47.3 [43.3, 51.5] ∗∗∗	46.4 [41.9, 51.2] ∗∗∗	43.8 [31.2, 56.2] ∗∗∗
GPT-4o	58.2 [55.1, 61.3]	51.8 [48.4, 55.3] ∗∗	50.2 [46.5, 53.9] ∗∗∗	52.6 [48.8, 56.3] ∗	50.0 [41.7, 58.3] ∗∗ ns
Gemini 2.5 Flash	51.3 [48.1, 54.6]	44.7 [41.0, 48.3] ∗∗	42.6 [38.9, 46.4] ∗∗∗	43.6 [39.5, 47.6] ∗∗	47.9 [37.5, 58.3] ∗∗ ns
Llama 3.3 70B	47.1 [44.5, 49.6]	40.5 [37.4, 43.7] ∗∗∗	40.1 [36.8, 43.5] ∗∗∗	41.0 [37.1, 44.5] ∗∗	43.8 [35.4, 52.1] ∗∗ ns

Appendix

Table 7: Mean judge score (0–100 scale) with bootstrap 95% CIs by turn group, with statistical significance of turn 0 vs. each later group (Mann-Whitney U, one-sided: turn 0

>

later group). †: turns 6+ are indicative only (

n=24

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

, ns: not significant.

Model	Input ($)	Output ($)	Total ($)
claude-haiku	0.45	2.43	2.88
gemini-2.5-flash	0.14	1.21	1.35
llama33_70b	0.40	0.43	0.83
gpt-4o	2.26	7.28	9.55
gpt-5	0.57	4.85	5.42

Appendix

Table 8: Inference cost for 5 models on 238 threads (948 QA pairs).

Judge model	Cost ($)
GPT-4o	30.98

Appendix

Table 9: LLM-as-judge cost (GPT-4o) for all 5 models’ outputs (4740 judgments).

Medical QA

Medical question answering develops and evaluates AI systems that answer clinical, patient-facing, and health-information questions.

See all 84 papers

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Sep 11, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Medical Diagnosis Multi-Turn Dialogue Evaluation
MBZUAI · Cairo University · Ain Shams University +2
Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations
Jul 14, 2026Monica Munnangi, Saiph SavageMulti-Turn Dialogue Evaluation Clinical Language Model Evaluation
Khoury College of Computer Sciences Northeastern University
PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue
Sep 23, 2026Chenxuan Li, Jiayi Wan, Xinrong Chen +3Clinical Reasoning Medical LLMs
Peking University, Beijing, China · Department of Medical Bioinformatics, School of Basic Medical Sciences, Peking University, Beijing, China

Sep 11, 2026J/K move · Enter open · S save

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang+12

MBZUAI · Cairo University · Ain Shams University+2

Medical Diagnosis Multi-Turn Dialogue Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss.

Abstract

Medical QA

Explore similar work

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations

PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue