CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations

Yusuf Kesmen; Aniruddha Mukherjee; Yena Chang; David Sasu; Trevor Brokowski; Alexandra V. Kulinkina; Kristina Keitel; Akhil Arora

cs.LGSep 28, 2026

CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations

Authors: Yusuf Kesmen, Aniruddha Mukherjee, Yena Chang, David Sasu, Trevor Brokowski, Alexandra V. Kulinkina, Kristina Keitel, Akhil Arora, +2 more

Organizations: EPFL · Aarhus University

Abstract

Patients often have several co-occurring clinical conditions, and the findings needed to identify and disambiguate them emerge over the course of a consultation. Evaluating clinical reasoning in this setting requires both multi-turn interaction and multi-label diagnosis. We introduce CLIMB, a benchmark in which a doctor model interviews a simulated patient to recover a ground truth set of co-occurring clinical conditions. Cases are synthesized from clinical decision algorithms and diagnostic datasets, grounding multimorbid presentations in structured clinical knowledge. Across six frontier and open models, none recovers the exact set of conditions in more than 10% of interactive cases. Diagnostic performance declines when conditions co-occur, even when models receive the full clinical record and the true number of conditions. Interaction reduces performance further. In controlled experiments, models behave like single-hypothesis trackers: they anchor on the diagnosis suggested by the opening findings, keep questioning around it, and recover a second condition mainly when a finding in view points to it. Questioning them further does not complete the set but adds mostly wrong diagnoses. We formalise this pattern with a theoretical reference model of single-hypothesis tracking. The benchmark, generator, and evaluation code are available at https://anonymous.4open.science/r/CLIMB-8340.

Figures & tables

Figure 1: Static and single-label settings against ours. Panels (a) and (b) provide all evidence upfront or elicit it for a single correct diagnosis, while CLIMB (c) elicits evidence over a multi-turn dialogue for a patient whose ground truth is a set of co-occurring diagnoses, so the model must hold several hypotheses at once.

Figure 2: Case construction. Diagnoses are sampled from cohort co-occurrence, their findings are taken from ePOCT+ paths or DDXPlus records and merged, and one or two unrelated complaints are added. A conflict resamples the diagnosis set.

	DDXPlus				ePOCT+
	Exact $\uparrow$		Jaccard $\uparrow$		Exact $\uparrow$		Jaccard $\uparrow$
Model	F	I	F	I	F	I	F	I
GPT-5.6	0.11	0.08	0.46	0.29	0.21	0.08	0.57	0.30
Gemini-3.8-Flash	0.26	0.08	0.47	0.32	0.21	0.10	0.51	0.39
Qwen3.8-Flash	0.17	0.06	0.46	0.27	0.18	0.06	0.49	0.24
GLM-5.3-Flash	0.11	0.02	0.46	0.24	0.15	0.05	0.52	0.26

Table 1: Pooled exact-set recovery and Jaccard under full information (F) and interaction (I). Rows are sorted by interactive exact-set recovery on DDXPlus; shading of the interactive exact-set columns is proportional to the value. Intervals are in Appendix E.1 .

Figure 3: Multiplicity on DDXPlus (six models, 200 cases per

k

). (a) Exact-set recovery, interactive (solid) and full information (dashed). (b) Mean returned and correct diagnoses per interactive consultation, models pooled. (c) Questions asked.

		$k{=}1$		$k{=}2$		$k{=}3$		$k{=}4$
Source	Model	F	I	F	I	F	I	F	I
DDXPlus	GPT-5.6	30	27	10	6	4	0	0	0
	Gemini-3.8-Flash	76	22	23	8	4	1	0	0
	Qwen3.8-Flash	44	14	22	9	4	1	0	0
	GLM-5.3-Flash	21	8	11	0	8	0	2	0
	DeepSeek-V4.1-Flash	9	4	4	1	1	0	0	0

Table 2: Exact-set recovery (%,

\uparrow

) by comorbidity count

k

. F is full information, I is interactive; shading of the interactive columns is proportional to the value. Intervals are in Appendix E.1 .

Figure 4: Single-answer gap in interaction (six models, 200 cases per

k

). Unlike the exact set, all true diagnoses may come with wrong ones. Bands are 95% bootstrap intervals.

Figure 5: GPT-5.6 and the reference tree on the matched respiratory cohort. (a) Exact-set recovery when each system asks and diagnoses. (b) Exact-set recovery when GPT-5.6 reads evidence of increasing quality. (c) Information gained, the drop in uncertainty about the diagnosis set after each question, scored by one fixed model for both systems (Appendix D.7 ). The dot marks where the tree stops on average, and the diamonds show the full record.

Figure 6: Anchoring on the designed cohort, GPT-5.6. A is the diagnosis whose findings open the consultation and B the other. (a) Recall of A and B. (b) Which true diagnoses the final set contains.

Figure 7: Single probes by question turn (DDXPlus, GPT-5.6, 50 cases per

k

). (a) Correct and (b) wrong diagnoses. (c) Jaccard; diamonds mark the committed answer at the median natural stop. Dotted lines mark

k

in (a) and the median stop in (b).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Answer type	Symptoms	Antecedents	Total
Binary	96	112	208
Categorical	9	1	10
Multi-select	5	0	5
Total	110	113	223

Appendix

Table 3: Evidence-variable schema in the local DDXPlus release. Antecedents encode medical history and risk factors. Counts refer to variables, not answer tokens.

Population	$k$	Cases	Unique labels	Distinct sets	Signal findings
ePOCT+	1	200	33	33	3 [3, 4]
	2	200	47	92	6 [5, 10]
	3	200	68	134	10 [8, 14]
	4	200	72	159	14 [11, 16]
DDXPlus merged	1	200	44	44	15 [11, 18]
	2	200	46	163	24 [21, 29]

Appendix

Table 4: Stored populations. Finding counts exclude added background and show median [first, third quartile]. Diagnosis-set counts ignore ordering. Each stratum has 200 cases.

Population	Cases	Unique labels	Age, median [Q1, Q3]	F (%)
ePOCT+	800	96	7.8 [3.8, 11.5]	49.5
DDXPlus merged	800	47	44 [28, 61]	51.2

Appendix

Table 5: Evaluated cases: all 200 stored cases at each

k

of each population. Label counts describe this subset, not the candidate vocabulary. Ages are generated ages in years; F is the recorded female category.

Figure 8: Finding counts in the ePOCT+ and merged DDXPlus populations, with 200 cases per box. Boxes show the interquartile range, central lines the median, and whiskers the full observed range. Separately injected background complaints are excluded.

$k$	Reader	Reader, $k$ given	Best model	GPT-5.6
1	99.5 [98.5, 100]	100	76.5	30.0
2	91.0 [87.0, 94.5]	96.5	23.0	10.0
3	79.5 [74.5, 85.5]	91.0	8.5	3.5
4	79.0 [73.0, 84.5]	89.5	2.5	0.5

Appendix

Table 6: Exact-set recovery (%) from the full DDXPlus record, 200 cases per

k

: reference reader without and with the true

k

, the best of the six models, and GPT-5.6. Brackets are 95% case-bootstrap intervals.

Panel of Figure 9 — Figure 9: A stored ePOCT+ case ( epoct_k2_81 ). Each diagnosis keeps at least one finding of its own; shared findings are retained once. Hidden routing and age conditions are omitted.

Model	DDXPlus	ePOCT+
GPT-5.6	none	none
Gemini-3.8-Flash	minimal	minimal
Qwen3.8-Flash	disabled	disabled
Gemma-4	disabled	disabled
GLM-5.3-Flash	minimal	minimal
DeepSeek-V4.1-Flash	disabled	disabled

Appendix

Table 7: Reasoning settings of the doctor models in the main evaluation.

Candidate	Present	Absent	Score	USD / probe
Gemini-3.8-Flash	1.00	0.93	0.97	$1.4\times 10^{-3}$
Qwen3.8-Flash	1.00	0.87	0.93	$2.2\times 10^{-4}$
DeepSeek-V4.1-Flash	1.00	0.83	0.92	$1.5\times 10^{-4}$
Qwen3.8-27B	1.00	0.83	0.92	$1.6\times 10^{-3}$
Ling-3.0-Flash	1.00	0.73	0.87	$1.8\times 10^{-5}$
GPT-5.6-Luna	0.97	0.77	0.87	$1.3\times 10^{-4}$

Appendix

Table 8: Patient-selection screen: accuracy on present and absent findings, their mean, and cost per probe. No candidate named an unsupported diagnosis.

Task	Doctor prompt	Evidence supplied
full	DX_SYSTEM , dx_user	Opening and all retained findings.
tree	Same as full	Opening and answers along the stored tree path.
replay	Same as full	Opening and recorded question–answer pairs.
interactive	free_doctor_sys	Dialogue with patient_sys .
interactive_single	single_doctor_sys	Same patient; single-diagnosis instruction.
interactive_count	count_doctor_sys	Same patient; true count disclosed.

Appendix

Table 9: Task entry points in experiments/tasks.py . All diagnosis outputs use the shared candidate-name scorer.

		Tree leaf		Fixed reader		Questions
Max depth	Leaves	$k=1$	$k=2$	$k=1$	$k=2$	$k=1$	$k=2$
3	8	58.0	11.3	49.7	11.6	3.0	3.0
5	28	83.3	40.5	87.5	45.5	4.9	4.9
6	47	87.8	54.8	88.4	55.4	5.4	5.5
8	109	92.9	68.5	90.2	64.9	6.6	6.3
10	203	95.5	77.4	90.5	66.7	8.0	7.0

Appendix

Table 10: Tree depth on the validation partition with a 20-question budget. Exact-set recovery (%) of the tree’s leaf and of the fixed reader on the tree’s answers, and mean questions asked. The reference tree is the unrestricted fit.

Evidence from	Diagnosed by	Reported as
GPT-5.6’s questions	GPT-5.6 in the consultation	Figure 5 a
Tree’s questions	Tree leaf	Figure 5 a
GPT-5.6’s questions	Fixed reader	Figure 5 b
Tree’s questions	Fixed reader	Figure 5 b
GPT-5.6’s questions, as a list	Fresh GPT-5.6 readout	Figure 5 c
Tree’s questions, as a list	Fresh GPT-5.6 readout	Figure 5 c

Appendix

Table 11: Cells of the evidence cross-over. Rows one and two are the direct comparison; rows three and four vary the evidence with the reader fixed; rows five to seven vary the evidence with GPT-5.6 as the reader.

		$k=1$			$k=2$
Evidence from	Reader	Exact	Recall	FP	Exact	Recall	FP
GPT-5.6’s questions	GPT-5.6, in consultation	35.7	69.6	0.75	17.3	58.3	0.77
Tree’s questions	Tree leaf	92.9	–	–	81.5	–	–
GPT-5.6’s questions	Fixed reader	73.8	91.7	0.21	45.8	77.1	0.15
Tree’s questions	Fixed reader	89.3	92.9	0.04	73.2	90.5	0.14
Full record	Fixed reader	85.7	96.4	0.11	78.6	94.0	0.11

Appendix

Table 12: Exact-set recovery (%), label recall (%), and false positives per consultation for every cell of the cross-over; 168 consultations per

k

. The tree’s leaf prediction is scored on exact-set recovery only.

Figure 10: Exact-set recovery by the fixed reader from GPT-5.6’s first

n

answers and from the tree capped at

n

questions. Bands are 95% intervals.

Figure 11: GPT-5.6 and the tree on the matched respiratory cohort. (a) Exact-set recovery when a fixed classifier reads each system’s evidence or the full record. (b) The observer’s belief on the true set after each question. After the tree stops, its last value is carried forward, and the diamonds show the full record.

		Exact $\uparrow$		Jaccard $\uparrow$
Source	Model	F	I	F	I
DDXPlus	GPT-5.6	0.11 [0.09, 0.13]	0.08 [0.07, 0.10]	0.46 [0.44, 0.48]	0.29 [0.27, 0.31]
	Gemini-3.8-Flash	0.26 [0.24, 0.28]	0.08 [0.06, 0.10]	0.47 [0.45, 0.50]	0.32 [0.30, 0.34]
	Qwen3.8-Flash	0.17 [0.15, 0.20]	0.06 [0.05, 0.08]	0.46 [0.44, 0.48]	0.27 [0.25, 0.29]
	GLM-5.3-Flash	0.11 [0.09, 0.13]	0.02 [0.01, 0.03]	0.46 [0.44, 0.48]	0.24 [0.23, 0.26]
	DeepSeek-V4.1-Flash	0.04 [0.03, 0.05]	0.02 [0.01, 0.03]	0.37 [0.35, 0.38]	0.25 [0.23, 0.26]

Appendix

Table 13: Table 1 with 95% bootstrap intervals, 800 cases per source.

Source	Model		$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$
DDXPlus	GPT-5.6	F	30 [24, 36]	10 [6, 14]	4 [2, 6]	0 [0, 2]
		I	27 [21, 33]	6 [3, 10]	0 [0, 2]	0 [0, 0]
	Gemini-3.8-Flash	F	76 [70, 82]	23 [18, 29]	4 [2, 8]	0 [0, 2]
		I	22 [16, 28]	8 [4, 12]	1 [0, 2]	0 [0, 2]
	Qwen3.8-Flash	F	44 [37, 51]	22 [16, 28]	4 [2, 7]	0 [0, 2]
		I	14 [10, 20]	9 [6, 13]	1 [0, 2]	0 [0, 0]

Appendix

Table 14: Table 2 with 95% bootstrap intervals: exact-set recovery (%) under full information (F) and interaction (I), 200 cases per

k

Condition	Model	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$
full	GPT-5.6	0.65/0.55/0.30	0.58/0.45/0.10	0.57/0.43/0.04	0.56/0.41/0.01
full	Gemini-3.8-Flash	0.79/0.79/0.77	0.58/0.48/0.23	0.47/0.35/0.04	0.40/0.28/0.01
full	Qwen3.8-Flash	0.70/0.64/0.44	0.57/0.47/0.21	0.53/0.41/0.04	0.48/0.34/0.01
full	GLM-5.3-Flash	0.61/0.50/0.21	0.57/0.45/0.11	0.59/0.46/0.09	0.57/0.43/0.03
full	DeepSeek-V4.1-Flash	0.50/0.37/0.09	0.51/0.37/0.04	0.50/0.36/0.01	0.50/0.36/0.01
full	Gemma-4	0.63/0.53/0.26	0.56/0.43/0.10	0.52/0.38/0.04	0.53/0.39/0.01

Appendix

Table 15: Per-count scores on DDXPlus for the six models. Each cell reads F1 / Jaccard / exact over 200 cases.

Figure 12: Interactive outcomes by

k

(DDXPlus, 200 cases per

k

). A partially correct set is a subset of the truth; a partially incorrect one adds wrong labels.

Figure 13: Set size (DDXPlus, six models, 200 cases per

k

). (a) Predicted count against

k

. (b) Exact set, right count with a wrong set, or wrong count.

Panel of Figure 14 — Figure 14: Oracle count (50 cases per $k$ ). Left, exact-set recovery with the true $k$ supplied. Right, change in Jaccard against standard interaction.

Figure 15: Precision and recall by

k

on DDXPlus, mean of six models (200 cases per

k

Figure 16: Jaccard by

k

on DDXPlus, mean of six models (200 cases per

k

Model	Interaction [95% CI]
GPT-5.6	$+0.108$ $[+0.049,+0.166]$
Gemini-3.8-Flash	$-0.242$ $[-0.303,-0.185]$
GLM-5.3-Flash	$+0.033$ $[-0.015,+0.081]$
Gemma-4	$-0.038$ $[-0.087,+0.010]$
Qwen3.8-Flash	$-0.076$ $[-0.134,-0.021]$
DeepSeek-V4.1-Flash	$+0.065$ $[+0.027,+0.104]$

Appendix

Table 16: Count

\times

condition interaction on Jaccard, DDXPlus, 200 cases per

k

	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$
hit@1	0.54	0.56	0.34	0.42
mean count	1.00	1.00	1.00	1.00
chief selected	–	0.38	0.28	0.18
$\tau$	11.4	11.5	10.9	11.3

Appendix

Table 17: Single-diagnosis instruction for Gemma across

k

, 50 cases per

k

. hit@1 is the fraction of cases where the one guess names any true diagnosis. chief is the fraction where the guess is the presenting-complaint disease.

Figure 17: Gemma on DDXPlus (50 cases per

k

). Single-answer hits versus exact sets with full information, in interaction and with the oracle count.

Evidence	Reader	Alone	With second	Change
Full record	GPT-5.6	85.7	77.4	$-8.3\;[-13.7,-3.0]$
Full record	Fixed reader	96.4	92.9	$-3.6\;[-6.5,-0.6]$
Tree’s findings	GPT-5.6	81.0	69.6	$-11.3\;[-17.9,-4.8]$
Tree’s findings	Fixed reader	92.9	90.5	$-2.4\;[-6.0,1.2]$
GPT-5.6’s findings	GPT-5.6	70.2	61.3	$-8.9\;[-15.5,-2.4]$
GPT-5.6’s findings	Fixed reader	91.7	75.6	$-16.1\;[-21.4,-10.7]$

Appendix

Table 18: Recovery (%) of the same focal diagnosis alone (

k=1

) and with a second disease (

k=2

, opening on the focal diagnosis), with the paired change in points.

Contrast	$k=2$ minus $k=1$
Fixed reader, exact set (Figure 5 b)	$+11.9$ pp $[3.6,20.2]$
Fixed reader, label recall	$+12.2$ pp $[7.1,17.6]$
Fixed reader, label log loss	$0.62$ bits $[0.31,0.94]$
Fixed reader, exact set, without rhinosinusitis	$+13.8$ pp $[2.5,23.8]$
Direct tree versus GPT-5.6, exact set	$+7.1$ pp $[-3.0,16.7]$

Appendix

Table 19: Change from

k=1

k=2

in the tree’s advantage over GPT-5.6.

	GPT-5.6	Gemma-4
Recall of A	0.65 $[0.60,0.69]$	0.58 $[0.52,0.62]$
Recall of B	0.32 $[0.27,0.36]$	0.41 $[0.37,0.46]$
A minus B, paired, points	$+33$ $[27,38]$	$+16$ $[12,21]$
Final set holds A and B	0.17 $[0.13,0.22]$	0.24 $[0.19,0.29]$
Final set holds A only	0.47 $[0.43,0.52]$	0.33 $[0.29,0.37]$
Final set holds B only	0.15 $[0.12,0.18]$	0.17 $[0.14,0.20]$

Appendix

Table 20: Designed

k=2

cohort, both openings pooled, with 95% intervals.

Opening contrast	Patients	B named	A named	Both named
Shared minus A’s symptoms	97	$+13$ $[4,24]$	$-4$ $[-12,4]$	$+5$ $[-4,14]$
Shared minus B’s symptoms	65	$-5$ $[-17,8]$	$+9$ $[2,17]$	$+2$ $[-7,11]$
B’s minus A’s symptoms	65	$+17$ $[8,27]$	$-12$ $[-22,-3]$	$+2$ $[-3,7]$

Appendix

Table 21: Symptom-sharing cohort, GPT-5.6, diagnoses of severity 3 to 5. Differences between openings in points, with 95% intervals.

Figure 18: Symptom-sharing cohort, GPT-5.6, diagnoses of severity 3 to 5. The same patients open with A’s first finding and two symptoms of A alone (not shared) or two that both diagnoses share. Share of consultations naming A, B and both.

Figure 19: Symptom-sharing cohort, GPT-5.6. Change from the opening with A’s symptoms within the same patients, for shared symptoms (the patients of Figure 18 ) and for B’s symptoms as a control (the patients who also had that opening), with 95% intervals over disease pairs.

Figure 20: Share of probe sets holding the opening diagnosis and an equally severe second one (DDXPlus, four open models, 50 cases per

k

). Bands are 95% bootstrap intervals.

Model	natural	probe at stop	probe at cap	refusals	forced turns
Gemma-4	0.426	0.457	0.424	404	7.6
GLM	0.401	0.387	0.363	1126	7.1
Qwen	0.309	0.357	0.373	64	5.5
DeepSeek	0.354	0.381	0.359	682	8.9

Appendix

Table 22: Post-stop readouts. natural is the F1 of the voluntary committed answer; stop and cap are the F1 of the single probe at the natural stop and at the last forced question. Refusals count post-stop turns where the model committed instead of asking. Four open models, 50 cases per

k

Figure 21: Composition of a single probe by question turn (four open models, 50 cases per

k

). Dashed lines mark

k

, dotted lines the median stop.

Figure 22: Left, single-probe exact-set recovery by

k

. Right, probes aligned at the natural stop

\tau

(586 consultations). Four open models, 50 cases per

k

Figure 23: Figure 7 per model (50 cases per

k

). Probes listing 40 or more labels are excluded.

Explore similar work

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Jul 28, 2026Rui Yang, Weihao Xuan, Yi Lin +23Multimodal Clinical Data Diagnosis
Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA +24
MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Sep 14, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Diagnostic Benchmark
MBZUAI · Cairo University · Ain Shams University +2
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
Sep 29, 2026Xueting Fang, Zehui Li, Yang Yang +5Clinician Trust Diagnostic Benchmark
Zhejiang University · Imperial College London · Nanchang University +2

Jul 28, 2026, cs.CLJ/K move · Enter open · S save

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

Rui Yang, Weihao Xuan, Yi Lin, Zhuhan Bao+22

Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA+24

Multimodal Clinical Data Diagnosis

Open paper AskPDF arXiv

Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating of diagnostic hypotheses, and continuous refinement of clinical reasoning. However, existing evaluations of multimodal large language models (MLLMs) typically rely on single-turn or isolated tasks, making it difficult to fully capture the complexity of real-world clinical diagnosis. To bridge this gap, we developed ClinMM-Bench, the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date. ClinMM-Bench contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties. We systematically evaluated 15 representative MLLMs using a two-level evaluation framework that assessed both diagnostic accuracy and diagnostic reasoning quality. Results showed that proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited across all models. In terms of diagnostic reasoning quality, current models can identify plausible diagnostic directions but still have considerable limitations in generating reliable diagnostic reasoning. Error analysis further identified five representative failure modes: information synthesis failure, knowledge mapping error, perception error, premature closure, and visual hallucination.

Abstract

Explore similar work

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy