AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

Alexey A. Bokov; Ivan I. Panin; Nikolay B. Nechaev; Mikhail Yu. Valkov; Valeria Yu. Chernina; Natalia Gombolevskaya; Ayyaz Hussain; Vladimir M. Agafonov

cs.CLOct 15, 2026

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

Authors: Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev, Mikhail Yu. Valkov, Valeria Yu. Chernina, Natalia Gombolevskaya, Ayyaz Hussain, Vladimir M. Agafonov,

Organizations: AIRI, Moscow, Russian Federation · Northern State Medical University, Arkhangelsk, Russian Federation · Arkhangelsk Clinical Oncological Dispensary, Arkhangelsk, Russian Federation · Intelligent Radiology Assistance Laboratories LLC (IRA LABS), Moscow, Russian Federation ·

Abstract

Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, and safe. We developed a multidimensional benchmark using 300 diagnostically resolved NEJM--MedQA cases: 110 for physician reference ratings and Judge calibration and 190 for held-out evaluation. Eight single-model pipelines generated independent role-specific opinions and interactive-consilium reports under free-form and structured clinical-reasoning conditions. A physician-calibrated LLM Judge evaluated 3,040 final reports and the corresponding structured primary-role outputs. Model identity separated report quality more strongly than response format. Structured final reports had higher overall mean scores in seven of eight model pipelines, with their clearest advantage in management (2.740 vs. 2.436). Compared with the structured pre-consilium primary-role opinion, the complete structured synthesis had higher pooled scores across all five shared criteria, with the largest increases in problem prioritization, factual grounding, and reasoning coherence. GPT-5.6-Sol led every evaluated component and reached consensus in the fewest review rounds. These findings show that the benchmark can distinguish model capability, the contribution of multidisciplinary synthesis, criterion-specific effects of response structure, deliberation dynamics, and inference requirements within one common-case evaluation.

Figures & tables

Panel of Figure 1 — Figure 1: Generation and physician-evaluation workflow. Case-specific roles produced independent pre-consilium opinions before interactive multidisciplinary synthesis. Physicians scored role-specific clinical reasoning, role-specific clinical opinion, and the final consilium conclusion as three separate components using the same eight 0–3 criteria. Panel adequacy, case complexity, and overall quality across all stages were rated separately.

No.	Criterion	Evaluation target
1	Factual Correctness and Source Grounding	Agreement of clinically material claims with the supplied case, without fabricated patient facts or consequential distortion.
2	Identification and Prioritization of Clinical Problems	Recognition of the principal and decision-relevant problems and their clinically justified ordering.
3	Completeness and Relevance of Data Use	Integration of supplied information that affects conclusions or management, without penalizing information absent from the input.
4	Coherence of Clinical Reasoning	Validity of links from confirmed facts to conclusions, including mechanisms, chronology, alternatives, and uncertainty.
5	Quality of the Differential Diagnosis	Relevance and prioritization of hypotheses, evidence for and against them, and concrete discriminating checks.
6	Quality of the Final Clinical Conclusion	Clarity, responsiveness, evidence grounding, agreement with the reference diagnosis, and calibration to uncertainty.

Table 1: Prespecified evaluation criteria.

Characteristic	Annotation/Judge calibration	Held-out evaluation	$\|$ SMD $\|$
Case records / patients	110 / 110	190 / 191	-
Age, median [IQR]; range, years	44.0 [26.3–60.8]; 0–89	48.0 [29.0–64.5]; 0–84	0.107
Younger than 18 years	15/110 (13.6%)	23/191 (12.0%)	0.048
18–64 years	76/110 (69.1%)	120/191 (62.8%)	0.132
65 years or older	19/110 (17.3%)	48/191 (25.1%)	0.193
Male	59/110 (53.6%)	105/191 (55.0%)	0.027

Table 2: Characteristics of the physician-annotation/Judge-calibration and held-out evaluation cohorts.

Panel of Figure 2 — Figure 2: Held-out cohort flow and model-generated panel architecture. (a) Only the 190 held-out cases entered the eight-model evaluation. Shared role assignments fed separate free-form and structured outputs; counts denote cases and artifacts, not enrolled participants. (b) Cell values are percentages within each model; the pooled row summarizes all 1,520 panels. (c) Categories are ranked by all 7,271 assignments and use separate primary and supporting denominators. The complete exact-label mapping is retained with the generated artifacts.

Panel of Figure 3 — Figure 3: Interactive-consilium dynamics in 3,040 case–model–pipeline arms. (a) Cumulative percentage of 1,520 arms per response format with unanimous acceptance of the reviewed report; round 1 evaluates the initial report. Counts show arms entering each round. (b) Mean reduction in dissent among the same arms before and after each revision, with 95% CIs from 10,000 DOI-clustered bootstrap resamples; the right column gives the number of arms compared. (c) Free-form (blue circles) and structured (orange squares) outcomes by model; lines connect response formats within models, and the final row pools the fixed eight-model set.

Model	Pipeline	R1 dissent, mean %	R1 unanimous, $n$ (%)	Mean rounds	Report consensus, $n$ (%)	No-change stop, $n$ (%)	Round-limit stop, $n$ (%)
DeepSeek-V4-Pro	Free-form	29.7	58 (30.5%)	1.77	189 (99.5%)	1 (0.5%)	0 (0.0%)
	Structured	7.5	144 (75.8%)	1.27	189 (99.5%)	1 (0.5%)	0 (0.0%)
Gemma-4-31B	Free-form	20.4	110 (57.9%)	1.44	190 (100.0%)	0 (0.0%)	0 (0.0%)
	Structured	30.9	87 (45.8%)	1.56	190 (100.0%)	0 (0.0%)	0 (0.0%)
GLM-5.2	Free-form	41.5	33 (17.4%)	1.95	189 (99.5%)	1 (0.5%)	0 (0.0%)
	Structured	38.8	40 (21.1%)	1.83	186 (97.9%)	4 (2.1%)	0 (0.0%)

Table 3: Model-specific controller-recorded interactive-consilium outcomes in the held-out evaluation cohort.

Transition	R1 $\rightarrow$ R2	R2 $\rightarrow$ R3	R3 $\rightarrow$ R4	R4 $\rightarrow$ R5
no $\rightarrow$ yes	4,582 (83.1%)	937 (79.9%)	278 (66.0%)	153 (62.4%)
no $\rightarrow$ no	932 (16.9%)	235 (20.1%)	143 (34.0%)	92 (37.6%)
yes $\rightarrow$ no	246 (7.4%)	194 (18.5%)	109 (29.7%)	31 (17.9%)
yes $\rightarrow$ yes	3,064 (92.6%)	854 (81.5%)	258 (70.3%)	142 (82.1%)

Table 4: Role-level vote transitions among interactive consilia continuing to the next review round.

Panel of Figure 4 — Figure 4: Controller-recorded stop reasons by response format. Each continuous 100% stacked bar partitions 1,520 interactive consilia into mutually exclusive stop reasons; labels give counts and percentages. Values for the narrow round-limit and no-change segments are repeated at right with matching color keys. Consensus indicates unanimous acceptance of the reviewed shared report. A no-substantive-change stop indicates that the moderator identified no further clinically meaningful revision; a round-limit stop indicates persistent dissent through review round 5.

Panel of Figure 5 — Figure 5: Provider-reported token use in the eight single-model evaluation pipelines. Blue and orange segments consistently denote prompt/input and completion/output tokens, respectively, in both panels; end labels give their summed mean in thousands. a , Mean complete-pipeline token use per case by model and response format. Each format-specific path includes role assignment; the same case–model role assignment was reused by both formats. b , Mean token use per case by pipeline stage and response format, averaged over the fixed eight-model set. The shared role-assignment value is repeated to show the resource requirement of a standalone format-specific path, but is counted only once in the experiment-wide total. Token units are model-tokenizer-specific and do not represent model-independent computational operations. F denotes free-form and S structured.

	Training re-evaluation MAE		Validation MAE
Criterion	Best initial	Best descendant	Best initial	Best descendant
1	$1.0494\pm 0.1132$	$0.6296\pm 0.1481$	$1.2222\pm 0.1925$	$0.6296\pm 0.1283$
2	$0.5556\pm 0.0980$	$0.5556\pm 0.0980$	$0.6296\pm 0.0642$	$0.7037\pm 0.0642$
3	$0.7963\pm 0.1604$	$0.6667\pm 0.0556$	$0.7778\pm 0.0962$	$0.6667\pm 0.1667$
4	$0.8642\pm 0.0855$	$0.8395\pm 0.0771$	$0.8889\pm 0.0000$	$0.8889\pm 0.0000$
5	$0.8889\pm 0.1111$	$0.5926\pm 0.0642$	$0.7222\pm 0.0962$	$0.5556\pm 0.0962$

Table 5: Criterion-specific MAE of the best initial candidates and evolved descendants in the criterion-local calibration run.

Model	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
GPT-5.6-Sol	$2.892\pm 0.409$	$2.486\pm 0.906$	$2.708\pm 0.700$	$2.772\pm 0.643$
GPT-OSS-120B	$2.417\pm 0.887$	$1.983\pm 1.108$	$2.543\pm 0.861$	$2.569\pm 0.839$
DeepSeek-V4-Pro	$2.441\pm 0.841$	$2.033\pm 1.053$	$2.514\pm 0.833$	$2.526\pm 0.863$
GLM-5.2	$2.462\pm 0.806$	$2.082\pm 1.036$	$2.491\pm 0.793$	$2.516\pm 0.800$
Qwen3.5-27B	$2.259\pm 0.937$	$1.943\pm 1.086$	$2.448\pm 0.894$	$2.451\pm 0.885$
Qwen3-235B	$2.323\pm 0.948$	$1.863\pm 1.084$	$2.365\pm 0.929$	$2.389\pm 0.988$

Table 6: Descriptive Judge-score summaries across the three evaluation components in the eight-model evaluation on 190 cases per model.

Model	Format	Mean Judge score	Input tokens/case (thousands)	Output tokens/case (thousands)	Total tokens/case (thousands)
GPT-5.6-Sol	Free-form	2.708	55.3	9.9	65.2
	Structured	2.772	108.5	23.5	132.1
GPT-OSS-120B	Free-form	2.543	164.9	42.3	207.2
	Structured	2.569	163.8	45.5	209.3
DeepSeek-V4-Pro	Free-form	2.514	84.5	14.0	98.4
	Structured	2.526	87.0	17.0	103.9

Table 7: Judge-rated quality and observed inference requirements by model and response format.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$1.743\pm 1.251$	$2.016\pm 1.169$	$2.353\pm 1.052$	$2.393\pm 1.055$
2: Problem prioritization	$2.618\pm 0.570$	$2.313\pm 0.672$	$2.772\pm 0.477$	$2.828\pm 0.406$
3: Data use	$2.517\pm 0.607$	-	$2.692\pm 0.506$	$2.636\pm 0.533$
4: Reasoning coherence	$2.517\pm 0.851$	$2.462\pm 0.803$	$2.822\pm 0.564$	$2.819\pm 0.566$
5: Differential diagnosis	$2.735\pm 0.670$	-	$2.807\pm 0.509$	$2.765\pm 0.483$
6: Clinical conclusion	-	$1.270\pm 1.283$	$1.416\pm 1.149$	$1.381\pm 1.208$

Table 8: Criterion-specific profiles pooled across the eight models.

Figure 6: Distribution of Judge outcomes for multidisciplinary conclusions by criterion. Each horizontal bar represents all 3,040 model-format opportunities for one criterion; the score categories and not-evaluable outcomes therefore sum to 100%.

Figure 7: Distribution of case-level mean Judge scores for multidisciplinary conclusions. For each case and criterion, the mean was calculated over numeric scores from eight models and two response formats. Bars count cases in 0.5-point bins; vertical red lines show the median. Not-evaluable outcomes were excluded from the means.

Completed rounds	Outputs	C1–C8 output mean Mean $\pm$ population SD	Output mean $<2$ No. (%)	C10 complexity Mean $\pm$ population SD
1	1,214	2.525 $\pm$ 0.326	62 (5.1)	2.148 $\pm$ 0.543
2	1,377	2.419 $\pm$ 0.338	123 (8.9)	2.131 $\pm$ 0.537
$\geq$ 3	449	2.438 $\pm$ 0.330	33 (7.3)	2.158 $\pm$ 0.494

Table 9: Final-answer Judge scores and case complexity by completed review rounds.

Figure 8: Judge scores and case complexity by completed consilium review rounds. Panels show descriptive summaries for one, two, and at least three rounds. Error bars in panels A and C show population SD; panel B shows the proportion of outputs with a mean score below 2. The mean in panels A and B averages available numeric scores for criteria 1–8; criterion 10 was analyzed separately.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Format	Applicable criteria	No. of criteria	Opportunities per model
Role reasoning	Structured	1, 2, 3, 4, 5, 7, 8	7	1330
Role clinical opinion	Structured	1, 2, 4, 6, 8	5	950
Conclusion	Free-form	1, 2, 3, 4, 5, 6, 7, 8	8	1520
Conclusion	Structured	1, 2, 3, 4, 5, 6, 7, 8	8	1520

Appendix

Table 10: Criterion applicability by evaluation component.

Model	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
GPT-OSS-120B	1330/1330	949/950	1498/1520	1491/1520
Qwen3.5-27B	1330/1330	950/950	1502/1520	1503/1520
Qwen3-235B	1330/1330	950/950	1513/1520	1509/1520
Gemma-4-31B	1330/1330	950/950	1418/1520	1503/1520
Qwen3-Next-80B	1330/1330	950/950	1501/1520	1500/1520
GPT-5.6-Sol	1330/1330	949/950	1518/1520	1519/1520

Appendix

Table 11: Numeric Judge-score coverage by model and evaluation component.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	1520/1520	1520/1520	1520/1520	1520/1520
2: Problem prioritization	1520/1520	1520/1520	1520/1520	1520/1520
3: Data use	1520/1520	-	1520/1520	1520/1520
4: Reasoning coherence	1520/1520	1518/1520	1520/1520	1520/1520
5: Differential diagnosis	1520/1520	-	1480/1520	1518/1520
6: Clinical conclusion	-	1520/1520	1355/1520	1387/1520

Appendix

Table 12: Criterion-specific numeric Judge-score coverage pooled across the eight models.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$1.789\pm 1.196$	$2.021\pm 1.147$	$2.468\pm 0.868$	$2.674\pm 0.807$
2: Problem prioritization	$2.616\pm 0.547$	$2.463\pm 0.646$	$2.879\pm 0.326$	$2.895\pm 0.339$
3: Data use	$2.642\pm 0.560$	-	$2.805\pm 0.396$	$2.732\pm 0.455$
4: Reasoning coherence	$2.584\pm 0.795$	$2.593\pm 0.719$	$2.937\pm 0.334$	$2.911\pm 0.394$
5: Differential diagnosis	$2.732\pm 0.686$	-	$2.878\pm 0.403$	$2.718\pm 0.462$
6: Clinical conclusion	-	$1.000\pm 1.240$	$1.294\pm 1.209$	$1.258\pm 1.216$

Appendix

Table 13: Model-specific criterion profile for GPT-OSS-120B.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$1.463\pm 1.208$	$1.953\pm 1.153$	$2.442\pm 0.997$	$2.516\pm 0.922$
2: Problem prioritization	$2.568\pm 0.601$	$2.226\pm 0.708$	$2.795\pm 0.441$	$2.811\pm 0.430$
3: Data use	$2.368\pm 0.642$	-	$2.663\pm 0.525$	$2.505\pm 0.578$
4: Reasoning coherence	$2.363\pm 0.906$	$2.437\pm 0.770$	$2.905\pm 0.425$	$2.884\pm 0.456$
5: Differential diagnosis	$2.595\pm 0.794$	-	$2.788\pm 0.535$	$2.816\pm 0.450$
6: Clinical conclusion	-	$1.174\pm 1.296$	$1.331\pm 1.171$	$1.283\pm 1.270$

Appendix

Table 14: Model-specific criterion profile for Qwen3.5-27B.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$1.332\pm 1.257$	$1.689\pm 1.207$	$1.889\pm 1.202$	$1.895\pm 1.273$
2: Problem prioritization	$2.679\pm 0.510$	$2.237\pm 0.617$	$2.811\pm 0.405$	$2.874\pm 0.332$
3: Data use	$2.547\pm 0.558$	-	$2.747\pm 0.481$	$2.711\pm 0.498$
4: Reasoning coherence	$2.379\pm 0.925$	$2.305\pm 0.907$	$2.663\pm 0.748$	$2.647\pm 0.779$
5: Differential diagnosis	$2.789\pm 0.639$	-	$2.847\pm 0.507$	$2.879\pm 0.424$
6: Clinical conclusion	-	$1.079\pm 1.169$	$1.228\pm 1.012$	$1.257\pm 1.168$

Appendix

Table 15: Model-specific criterion profile for Qwen3-235B.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$1.737\pm 1.220$	$2.132\pm 1.142$	$2.789\pm 0.587$	$2.647\pm 0.825$
2: Problem prioritization	$2.416\pm 0.689$	$2.105\pm 0.695$	$2.442\pm 0.707$	$2.695\pm 0.504$
3: Data use	$2.179\pm 0.665$	-	$2.532\pm 0.577$	$2.468\pm 0.559$
4: Reasoning coherence	$2.621\pm 0.756$	$2.316\pm 0.765$	$2.768\pm 0.623$	$2.889\pm 0.415$
5: Differential diagnosis	$2.674\pm 0.680$	-	$2.480\pm 0.716$	$2.584\pm 0.514$
6: Clinical conclusion	-	$1.332\pm 1.306$	$1.450\pm 1.071$	$1.387\pm 1.265$

Appendix

Table 16: Model-specific criterion profile for Gemma-4-31B.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$0.916\pm 1.068$	$1.568\pm 1.241$	$1.621\pm 1.258$	$1.579\pm 1.286$
2: Problem prioritization	$2.532\pm 0.540$	$2.174\pm 0.708$	$2.716\pm 0.546$	$2.732\pm 0.499$
3: Data use	$2.458\pm 0.612$	-	$2.705\pm 0.500$	$2.516\pm 0.596$
4: Reasoning coherence	$1.953\pm 1.012$	$2.268\pm 0.933$	$2.532\pm 0.844$	$2.500\pm 0.844$
5: Differential diagnosis	$2.505\pm 0.851$	-	$2.809\pm 0.521$	$2.637\pm 0.657$
6: Clinical conclusion	-	$1.021\pm 1.261$	$1.189\pm 1.168$	$1.123\pm 1.181$

Appendix

Table 17: Model-specific criterion profile for Qwen3-Next-80B.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$2.737\pm 0.721$	$2.532\pm 0.933$	$2.700\pm 0.801$	$2.853\pm 0.571$
2: Problem prioritization	$2.874\pm 0.417$	$2.732\pm 0.455$	$2.937\pm 0.243$	$2.947\pm 0.223$
3: Data use	$2.900\pm 0.300$	-	$2.758\pm 0.440$	$2.858\pm 0.378$
4: Reasoning coherence	$2.947\pm 0.320$	$2.810\pm 0.587$	$2.958\pm 0.287$	$2.968\pm 0.249$
5: Differential diagnosis	$3.000\pm 0.000$	-	$2.947\pm 0.246$	$2.953\pm 0.212$
6: Clinical conclusion	-	$1.879\pm 1.249$	$1.766\pm 1.202$	$1.688\pm 1.114$

Appendix

Table 18: Model-specific criterion profile for GPT-5.6-Sol.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$1.947\pm 1.243$	$2.084\pm 1.135$	$2.395\pm 1.035$	$2.263\pm 1.107$
2: Problem prioritization	$2.626\pm 0.535$	$2.216\pm 0.641$	$2.800\pm 0.400$	$2.889\pm 0.314$
3: Data use	$2.547\pm 0.567$	-	$2.700\pm 0.502$	$2.653\pm 0.529$
4: Reasoning coherence	$2.653\pm 0.771$	$2.468\pm 0.779$	$2.900\pm 0.430$	$2.805\pm 0.606$
5: Differential diagnosis	$2.784\pm 0.599$	-	$2.914\pm 0.363$	$2.900\pm 0.391$
6: Clinical conclusion	-	$1.411\pm 1.310$	$1.574\pm 1.141$	$1.598\pm 1.241$

Appendix

Table 19: Model-specific criterion profile for DeepSeek-V4-Pro.

Criterion	Role reasoning Structured	Role clinical opinion Structured	Conclusion Free-form	Conclusion Structured
1: Factual grounding	$2.026\pm 1.149$	$2.153\pm 1.102$	$2.516\pm 0.950$	$2.716\pm 0.698$
2: Problem prioritization	$2.637\pm 0.580$	$2.353\pm 0.654$	$2.795\pm 0.429$	$2.784\pm 0.448$
3: Data use	$2.495\pm 0.613$	-	$2.626\pm 0.555$	$2.647\pm 0.520$
4: Reasoning coherence	$2.637\pm 0.768$	$2.505\pm 0.773$	$2.916\pm 0.402$	$2.947\pm 0.303$
5: Differential diagnosis	$2.800\pm 0.617$	-	$2.769\pm 0.514$	$2.637\pm 0.492$
6: Clinical conclusion	-	$1.268\pm 1.199$	$1.510\pm 1.045$	$1.418\pm 1.096$

Appendix

Table 20: Model-specific criterion profile for GLM-5.2.

Spreadsheet component	Columns	Material to review	Scores to assign
Reasoning	B–I	In the pre-consensus stage, identify the block corresponding to the rater’s specialty. Evaluate the sections describing clinical data, syndromes, differential diagnosis, diagnostic strategy, and management strategy. Do not include the section titled “Final conclusion.”	Eight scores, one for each criterion.
First-stage final judgment	J–Q	In the same specialty-specific block, evaluate only the section titled “Final conclusion.” This represents the specialist’s final position before the overall consensus discussion.	Eight scores, one for each criterion.
Consensus conclusion	R–Y	Evaluate the overall consensus stage, including the final opinion, summary, final differential diagnosis, diagnostic plan, management plan, and safety warnings.	Eight scores, one for each criterion.
Whole-case assessment	Z–AB	Evaluate the consensus report as a whole, including the adequacy of the specialist composition, clinical complexity, and overall quality of all stages.	Three scores: specialist composition, complexity, and overall quality.
Comment	AC	Briefly explain, if possible, why low scores were assigned or identify the most important errors.	One to four sentences.

Appendix

Table 29

Columns	Criterion	Operational meaning
B / J / R	Factual accuracy and sources	Facts, numerical values, dates, investigations, and treatments are not fabricated or distorted.
C / K / S	Identification and prioritization of clinical problems	The main clinical problems are identified and prioritized appropriately.
D / L / T	Completeness and relevance of the data used	Relevant data are used, while irrelevant or incidental details do not drive the assessment.
E / M / U	Coherence of clinical reasoning	The report contains a coherent chain linking data, explanation, diagnoses, conclusions, and management.
F / N / V	Quality of differential diagnosis	Plausible alternatives are considered, with arguments for and against them and methods for verification.
G / O / W	Quality of the final clinical conclusion	The conclusion is clear, clinically appropriate, and answers the main clinical question.

Appendix

Table 30

Score	Assign this score when	Interpretation
0	The criterion is not met. There is a major error, an unsafe recommendation, or fabricated information that invalidates the conclusion.	The response cannot be trusted.
1	Some useful content is present, but there is a substantial deficiency: important information is missing, the logic is weak, the conclusion is incomplete, or factual reliability is poor.	The response requires substantial revision.
2	The criterion is generally met, but there is a noticeable deficiency, such as missing data, incomplete arguments, absent timeframes, unclear conditions, or insufficient specification.	The response may be acceptable after revision.
3	The criterion is fully met and clinically convincing. Important facts are accurate, the logic is clear, and the conclusion and management plan are specific.	The response is strong.

Appendix

Table 31

Example	Scoring implication
The source case contains no leukocytosis, C-reactive protein value, procalcitonin level, creatine kinase value, troponin value, or acute kidney injury, but the consensus report presents these findings as facts and uses them to support a diagnosis.	This is not a minor inaccuracy. Factual accuracy should receive 0 points. Reasoning, differential diagnosis, the final conclusion, and management strategy should receive no more than 1 point if they rely on these data.
The consensus report states that C-reactive protein may be elevated and should be checked, although no C-reactive protein result is available.	This is a hypothesis rather than a fact. It is permissible if it is clearly framed as a hypothesis and not presented as an already available test result.

Appendix

Table 32

Score	Assign this score when
0	A fabricated or distorted clinically significant fact is present and could alter the diagnosis, risk assessment, or management strategy; or the conclusion directly contradicts the source data.
1	There are several factual errors or one important unsupported interpretation; the sources are unclear; however, the entire conclusion does not depend exclusively on fabricated data.
2	The main facts are reported correctly and no clinically significant fabricated facts are present, but there are minor inaccuracies or incomplete source references.
3	All important facts, numerical values, dates, and investigations are accurate; their sources are clear; and hypotheses are distinguished from established facts.

Appendix

Table 33

Score	Assign this score when
0	The main clinical problem is missed or incorrectly stated.
1	The main problem is only partly recognized, but priorities are incorrect or an important risk is omitted.
2	The main problem and most important associated problems are identified correctly, but there is a limited omission or weak prioritization.
3	All key clinical problems and risks are identified and arranged in an appropriate clinical order.

Appendix

Table 34

Score	Assign this score when
0	Key data are omitted, making the conclusion unreliable or incorrect.
1	Only part of the available data is used; several important domains are omitted, such as history, physical examination, clinical course, laboratory results, imaging, treatment, or risks.
2	Most important data are considered, but one or two meaningful omissions remain, or excessive irrelevant detail reduces focus.
3	All data affecting diagnosis, risk assessment, and management are considered; irrelevant information does not drive the conclusion.

Appendix

Table 35

Score	Assign this score when
0	Conclusions do not follow from the data, contradict the data, or are based on fabricated facts.
1	Logic is present only in fragments; important conclusions are not explained, and causal links are superficial.
2	Reasoning is generally coherent, but there are limited logical gaps, weak mechanisms, or incomplete chronology.
3	There is a clear chain from data to syndromes or mechanisms, differential diagnosis, conclusion, and management. Chronology and clinical dynamics are considered.

Appendix

Table 36

Score	Assign this score when
0	Differential diagnosis is absent, irrelevant, omits an obvious dangerous alternative, or is based on fabricated data.
1	Too few alternatives are considered, probabilities are not ordered, and arguments for and against the alternatives are largely absent.
2	The main alternatives are identified, but arguments, prioritization, or verification methods are incomplete.
3	Diagnoses are relevant and prioritized; for each important alternative, there are arguments for and against it and a way to confirm or exclude it.

Appendix

Table 37

Score	Assign this score when
0	The conclusion is absent, clinically incorrect, contradictory, or based on fabricated clinically significant facts.
1	The conclusion is overly general, incomplete, does not answer the main question, or merely repeats a list of possible diagnoses without a clear position.
2	The main conclusion is generally correct and understandable, but requires clarification regarding probability, conditions, limitations, or links to key data.
3	The conclusion is clear, specific, and justified. It names the main diagnosis, syndrome, or working hypothesis, states the level of confidence, and identifies data that could change the decision.

Appendix

Table 38

Score	Assign this score when
0	Management strategy is absent, clearly inappropriate, unsafe, or based on fabricated clinically significant facts.
1	Some appropriate actions are present, but the plan is generic: priorities, timeframes, reassessment criteria, monitoring, or links to the diagnosis are missing.
2	The main actions are appropriate, but specific details are missing, such as timeframes, monitoring parameters, responsible specialties, or escalation criteria.
3	The plan is specific and prioritized. Diagnostic actions, treatment, monitoring, timeframes, reassessment criteria, and next steps are linked to the clinical reasoning.

Appendix

Table 39

Score	Assign this score when
0	There is a potentially unsafe recommendation, an omitted life-threatening condition, or an ignored contraindication.
1	Some risks are recognized, but there is a substantial gap in monitoring, contraindications, drug interactions, or actions in case of deterioration.
2	The management strategy is generally safe, but one important precaution or one deterioration scenario is not addressed.
3	Urgent risks, contraindications, interactions, complications, monitoring requirements, and an action plan for deterioration are addressed.

Appendix

Table 40

Score	Assign this score when
0	The specialist composition does not cover the main clinical problem; a key specialty is absent, and the consensus cannot reliably address the case.
1	Part of the composition is appropriate, but one important specialty is absent or the roles of the participants do not match the main clinical tasks.
2	Most necessary specialties are present, but the composition does not fully cover diagnosis, treatment, or safety.
3	The composition is appropriate: key specialists are present, and their roles correspond to the main problems and patient risks.

Appendix

Table 41

Score	Assign this score when
0	The case is simple for the rater’s specialty, with one obvious problem, a standard diagnostic pathway, limited need for additional analysis, and low risk of error.
1	The case is moderately complex, with several relevant data points and simple alternatives, but the expected decision is largely standard.
2	The case is complex, with involvement of several systems, ambiguous data, important clinical dynamics, or several plausible differential diagnoses.
3	The case is very complex, with a rare or atypical presentation, high risk, incomplete or conflicting data, and a clear need for multidisciplinary input.

Appendix

Table 42

Score	Assign this score when
0	The consensus report as a whole is unusable: there are many major errors, and fabricated data lead to an incorrect conclusion or unsafe management.
1	Some useful fragments are present, but overall quality is low: substantial errors occur across several stages, and the result requires major revision.
2	The consensus report is generally acceptable, but there are noticeable deficiencies in facts, reasoning, conclusion, management, or consistency between stages.
3	All stages are consistent, factually accurate, logically coherent, and clinically useful; the result can be accepted without major revision.

Appendix

Table 43

Non-specific comment	Acceptable comment
Many errors.	The reasoning cites a non-existent C-reactive protein value and uses it to support an infectious diagnosis; factual accuracy is 0, and reasoning and differential diagnosis should not exceed 1.
Acceptable, but could be better.	The main diagnosis is selected appropriately, but the final plan lacks timeframes for reassessment, escalation criteria, and safety monitoring.
Poor conclusion.	The final conclusion lists three diagnoses without prioritization and does not identify the leading hypothesis at the current stage.

Appendix

Table 44

Question	If the answer is no
Do the facts match the source case?	Reduce the factual accuracy score. If a fabricated fact affects the conclusion, assign 0 for factual accuracy.
Is the main clinical problem identified?	Reduce the scores for problem prioritization and the final clinical conclusion.
Are key laboratory tests, imaging, history, and clinical dynamics used?	Reduce the completeness score.
Do conclusions follow logically from the facts?	Reduce the coherence of reasoning score.
Is the differential diagnosis adequate?	Reduce the differential diagnosis score.
Does the final conclusion answer the main clinical question?	Reduce the final clinical conclusion score.

Appendix

Table 45

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 419 papers

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-Judge Healthcare
aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa +3
Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency
Jun 29, 2026Nisarg A. PatelLLM Evaluation Reasoning Consistency in Language Models
University of California, San Francisco · Department of Oral and Maxillofacial Surgery, University of California, San Francisco, CA, USA · Department of Medicine, Division of Clinical Informatics and Digital Transformation, University of California, San Francisco, CA, USA
ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs
Mar 2, 2026Xiang Zheng, Han Li, Wenjie Luo +15Clinical Language Model Evaluation Medical LLMs
Alibaba Group, China

Apr 16, 2026J/K move · Enter open · S save

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor+7

aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa+3

LLM-as-a-Judge Healthcare Clinical Reasoning Clinical Language Model Evaluation

Open paper AskPDF arXiv

Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators. Here, we evaluate an LLM Jury, composed of three frontier AI models, for scoring 3334 diagnoses on 300 real-world low- and middle-income country (LMIC) hospital cases. Both LLM- and clinician-generated diagnoses are scored against expert panel diagnoses across four dimensions: diagnosis, differential diagnosis, clinical reasoning, and negative treatment risk. The LLM Jury scores are compared with expert and independent re-scoring panel scores to assess error metrics, inter-rater agreement, severe-risk errors, and the effect of post hoc calibration using isotonic regression. In our data, we find that: (i) the uncalibrated LLM Jury scores preserve ordinal agreement with the expert clinician panel scores, but are systematically lower; (ii) the probability of severe-risk errors is lower for the LLM Jury than the human expert re-score panels; (iii) the LLM Jury combined with LLM diagnoses can be used to identify diagnoses at high risk of error, enabling targeted expert review and improved panel efficiency; (iv) the calibrated LLM Jury scores and rankings of diagnosing agents show excellent agreement with those of the primary expert panels; (v) LLM Jury models show no self-preference bias, they did not score diagnoses generated by their own underlying model or models from the same vendor more (or less) favourably than those generated by other models. Together, these results provide evidence that a calibrated LLM Jury is a trustworthy and reliable proxy for expert clinician evaluation in medical AI benchmarking. Confirming these findings in other clinical settings is an important direction for future work.

Abstract

Clinical Language Model Evaluation

Explore similar work

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs