ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs

Xiang Zheng; Han Li; Wenjie Luo; Weiqi Zhai; Yiyuan Li; Chuanmiao Yan; Xue Yang; Kailuan Wu

cs.CLMar 2, 2026

ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs

Authors: Xiang Zheng, Han Li, Wenjie Luo, Weiqi Zhai, Yiyuan Li, Chuanmiao Yan, Xue Yang, Kailuan Wu,

Organizations: Alibaba Group, China

Abstract

Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce ClinConsensus, a Chinese medical benchmark of 2{,}500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose Clinician-Anchored Coverage Score (CACS), a physician-calibrated threshold metric instantiated at $k=10$ , and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6% to 52.1%, whereas CACS@10 ranges from 17.8% to 32.9%, leaving a 19.2--21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness.

Figures & tables

Benchmark	Chinese / localized	Open- ended	Multi- specialty	Rubric / checklist	Physician validation	Interactive / multi-turn	Difficulty tiers
HealthBench ( Arora et al., 2025 )	–	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–
LLMEval-Med ( Zhang et al., 2025 )	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	–
CSEDB ( Wang et al., 2025a )	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	–
MedThink-Bench ( Zhou et al., 2025 )	–	$\checkmark$	$\checkmark$	$\circ$	$\checkmark$	–	–
AgentClinic ( Schmidgall et al., 2024 )	–	$\checkmark$	$\checkmark$	–	–	$\checkmark$	–
ClinicalLab ( Yan et al., 2024 )	–	$\checkmark$	$\checkmark$	–	$\circ$	$\checkmark$	–

Table 1: Positioning ClinConsensus among medical LLM benchmarks.

\checkmark

\circ

, and – denote explicit, partial, and absent support. Physician validation includes authorship, review, or grading by physicians.

Figure 1: ClinConsensus pipeline. Expert-curated clinical cases are transformed into structured benchmark items, paired with case-specific rubrics, curated and audited through quality control, and evaluated under a rubric-level protocol for CACS@ $k$ scoring and judge validation.

Axis	Strata	Purpose
Task theme	12 clinical task themes	Identifies capability-specific strengths and failures
Medical specialty	8 macro-categories	Tests domain generalization and specialty-level blind spots
Difficulty	Low, Medium, High	Measures robustness across curated difficulty strata
Dialogue register	Lay-facing, professional-facing	Compares responses for lay and professional users

Table 2: Stratified evaluation axes. Task and specialty labels are multi-label; difficulty and dialogue register are mutually exclusive.

Figure 2: Task distribution and theme-wise CACS@10. (a) Case distribution across 12 task themes; task themes are multi-label, so case counts across themes do not sum to 2,500. (b) Model performance by task theme.

Figure 3: Inter-expert group agreement across 11 models. Case-level Macro-F1 (MF1) distributions and pooled MF1 summarize agreement between two independent expert groups after model–case alignment.

Model	Annotators	Checkers	Pooled MF1
DeepSeek V3.2	16 / 31	1 / 4	0.837
GPT-5.2	8 / 21	1 / 4	0.812
Grok 4.1 Fast	9 / 25	1 / 3	0.803
Baichuan M3	15 / 26	7 / 5	0.791
GLM-5	10 / 5	5 / 1	0.783
Gemini 3.1 Pro	10 / 4	1 / 1	0.766

Table 3: Inter-expert agreement by model. Counts are Expert Group 1 / Expert Group 2; pooled MF1 is computed after model–case alignment.

Judge	Expert Group 1 MF1	Expert Group 2 MF1	Average MF1
GPT-5.1 strict judge	0.779	0.738	0.759
Base Qwen3-8B judge	0.782	0.771	0.777
Our judge model	0.844	0.808	0.826

Table 4: Held-out automated judge fidelity. MF1 is computed against each expert group on 82,500 held-out rubric decisions.

Figure 4: Automated judge fidelity relative to expert agreement. Human–human MF1 is the reference. Open markers denote GPT-5.1 strict and base Qwen3-8B judges; filled markers denote our physician-supervised judge, each compared with Expert Groups 1/2.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Access	Purpose
De-identified rubric-only public release (prompts and metadata)	Public: Hugging Face ( $900$ low-difficulty cases)	Score regenerated or new model responses without reference answers
Case-specific binary rubrics	Public: Hugging Face ( $30$ per released case)	Recompute rubric hits, Rubric Accuracy, Pass@10, and CACS@ $k$
Schema, JSONL data, counts, and SHA256 manifest	Public: Hugging Face	Verify file integrity and reproduce the released tier
Evaluation and validation code	Public: GitHub	Reproduce the scoring pipeline end to end
Full $2{,}500$ -case benchmark and expert-reviewed reference answers	On request; non-commercial DUA	Full-scale evaluation and reference-answer analysis
Judge prompts and output schemas	Appendix and supplement	Audit the rubric-level grading interface and JSON format

Appendix

Table 5: Release components and access. The public release ships rubric-only cases for direct scoring; the full benchmark and reference answers are available on request under a non-commercial data-use agreement; restricted source files are withheld for privacy and provenance.

Axis	HealthBench	ClinConsensus
Scoring mechanism	Continuous and compensatory: signed per-criterion point weights (positive for desirable, negative for undesirable behaviors) are summed and normalized, with no minimum-coverage threshold.	Thresholded and conjunctive in spirit: binary criteria feed CACS@ $k$ ( $k{=}10$ of 30), assigning zero credit below the physician-calibrated coverage threshold.
Judge / grader	Single automated grader (GPT-4.1), validated post-hoc against physician agreement.	Dual judge: a GPT-5.1 grader paired with a physician-supervised Qwen3-8B judge.
Rubric scale	262 physicians authored rubrics over 5,000 conversations, 48,562 criteria in total, with a median of 11 criteria per example.	Physician-curated rubrics over 2,500 cases, 75,000 criteria in total, with a fixed 30 case-specific criteria per case.
Case source	Largely model-generated conversations (with red-teaming and search-seeded prompts), filtered for quality.	Expert-curated from de-identified real clinical practice, with quality-control review.
Localization / audience	Globally framed and English-authored, with no China-specific guideline coverage; conversation-language distribution and professional-vs.-lay split not reported.	Chinese and localized to Chinese clinical practice, with an explicit dialogue register (1,496 professional-facing vs. 1,004 lay-facing cases).

Appendix

Table 6: Design-axis comparison between HealthBench and ClinConsensus . Entries are drawn from the HealthBench paper and public release ( Arora et al., 2025 ) ; “not reported” marks quantities the source does not state.

Slice	Count	% cases
Difficulty
Low	900	36.00
Medium	800	32.00
High	800	32.00
Dialogue register
Professional-facing	1,496	59.84

Appendix

Table 7: Retained-case distribution by metadata slice. Counts summarize the 2,500 retained cases. Difficulty, dialogue register, and subject macro-category are mutually exclusive axes. Task themes are multi-label, so their counts and case percentages do not sum to 2,500 or 100%.

Difficulty	Task-theme profile	Subject profile
Low	Patient education (30.4%, 2.20 $\times$ ); clinical summarization (14.1%, 1.67 $\times$ ); IE (18.1%, 1.65 $\times$ ); test/lab interpretation (14.9%, 1.51 $\times$ )	ENT / Derm / Ophthal / Dental (18.7%, 1.70 $\times$ ); Surgery & Orthopedics (22.7%, 1.20 $\times$ ); OB/GYN & Pediatrics (12.9%, 1.15 $\times$ )
Medium	Personalized treatment planning (64.9%, 1.21 $\times$ ); causal/logical reasoning (41.6%, 1.20 $\times$ ); chart understanding (46.1%, 1.20 $\times$ ); differential diagnosis (51.7%, 1.11 $\times$ )	IM: Cardio-Pulmonary (12.0%, 1.33 $\times$ ); Geriatrics / Acute-Critical / Rehab (19.3%, 1.23 $\times$ ); Surgery & Orthopedics (20.8%, 1.10 $\times$ )
High	Medication instructions (35.9%, 1.88 $\times$ ); evidence retrieval (26.8%, 1.56 $\times$ ); chart understanding (54.5%, 1.42 $\times$ ); causal/logical reasoning (45.0%, 1.30 $\times$ ); personalized treatment planning (65.9%, 1.23 $\times$ )	Neurology & Psychiatry (21.6%, 2.13 $\times$ ); IM: Immunology & Infection (20.6%, 1.66 $\times$ ); IM: Gastro-Metabolic (16.4%, 1.41 $\times$ )

Appendix

Table 8: Metadata profile of difficulty strata. Entries show task themes and subject macro-categories over-represented within each difficulty stratum relative to their overall retained-set rates. Values are within-stratum case percentages and enrichment ratios; task themes are multi-label.

Panel of Table 9 — Table 9: Example of guideline-grounded rubric construction. A de-identified acute chest-pain case illustrates how selected physician-written criteria map to guideline-aligned clinical expectations.

Figure 5: CACS@10 by case difficulty.

Figure 6: Motivating example for CACS@10. Only the expert response crosses the physician-calibrated coverage threshold.

Figure 7: Distribution of per-case rubric accuracy around the CACS@10 threshold. Each violin shows per-case rubric accuracy for one evaluated model. The hatched gray region marks cases below the 10/30 CACS@10 threshold; these cases are not excluded from evaluation, but contribute zero to CACS@10. Colored densities summarize threshold-reaching cases, and diamonds denote the mean rubric accuracy among threshold-reaching cases only, not the model-level CACS@10 score.

Figure 8: Aggregated clinical capability profile. Numbers report CACS@10 after aggregating 12 task themes into five capability families; model-axis labels give each model’s overall CACS@10. Cell color encodes deviation from that model’s overall score, highlighting relative strengths and weaknesses.

Figure 9: CACS@10 by subject macro-category. Cases are assigned by the first observed specialty label to one of eight macro-categories; raw specialty labels are retained for audit.

Figure 10: Rubric accuracy vs. CACS@10. Circles denote CACS@10 and squares denote Rubric Accuracy; models are ordered by CACS@10.

Rubric hits $s$	Accuracy	Pass $s\geq 10$	CACS@10
9	30.0%	0	0
10	33.3%	100%	4.8%
15	50.0%	100%	28.6%
30	100.0%	100%	100.0%

Appendix

Table 10: Behavior of alternative metrics for

N=30

k=10

. CACS@10 assigns zero credit to below-threshold responses while preserving coverage differences above threshold.

Metric	Spearman	Top-3	Top-5
CACS@6	0.991	3/3	5/5
CACS@7	1.000	3/3	5/5
CACS@8	0.998	3/3	5/5
CACS@9	1.000	3/3	5/5
CACS@12	0.991	3/3	5/5
CACS@15	0.991	3/3	5/5

Appendix

Table 11: Threshold sensitivity of model-level rankings. Spearman correlations and top-

k

overlaps compare each metric against the CACS@10 ranking over the 11 evaluated models; CACS@10 is the main setting.

Diagnostic	Value
Pearson corr. length vs CACS@10	0.056
Spearman corr. length vs CACS@10	0.050
Pearson corr. length vs Rubric Accuracy	0.038
Spearman corr. length vs Rubric Accuracy	0.046
Partial corr. length vs CACS@10	0.212

Appendix

Table 12: Response length and CACS@10.

Model	Lay-facing	Professional-facing	$\Delta$
GPT-5.2	33.9	32.2	-1.7
Baichuan M3	31.0	25.7	-5.3
Claude Opus 4.6	25.2	28.0	2.8
Qwen3.5 Plus	26.1	27.1	1.0
Kimi K2.5	23.0	24.4	1.4
Qwen3-Max	22.3	23.7	1.4

Appendix

Table 13: CACS@10 by dialogue register. Cases are split by keyword match on the stated user role: professional-facing roles cover clinicians, nurses, medical trainees, care and nursing staff, documentation reviewers, and medical educators (1,496 cases); all remaining roles—patients, family, friends, and general users—are lay-facing (1,004 cases). Slice scores are means over the cases in each register; the overall CACS@10 (Figure 10 ) equals their case-weighted average up to rounding.

\Delta

denotes professional-facing minus lay-facing CACS@10, computed from the displayed one-decimal values.

Failure type	Description	Common slices	Benchmark implication
Missing emergency trigger	The response omits explicit red flags, urgent escalation criteria, or emergency referral conditions required by the rubric.	Differential diagnosis; causal mechanism reasoning; medication safety.	Time-sensitive risk handling may remain implicit rather than operationally checkable.
Non-operational treatment plan	The response gives a generic plan but omits sequencing, personalization, or decision conditions.	Personalized treatment planning; differential diagnosis; causal / logical reasoning.	The answer may sound plausible while failing to support case-specific planning under the rubric.
Implicit but unverifiable reasoning	The response is directionally plausible but does not make reasoning steps, evidence, or decision criteria explicit enough for verification.	Differential diagnosis; medication safety; evidence retrieval and guideline alignment.	Independent graders may be unable to verify clinically important reasoning steps.
Structured extraction loss	The response misses entities, fields, chart facts, summary elements, or documentation constraints requested by the rubric.	Chart understanding; medical information extraction; clinical documentation.	Information-dense workflows may lose clinically relevant details even when the response is fluent.
Medication boundary omission	The response omits dose, interaction, contraindication, adverse-reaction, monitoring, or adjustment boundaries.	Medication instructions; medication safety; treatment planning.	Medication-related coverage can remain incomplete under thresholded rubric scoring.
Follow-up incompleteness	The response omits follow-up timing, monitoring indicators, return precautions, or longitudinal management details.	Follow-up and monitoring; chronic monitoring; rehabilitation and lifestyle management.	The answer may address the immediate question but fail to specify continuity-of-care requirements.

Appendix

Table 14: Qualitative failure taxonomy from low-CACS responses.

Panel of Figure 11 — Figure 11: Low-agreement tail cases. Each example compares a low-agreement response with a higher-agreement response on the same case.

Item	Configuration
Base model	Qwen3-8B
Training data	52,209 train / 5,933 development examples
Held-out data	82,500 rubric decisions from 11 unseen models
Training setup	Full-parameter SFT; BF16; Flash Attention 2; PyTorch FSDP full sharding
Hardware	8 NVIDIA H20-3e GPUs
Batching	Effective batch size 64

Appendix

Table 15: Training configuration for our judge model.

Figure 12: Cross-grader model ranking robustness. The left panel compares criterion-positive rates from our physician-aligned judge and the GPT-5.1 strict judge; the right panel visualizes rank shifts across the same 11 models.

Metric	Value
Spearman $\rho$	0.791
Kendall $\tau_{b}$	0.636
Mean absolute difference	8.72 pp
Top-3 overlap	2/3 (66.7%)
Top-5 overlap	4/5 (80.0%)

Appendix

Table 16: Summary diagnostics for model-level cross-grader robustness. Metrics compare model-level criterion-positive rates from our physician-aligned judge and the GPT-5.1 strict judge.

Panel of Figure 13 — Figure 13: LLM-as-Judge prompt template. Schema-constrained prompt used for rubric-level grading.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 418 papers

Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness
Mar 26, 2026Alexandra DeLucia, Heyuan Huang, Sonal Joshi +3LLM-as-a-Judge LLM Reliability
Data Science and AI Institute, Johns Hopkins University
When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation
Sep 11, 2026Griffin Farrow, Lily Sijia Li, Jack Johnson +4Hallucination in RAG LLM Hallucination Detection
University of Oxford
Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters
Apr 27, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6LLM-as-a-Judge Human-AI Agreement
1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2

Mar 26, 2026J/K move · Enter open · S save

Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness

Alexandra DeLucia, Heyuan Huang, Sonal Joshi, Mahsa Yarmohammadi+2

Data Science and AI Institute, Johns Hopkins University

LLM-as-a-Judge LLM Reliability Clinical Language Model Evaluation

Open paper AskPDF arXiv

LLM-as-a-Judge frameworks are increasingly trusted to automate evaluation in place of human experts, yet their reliability in high-stakes medical contexts remains unproven. We stress-test this assumption for detecting incomplete patient-facing medical responses, evaluating three rubric granularities (General-Likert, Analytical-Rubric, Dynamic-Checklist) and three backbone models across two clinician-annotated datasets, including HealthBench, the largest publicly available benchmark for medical response evaluation. LLM Judges discriminate complete from incomplete responses at and slightly above near chance (AUC

0.49

0.66

); at the threshold required to recall

90\%

of incomplete responses, clinicians must still review the vast majority of the dataset, offering no triage utility. Even when model and clinician verdicts agree, they rarely cite the same explanation; and when they diverge, false positives stem from over-flagging non-essential gaps while false negatives reflect outright detection failures. These results reveal that LLM Judges and clinicians apply fundamentally different completeness standards; a finding that undermines their use as autonomous evaluators or triage filters in clinical settings.

Abstract

Clinical Language Model Evaluation

Explore similar work

Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness

When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation

Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters