QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models

Yao Wu; Kangping Yin; Liang Dong; Zhenxin Ma; Shuting Xu; Xuehai Wang; Yuxuan Jiang; Tingting Yu

cs.CLMar 14, 2026

QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models

Authors: Yao Wu, Kangping Yin, Liang Dong, Zhenxin Ma, Shuting Xu, Xuehai Wang, Yuxuan Jiang, Tingting Yu,

Organizations: Quark Medical Team, Alibaba Group

Abstract

While Large Language Models (LLMs) excel on standardized medical exams, high scores often fail to translate to high-quality responses for real-world medical queries. Current evaluations rely heavily on multiple-choice questions, failing to capture the unstructured, ambiguous, and long-tail complexities inherent in genuine user inquiries. To bridge this gap, we introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment. We compiled a massive dataset spanning Clinical Care, Wellness Health, and Professional Inquiry, comprising 20,821 single-turn queries and 3,853 multi-turn sessions. To objectively evaluate open-ended answers, we propose an automated scoring framework that integrates multi-model consensus with evidence-based retrieval to dynamically generate 220,617 fine-grained scoring rubrics (~9.8 per query). During evaluation, hierarchical weighting and safety constraints structurally quantify medical accuracy, key-point coverage, and risk interception, effectively mitigating the high costs and subjectivity of human grading. Experimental results demonstrate that the generated rubrics achieve a 91.8% concordance rate with clinical expert blind audits, establishing highly dependable medical reliability. Crucially, baseline evaluations on this benchmark reveal significant performance disparities among state-of-the-art models when navigating real-world clinical nuances, highlighting the limitations of conventional exam-based metrics. Ultimately, QuarkMedBench establishes a rigorous, reproducible yardstick for measuring LLM performance on complex health issues, while its framework inherently supports dynamic knowledge updates to prevent benchmark obsolescence.

Figures & tables

Panel of Figure 1 — Figure 1: Performance of mainstream LLMs on QuarkMedBench. This grouped bar chart compares the scores of 14 models without length constraints (hatched bars) versus those with a strict length constraint of $\leq$ 1000 words (solid bars).

Figure 2: Automated Scoring Rubric (Auto-Rubrics) Generation Pipeline

Tier	Rubric Description	Difficulty Interpretation
Basic	States that the diagnostic criterion for hypertension is a systolic blood pressure $\geq$ 130 mmHg.	Verifies the factual accuracy of the core criterion based on recent standards.
Pro	Explicitly indicates that the 2023 Chinese Guidelines for the Management of Hypertension adjusted the diagnostic criteria.	Assesses the model’s adherence to clinical authority and information recency by referencing the latest guideline updates.
Expert	Elucidates that the threshold adjustment aims for earlier intervention, which mitigates hypertension-related risks, extends life expectancy, and reduces healthcare expenditures.	Demands an explanation of the underlying rationale for the criteria revision, accompanied by relevant clinical evidence, transcending mere factual recall.

Table 1: Example of Diagnostic Criteria for Hypertension (Difficulty Tiers)

Perspective	Rubric Description	Cognitive Perspective Explanation
Direct	States the diagnostic threshold for hypertension is $\geq$ 130 mmHg.	Provides information directly addressing the explicit question.
Depth	Elucidates that the threshold was adjusted for early control, mitigating risks, extending life expectancy, and reducing healthcare costs.	Probes the essential reasoning to understand why the standard was established at 130 mmHg.
Breadth	Provides common symptoms of hypertension and daily precautions.	Extends laterally to address typical symptoms and lifestyle interventions the user might implicitly care about.

Table 2: Example of Diagnostic Criteria for Hypertension (Response Perspectives)

Experimental Configuration	Weight Distribution	Concordance Ratio
	(Essential : Important : Highlight : Pitfall)	(Correct : Reversed)
Baseline (Equal Weights)	1 : 1 : 1 : 1	2:1
V1 (Enhanced Essential & Penalty)	2 : 1 : 1 : 2	3:1
V2 (Over-enhanced Core)	3 : 1 : 1 : 3	Slightly $<$ 3:1
V3 (Differentiated Important & Highlight)	3 : 2 : 1 : 3	Slightly $>$ 3:1

Table 3: Comparison of Experimental Outcomes under Various Weight Configurations

Main Category	Subcategory	Percentage (%)	Overall (%)
Clinical Care	Digestive System	12.87	66.22
	Reproductive System	10.88
	Integumentary System	7.88
	Respiratory System	6.92
	Nervous System	6.67
	Circulatory System	5.59

Table 4: Query Scenario Distribution in QuarkMedBench

Model	Essential ( $\bar{y}=7.85$ )		Important ( $\bar{y}=4.28$ )		Highlight ( $\bar{y}=2.71$ )		Pitfall ( $\bar{y}=2.71$ )
Model	Mean ( $\bar{x}$ )	$r$	Mean ( $\bar{x}$ )	$r$	Mean ( $\bar{x}$ )	$r$	Mean ( $\bar{x}$ )	$r$
Qwen3-235B	8.57	0.696***	5.39	0.610***	3.21	0.585**	-0.28	0.375*
DeepSeekR1-0528	6.64	0.437*	4.07	0.566**	2.39	0.348	-0.35	0.105
DeepSeekR1-671B	8.50	0.751***	5.03	0.675***	3.07	0.576**	-0.64	0.395*
Doubao-Seed-1.6	8.28	0.652***	5.42	0.570**	2.96	0.425*	-0.35	0.559**
GPT-5	5.78	0.130	3.50	0.154	2.46	0.211	-0.57	-0.181

Table 5: Comparison of Pearson Correlation Coefficients (

r

) and Means (

\bar{x}

) Between Candidate Models and Human Expert Scores

Total Score
Model	Exact Match	Acceptable Deviation	Severe Deviation	MAE	Pearson
Model	(%)	(%)	(%)	MAE	( $r$ )
Gemini 2.5 Pro	78.6	14.3	7.1	3.57	0.56
Qwen3-235B	78.6	14.3	7.1	3.68	0.58
DeepSeekR1-0528	75.0	14.3	10.7	4.25	0.34
Doubao-Seed-1.6	71.4	21.4	7.1	3.82	0.45

Table 6: Distribution of Scoring Deviations and Consistency Metrics Across Evaluation Dimensions

Evaluation Dimension	Rubric Level	Total	Preliminary	Confirmed	Confirmed	Severity
Evaluation Dimension	Rubric Level	Rubrics (A)	Issues (B)	Issues (C)	Error Rate (C/A)	Minor	Major
Correctness	Essential	719	87	59	8.21%	44	15
	Important	698	64	49	7.02%	41	8
	Highlight	735	59	42	5.71%	36	6
	Pitfalls	428	15	10	2.34%	8	2
	Subtotal	2580	225	160	6.20%	129	31

Table 7: Clinical Expert Evaluation of Rubric Content Quality

Rubric Level	Total	Preliminary	Confirmed	Confirmed	Discrepancy Type
Rubric Level	Rubrics (A)	Discrepancies (B)	Discrepancies (C)	Rate (C/A)	Overestimated	Underestimated	Redundant
Essential	719	58	13	1.81%	13	0	0
Important	699	47	16	2.29%	12	3	1
Highlight	734	32	11	1.50%	0	3	8
Pitfalls	428	8	2	0.47%	0	0	2
Total	2580	145	42	1.63%	25	6	11

Table 8: Clinical Expert Evaluation of Rubric Weight Rationality

Rank	Model	Avg.	Base	Ablation Study (Score %)
Rank	Model	Length	Score (%)	w/ Saturation	w/ Truncation*	w/ Both*
1	GPT-5	1,007	79.84	68.88	78.30	67.63
2	Gemini 2.5 Pro	1,935	74.57	66.87	72.42	65.10
3	Qwen3-30B-A3B-Thinking-2507	2,044	74.54	66.15	71.64	63.72
4	DeepSeek-V3.2	1,629	71.43	64.51	68.37	61.93
5	GPT-5.1	1,473	70.90	63.99	68.07	61.62

Table 9: Comprehensive Performance of Mainstream LLMs without Output Length Constraints

Rank	Model	Avg.	Base	Ablation Study (Score %)
Rank	Model	Length	Score (%)	w/ Saturation	w/ Truncation*	w/ Both*
1	GPT-5	1,029	82.17	70.38	80.71	69.26
2	GPT-5.2	944	72.52	65.59	67.42	61.77
3	GPT-5.1	1,063	69.02	63.42	65.91	60.58
4	Qwen3-235B-A22B	980	67.71	60.94	63.28	57.26
5	Qwen3-30B-A3B-Thinking-2507	977	66.23	60.10	61.59	56.14

Table 10: Comprehensive Performance of Mainstream LLMs with Strict Length Constraint (

\leq

1000 words)

Rank	Model	Avg.	Overall	Dimensional Score Rate (%)
Rank	Model	Length	Score (%)	Essential	Important	Highlight	Pitfall
1	GPT-5	1,007	79.84	92.23	81.54	56.04	-4.30
2	Gemini 2.5 Pro	1,935	74.57	91.58	74.64	45.01	-6.39
3	Qwen3-30B-A3B-Thinking-2507	2,044	74.54	89.69	74.37	49.11	-6.23
4	DeepSeek-V3.2	1,629	71.43	89.48	72.29	41.86	-8.66
5	GPT-5.1	1,473	70.90	88.40	72.32	40.37	-7.78

Table 11: Dimensional Score Distribution under Unconstrained Conditions

Rank	Model	Avg.	Overall	Dimensional Score Rate (%)
Rank	Model	Length	Score (%)	Essential	Important	Highlight	Pitfall
1	GPT-5	1,029	82.17	93.90	85.09	58.36	-4.21
2	GPT-5.2	944	72.52	89.35	73.33	41.22	-5.15
3	GPT-5.1	1,063	69.02	87.92	69.21	36.63	-7.38
4	Qwen3-235B-A22B	980	67.71	84.74	69.62	45.11	-13.53
5	Qwen3-30B-A3B-Thinking-2507	977	66.23	83.49	66.57	42.93	-11.81

Table 12: Dimensional Score Distribution with Strict Length Constraint (

\leq

1000 words)

Model	HealthBench (HB)			QuarkMedBench (QB)			$\Delta$ Rank
Model	Score (%)	Avg. Length	Rank	Score (%)	Avg. Length	Rank	$\Delta$ Rank
GPT-5	65.56	1,364	1	79.84	1,007	1	–
GPT-5.2	62.72	1,267	2	66.19	788	7	$\downarrow$ 5
GPT-5.1	62.20	1,802	3	70.90	1,473	4	$\downarrow$ 1
Gemini 2.5 Pro	55.90	1,959	4	74.57	1,935	2	$\uparrow$ 2
Doubao Seed 1.6	54.78	1,425	5	61.91	1,052	9	$\downarrow$ 4

Table 13: Performance Distribution Comparison of Mainstream Models on HealthBench and QuarkMedBench

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Main Category	Subcategory	Percentage (%)	Overall (%)
Clinical Care	Nervous System	11.79	55.84
	Respiratory System	7.69
	Circulatory System	6.14
	Integumentary System	6.03
	Digestive System	5.87
	Musculoskeletal System	5.27

Appendix

Table 14: Query Scenario Distribution in Healthbench

Figure 4: Task classification distribution comparison between QuarkMedBench and Healthbench datasets. Left panel (1) shows the distribution across six main categories: Basic Inquiry, Pre-consultation, During-treatment, Post-treatment, Professional Medical, and Special Cases. Right panel (2) displays the top 15 specific task labels ranked by combined frequency. Percentages are calculated relative to the total number of samples in each dataset. QuarkMedBench demonstrates a balanced distribution across foundational categories including Basic Inquiry (30.0% vs 15.8%) and Pre-consultation (18.0% vs 12.1%), aligning with the full medical consultation lifecycle.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4
Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Feb 16, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2Medical QA Medical LLMs
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences
Mar 13, 2026Eric Wu, Kevin Wu, Jason Hom +11Clinical Language Model Evaluation Medical LLMs
Department of Electrical Engineering, Stanford University, Stanford, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Division of Hospital Medicine, Department of Medicine, Stanford School of Medicine, Stanford, CA, USA +8

Jun 10, 2026J/K move · Enter open · S save

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski, Jeremi Ignacy Kaczmarek+6

NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland+4

LLM Evaluation LLM Reliability Medical QA Language Model Bias Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.

Abstract

LLM Evaluation

Explore similar work

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences