QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models

Yao Wu; Kangping Yin; Liang Dong; Zhenxin Ma; Shuting Xu; Xuehai Wang; Yuxuan Jiang; Tingting Yu

cs.CLMar 14, 2026

QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models

Authors: Yao Wu, Kangping Yin, Liang Dong, Zhenxin Ma, Shuting Xu, Xuehai Wang, Yuxuan Jiang, Tingting Yu,

Organizations: Quark Medical Team, Alibaba Group

Abstract

While Large Language Models (LLMs) excel on standardized medical exams, high scores often fail to translate to high-quality responses for real-world medical queries. Current evaluations rely heavily on multiple-choice questions, failing to capture the unstructured, ambiguous, and long-tail complexities inherent in genuine user inquiries. To bridge this gap, we introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment. We compiled a massive dataset spanning Clinical Care, Wellness Health, and Professional Inquiry, comprising 20,821 single-turn queries and 3,853 multi-turn sessions. To objectively evaluate open-ended answers, we propose an automated scoring framework that integrates multi-model consensus with evidence-based retrieval to dynamically generate 220,617 fine-grained scoring rubrics (~9.8 per query). During evaluation, hierarchical weighting and safety constraints structurally quantify medical accuracy, key-point coverage, and risk interception, effectively mitigating the high costs and subjectivity of human grading. Experimental results demonstrate that the generated rubrics achieve a 91.8% concordance rate with clinical expert blind audits, establishing highly dependable medical reliability. Crucially, baseline evaluations on this benchmark reveal significant performance disparities among state-of-the-art models when navigating real-world clinical nuances, highlighting the limitations of conventional exam-based metrics. Ultimately, QuarkMedBench establishes a rigorous, reproducible yardstick for measuring LLM performance on complex health issues, while its framework inherently supports dynamic knowledge updates to prevent benchmark obsolescence.

Figures & tables

Panel of Figure 1 — Figure 1: Performance of mainstream LLMs on QuarkMedBench. This grouped bar chart compares the scores of 14 models without length constraints (hatched bars) versus those with a strict length constraint of $\leq$ 1000 words (solid bars).

Figure 2: Automated Scoring Rubric (Auto-Rubrics) Generation Pipeline

Tier	Rubric Description	Difficulty Interpretation
Basic	States that the diagnostic criterion for hypertension is a systolic blood pressure $\geq$ 130 mmHg.	Verifies the factual accuracy of the core criterion based on recent standards.
Pro	Explicitly indicates that the 2023 Chinese Guidelines for the Management of Hypertension adjusted the diagnostic criteria.	Assesses the model’s adherence to clinical authority and information recency by referencing the latest guideline updates.
Expert	Elucidates that the threshold adjustment aims for earlier intervention, which mitigates hypertension-related risks, extends life expectancy, and reduces healthcare expenditures.	Demands an explanation of the underlying rationale for the criteria revision, accompanied by relevant clinical evidence, transcending mere factual recall.

Table 1: Example of Diagnostic Criteria for Hypertension (Difficulty Tiers)

Perspective	Rubric Description	Cognitive Perspective Explanation
Direct	States the diagnostic threshold for hypertension is $\geq$ 130 mmHg.	Provides information directly addressing the explicit question.
Depth	Elucidates that the threshold was adjusted for early control, mitigating risks, extending life expectancy, and reducing healthcare costs.	Probes the essential reasoning to understand why the standard was established at 130 mmHg.
Breadth	Provides common symptoms of hypertension and daily precautions.	Extends laterally to address typical symptoms and lifestyle interventions the user might implicitly care about.

Table 2: Example of Diagnostic Criteria for Hypertension (Response Perspectives)

Experimental Configuration	Weight Distribution	Concordance Ratio
	(Essential : Important : Highlight : Pitfall)	(Correct : Reversed)
Baseline (Equal Weights)	1 : 1 : 1 : 1	2:1
V1 (Enhanced Essential & Penalty)	2 : 1 : 1 : 2	3:1
V2 (Over-enhanced Core)	3 : 1 : 1 : 3	Slightly $<$ 3:1
V3 (Differentiated Important & Highlight)	3 : 2 : 1 : 3	Slightly $>$ 3:1

Table 3: Comparison of Experimental Outcomes under Various Weight Configurations

Main Category	Subcategory	Percentage (%)	Overall (%)
Clinical Care	Digestive System	12.87	66.22
	Reproductive System	10.88
	Integumentary System	7.88
	Respiratory System	6.92
	Nervous System	6.67
	Circulatory System	5.59

Table 4: Query Scenario Distribution in QuarkMedBench

Model	Essential ( $\bar{y}=7.85$ )		Important ( $\bar{y}=4.28$ )		Highlight ( $\bar{y}=2.71$ )		Pitfall ( $\bar{y}=2.71$ )
Model	Mean ( $\bar{x}$ )	$r$	Mean ( $\bar{x}$ )	$r$	Mean ( $\bar{x}$ )	$r$	Mean ( $\bar{x}$ )	$r$
Qwen3-235B	8.57	0.696***	5.39	0.610***	3.21	0.585**	-0.28	0.375*
DeepSeekR1-0528	6.64	0.437*	4.07	0.566**	2.39	0.348	-0.35	0.105
DeepSeekR1-671B	8.50	0.751***	5.03	0.675***	3.07	0.576**	-0.64	0.395*
Doubao-Seed-1.6	8.28	0.652***	5.42	0.570**	2.96	0.425*	-0.35	0.559**
GPT-5	5.78	0.130	3.50	0.154	2.46	0.211	-0.57	-0.181

Table 5: Comparison of Pearson Correlation Coefficients (

r

) and Means (

\bar{x}

) Between Candidate Models and Human Expert Scores

Total Score
Model	Exact Match	Acceptable Deviation	Severe Deviation	MAE	Pearson
Model	(%)	(%)	(%)	MAE	( $r$ )
Gemini 2.5 Pro	78.6	14.3	7.1	3.57	0.56
Qwen3-235B	78.6	14.3	7.1	3.68	0.58
DeepSeekR1-0528	75.0	14.3	10.7	4.25	0.34
Doubao-Seed-1.6	71.4	21.4	7.1	3.82	0.45

Table 6: Distribution of Scoring Deviations and Consistency Metrics Across Evaluation Dimensions

Evaluation Dimension	Rubric Level	Total	Preliminary	Confirmed	Confirmed	Severity
Evaluation Dimension	Rubric Level	Rubrics (A)	Issues (B)	Issues (C)	Error Rate (C/A)	Minor	Major
Correctness	Essential	719	87	59	8.21%	44	15
	Important	698	64	49	7.02%	41	8
	Highlight	735	59	42	5.71%	36	6
	Pitfalls	428	15	10	2.34%	8	2
	Subtotal	2580	225	160	6.20%	129	31

Table 7: Clinical Expert Evaluation of Rubric Content Quality

Rubric Level	Total	Preliminary	Confirmed	Confirmed	Discrepancy Type
Rubric Level	Rubrics (A)	Discrepancies (B)	Discrepancies (C)	Rate (C/A)	Overestimated	Underestimated	Redundant
Essential	719	58	13	1.81%	13	0	0
Important	699	47	16	2.29%	12	3	1
Highlight	734	32	11	1.50%	0	3	8
Pitfalls	428	8	2	0.47%	0	0	2
Total	2580	145	42	1.63%	25	6	11

Table 8: Clinical Expert Evaluation of Rubric Weight Rationality

Rank	Model	Avg.	Base	Ablation Study (Score %)
Rank	Model	Length	Score (%)	w/ Saturation	w/ Truncation*	w/ Both*
1	GPT-5	1,007	79.84	68.88	78.30	67.63
2	Gemini 2.5 Pro	1,935	74.57	66.87	72.42	65.10
3	Qwen3-30B-A3B-Thinking-2507	2,044	74.54	66.15	71.64	63.72
4	DeepSeek-V3.2	1,629	71.43	64.51	68.37	61.93
5	GPT-5.1	1,473	70.90	63.99	68.07	61.62

Table 9: Comprehensive Performance of Mainstream LLMs without Output Length Constraints

Rank	Model	Avg.	Base	Ablation Study (Score %)
Rank	Model	Length	Score (%)	w/ Saturation	w/ Truncation*	w/ Both*
1	GPT-5	1,029	82.17	70.38	80.71	69.26
2	GPT-5.2	944	72.52	65.59	67.42	61.77
3	GPT-5.1	1,063	69.02	63.42	65.91	60.58
4	Qwen3-235B-A22B	980	67.71	60.94	63.28	57.26
5	Qwen3-30B-A3B-Thinking-2507	977	66.23	60.10	61.59	56.14

Table 10: Comprehensive Performance of Mainstream LLMs with Strict Length Constraint (

\leq

1000 words)

Rank	Model	Avg.	Overall	Dimensional Score Rate (%)
Rank	Model	Length	Score (%)	Essential	Important	Highlight	Pitfall
1	GPT-5	1,007	79.84	92.23	81.54	56.04	-4.30
2	Gemini 2.5 Pro	1,935	74.57	91.58	74.64	45.01	-6.39
3	Qwen3-30B-A3B-Thinking-2507	2,044	74.54	89.69	74.37	49.11	-6.23
4	DeepSeek-V3.2	1,629	71.43	89.48	72.29	41.86	-8.66
5	GPT-5.1	1,473	70.90	88.40	72.32	40.37	-7.78

Table 11: Dimensional Score Distribution under Unconstrained Conditions

Rank	Model	Avg.	Overall	Dimensional Score Rate (%)
Rank	Model	Length	Score (%)	Essential	Important	Highlight	Pitfall
1	GPT-5	1,029	82.17	93.90	85.09	58.36	-4.21
2	GPT-5.2	944	72.52	89.35	73.33	41.22	-5.15
3	GPT-5.1	1,063	69.02	87.92	69.21	36.63	-7.38
4	Qwen3-235B-A22B	980	67.71	84.74	69.62	45.11	-13.53
5	Qwen3-30B-A3B-Thinking-2507	977	66.23	83.49	66.57	42.93	-11.81

Table 12: Dimensional Score Distribution with Strict Length Constraint (

\leq

1000 words)

Model	HealthBench (HB)			QuarkMedBench (QB)			$\Delta$ Rank
Model	Score (%)	Avg. Length	Rank	Score (%)	Avg. Length	Rank	$\Delta$ Rank
GPT-5	65.56	1,364	1	79.84	1,007	1	–
GPT-5.2	62.72	1,267	2	66.19	788	7	$\downarrow$ 5
GPT-5.1	62.20	1,802	3	70.90	1,473	4	$\downarrow$ 1
Gemini 2.5 Pro	55.90	1,959	4	74.57	1,935	2	$\uparrow$ 2
Doubao Seed 1.6	54.78	1,425	5	61.91	1,052	9	$\downarrow$ 4

Table 13: Performance Distribution Comparison of Mainstream Models on HealthBench and QuarkMedBench

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Main Category	Subcategory	Percentage (%)	Overall (%)
Clinical Care	Nervous System	11.79	55.84
	Respiratory System	7.69
	Circulatory System	6.14
	Integumentary System	6.03
	Digestive System	5.87
	Musculoskeletal System	5.27

Appendix

Table 14: Query Scenario Distribution in Healthbench

Figure 4: Task classification distribution comparison between QuarkMedBench and Healthbench datasets. Left panel (1) shows the distribution across six main categories: Basic Inquiry, Pre-consultation, During-treatment, Post-treatment, Professional Medical, and Special Cases. Right panel (2) displays the top 15 specific task labels ranked by combined frequency. Percentages are calculated relative to the total number of samples in each dataset. QuarkMedBench demonstrates a balanced distribution across foundational categories including Basic Inquiry (30.0% vs 15.8%) and Pre-consultation (18.0% vs 12.1%), aligning with the full medical consultation lifecycle.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Jun 10, 2026

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.

Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7

NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4

LLM Evaluation LLM Reliability Medical QA Language Model Bias Evaluation Clinical Language Model Evaluation

Ask PDF

Feb 16, 2026

Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.

Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2

Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh

Medical QA Medical LLMs Clinical Language Model Evaluation LLM Evaluation

Ask PDF

Mar 13, 2026

MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences

Large language models (LLMs) are increasingly central to clinician workflows, spanning clinical decision support, medical education, and patient communication. However, current evaluation methods for medical LLMs rely heavily on static, templated benchmarks that fail to capture the complexity and dynamics of real-world clinical practice, creating a dissonance between benchmark performance and clinical utility. To address these limitations, we present MedArena, an interactive evaluation platform that enables clinicians to directly test and compare leading LLMs using their own medical queries. Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response. Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating. Only one-third of clinician-submitted questions resembled factual recall tasks (e.g., MedQA), whereas the majority addressed topics such as treatment selection, clinical documentation, or patient communication, with ~20% involving multi-turn conversations. Additionally, clinicians cited depth and detail and clarity of presentation more often than raw factual accuracy when explaining their preferences, highlighting the importance of readability and clinical nuance. We also confirm that the model rankings remain stable even after controlling for style-related factors like response length and formatting. By grounding evaluation in real-world clinical questions and preferences, MedArena offers a scalable platform for measuring and improving the utility and efficacy of medical LLMs.

Eric Wu, Kevin Wu, Jason Hom +11

Department of Electrical Engineering, Stanford University, Stanford, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Division of Hospital Medicine, Department of Medicine, Stanford School of Medicine, Stanford, CA, USA +8

Clinical Language Model Evaluation Medical LLMs Human Preference Evaluation Pairwise Comparison

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences