BenchBench: Benchmarking Automated Benchmark Generation

Yandan Zheng; Haoran Luo; Zhenghong Lin; Wenjin Liu; Luu Anh Tuan

cs.CLMar 21, 2026

BenchBench: Benchmarking Automated Benchmark Generation

Authors: Yandan Zheng, Haoran Luo, Zhenghong Lin, Wenjin Liu, Luu Anh Tuan

Organizations: Nanyang Technological University Singapore, Singapore

Abstract

Benchmarks are the de facto standard for tracking progress in large language models (LLMs), yet static test sets can rapidly saturate, become vulnerable to contamination, and are costly to refresh. Scalable evaluation of open-ended items often relies on LLM judges, introducing additional sources of bias and prompt sensitivity. We argue that evaluation must extend beyond how well models answer benchmarks to how well models design them. We introduce BenchBench, a three-stage pipeline and dataset for benchmarking automated benchmark generation: (i) extract structured domain cards from seed benchmarks, (ii) prompt multiple designer LLMs to generate quota-controlled suites, and (iii) validate items with a multi-model answerer panel using exact/numeric/symbolic verifiers when possible and rubric-guided judging otherwise, yielding designer--answerer matrices with item-level quality flags and psychometric diagnostics. Across nine variants spanning computer science, mathematics, medicine, and theory-of-mind reasoning (including multilingual and multimodal settings), we generate 16.7K items, retain ~15K core items post-filtering, and produce ~152K graded model--item responses. BenchBench shows that benchmark-design ability is only moderately correlated with answer-time strength (Spearman rho ~~0.37), invalidity is negatively associated with discrimination (Pearson r~~0.62), and the resulting designer--answerer matrices enable scalable audits of format/modality/language fidelity and suite-dependent self/family interactions. The project is available at: https://github.com/koanatakiyo/BenchBench.

Figures & tables

Figure 1. Benchmark creation is shifting from expert curation to human+LLM co-creation; BenchBench benchmarks the next step: auto-benchmarks . It makes generated suites measurable via domain cards, quota-controlled generation, and panel-based validation, yielding designer–answerer matrices for auditing validity, diagnostic utility, and interaction effects. Saturation trends adapted from ( Hendrycks et al., 2021 ; Epoch AI, 2025 ) .

Figure 2. BenchBench knowledge-guided pipeline. Stage 1 extracts deterministic domain cards from seed benchmarks using text (and vision when applicable) oracles plus offline canonicalization. Stage 2 generates quota-controlled benchmark suites conditioned on the domain card and closes coverage gaps via deficit-driven top-ups. Stage 3 performs post-hoc cleaning, runs a multi-model answerer panel, routes scoring via exact/numeric/symbolic matching or LLM judging, and applies static/dynamic quality gates to produce a core designer $\times$ answerer response matrix and downstream evaluation metrics.

Variant	#Designers	#Answerers	#Core / #Items
CSBench (EN)	8	12	2,051 / 2,390
CSBench (ZH)	6	10	1,643 / 1,780
ToMBench (EN)	6	10	1,594 / 1,790
ToMBench (ZH)	6	10	1,513 / 1,796
MedXpertQA (Text)	6	10	1,684 / 1,802
MedXpertQA (MM-T))	6	10	1,645 / 1,800

Table 1. BenchBench overview (evaluated variants). #Items/#Core count unique generated questions aggregated across designers within each variant. CSBench-EN uses an extended designer/answerer panel; all other variants use 6 designers and 10 answerers. Shorthand dataset labels correspond to full identifiers as follows: MedXpertQA (Text) is medxpertqa_text; MedXpertQA (MM-T) is medxpertqa_mm_stage2_textonly; MedXpertQA (MM-V) is medxpertqa_mm_stage2_visualprimed; WeMath (Text) is wemath_stage2_textonly; and WeMath (Visual) is wemath_stage2_visualprimed.

Designer	Broken%	MeanDiscr
gpt-5-mini	3.5	0.301
llama-4-maverick	10.9	0.241
deepseek-chat	7.8	0.213
doubao-seed-1-6-flash-250828	12.6	0.209
gemini-2.5-flash	5.0	0.205
qwen3-next-80b-a3b-instruct	14.1	0.189

Table 2. Aggregated designer leaderboard across all variants (core set). Broken% is the non-core rate (lower is better); MeanDiscr is mean item discrimination (point-biserial) on hard-scored core items (higher the better). Full metrics are reported in the supplementary material Table 10 .

Figure 3. Validity–discrimination tradeoff across designers (pooled across variants). Broken% is the non-core rate; MeanDiscr is averaged over hard-scored core items.

Figure 4. Ranking preservation across variants (Kendall’s $\tau$ ), comparing designer-induced answerer rankings against a reference ranking. Higher values indicate greater preservation of established model order.Variant labels follow Table 1 .

Figure 5. Self/family bias in the designer–answerer matrix. Left: own-family advantage (accuracy on own-family items minus accuracy on other-family items, percentage points). Right: family-level accuracy matrix (darker = higher accuracy). Cells are aggregated over answerers; rows/columns grouped by model family.

Figure 6. Adversarial effect on strong-panel accuracy. Each cell shows the drop in mean accuracy (standard minus adversarial subset, the higher the better adversarial design) on the strong answerer subset. e.g., $+$ 62.1% for Qwen3-Next on TomBench (CN) indicates effective adversarial design.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Intent ( design_type )
Value	Items		Core
	n	%	n	%
standard	12,620	75.7	11,597	77.9
adversarial	4,038	24.2	3,289	22.1
other/invalid	11	$<$ 0.1	7	$<$ 0.1
Difficulty ( declared_difficulty )

Appendix

Table 3. Pooled suite composition (Items vs. Core). Items count unique generated questions across the nine evaluated variants (Table 1 ). Core retains items with item_status

\in

{ clean , suspect_static } and scoring_method

\neq

skip_core . Percentages are within Items (

N=16{,}669

) and within Core (

N=14{,}893

Label	Scale	$\alpha$
fatal_flaw	nominal (0/1)	0.32
ambiguity	ordinal (1–5)	-0.88
key correctness	ordinal (1–5)	0.02
solvability	ordinal (1–5)	-0.43
format compliance	ordinal (1–5)	-0.22

Appendix

Table 4. Inter-rater agreement in E4 (Krippendorff’s

\alpha

N=146

). We compute

\alpha

for fatal_flaw using nominal distance, and for the Likert (1–5) dimensions using ordinal distance. Negative

\alpha

indicates systematic disagreement and can arise under scale compression or rater calibration differences.

Designer	n	Fatal%	Broken%	NegDiscr%
gpt-5-mini	25	0.0	3.5	7.0
gemini-2.5-flash	24	4.2	5.0	9.8
deepseek-chat	25	0.0	7.8	9.5
doubao-seed-1-6-flash-250828	22	0.0	12.6	8.8
llama-4-maverick	25	8.0	10.9	7.8
qwen3-next-80b-a3b-instruct	25	8.0	14.1	12.3

Appendix

Table 5. Human audit vs. automatic metrics (per designer). fatal_flaw uses the union-of-raters definition; automatic metrics are computed over all generated suites (per-designer summaries are included in the released artifact).

Panel of Figure 7 — Figure 7. Example rendered domain cards. Each plot summarizes the extracted topic ontology (super-parent hierarchy) from a seed benchmark; the domain card also includes representative terms and language/modality constraints.

Designer	Broken%	MeanDiscr	NegDiscr% (all)	Mean $p(\mathrm{correct})$	#Core
qwen3-next-80b-a3b-instruct	5.0	0.434	5.7	0.803	285
gpt-5-mini	1.7	0.391	4.3	0.865	294
doubao-seed-1-6-flash-250828	6.7	0.366	8.0	0.833	280
gemini-2.5-flash	3.9	0.346	8.8	0.830	274
deepseek-chat	3.7	0.277	8.7	0.839	287
llama-4-maverick	23.9	0.270	9.1	0.842	226

Appendix

Table 6. Designer leaderboard for dataset csbench_cn.

Designer	Broken%	MeanDiscr	NegDiscr% (all)	Mean $p(\mathrm{correct})$	#Core
gpt-5-mini	2.1	0.235	7.1	0.887	275
llama-4-maverick	15.7	0.225	8.3	0.841	253
qwen3-next-80b-a3b-instruct	13.7	0.222	6.8	0.819	253
gemini-2.5-flash	5.0	0.220	7.0	0.866	284
deepseek-chat	10.3	0.128	12.7	0.859	269
doubao-seed-1-6-flash-250828	19.6	0.105	11.3	0.766	221

Appendix

Table 7. Designer leaderboard for dataset medxpertqa_mm_stage2_visualprimed.

Designer	Broken%	MeanDiscr	NegDiscr% (all)	Mean $p(\mathrm{correct})$	#Core
grok-4-1-fast	6.8	0.307	5.1	0.880	274
llama-4-maverick	6.4	0.298	5.4	0.902	276
gpt-5-mini	4.3	0.291	5.1	0.896	265
qwen3-next-80b-a3b-instruct	8.1	0.242	8.1	0.846	262
claude-sonnet-4-5-20250929	9.6	0.229	7.8	0.857	244
doubao-seed-1-6-flash-250828	10.4	0.209	6.2	0.894	232

Appendix

Table 8. Designer leaderboard for dataset csbench_en.

Figure 8. Family bias on CSBench-EN with extended designer families. Left: own-family advantage $\Delta$ Acc (percentage points), comparing answerer accuracy on items authored by its own family vs. other families. Right: family-level designer–answerer accuracy matrix (darker indicates higher accuracy).

Answerer family	Acc own	Acc other	$\Delta$ Acc (pp)	$n_{\mathrm{own}}$	$n_{\mathrm{other}}$
gpt	0.936	0.910	+2.61	529	3587
claude	0.790	0.881	-9.10	243	1540
gemini	0.888	0.870	+1.77	498	3619
grok	0.927	0.926	+0.15	274	1786
deepseek	0.915	0.908	+0.70	258	1800
qwen3	0.864	0.895	-3.14	521	3594

Appendix

Table 9. CSBench-EN family bias statistics. Acc own / Acc other denote answerer-family accuracy on items designed by its own family vs other families;

\Delta

Acc is the difference in percentage points.

n_{\mathrm{own}}

and

n_{\mathrm{other}}

count evaluated (answerer, item) pairs under each condition.

Figure 9. Format distribution across designers (pooled across all generated suites). Each bar shows the proportion of items by question type (mcq_single, mcq_multi, open_ended, structured, judgment).

n

= total unique items after deduplication.

Figure 10. Visual-primed vs. text-only effects in MedXpertQA-MM. Each bar shows the change in mean discrimination and mean

p(\mathrm{correct})

when switching from textonly to visualprimed generation (per designer).

Figure 11. Visual-primed vs. text-only effects in WeMath. Each bar shows

\Delta

mean discrimination (left) and

\Delta

mean

p(\mathrm{correct})

(right) when switching from text-only to visual-primed generation (per designer).

Figure 12. Correspondence between Model Capability and Benchmark Design Quality for models appearing in both roles. The Design Quality Index (DQI) is a composite metric defined as

\text{DQI}=Z(\text{Discrimination})-Z(\text{Broken Rate})-Z(\text{Negative Rate})

, which rewards the generation of valid, discriminating questions while penalizing broken or ambiguous items.

Designer	Broken%	MeanDiscr	NegDiscr% (all)	Mean $p(\mathrm{correct})$	#Core
gpt-5-mini	3.5	0.301	7.0	0.826	707
llama-4-maverick	10.9	0.241	7.8	0.840	675
deepseek-chat	7.8	0.213	9.5	0.812	702
doubao-seed-1-6-flash-250828	12.6	0.209	8.8	0.857	688
gemini-2.5-flash	5.0	0.205	9.8	0.836	1281
qwen3-next-80b-a3b-instruct	14.1	0.189	12.3	0.808	1084

Appendix

Table 10. Aggregated designer leaderboard via auto-metrics across all variants. Broken% represents the fraction of invalid (non-core) items (lower is better for validity); MeanDiscr is the average point-biserial discrimination on core items (higher is better for diagnostic utility); NegDiscr% (all) denotes the percentage of negatively discriminating items (lower is better to minimize biases); Mean p(correct) proxies item difficulty as the average proportion correct (lower is better for challenge and anti-saturation); #Core counts retained core items (higher is better for yield). These psychometrics underscore BenchBench’s ability to quantify generation tradeoffs, enabling scalable audits of LLM designer prowess.

Explore similar work

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Feb 25, 2026Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1Benchmark Validity Benchmark Design
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States
Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM
Aug 17, 2026Atul AnandLLM-as-a-Judge Language Model Generation Evaluation
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM Evaluation Benchmark Auditing
Aleph Alpha Research, Heidelberg, Germany.

Feb 25, 2026J/K move · Enter open · S save

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Harshita Diddee, Gregory Yauney, Swabha Swayamdipta, Daphne Ippolito

Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Benchmark Validity Benchmark Design

Open paper AskPDF arXiv

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Abstract

Explore similar work

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation