EvalBuilder: Can LLMs Build Efficient Benchmarks?

Huaisheng Zhu; Hangfan Zhang; Teng Xiao; Siyuan Xu; Shijie Zhou; Zhimeng Guo; Zhengyu Chen; Jingang Wang

cs.AIOct 14, 2026

EvalBuilder: Can LLMs Build Efficient Benchmarks?

Authors: Huaisheng Zhu, Hangfan Zhang, Teng Xiao, Siyuan Xu, Shijie Zhou, Zhimeng Guo, Zhengyu Chen, Jingang Wang

Organizations: Meituan · Pennsylvania State University · Allen Institute for AI (AI2) · University of Washington ·

Abstract

Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.

Figures & tables

Figure 1: Overview of EvalBuilder for building efficient and effective Benchmarks.

Construction agent	$N$	Pairwise accuracy $\uparrow$			$\rho\uparrow$	MAE $\downarrow$	Bias $\to 0$	Spread $\to 1$
		All	Visible	Held-out
Terminal-Bench 2.1
GLM-5.2	50	0.489	0.381	0.583	0.042	0.277	$+0.277$	0.263
DeepSeek-V4-Pro	50	0.622	0.667	0.583	0.297	0.153	$\mathbf{+0.134}$	0.877
DeepSeek-V4-Flash	65	0.444	0.571	0.333	$-0.176$	0.336	$+0.336$	0.121
GPT-5.4	50	0.467	0.524	0.417	$-0.055$	0.158	$+0.158$	0.057

Table 1: Results from confirmed evaluations. All, Visible, and Held-out report pairwise accuracy on

\mathcal{P}

\mathcal{P}_{\mathrm{vis}}

, and

\mathcal{P}_{\mathrm{held}}

, respectively.

N

is the number of tasks evaluated;

\rho

, MAE, Bias, and Spread are computed across all ten candidates. Bold marks the best value in each metric column within each reference, including ties; Bias and Spread are best at zero and one, respectively.

Figure 2: Pairwise accuracy relative to random ordering for each result(accuracy minus 50%). The zero line denotes chance performance. Visible-only bars show visible–visible pairs; held-out bars aggregate visible–held-out and held-out–held-out pairs by total correct and total comparisons.

Figure 3: Evaluation Time of Agent Benchmarks and Generated Lightweight Benchmarks.

Figure 4: Case Study of Model’s Strategy to improve performance of building efficient benchmarks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Construction agent	Iter./Final.	Invalid	$N$	Time	Stop
SWE-bench Verified
GLM-5.2	6/1	1	50	8h 07m	Agent submit
DeepSeek-V4-Pro	10/1	1	50	4h 33m	Agent submit
DeepSeek-V4-Flash	4/1	2	50	5h 37m	Agent submit
GPT-5.4	6/1	1	41	7h 00m	Agent submit
Claude-Opus-4.7	10/1	–	50	13h 07m	Agent submit

Appendix

Table 2: Construction histories for the selected runs. Iter./Final. counts refinement and finalization rounds; Invalid lists rejected saved rounds.

N

is the independently confirmed task count. Times include recorded construction checks.

Figure 5: Construction time for the selected runs. Dashed lines mark the 16-hour budget. The arrow for GLM-5.2 on Terminal-Bench denotes a recorded lower bound because its final timing is unavailable.

$w_{p}$	$w_{j}$	$w_{s}$	PAcc	Pearson
50	20	30	0.70	0.563
55	20	25	0.75	0.577
60	20	20	0.70	0.589
50	25	25	0.75	0.556
45	25	30	0.70	0.541
50	15	35	0.60	0.564

Appendix

Table 3: Weight combinations tried by GLM-5.2 in iteration 3, as reported in its revision summary. Weights are percentages. Bold marks the selected mixture; these are visible-set development results.

Benchmark Construction

Benchmark construction develops tasks, datasets, environments, and protocols that make AI capabilities measurable and comparisons reliable.

See all 67 papers

Benchmark Everything Everywhere All at Once
Jun 4, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5LLM Evaluation Benchmark Design
MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen +3
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
Apr 27, 2026Xinming Tu, Tianze Wang, Yingzhou +4Benchmark Auditing AI Agent Benchmarks
Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1

Jun 4, 2026J/K move · Enter open · S save

Benchmark Everything Everywhere All at Once

Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai+4

MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen+3

LLM Evaluation Benchmark Design Synthetic Benchmark Generation

Open paper AskPDF arXiv

Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.

Abstract

Benchmark Construction

Explore similar work

Benchmark Everything Everywhere All at Once

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks