MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

Xuanjun Chen; Hua-Hsuan Chen; Wei-Chung Lu; Yinghao Ma; Jyh-Shing Roger Jang; Hung-yi Lee

cs.AISep 27, 2026

MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

Authors: Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu, Yinghao Ma, Jyh-Shing Roger Jang, Hung-yi Lee

Organizations: National Taiwan University · NTU AI-Core · Queen Mary University of London

Abstract

Rapid progress in Large Language Models (LLMs) is saturating static benchmarks faster than they can be designed. While existing automated evolution frameworks attempt to generate harder questions by perturbing individual tasks, they remain constrained by rigid, hard-coded generation rules. Moving beyond the evolution of isolated tasks, we propose to optimize the benchmark generation workflow itself end to end with MetaBench-Harness, a dual-loop search framework. Specifically, the inner loop utilizes a benchmark harness to generate a new benchmark in each round, while the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical evolution trajectories. By applying MetaBench-Harness to the competitive programming CodeContests and Olympiad mathematics AIME-2024 datasets, we demonstrate that the evolved benchmarks are challenging and discriminative for frontier models. Trajectory and quality analyses verify that MetaBench-Harness enables multi-dimensional evolution, steadily improving evolution reasonableness, benchmark competency, and evaluator robustness across successive rounds. Furthermore, case studies reveal its effective utilization of diverse difficulty levers to reframe problems and elevate required capabilities. Ultimately, this work provides a solution to the pressing challenge of benchmark saturation.

Figures & tables

Figure 1: Overview of the MetaBench-Harness framework.

Figure 2: The dual-loop procedure of MetaBench-Harness . (a) Meta-Harness Orchestration: The outer loop where the LLM agent, conditioned on trajectory memory, proposes harness updates. (b) Benchmark Generation: The inner loop where the evolved harness drives benchmark synthesis.

Evolution Methods	Note	DeepSeek	Kimi	Gemini	Qwen-32B	Qwen-7B
Baseline	AIME-2024	0.900	0.900	0.833	0.700	0.533
TRACE	Reported	N/A	N/A	N/A	0.533	0.393
AutoEvoEval	Reproduced	0.700	0.633	0.500	0.400	0.367
Naïve Prompting	Ablation	1.000	1.000	1.000	0.875	0.750
Fixed Harness	Ablation	0.817	0.767	0.633	0.600	0.133
		0.322	0.544	0.144	0.089	0.044

Table 1: Model performance on benchmarks evolved via different evolution methods.

Model	Type	Loop	MB-CodeContests			MB-AIME
Model	Type	Loop	Base.	Early	Late	Base.	Early	Late
Claude-Sonnet-5	AR	✓	0.835	0.948	0.839	0.733	0.750	0.633
GPT-5.4	NR	✓	0.586	0.591	0.293	0.067	0.125	0.100
DeepSeek-V3.1	NR	$\times$	0.736	0.845	0.396	0.667	0.125	0.133
Kimi-K2.5	NR	$\times$	0.694	0.834	0.739	0.833	0.333	0.167
Gemini-2.5-Flash	NR	$\times$	0.458	0.484	0.278	0.767	0.167	0.100

Table 2: Model performance across evolved benchmarks. Models are grouped into reasoning (R), non-reasoning (NR), and adaptive (AR) types. “Involve.” denotes whether the LLM was involved in the evolution loop; “Base.” is the seed benchmark, with “Early/Late” as different evolution rounds.

Model	Type	MB-CodeContests		MB-AIME
Model	Type	$\text{Base}\rightarrow\text{Late}$	$\Delta$ Tokens	$\text{Base}\rightarrow\text{Late}$	$\Delta$ Tokens
DeepSeek-V3.1	R	$11{,}223\rightarrow 13{,}382$	$(+2{,}159)$	$10{,}208\rightarrow 15{,}744$	$(+5{,}536)$
Kimi-K2.5	R	$15{,}879\rightarrow 16{,}732$	$(+853)$	$14{,}218\rightarrow 28{,}982$	$(+14{,}764)$
Gemini-2.5-Flash	R	$15{,}537\rightarrow 16{,}562$	$(+1{,}025)$	$11{,}254\rightarrow 17{,}020$	$(+5{,}766)$

Table 3: Comparison of model reasoning lengths under the base and late evolution datasets.

	MB-CodeContests	MB-AIME
Meta-Harness Orchestrated Proposals
Optimization rounds	10	5
Proposals applied	20/23 (87%)	10/12 (83%)
Edits Applied by the Harness
Measurement	11 (39%)	2 (20%)
Generation	9 (32%)	6 (60%)

Table 4: Harness-Level Evolutionary Changes.

	MB-CodeContests	MB-AIME
Meta-Harness Orchestrated Proposals
Optimization rounds	10	5
Proposals applied	20/23 (87%)	10/12 (83%)
Edits Applied by the Harness
Measurement	11 (39%)	2 (20%)
Generation	9 (32%)	6 (60%)

Table 4: Harness-Level Evolutionary Changes.

	#. Ori.	#. Evo.
MB-CodeContests
Total Questions	165	100
Solving Techniques	5.02	5.49
Capability	3.55	3.90
MB-AIME
Total Questions	30	30

Table 5: Benchmark Evolution Statistics.

Metric	MB-CC ∗	MB-AIME	Human
Reasonableness	95% / 91%	90% / 83%	85%
Competency	78% / 61%	80% / 78%	68%
Robustness	90% / 62%	93% / 69%	85%
Overall	70% / 44%	70% / 49%	80%
∗ MB-CC denotes MB-CodeContests.

Table 6: Evolution Quality Assessment Results: Final and overall average success rates. “Human” denotes the Human–AI Agreement, measuring consistency between human and agent.

Part I: Case Study of MB-CodeContests Evolution
Seed (Codeforces 1582D)	Round 5 (Evolved Emergence)	Round 9 (Evolved Deepening)
Task Formulation
Constructive Math : Given an array $a$ ( $a_{i}\neq 0$ ), construct $b$ such that $\sum a_{i}b_{i}=0$ and $\sum\|b_{i}\|\leq 10^{9}$ .	Single Gap Audit : Exploit a planted gap ( range(n-1) ) to trick a Python checker with an invalid $(a,b)$ pair.	Compound Audit : Trace complex logic to exploit a vulnerability gated by 3 conditions ( $n>1000$ , duplicate $a$ , odd $n$ ).
Evolution Summary
Capability: 3 Technique: 3 Lever: Baseline specification Baseline: The root foundation originating from a standard coding task.	Capability: 3 Technique: 5 Key Lever: Meta level shift, Output contract tightening Moderate: Shifts to basic code auditing and static inspection.	Capability: 4 Technique: 6 Key Lever: Compound conjunction, Scaffolding removal High: Mandates multi-technique fusion without hints.

Table 7: MetaBench-Harness Evolution Case Study: From Initial Seed to Deepening

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Part I: Case Study of MB-CodeContests Evolution
Seed (Codeforces 1582D)	Round 5 (Evolved Emergence)	Round 9 (Evolved Deepening)
Task Formulation
Constructive Math : Given an array $a$ ( $a_{i}\neq 0$ ), construct $b$ such that $\sum a_{i}b_{i}=0$ and $\sum\|b_{i}\|\leq 10^{9}$ .	Single Gap Audit : Exploit a planted gap ( range(n-1) ) to trick a Python checker with an invalid $(a,b)$ pair.	Compound Audit : Trace complex logic to exploit a vulnerability gated by 3 conditions ( $n>1000$ , duplicate $a$ , odd $n$ ).
Capabilities and Solving Techniques
Capabilities: Construction search, Domain reasoning, Precision scale control Solving Techniques: Constructive algorithm design, Exact integer arithmetic, Magnitude bound control	Capabilities: Adversarial analysis, Construction search, Output contract Solving Techniques: Code reading static analysis, Checker vulnerability exploitation, Counterexample construction, Program execution simulation, Protocol compliance	Capabilities: Adversarial analysis, Construction search, Precision scale control, Output contract Solving Techniques: Code reading static analysis, Checker vulnerability exploitation, Counterexample construction, Long horizon arithmetic stamina, Casework exhaustiveness, Protocol compliance
Applied Difficulty Levers

Appendix

Table 8: MetaBench-Harness Evolution Case Study: From Initial Seed to Deepening

Explore similar work

BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
May 31, 2026Yangzhen Wu, Aaron J. Li, Wenjie Ma +10Large Language Model Benchmarks Synthetic Task
University of California, Berkeley · Institute for Interdisciplinary Information Sciences, Tsinghua University
An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models
Jan 19, 2026Yuhiro Ono, Tomohiro Harada, Yukiya MiuraEvolutionary Optimization Methods
Graduate School of Systems Design, Tokyo Metropolitan University, 6-6, Asahigaoka, Hino, 1910065, Tokyo, Japan. · Graduate School of Science and Engineering, Saitama University, 2552026 Shimo-okubo, Sakura-ku, Saitama, 3388570, Saitama, Japan.
AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation
Jun 30, 2026Xinyuan Song, Zekun Cai, Liang ZhaoCode Generation
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan

May 31, 2026, cs.SEJ/K move · Enter open · S save

BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

Yangzhen Wu, Aaron J. Li, Wenjie Ma, Li Cao+9

University of California, Berkeley · Institute for Interdisciplinary Information Sciences, Tsinghua University

Large Language Model Benchmarks Synthetic Task Synthesis

Open paper AskPDF arXiv

The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.

Abstract

Explore similar work

BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models

AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation