More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

Ziyang Xu; Haitian Zhong; Hao Zhou; Hao Qin; Chenhan Jin; Te Qi; Shengze Xu; Tieyong Zeng

cs.AISep 26, 2026

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

Authors: Ziyang Xu, Haitian Zhong, Hao Zhou, Hao Qin, Chenhan Jin, Te Qi, Shengze Xu, Tieyong Zeng

Organizations: Department of Mathematics, The Chinese University of Hong Kong Hong Kong, China · New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy · Department of Basic Research, SINOPEC Research Institute of Petroleum Processing ·

Abstract

Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre-execution selection. On 386 MATH-500 tasks, we compare eight generated harnesses plus a baseline with nine byte-identical baseline copies, using three executions per member. Identical programs yield 2.16 percentage points of repeat-averaged oracle headroom. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on 100 tasks, while persistent wins occur on only one task and are sensitive to answer extraction. The frozen selector gains 0.00 percentage points, and both populations reach 98.70% oracle coverage at 27 harness executions. Stable complementarity remains unresolved at three repeats. Supporting BIRD traces locate failures in mechanism implementation, activation, and output validity. Together, these findings establish why coverage and repeatability alone cannot justify claims of useful specialization. They motivate an evaluation standard for harness diversity: task advantages should persist across executions, guide usable decisions, and improve on additional fixed-program executions under matched inference budgets.

Figures & tables

Figure 1: Coverage, repeatability, and selection on $386$ complete MATH tasks. Oracle coverage matches executions, not tokens/calls; headroom uses repeat means. Persistent task counts require at least one generated member to win/lose against baseline in all three repeats; the sole win is extraction-sensitive. Here $G$ is held-out task-wise selection gain over the discovery-selected fixed member; panel (b) labels residual Pearson correlation as $r$ . R1–R3 are repeat labels; pp denotes percentage points and CI denotes confidence interval. Intervals are 95% bootstrap CIs. Program drawings are schematic.

Question	Key comparison	Readout
Additional coverage	Full pool vs. best fixed	Oracle headroom $H(P)$
Repeat transfer	Generated vs. same-code	Held-out gain $D$
Usable selection	Frozen selector vs. dev-fixed	Selection gain $S(\pi_{Z})$
Budgeted coverage	Portfolio vs. repeated baseline	Oracle coverage

Table 1: Comparisons separating coverage from usable specialization. Oracle comparisons use correctness after execution; the frozen selector chooses from the question before execution. Dev-fixed denotes the development-selected fixed member.

D

is generated-panel minus clone held-out selection gain (Equation 3 );

S(\pi_{Z})

is the frozen selector’s accuracy gain over dev-fixed.

Clone-adjusted gain: $D=-0.26$ pp; 95% CI $[-1.45,0.95]$ pp
Measure	Generated + single-call	Same-code clones
Oracle headroom $\widehat{H}$ (pp)	$2.33$	$2.16$
Residual correlation $\rho_{\rm res}$	$0.789$	$0.006$
Held-out selection gain $G$ (pp)	$0.10$	$0.35$

Table 2: Coverage, repeatability, and selection gain separate. MATH-500, selected panel versus the same-code control (§ 4.1 ).

\widehat{H}

uses repeat means;

G

tests choices on held-out repeats.

D

is their paired, unrounded between-arm difference, so displayed means may not subtract exactly.

Harness executions per task $b$	3	9	15	27
Panel members, including single-call	1	3	5	9
Panel oracle coverage (%)	98.45	98.70	98.70	98.70
Clone oracle coverage (%)	98.24	98.63	98.69	98.70
Panel minus clones (pp)	$+0.20$	$+0.08$	$+0.02$	$+0.00$

Table 3: Full-budget oracle coverage ties in the recorded executions. MATH-500 coverage from the development-ranked panel and uniform same-code sampling. Clone values are exact subset means (§ 4.3 ). Budgets count harness executions; model-call and token costs vary by program.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Meaning
Members, tasks, and outcomes (§ 3 )
$P$ , $h$ , $M$	Harness population, an indexed member $h\in P$ , and population size $M=\|P\|$ . Identical clone slots have distinct member identities.
$H_{0}$ , $H_{1}$	Single-call baseline and the other member in the two-member identity.
$x$ , $x_{i}$ , $i$ , $n$	A task, evaluated task $i$ , its index, and the number of evaluated tasks.
$r$ , $R$	Execution-repeat index and number of repeats per member–task pair; empirical repeat comparisons use $R=3$ .
$Y(h,x,r)$	Binary correctness of member $h$ on task $x$ in repeat $r$ .

Appendix

Table 4: Symbols, definitions, and scope.

Frozen member	Source-defined execution path
single-call	One temperature- $0$ solution with the final-answer format.
H1 (DeepSeek)	Three temperature- $0.4$ solutions; normalized-answer vote; a greedy fallback when no answer or a tie remains.
H2 (ERNIE)	Draft a solution, then ask the solver to verify/correct the extracted answer; return the checked answer.
H3 (GLM)	Up to five differently prompted temperature- $0$ solutions; stop when three agree; adjudicate unresolved disagreement, with format fallbacks.
H4 (Kimi)	Request five temperature- $0.7$ samples in one call; extract answers and vote.
H5 (Kimi)	Initial solution, verification, and an alternative solution; vote, with an additional resolution call if no majority emerges.

Appendix

Table 5: What was selected for the repeat study. All calls use the same frozen solver; verification and adjudication are solver prompts, not access to benchmark labels. Six seats are builder-stratified; H5 (Kimi) and H7 (MiniMax) are the two at-large seats.

Role	Model (exact version string)	Used in
Frozen target solver	GLM-5.3-Flash (baseline temperature $0$ )	MATH / BIRD
BIRD discovery builder	GLM-5.3-Flash	App. F.1
MATH / BIRD admission builders	GLM-5.3 , Qwen3.8-Max , DeepSeek-V4-Pro , Kimi-K3 , MiniMax-M3 , ERNIE-5.0-Thinking-Preview	MATH / BIRD
Judge	proxy / official BIRD execution scorer; MATH answer comparator	see text

Appendix

Table 6: Model inventory for the reported benchmark studies. The target is held frozen within each phase; Phase-II builders come from six independent labs.

Observed characteristic	Included ( $386$ )	Excluded ( $14$ )
Baseline repeat-mean accuracy (%)	95.85	71.43
Mean benchmark difficulty level	3.47	4.29
Level-5 tasks (%)	26.42	64.29
Mean question length (characters)	198.75	284.71

Appendix

Table 7: Observed characteristics of included and excluded tasks. Excluded tasks have higher benchmark difficulty, longer questions, and lower baseline accuracy.

regime	plug	v3 ∗	plug	v3 ∗	plug	v3 ∗	$R{=}3$
	$R{=}3$		$R{=}10$		$R{=}20$		abst.
equal $p{=}0.9$ †	100	0	100	0	100	1	100
equal + difficulty †	100	0	100	1	100	1	100
$0.5/0.9$ dominance †	100	0	100	0	13	0	100
dominance †	100	0	100	0	100	0	100
cross $H{\approx}0.9$ pp	100	1	100	0	100	0	99

Appendix

Table 8: Three repeats can miss even a strong positive case. At

M=9,T=400

, null rows report false-support % and alternative rows report power %; the final column gives

R=3

abstention. The simulated clone-calibrated rule (v3 ∗ ) rarely declares positive support when stable complementarity is zero, but has little sensitivity to the tested small and moderate crossovers.

Policy	Eval acc. (%)	$S$ vs dev-fixed (pp) [95% CI]
single-call (fixed)	95.85	0.00 [0.00, 0.00]
dev-fixed	95.85	—
One randomly drawn fixed member	86.01	-9.84 [-13.04, -6.91]
Dev-fixed + baseline fallback	95.85	0.00 [0.00, 0.00]
$\pi_{Z}$ (frozen selector)	95.85	0.00 [0.00, 0.00]

Appendix

Table 9: Frozen selection on MATH-500. Accuracy is repeat-mean correctness on fresh executions;

S

is the paired gain over the development-selected fixed member, with 95% task-bootstrap intervals.

Figure 2: Source differences need not change observed correctness. BIRD discovery:

14

harnesses,

60

tasks, one execution each. Across

91

pairs including baselines, token-sequence source similarity has

\rho=-0.010

rank association with correctness disagreement;

36/91

pairs have identical correctness vectors. This is an outcome comparison on the observed tasks, not a test of program equivalence.

Figure 3: BIRD admission design. Left: data splits and disclosed development exposure. Middle: the

2{\times}2

four-arm comparison with a common attempt cap. Right: neutral validity and the implemented structural-screening/conformance bundle in gated arms. The additional open-mechanism arm II-E is development-only.

Metric	II-A	II-D
Candidate count	5.78	3.83
Oracle accuracy (%)	76.14	75.70
Best-fixed accuracy (%)	67.58	67.30
Headroom (pp)	8.56	8.40
D $-$ A headroom (pp)	$-0.16$
Unadjusted bootstrap 95% CI	$[-0.83,\ +0.87]$

Appendix

Table 10: BIRD admission: full-set headroom. Means over builder–seed cells on

1{,}169

questions; single-call is included in both the oracle and best-fixed comparators.

Figure 4: BIRD admission contrasts. Left: full-set D

-

A cell contrasts with the pooled 95% interval. Right: four-arm headroom on the shared core-

400

tasks with a common single-call baseline. Headroom and contrast axes show percentage points.

Contrast	Effect (pp)	95% CI (pp)	Raw $p$	Holm $p$
Gate	$-0.69$	$[-1.56,+0.20]$	0.026	0.079
Strategy	$+0.42$	$[-0.32,+1.51]$	0.263	0.428
Interaction	$-0.87$	$[-2.83,+0.65]$	0.214	0.428

Appendix

Table 11: Corrected core-

400

factorial reanalysis. Here A–D denote arm mean headrooms. Gate is

[(B{-}A)+(D{-}C)]/2

, strategy

[(C{-}A)+(D{-}B)]/2

, interaction

(D{-}C)-(B{-}A)

; raw and Holm-adjusted

p

-values.

Arm	$K_{\rm cand}$	$K_{\rm total}$	Single-call (%)	Best (%)	Oracle (%)	$H$ (pp)
A	5.78	6.78	63.25	66.00	74.51	8.51
B	6.11	7.11	63.25	66.12	74.39	8.26
C	5.61	6.61	63.25	65.64	75.01	9.38
D	3.83	4.83	63.25	65.65	73.90	8.25

Appendix

Table 12: Same-core headroom decomposition. All arms use identical 400 tasks and identical single-call outcomes.

K_{\rm cand}

counts admitted candidates, excluding single-call, and

K_{\rm total}=K_{\rm cand}+1

; counts and accuracies are cell means. Headroom is oracle minus best-fixed accuracy, not realized routing gain.

Sensitivity estimand	Estimate (pp)	95% CI (pp)
K-matched B $-$ A headroom	$-0.55$	$[-1.53,+0.63]$
K-matched D $-$ C headroom	$-0.08$	$[-1.26,+0.80]$
K-matched gate mean	$-0.31$	$[-1.07,+0.40]$
R2 subset D $-$ A (cached)	$+0.42$	$[-0.61,+1.48]$
R2 subset D $-$ A (cache-off)	$+0.81$	$[-0.35,+1.84]$
R2 change in subset D $-$ A	$+0.39$	$[-0.95,+1.63]$

Appendix

Table 13: BIRD sensitivity on core-400. CIs are unadjusted. K-matched comparisons always retain single-call; R2 comparisons exclude single-call and are conditional on the observed acquisition groups.

Arm	Candidate-task records	Logged solver calls	Calls/task/population
A	41,600	67,258	9.34
B	44,000	64,399	8.94
C	40,400	75,604	10.50
D	27,600	52,043	7.23

Appendix

Table 14: BIRD generation and execution cost on core-400. Logged logical calls are not billable requests or token counts. The last column averages candidate-population calls per task over cells.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

Evaluating Multiple LLM Generations with Validated Task Coverage
Aug 25, 2026Florian Le Bronnec, Rio YokotaOutput Diversity in Language Models LLM Evaluation
RIKEN Center for Computational Science, Tokyo, Japan
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI
How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation
May 30, 2026Yongxi Zhou, Lai Yun Choi, Jiaxi Wen +2LLM Evaluation Language Model Generation Evaluation
Northeastern University, Massachusetts, USA · University of Southern California, California, USA

Aug 25, 2026J/K move · Enter open · S save

Evaluating Multiple LLM Generations with Validated Task Coverage

Florian Le Bronnec, Rio Yokota

RIKEN Center for Computational Science, Tokyo, Japan

Output Diversity in Language Models LLM Evaluation Language Model Generation Evaluation

Open paper AskPDF arXiv

Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation settings, however, still focus on individual outputs or reduce multiple samples to a single success or selected answer. This can miss whether the outputs include several genuinely different useful results. We introduce VTC-Bench, a five-domain benchmark for this setting, together with Validated Task Coverage (VTC) as its core evaluation quantity. The benchmark is built from carefully selected real-data tasks where both output quality and task-relevant distinctness can be checked automatically and reproducibly, without model-based judges. VTC measures how many distinct useful results are obtained within

k

attempts. Across multiple models and inference settings, the benchmark leads to different conclusions from conventional evaluation: configurations that look strongest from single-draw quality are not necessarily those with the best coverage, and simple measures of output variation do not reliably recover task-relevant coverage. These results show that finite candidate sets can be evaluated directly as objects of interest, revealing differences in model behavior that are not apparent from conventional per-output evaluation.

Abstract

LLM Agent Harnesses

Explore similar work

Evaluating Multiple LLM Generations with Validated Task Coverage

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation