What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

Bowen Xu; Boyu Chen

cs.LGOct 4, 2026

What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

Authors: Bowen Xu, Boyu Chen

Organizations: Department of Biology, Stanford University, Stanford, CA, USA · Institute of Health Informatics, University College London, London, UK

Abstract

Harness search keeps a change to the prompts, reasoning switches, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison, or a defect in the evaluation. We preregistered a study of where these gains come from, with three small models, three benchmarks, replication and test partitions, a GEPA search arm and six evaluation defects injected one at a time, and we report all 47 primary endpoints. Turning thinking off raised accuracy over a capped thinking setting in 5 of 9 model-benchmark cells, and in each the gain came mostly from questions where the capped setting gave no readable answer. The thinking-off setting was not meaningfully worse than a rescue configuration or four GEPA-selected harnesses in 11 of 13 comparisons, and lost to the rescue on GSM8K for two models. GEPA repaired its broken starting points, but none of its selected harnesses was more accurate than the thinking-off setting. A thinking budget in the serving engine, which also allows a longer answer, lowered truncation and raised the parse rate in 6 of 9 cells. In 6 of 15 evaluable defect-model pairs, replication through the same pipeline reproduced the defect's distortion instead of revealing it. On the LongevityBench multiple-choice tasks, only the longevity-tuned model beat the strongest constant-label baseline.

Figures & tables

Configuration	Request
THINK-64 , THINK-512 , THINK-1536	thinking on; 64, 512 or 1,536 tokens for thinking and answer together ( THINK-64 is the case study’s parent)
AUTHORS-1500	thinking off; 1,500 tokens (the benchmark authors’ serving protocol)
BUDGET-512-1500	thinking on; vLLM ends thinking after 512 tokens ( Muennighoff et al., 2025 ) ; up to 1,500 tokens of answer follow, so 2,012 tokens in all against 512 for THINK-512
RESCUE-1500	THINK-512 ; where its call ended at the token limit, the AUTHORS-1500 call on the same row (the call that gives the answer decides the properties of the row)
GEPA harnesses	four runs (clean or broken start, two seeds), 6,000 metric calls on 300 search rows, Intern-S1-mini as proposer; the selected harness edits the system prompt and appends an instruction, so the arm searches prompt text only (Appendix J )

Table 1: Configurations (“tokens” is max_tokens ; thinking on or off is set in every request). RESCUE-1500 is derived from the two grid runs and verified against the real two-call sequence on 300 items per model.

	Claim and estimand	Verdict	Endpoints (family)
H1	Most of the accuracy that AUTHORS-1500 gains over THINK-512 comes from rows THINK-512 leaves unparsed. Gate: $\Delta$ ; statistic: $T$ .	no gain: not tested unless the lower bound of $\Delta$ is above zero; if it is, visibility dominates if the lower bound of $T$ is above zero, quality matters if the upper bound of $T$ is below zero, inconclusive otherwise.	9: one per cell
H2	AUTHORS-1500 is not meaningfully worse than a comparator: $\theta=\mathrm{acc}(\text{\mbox{{\small AUTHORS-1500}}})-\mathrm{acc}(\text{comparator})$ .	holds if the lower bound of $\theta>-\delta$ ; fails if the upper bound $<-\delta$ ; else inconclusive .	13: RESCUE-1500 in nine cells; four GEPA harnesses ( Qwen3-1.7B-Longevity , letters)
H3	Against THINK-512 , BUDGET-512-1500 (a forced end of thinking and a larger allowance) lowers truncation ( $D_{T}$ ) and raises parsing ( $D_{P}$ ).	holds if both lower bounds $>m$ ; fails if either upper bound $<m$ ; no room if the upper bound of THINK-512 ’s truncation is $<m$ (applied first); else inconclusive .	9: one per cell
H4	For each defect and model, prospective replication confirms the defect’s distortion (Section 3.4 ).	holds , fails or inconclusive from three intervals; not evaluable if the defect cannot move the start by more than $m$ .	16 pairs, fixed

Table 2: Hypotheses, estimands and verdict rules.

m

is the minimal difference and

\delta

the non-inferiority margin, both 0.05. Every bound in a verdict is a bound of the family-adjusted interval, and H1’s gate uses the same resamples and the same adjusted level as

T

Model	Pool	H1 AUTHORS-1500 vs THINK-512	H2 AUTHORS-1500 vs RESCUE-1500	H3 BUDGET-512-1500 vs THINK-512
Qwen3-1.7B-Longevity	LB letters	no gain: not tested	holds	no room
	MMLU-Pro	no gain: not tested	holds	inconclusive
	GSM8K	no gain: not tested	holds	no room
SmolLM3-3B	LB letters	visibility dominates	holds	holds
	MMLU-Pro	visibility dominates	holds	holds
	GSM8K	visibility dominates	fails †	holds

Table 3: Verdicts of the 31 endpoints of H1–H3 on the test partition. A dagger marks the one endpoint whose replication label is not replicated : an inconclusive replication ( SmolLM3-3B on GSM8K, H2: fails on test, inconclusive on replication). The last row is H2 against the four GEPA harnesses.

Figure 1: Adjusted intervals of the primary endpoints of H1–H3 on the test partition. (a) H1:

\Delta

, the gain gate; (b) H2:

\theta

against the non-inferiority margin

-\delta

; (c) H3:

D_{T}

and

D_{P}

against the minimal difference

m

. Cells are model

\times

pool; filled markers pass the verdict’s boundary, open markers do not.

Defect	Model	Bound	$D_{S}$	$D_{R}$	$\Gamma$	Claim
A reference answer as assistant turn	Qwen3	1.000	$-0.052$	$-0.030$	$+0.022$	inconclusive
	SmolLM3	1.000	$+0.097$	$+0.119$	$+0.022$	inconclusive
	Intern-S1	1.000	$+0.142$	$+0.164$	$+0.022$	inconclusive
B1 gold fallback	Qwen3	0.205	$-0.145$	$-0.148$	$-0.003$	holds
	SmolLM3	0.083	$+0.001$	$+0.007$	$+0.006$	fails
	Intern-S1	0.057	$-0.012$	$-0.019$	$-0.007$	fails

Table 4: H4: the 16 (defect, model) pairs, with the letters of Section 3.4 . The tied output layer (E) exists for Qwen3-1.7B-Longevity only, the one model with two served variants. The bound is the largest distortion the mechanism can cause on

S

, computed without accuracy; a pair is evaluable only if it exceeds 0.05.

D_{S}

D_{R}

and

\Gamma=D_{R}-D_{S}

are point estimates of the distortion of the start’s accuracy (for the broken start, of the gain over the start, which equals the accuracy of THINK-1536 minus that of THINK-64 ); intervals are in Figure 2 and Appendix B . A dagger marks the claim of the skewed split in SmolLM3-3B , which fails by the general rule of the preregistration (Section 5 ).

Figure 2: H4: the defect’s distortion on the selection sample (

D_{S}

), on the check sample (

D_{R}

) and their difference (

\Gamma

), with adjusted intervals (family of 16) and the band

\pm m

(shaded). Filled markers are claims that hold; arrows mark intervals cut at the axis limit. The unparsed-as-A scoring (B2) in Qwen3-1.7B-Longevity is not evaluable and is not drawn.

Check	Reads	Detects	Uninformative	Misses	Fires on the clean instances
PR	gain and level rules of prospective replication	5	1	10	PR gain 1/7
TR	per-call prompt-token reconciliation	3	0	13	none
TB	trivial baselines	6	3	7	TB2 3/7
SCF	the simple configuration family	3	12	1	SCF1 4/7, SCF2 4/7, SCF3 2/7
IR	reading of the rendered inputs	0	16	0	IR4 7/7
SMI	served-model identity (defect E)	1	0	0	–

Table 5: The audit checks of H4: what each reads, its outcome on the 16 defective instances of the detection matrix ( detects : fires on the defective instance and not on the clean one; uninformative : fires on both; misses : does not fire; descriptive) and the rules that fire on the clean instances (what is empirical).

Model	THINK-512	THINK-1536	AUTHORS-1500	BUDGET-512-1500	RESCUE-1500
LongevityBench letter pool (3,942 rows). Chance 0.380; constant letter 0.383 (chosen on replication), 0.389 (oracle); constant label 0.530 (chosen on replication), 0.539 (oracle). Strongest, the oracle constant label: 0.539 [0.527, 0.551].
Qwen3	0.635 ( $+0.096$ )	0.635 ( $+0.096$ )	0.632 ( $+0.093$ )	0.635 ( $+0.096$ )	0.635 ( $+0.096$ )
SmolLM3	0.070 ( $-0.470$ )	0.157 ( $-0.382$ )	0.129 ( $-0.410$ )	0.121 ( $-0.418$ )	0.128 ( $-0.411$ )
Intern-S1	0.000 ( $-0.539$ )	0.125 ( $-0.414$ )	0.314 ( $-0.225$ )	0.253 ( $-0.286$ )	0.314 ( $-0.225$ )
LB-0030 (1,577 rows). Chance 0.200; constant letter 0.216 (oracle); constant label 0.250 (oracle). Strongest, the oracle constant label: 0.250 [0.230, 0.269].
Qwen3	0.369 ( $+0.119$ )	0.369 ( $+0.119$ )	0.344 ( $+0.094$ )	0.369 ( $+0.119$ )	0.369 ( $+0.119$ )

Table 6: Accuracy in the three pools and in each LongevityBench task, test partition, next to the trivial baselines; in parentheses the paired difference to the strongest baseline (the oracle constant, chosen once and held fixed in the bootstrap). Unparsed answers count as wrong. THINK-64 , the GEPA harnesses and the replication partition are in Appendix B . Computed after the outputs existed (Appendix C ).

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Lesson	Tested by
The parent parsed none of the prospective rows; the rescue made answers visible	H1, H3
A thinking-off setting that the search never tried was non-inferior to the rescue	H2
The caller sent the reference answer	H4, defect A
LB-0042’s gold came from a fallback rule; the scorer could fail open	H4, defects B1, B2
The search’s split put a giant component into one partition	H4, defect C
The parent was a broken start	H4, defect D

Appendix

Table 7: Lessons of the case study and where this study tests them.

Cell	$\Delta$	$\Delta_{V}$	$\Delta_{B}$	$\Delta_{L}$	$T$	$V$ / $B$ / $L$	Verdict	Repl.
Qwen3 LB letters	$-0.003$ [ $-0.018$ , $+0.012$ ]	$0.000$	$-0.003$	$0.000$	$+0.002$ [ $-0.009$ , $+0.014$ ]	1 / 3,941 / 0	no gain: not tested	R
Qwen3 MMLU-Pro	$+0.013$ [ $-0.006$ , $+0.033$ ]	$+0.015$	$-0.003$	$0.000$	$+0.005$ [ $-0.008$ , $+0.018$ ]	48 / 751 / 0	no gain: not tested	R
Qwen3 GSM8K	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$	$0.000$	$0.000$	$0.000$ [ $0.000$ , $0.000$ ]	0 / 523 / 0	no gain: not tested	R
SmolLM3 LB letters	$+0.060$ [ $+0.046$ , $+0.074$ ]	$+0.071$	$+0.001$	$-0.012$	$+0.013$ [ $+0.009$ , $+0.018$ ]	749 / 448 / 100	visibility dominates	R
SmolLM3 MMLU-Pro	$+0.255$ [ $+0.207$ , $+0.304$ ]	$+0.282$	$-0.014$	$-0.013$	$+0.067$ [ $+0.054$ , $+0.081$ ]	529 / 81 / 13	visibility dominates	R
SmolLM3 GSM8K	$+0.108$ [ $+0.046$ , $+0.173$ ]	$+0.197$	$-0.036$	$-0.053$	$+0.068$ [ $+0.047$ , $+0.090$ ]	141 / 340 / 29	visibility dominates	R

Appendix

Table 8: H1, primary endpoints: THINK-512 against AUTHORS-1500 , test partition (family of 9). Repl.: R replicated, I inconclusive replication, N not replicated.

Cell	$\Delta$	$\Delta_{V}$	$\Delta_{B}$	$\Delta_{L}$	$T$	$V$ / $B$ / $L$	Verdict
Qwen3 LB letters	$+0.632$ [ $+0.613$ , $+0.651$ ]	$+0.632$	$0.000$	$0.000$	$+0.126$ [ $+0.123$ , $+0.130$ ]	3,942 / 0 / 0	visibility dominates
Qwen3 MMLU-Pro	$+0.089$ [ $+0.063$ , $+0.119$ ]	$+0.089$	$0.000$	$0.000$	$+0.018$ [ $+0.013$ , $+0.024$ ]	274 / 525 / 0	visibility dominates
Qwen3 GSM8K	$+0.002$ [ $0.000$ , $+0.009$ ]	$+0.002$	$0.000$	$0.000$	$0.000$ [ $0.000$ , $+0.002$ ]	1 / 522 / 0	no gain: not tested
SmolLM3 LB letters	$+0.129$ [ $+0.116$ , $+0.143$ ]	$+0.129$	$0.000$	$0.000$	$+0.026$ [ $+0.023$ , $+0.029$ ]	1,197 / 0 / 0	visibility dominates
SmolLM3 MMLU-Pro	$+0.353$ [ $+0.307$ , $+0.401$ ]	$+0.353$	$0.000$	$0.000$	$+0.071$ [ $+0.061$ , $+0.080$ ]	610 / 0 / 0	visibility dominates
SmolLM3 GSM8K	$+0.789$ [ $+0.738$ , $+0.837$ ]	$+0.789$	$0.000$	$0.000$	$+0.158$ [ $+0.148$ , $+0.167$ ]	481 / 0 / 0	visibility dominates

Appendix

Table 9: H1, sanity checks: THINK-64 against AUTHORS-1500 , test partition, at the family’s adjusted level, outside the family and carrying no claim.

Endpoint	$\theta$ (adjusted interval)	Verdict	Replication verdict	Label
Qwen3 LB letters, RESCUE-1500	$-0.003$ [ $-0.018$ , $+0.012$ ]	holds	holds	replicated
Qwen3 MMLU-Pro, RESCUE-1500	$-0.003$ [ $-0.019$ , $+0.014$ ]	holds	holds	replicated
Qwen3 GSM8K, RESCUE-1500	$0.000$ [ $0.000$ , $0.000$ ]	holds	holds	replicated
SmolLM3 LB letters, RESCUE-1500	$+0.001$ [ $-0.007$ , $+0.009$ ]	holds	holds	replicated
SmolLM3 MMLU-Pro, RESCUE-1500	$-0.025$ [ $-0.044$ , $-0.009$ ]	holds	holds	replicated
SmolLM3 GSM8K, RESCUE-1500	$-0.089$ [ $-0.129$ , $-0.053$ ]	fails	inconclusive	inconclusive replication

Appendix

Table 10: H2, primary endpoints:

\theta

of AUTHORS-1500 against RESCUE-1500 in nine cells and against the four GEPA harnesses (family of 13), with the replication partition’s verdict and the label.

Cell	vs THINK-512	vs THINK-1536	vs BUDGET-512-1500
Qwen3 LB letters	$-0.003$ [ $-0.013$ , $+0.008$ ] holds	$-0.003$ [ $-0.013$ , $+0.007$ ] holds	$-0.003$ [ $-0.013$ , $+0.007$ ] holds
Qwen3 MMLU-Pro	$+0.013$ [ $-0.001$ , $+0.026$ ] holds	$-0.005$ [ $-0.018$ , $+0.008$ ] holds	$-0.003$ [ $-0.015$ , $+0.010$ ] holds
Qwen3 GSM8K	$0.000$ [ $0.000$ , $0.000$ ] holds	$0.000$ [ $0.000$ , $0.000$ ] holds	$0.000$ [ $0.000$ , $0.000$ ] holds
SmolLM3 LB letters	$+0.060$ [ $+0.050$ , $+0.070$ ] holds	$-0.028$ [ $-0.040$ , $-0.015$ ] holds	$+0.009$ [ $-0.003$ , $+0.020$ ] holds
SmolLM3 MMLU-Pro	$+0.255$ [ $+0.220$ , $+0.290$ ] holds	$+0.039$ [ $+0.003$ , $+0.075$ ] holds	$+0.095$ [ $+0.059$ , $+0.131$ ] holds
SmolLM3 GSM8K	$+0.108$ [ $+0.063$ , $+0.154$ ] holds	$-0.091$ [ $-0.127$ , $-0.055$ ] fails	$-0.046$ [ $-0.083$ , $-0.008$ ] inconclusive

Appendix

Table 11: H2, descriptive:

\theta

of AUTHORS-1500 against three other configurations, test partition, unadjusted intervals and the verdict rule applied to them.

Cell	$D_{T}$	$D_{P}$	THINK-512 truncation	Verdict	Replication verdict	Label
Qwen3 LB letters	$0.000$ [ $0.000$ , $+0.001$ ]	$0.000$ [ $0.000$ , $+0.001$ ]	$0.000$ [ $0.000$ , $+0.001$ ]	no room	no room	replicated
Qwen3 MMLU-Pro	$+0.060$ [ $+0.038$ , $+0.085$ ]	$+0.060$ [ $+0.038$ , $+0.085$ ]	$+0.060$ [ $+0.038$ , $+0.085$ ]	inconclusive	inconclusive	replicated
Qwen3 GSM8K	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]	no room	no room	replicated
SmolLM3 LB letters	$+0.428$ [ $+0.407$ , $+0.450$ ]	$+0.151$ [ $+0.136$ , $+0.167$ ]	$+0.725$ [ $+0.708$ , $+0.742$ ]	holds	holds	replicated
SmolLM3 MMLU-Pro	$+0.526$ [ $+0.476$ , $+0.574$ ]	$+0.248$ [ $+0.205$ , $+0.292$ ]	$+0.905$ [ $+0.875$ , $+0.932$ ]	holds	holds	replicated
SmolLM3 GSM8K	$+0.260$ [ $+0.209$ , $+0.315$ ]	$+0.190$ [ $+0.144$ , $+0.239$ ]	$+0.300$ [ $+0.245$ , $+0.357$ ]	holds	holds	replicated

Appendix

Table 12: H3, primary endpoints: THINK-512 against BUDGET-512-1500 , test partition (family of 9); the truncation rate of THINK-512 decides no room .

Pair	Bound	$D_{S}$	$D_{R}$	$\Gamma$	Stability of $\Gamma$	Claim
A Qwen3	1.000	$-0.052$ [ $-0.106$ , $+0.002$ ] (open)	$-0.030$ [ $-0.087$ , $+0.030$ ] (open)	$+0.022$ [ $-0.057$ , $+0.102$ ] (open)	inconclusive	inconclusive
A SmolLM3	1.000	$+0.097$ [ $+0.022$ , $+0.173$ ] (open)	$+0.119$ [ $+0.042$ , $+0.199$ ] (open)	$+0.022$ [ $-0.089$ , $+0.133$ ] (open)	inconclusive	inconclusive
A Intern-S1	1.000	$+0.142$ [ $+0.082$ , $+0.201$ ] (holds)	$+0.164$ [ $+0.102$ , $+0.226$ ] (holds)	$+0.022$ [ $-0.065$ , $+0.108$ ] (open)	inconclusive	inconclusive
B1 Qwen3	0.205	$-0.145$ [ $-0.170$ , $-0.118$ ] (holds)	$-0.148$ [ $-0.166$ , $-0.130$ ] (holds)	$-0.003$ [ $-0.035$ , $+0.028$ ] (holds)	stable	holds
B1 SmolLM3	0.083	$+0.001$ [ $-0.019$ , $+0.021$ ] (fails)	$+0.007$ [ $-0.006$ , $+0.021$ ] (fails)	$+0.006$ [ $-0.018$ , $+0.031$ ] (holds)	stable	fails
B1 Intern-S1	0.057	$-0.012$ [ $-0.028$ , $+0.004$ ] (fails)	$-0.019$ [ $-0.030$ , $-0.007$ ] (fails)	$-0.007$ [ $-0.027$ , $+0.013$ ] (holds)	stable	fails

Appendix

Table 13: H4, the 16 pairs with their adjusted intervals (family of 16, level 99.69%) and, in parentheses, which side of its boundary each interval lies on ( holds , fails or open ). Stability of

\Gamma

is a descriptive fact about the same interval.

Pair	PR	TR	TB	SCF	IR	SMI	Rules fired (expected ones in bold)
A Qwen3	U	D	U	U	U	–	IR1 , IR2 , IR4, PR.gain, SCF2, TB2, TB3 , TR
A SmolLM3	D	D	U	U	U	–	IR1 , IR2 , IR4, PR.gain, SCF1, SCF2, SCF3, TB2, TR ; expected, not fired: TB3
A Intern-S1	M	D	U	U	U	–	IR1 , IR2 , IR4, SCF1, SCF3, TB2, TB3 , TR
B1 Qwen3	M	M	M	M	U	–	IR3 , IR4
B1 SmolLM3	M	M	M	U	U	–	IR3 , IR4, SCF1
B1 Intern-S1	M	M	M	U	U	–	IR3 , IR4, SCF1, SCF2, SCF3

Appendix

Table 14: The detection matrix (descriptive; largely true by construction). Checks: PR prospective replication, TR token reconciliation, TB trivial baselines, SCF simple configuration family, IR input reading, SMI served-model identity.

Check	Rule	Fired on	Of
PR	PR.gain	1: clean/A / Qwen3	7
PR	PR.level	0: none	7
TR	TR	0: none	7
TB	TB1	0: none	7
TB	TB2	3: clean/A / Intern-S1, clean/A / Qwen3, clean/A / SmolLM3	7
TB	TB3	0: none	7

Appendix

Table 15: False alarms of each rule on the clean instances (the main clean instance and the clean counterparts of A and E, per model).

Cell	Configuration	Accuracy	Parse	Truncated	Non-committed
Qwen3 LB letters	THINK-64	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]	1.000 [1.000, 1.000]	0.000 [0.000, 0.000]
	THINK-512	0.635 [0.621, 0.649]	1.000 [0.999, 1.000]	0.000 [0.000, 0.001]	0.000 [0.000, 0.000]
	THINK-1536	0.635 [0.621, 0.649]	1.000 [1.000, 1.000]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]
	AUTHORS-1500	0.632 [0.618, 0.646]	1.000 [1.000, 1.000]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]
	BUDGET-512-1500	0.635 [0.621, 0.649]	1.000 [1.000, 1.000]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]
	RESCUE-1500	0.635 [0.621, 0.649]	1.000 [1.000, 1.000]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]

Appendix

Table 16: Accuracy, parse, truncation and non-commitment rates of the six configurations in the nine cells, test partition, with unadjusted 95% intervals. Unparsed answers count as wrong. The trivial baselines of the three pools are in Table 6 and Appendix B.4 .

LB-0030 (1,577 rows). Chance 0.200; constant letter 0.216 (non-oracle), 0.216 (oracle); constant label 0.242 (non-oracle), 0.250 (oracle). Strongest, oracle: 0.250 [0.230, 0.269].
Model	THINK-64	THINK-512	THINK-1536	AUTHORS-1500	BUDGET-512-1500	RESCUE-1500
Qwen3	0.000 ( $-0.250$ )	0.369 ( $+0.119$ )	0.369 ( $+0.119$ )	0.344 ( $+0.094$ )	0.369 ( $+0.119$ )	0.369 ( $+0.119$ )
SmolLM3	0.000 ( $-0.250$ )	0.000 ( $-0.250$ )	0.021 ( $-0.229$ )	0.053 ( $-0.197$ )	0.032 ( $-0.218$ )	0.053 ( $-0.197$ )
Intern-S1	0.000 ( $-0.250$ )	0.000 ( $-0.250$ )	0.004 ( $-0.245$ )	0.023 ( $-0.227$ )	0.055 ( $-0.195$ )	0.023 ( $-0.227$ )
Qwen3, GEPA	CLEAN-s0 0.344 ( $+0.094$ ); BROKEN-s0 0.353 ( $+0.103$ ); CLEAN-s1 0.388 ( $+0.138$ ); BROKEN-s1 0.321 ( $+0.072$ )
LB-0034 (788 rows). Chance 0.500; constant letter 0.485 (non-oracle), 0.515 (oracle); constant label 0.485 (non-oracle), 0.515 (oracle). Strongest, oracle: 0.515 [0.480, 0.551].

Appendix

Table 17: Accuracy and difference to the strongest baseline (oracle constant label), test partition.

LB-0030 (796 rows). Chance 0.200; constant letter 0.212 (non-oracle), 0.212 (oracle); constant label 0.255 (non-oracle), 0.270 (oracle). Strongest, oracle: 0.270 [0.242, 0.299].
Model	THINK-64	THINK-512	THINK-1536	AUTHORS-1500	BUDGET-512-1500	RESCUE-1500
Qwen3	0.000 ( $-0.270$ )	0.358 ( $+0.088$ )	0.358 ( $+0.088$ )	0.335 ( $+0.065$ )	0.358 ( $+0.088$ )	0.358 ( $+0.088$ )
SmolLM3	0.000 ( $-0.270$ )	0.000 ( $-0.270$ )	0.025 ( $-0.245$ )	0.041 ( $-0.229$ )	0.031 ( $-0.239$ )	0.041 ( $-0.229$ )
Intern-S1	0.000 ( $-0.270$ )	0.000 ( $-0.270$ )	0.005 ( $-0.265$ )	0.011 ( $-0.259$ )	0.046 ( $-0.224$ )	0.011 ( $-0.259$ )
Qwen3, GEPA	CLEAN-s0 0.335 ( $+0.065$ ); BROKEN-s0 0.345 ( $+0.075$ ); CLEAN-s1 0.389 ( $+0.119$ ); BROKEN-s1 0.307 ( $+0.036$ )
LB-0034 (398 rows). Chance 0.500; constant letter 0.490 (non-oracle), 0.510 (oracle); constant label 0.490 (non-oracle), 0.510 (oracle). Strongest, oracle: 0.510 [0.462, 0.558].

Appendix

Table 18: The same for the replication partition.

MMLU-Pro (799 items). Chance 0.110; constant letter B, chosen on the other partition and oracle alike, 0.119 [0.096, 0.141].
Model	THINK-64	THINK-512	THINK-1536	AUTHORS-1500	BUDGET-512-1500	RESCUE-1500
Qwen3	0.140 ( $+0.021$ )	0.217 ( $+0.098$ )	0.234 ( $+0.115$ )	0.229 ( $+0.110$ )	0.232 ( $+0.113$ )	0.232 ( $+0.113$ )
SmolLM3	0.000 ( $-0.119$ )	0.098 ( $-0.021$ )	0.314 ( $+0.195$ )	0.353 ( $+0.234$ )	0.258 ( $+0.139$ )	0.378 ( $+0.259$ )
Intern-S1	0.000 ( $-0.119$ )	0.074 ( $-0.045$ )	0.279 ( $+0.160$ )	0.477 ( $+0.358$ )	0.542 ( $+0.423$ )	0.493 ( $+0.374$ )
GSM8K (527 items). Constant answer 10 (chosen on the other partition) 0.019 [0.008, 0.032], 2 (oracle) 0.034 [0.019, 0.051]; no chance level.
Qwen3	0.068 ( $+0.034$ )	0.070 ( $+0.036$ )	0.070 ( $+0.036$ )	0.070 ( $+0.036$ )	0.070 ( $+0.036$ )	0.070 ( $+0.036$ )

Appendix

Table 19: MMLU-Pro and GSM8K, test partition: accuracy of the six configurations and, in parentheses, the paired difference to the strongest baseline (the oracle constant).

MMLU-Pro (399 items). Chance 0.111; constant letter B, chosen on the other partition and oracle alike, 0.133 [0.100, 0.168].
Model	THINK-64	THINK-512	THINK-1536	AUTHORS-1500	BUDGET-512-1500	RESCUE-1500
Qwen3	0.108 ( $-0.025$ )	0.180 ( $+0.048$ )	0.193 ( $+0.060$ )	0.180 ( $+0.048$ )	0.190 ( $+0.058$ )	0.185 ( $+0.053$ )
SmolLM3	0.000 ( $-0.133$ )	0.090 ( $-0.043$ )	0.333 ( $+0.201$ )	0.378 ( $+0.246$ )	0.291 ( $+0.158$ )	0.398 ( $+0.266$ )
Intern-S1	0.000 ( $-0.133$ )	0.105 ( $-0.028$ )	0.293 ( $+0.160$ )	0.446 ( $+0.313$ )	0.566 ( $+0.434$ )	0.454 ( $+0.321$ )
GSM8K (264 items). Constant answer 2 (chosen on the other partition) 0.030 [0.011, 0.053], 10 (oracle) 0.045 [0.023, 0.072]; no chance level.
Qwen3	0.057 ( $+0.011$ )	0.057 ( $+0.011$ )	0.057 ( $+0.011$ )	0.057 ( $+0.011$ )	0.057 ( $+0.011$ )	0.057 ( $+0.011$ )

Appendix

Table 20: MMLU-Pro and GSM8K, replication partition.

Model	Pool	Configuration	Parse (seeded)	Truncated (seeded)	$\Delta$ accuracy	$\Delta$ parse	$\Delta$ truncation
Qwen3	LB letters	THINK-64	0.000	1.000	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]
		THINK-512	0.999	0.001	$-0.013$ [ $-0.026$ , $-0.001$ ]	$0.000$ [ $-0.001$ , $+0.001$ ]	$0.000$ [ $-0.001$ , $+0.001$ ]
		THINK-1536	1.000	0.000	$-0.014$ [ $-0.027$ , $-0.002$ ]	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]
		BUDGET-512-1500	1.000	0.000	$-0.018$ [ $-0.030$ , $-0.005$ ]	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]
	MMLU-Pro	THINK-64	0.641	0.359	$-0.005$ [ $-0.026$ , $+0.016$ ]	$-0.016$ [ $-0.039$ , $+0.005$ ]	$+0.016$ [ $-0.005$ , $+0.039$ ]
		THINK-512	0.932	0.068	$-0.004$ [ $-0.029$ , $+0.021$ ]	$-0.008$ [ $-0.026$ , $+0.011$ ]	$+0.008$ [ $-0.011$ , $+0.026$ ]

Appendix

Table 21: Seeded sampling at the models’ card values (test partition): parse and truncation rates, and the difference seeded minus zero temperature with unadjusted intervals.

Partition	Model	Pool	$\tau_{g}$	$\tau_{s}$	$\tau_{s}-\tau_{g}$	$D_{T}$ (seeded)	$D_{P}$ (seeded)
Test	Qwen3	LB letters	0.000	0.001	$0.000$ [ $-0.001$ , $+0.001$ ]	$+0.001$ [ $0.000$ , $+0.001$ ]	$+0.001$ [ $0.000$ , $+0.001$ ]
		MMLU-Pro	0.060	0.068	$+0.008$ [ $-0.011$ , $+0.026$ ]	$+0.068$ [ $+0.051$ , $+0.085$ ]	$+0.066$ [ $+0.050$ , $+0.084$ ]
		GSM8K	0.000	0.000	$0.000$ [ $0.000$ , $0.000$ ]	$0.000$ [ $0.000$ , $0.000$ ]	$-0.002$ [ $-0.006$ , $0.000$ ]
	Intern-S1	LB letters	1.000	0.997	$-0.003$ [ $-0.005$ , $-0.001$ ]	$+0.963$ [ $+0.957$ , $+0.969$ ]	$+0.702$ [ $+0.689$ , $+0.716$ ]
		MMLU-Pro	0.915	0.902	$-0.013$ [ $-0.034$ , $+0.009$ ]	$+0.618$ [ $+0.584$ , $+0.652$ ]	$+0.589$ [ $+0.554$ , $+0.625$ ]
		GSM8K	0.522	0.619	$+0.097$ [ $+0.051$ , $+0.142$ ]	$+0.581$ [ $+0.539$ , $+0.622$ ]	$+0.567$ [ $+0.526$ , $+0.609$ ]

Appendix

Table 22: H3’s statistics recomputed on the seeded outputs (the cards’ sampling values, a seed per request) for the two models with a seeded arm; descriptive.

\tau_{g}

and

\tau_{s}

are the shares of THINK-512 calls cut at the token limit with greedy decoding and seeded, and

\tau_{s}-\tau_{g}

their paired difference.

D_{T}

and

D_{P}

are the seeded drop in truncation and gain in parsing of BUDGET-512-1500 against THINK-512 , with unadjusted intervals. The preregistered verdicts use family-adjusted intervals at another level and are not compared with these.

Configuration	Tied	Untied	Tied $-$ untied	Parse t / u	Same answers	Only t / u right	$p$
CFG-THINK-64-ORIG	0.000	0.000	$0.000$ [ $0.000$ , $0.000$ ]	0.000 / 0.000	1.000	0 / 0	1.000
B1	0.452	0.449	$+0.003$ [ $-0.001$ , $+0.007$ ]	1.000 / 1.000	0.741	182 / 157	0.192
CFG-THINKING-FALSE-64-ORIG	0.460	0.460	$0.000$ [ $-0.005$ , $+0.005$ ]	1.000 / 1.000	0.731	194 / 194	1.000
CFG-THINK-1536-ORIG	0.469	0.463	$+0.006$ [ $0.000$ , $+0.013$ ]	1.000 / 1.000	0.656	384 / 333	0.062
CFG-NOTHINK-SUFFIX-64-ORIG	0.452	0.449	$+0.003$ [ $-0.001$ , $+0.007$ ]	1.000 / 1.000	0.741	182 / 157	0.192
CFG-THINK-64-ANSWERONLY	0.479	0.470	$+0.009$ [ $+0.004$ , $+0.014$ ]	1.000 / 1.000	0.731	260 / 185	0.000

Appendix

Table 23: Tied against untied output layer of Qwen3-1.7B-Longevity on FreeToken, the case study’s six configurations, all four LongevityBench tasks pooled (8,305 rows in 1,188 participant components, unflagged rows of both partitions); computed after the outputs existed.

Harness	Proposals	Candidates seen	Refused	Kept	Metric calls	Selected / kept	Characters (system / instruction)	Valset	Replication acc.	Test acc.	Test parse
GEPA-CLEAN-s0	207	204	28	31	6,039	0 / 31	103 / 0	0.680	0.633	0.632 [0.618, 0.646]	1.000
GEPA-BROKEN-s0	377	377	24	24	6,147	1 / 24	103 / 1,146	0.660	0.635	0.629 [0.615, 0.643]	1.000
GEPA-CLEAN-s1	195	192	36	31	6,003	12 / 31	2,469 / 922	0.700	0.641	0.631 [0.616, 0.645]	1.000
GEPA-BROKEN-s1	243	238	67	29	6,003	1 / 29	103 / 2,797	0.667	0.619	0.620 [0.606, 0.634]	1.000

Appendix

Table 24: The four GEPA runs (computed after the outputs existed): proposals and candidates, refused by the interface check, kept by GEPA, metric calls, the selected candidate, its prompt lengths, its validation score and its accuracy on the letter rows of the replication and test partitions.

Model	Measure	THINK-64	THINK-512	THINK-1536	AUTHORS-1500	BUDGET-512-1500	RESCUE-1500
Qwen3	parse rate	0.000	1.000	1.000	1.000	1.000	1.000
	mean abs. error (years)	–	15.69	15.69	16.50	15.69	15.69
	exact match	0.000	0.027	0.027	0.030	0.027	0.027
SmolLM3	parse rate	0.000	0.000	0.004	0.001	0.003	0.001
	mean abs. error (years)	–	–	24.71	57.70	35.08	57.70
	exact match	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table 25: LB-0038 (age regression), test partition, descriptive.

Analysis	Endpoints	Changed	Which (primary verdict $\to$ here)
Strict scorer	31	2	H2 SmolLM3 MMLU-Pro, RESCUE: holds $\to$ inconclusive; H1 SmolLM3 GSM8K: visibility dominates $\to$ no gain: not tested
Never-called rows only	13	3	H2 Qwen3 letters, RESCUE: holds $\to$ inconclusive; H2 Qwen3 letters, GEPA-BROKEN-s1: holds $\to$ inconclusive; H1 SmolLM3 letters: visibility dominates $\to$ inconclusive
Flagged rows included (H1–H3)	31	0	none
Case study’s pool (LB-0030 and LB-0034)	13	1	H2 Qwen3 letters, GEPA-CLEAN-s1: holds $\to$ inconclusive
LB-0030 alone	13	2	H2 Qwen3 letters, RESCUE: holds $\to$ inconclusive; H2 Qwen3 letters, GEPA-CLEAN-s1: holds $\to$ inconclusive
LB-0034 alone	13	0	none

Appendix

Table 26: Sensitivity analyses: the number of the primary endpoints whose verdict changes, and which.

Rows	Partition	Rows	Letter rows	LB-0030 mean age	LB-0042 share deceased at follow-up
Never called	replication	187	103	36.2	0.022
Analysis rows	replication	1,990 (pool)	–	50.8	0.345
Never called	test	356	207	37.3	0.022
Analysis rows	test	3,942 (pool)	–	50.8	0.346

Appendix

Table 27: Composition of the never-called rows (computed after the outputs existed) next to the analysis rows. The share deceased at follow-up is the share of LB-0042 rows whose follow-up turn says that the participant was deceased, whenever the death occurred; it is not the ten-year label of the task, whose gold is “Yes” for a participant who was alive at ten years or who died later.

Entry	What happened	Bearing on the results	Seen
Before the freeze	The primary set of H1 changed twice (after the pilot’s parse rates, and after a review in Codex); the scoring defect was split in two; the tied-output-layer defect was redesigned; one evaluability rule was fixed; the family of H4 was fixed at 16; the margins were set to 0.05.	Defines the endpoints; no study accuracy was used.	pilot parse rates
Rows left out	147 replication and 371 test rows that the case study had called, and 7 rows whose answers could be recovered from an upload made by mistake, were called but left out.	Putting them back changes no verdict of H1–H3 and 2 H4 claims.	no outputs
A change of GPU	After a restart the platform assigned another GPU of the same model; the determinism check rerun on the pilot rows passed, with outputs equal to the old GPU’s.	None. 9.33 GPU-hours outside the budget.	pilot rows
Two script stops	The chain script stopped twice during the GEPA search, both times from a fault in the script.	None; no run was affected.	none
The thinking-switch check	A check on Intern-S1-mini ’s thinking switch stopped the run that sends the reference answer, and we let it record instead of stop. The repeat reproduced the first 3,700 calls token for token, and the check failed on 324 of the run’s 1,204 thinking-off calls and on none of 11,265 in the clean stages.	The reference-answer defect in Intern-S1-mini only through the descriptive selection, since the claim reads the start. 6.46 idle server-hours counted.	counts only
The checkpoint	15 of the 16 pairs were evaluable; the pilot had suggested 14.	The unparsed-as-A scoring in Qwen3-1.7B-Longevity is not evaluable ; the gold fallback in Intern-S1-mini (bound 0.057) is evaluable.	counts and bounds

Appendix

Table 28: Deviations and addenda that bear on a result; the entries below give the rest. “Seen” is what had been seen of the study’s outputs when the entry was made.

Item	Expected	Observed
H1 sanity checks ( THINK-64 )	$T>0$ in eight cells; Qwen3-1.7B-Longevity on GSM8K fails the gate	visibility dominates in 8 of nine cells; Qwen3-1.7B-Longevity on GSM8K fails the gate. As expected.
H1, Qwen3-1.7B-Longevity (three cells)	Fails the gate (on MMLU-Pro, or quality matters)	no gain: not tested in all three. As expected.
H1, Intern-S1-mini on the letters	$T>0$ nearly inevitable	visibility dominates , $T=+0.063$ . As expected.
H1, SmolLM3-3B and Intern-S1-mini on GSM8K	Inconclusive on $T$ (planned power 0.52 and 0.00 for SmolLM3-3B , 0.45 and 0.06 for Intern-S1-mini , in the two scenarios)	SmolLM3-3B : visibility dominates , $T=+0.068$ [ $+0.047$ , $+0.090$ ]. Intern-S1-mini : the gate fails, $\Delta=-0.287$ . Both differ.
H2, Intern-S1-mini on GSM8K against RESCUE-1500	Inconclusive (planned adjusted power 0.19)	fails , $\theta=-0.342$ [ $-0.402$ , $-0.281$ ]. Differs.
H2, GEPA harnesses	Power unknown in advance; may well be inconclusive	holds for all four; all intervals contain zero.

Appendix

Table 29: Expected in advance (preregistration, the sections on H1, H2 and H4) and observed.

#	Contact
#39	The ICML formatting draft of the case study copied to a synchronised folder for an upload.
#40	The LongevityBench split of this study, the earlier-contact flags and the first gold audit built.
#41	Fourteen files uploaded to claude.ai for a review of the reading export and the runner (item-exposure contact 1).
#42	The preregistration draft revised and uploaded for reviews in claude.ai and in Codex, after a check for holdout-derived values.
#43	The main item files built.
#44	The LB-0042 rows on which B1’s fallback gold differs from the derived gold counted.

Appendix

Table 30: Holdout contacts of this study (#39 to #58 and #60): what was read or done. Entries #50 to #58 and #60 are after the freeze.

Partition	Model	Configuration	Calls	Split disagreements	Prompt tokens	Completion tokens	Seconds
Replication	Qwen3	THINK-64	3,603	0	480	56	1.0
		THINK-512	3,603	3	480	257	3.7
		THINK-1536	3,603	3	480	258	3.7
		AUTHORS-1500	3,603	3	484	3	0.2
		BUDGET-512-1500	3,603	4	480	257	3.6
	SmolLM3	THINK-64	3,603	0	500	64	1.8

Appendix

Table 31: Serving counts of the zero-temperature grid.

	Construction	Samples $S$ and $R$	Clean counterpart and resampling	Evaluability bound
A	The row’s reference answer is sent as a final assistant turn, as the case study’s caller did: new calls, built by a separate module.	1,204 replication rows drawn by whole component (seed 20261012) in proportion to the task counts (344, 172, 344 and 344 rows of LB-0030, LB-0034, LB-0038 and LB-0042), halved by component into $S$ and $R$ (86 components and 602 rows each; after the flags 562 and 565 rows in 81 components each). Both halves are called in one pass, so $R$ is a disjoint half, not a later sample.	The grid’s own outputs on the same rows, in the same halves; one resample serves both.	One: the leaked reference can change any row.
B1	Offline: the clean outputs are rescored with LB-0042’s gold taken from the case study’s historical fallback, which with this release’s labels differs from the derived gold. Unparsed answers stay wrong; other tasks keep the release’s gold.	$S$ : the replication partition’s letter rows; $R$ : the test partition’s, called after the checkpoint. The fallback changes the gold of 408 of 796 LB-0042 rows of $S$ and 794 of 1,577 of $R$ .	The main clean instance on the same rows; one resample serves both.	The share of $S$ ’s letter rows that the start parses and whose gold B1 changes (exact; B1 changes the gold of 0.2050 of them).
B2	Offline: the clean outputs are rescored by a scorer that never fails closed: an unparsed or conflicting letter answer is scored as the letter A, on every letter task, with the release’s gold.	As B1.	As B1.	The share of $S$ ’s letter rows that the start leaves unparsed or conflicting and whose gold is A (exact; it equals B2’s distortion of the start’s accuracy).
C	The case study’s skewed split, re-implemented. The pooled LongevityBench rows of both partitions that are not flagged (8,305 of 8,830) are reassigned: components (from all six NHANES tasks, chaining into one giant component that goes to the evolution part) are sorted by size, largest first, and each goes to the part furthest below its target share (evolution 0.50, validation 0.30, holdout 0.20).	$S$ and $R$ are the two halves of the validation part: its components are sorted by identifier, shuffled (seed 20261016) and dealt alternately. Their rows come from both partitions, so they are not the replication and test rows of the other pairs, and C, computed after the test calls, is not temporally prospective.	The main clean instance, whose $S$ and $R$ are the study’s replication and test rows. The rows differ, so the two are resampled independently, each with its own seed.	None: C is evaluable by rule, because its mechanism changes the sample and has no row-level bound.
D	Offline: the start is THINK-64 in place of THINK-1536 ; the reported quantity is the gain over the start.	$S$ : the replication partition’s letter rows; $R$ : the test partition’s.	The main clean instance; one resample serves both.	The share of $S$ ’s letter rows on which the parsed answers of the two starts differ, an unparsed or conflicting answer counting as one value.
E	The tied output layer is served in place of the untied one ( Qwen3-1.7B-Longevity only): the case study’s six configurations on FreeToken, for both variants, with CFG-THINK-1536-ORIG as the start. The confirmatory instance serves the tied variant on $S$ and on $R$ .	$S$ : the replication rows; $R$ : the test rows.	The untied variant on both samples; one resample serves both. A second instance (tied on $S$ , untied on $R$ ) feeds only the descriptive detection matrix.	The share of $S$ ’s letter rows on which the tied and untied variants’ parsed answers of the start differ.

Appendix

Table 32: The six defects of H4: construction, samples, clean counterpart and evaluability bound. The bound is the largest distortion the defect’s mechanism can cause on

S

, computed without accuracy.

Rule	Fires when
PR, gain rule	$c^{}\neq s$ and the lower bound of the 95% percentile interval of $\mathrm{acc}_{R}(c^{})-\mathrm{acc}_{R}(s)$ is at or below zero: the reported gain does not reproduce on $R$ .
PR, level rule	The 95% percentile interval of $\mathrm{acc}_{R}(c^{})-\mathrm{acc}_{S}(c^{})$ excludes zero, whether or not $c^{*}=s$ ; the $b$ -th resample of $S$ is paired with the $b$ -th of $R$ , each drawn with its own seed.
TR	On any call of the instance the server’s prompt-token count differs from the local count of the clean render (the corrected caller’s render of the row’s messages before the final assistant turn), or, on vLLM, its prompt token IDs differ from the clean render’s. For FreeToken only the count is compared.
TB1	A strong trivial baseline (the oracle constant, the frequency of the most frequent gold letter in the sample itself) exceeds 0.90 on a task in $S$ or $R$ .
TB2	For some task, the trivial constant chosen on $S$ (the most frequent gold letter there, ties to the smallest letter) scores on $R$ more than 0.05 away from its score on $S$ .
TB3	On LB-0038, some configuration’s exact-match rate exceeds three times the best constant age’s, in $S$ or $R$ .

Appendix

Table 33: The rules of PR, TR and TB.

c^{*}

is the selected configuration,

s

the start,

g_{R}

the gain on

R

Rule	Fires when
SCF1	On $S$ the start’s parse rate is below 0.90 or its truncation rate above 0.10.
SCF2	$c^{}\neq s$ and $\mathrm{acc}_{S}(c^{})$ is less than 0.05 above the best of three panel members on $S$ : chance, the strong trivial baseline and AUTHORS-1500 (in the instances of E, CFG-THINKING-FALSE-64-ORIG). For the letter pool, chance and the baseline are the row-weighted means of the tasks’ values.
SCF3	$c^{*}\neq s$ , $g_{S}>0$ and $\Delta_{V}$ is at least 80% of $g_{S}$ .
IR1	Any request of the instance has an assistant turn, or a last message that is not a user turn.
IR2	The gold screen hits on any request: the row’s follow-up text, the LB-0038 age as a standalone integer, or another reference answer of more than one character appears in it.
IR3	On any row the scorer’s gold is not among the golds derived from the question and follow-up (apart from the LB-0030 band-endpoint exception).

Appendix

Table 34: The rules of SCF, IR and SMI.

g_{S}

is the reported gain on

S

and

\Delta_{V}

its visibility part (Section 3 , with the start as P and

c^{*}

as R, point estimate on

S

Endpoint	Assumed effect	Inputs from the pilot	Units	Family, $z$	Power
H2: Intern-S1-mini , GSM8K, $\theta$ against RESCUE-1500	$\theta$ zero	$d=0.325$	527 items	13; 2.89	0.19 (adjusted), 0.52 (unadjusted)
H1: SmolLM3-3B , GSM8K, $T>0$	S1, S2	$v/b/l=0.210/0.725/0.040$ , $d_{B}=0.055$	527 items	9; 2.77	S1 0.52, S2 0.00
H1: Intern-S1-mini , GSM8K, $T>0$	S1, S2	$v/b/l=0.550/0.450/0.000$ , $d_{B}=0.756$	527 items	9; 2.77	S1 0.45, S2 0.06

Appendix

Table 35: The planned-power figures quoted in this paper. All are for the test partition, with a normal approximation. S1 and S2 are the two assumed scenarios of H1.

Explore similar work

Oct 1, 2026cs.AI

Finding the Right Fit: Model-Harness Interactions across Agent Tasks

Choosing an agent system means choosing both a language model and the harness through which it acts. We ask whether a strong model, harness, or pairing stays strong when the setting changes. We evaluate 66 configurations: four configurable harnesses (OpenHands, DeepSeek Harness, PI, and openJiuwen) paired with five models on TUA-Bench, ALE-CLI, and Terminal-Bench 4, plus the native Codex-GPT and Claude Code-Claude pairings. Model rankings reverse across harnesses. On Terminal-Bench 4, Claude leads GPT by 7.94 points in OpenHands but trails it by 30.16 points in PI. For four of the five models, the best harness changes from one benchmark to another, yet some pairings hold: openJiuwen gives Kimi its highest score on all three benchmarks, by 5.61 to 11.11 points. A model's own vendor harness is not reliably its best, and higher cost does not reliably buy a higher score. On Terminal-Bench 4, GPT scores higher under PI than under DSH at less than a quarter of the cost per task. Matched trajectories suggest why fit varies. Models start almost all repairs themselves, so much depends on whether the harness hands failures back in a form the model can use. GPT does best with PI's lean scaffold, while Kimi, which often issues malformed tool calls, does best in openJiuwen. We argue that the model, the harness, and the task should be evaluated together, and we release the harness adapters, evaluation code, and all 6,204 scored trajectories at https://github.com/liyix/finding-the-right-fit and https://huggingface.co/datasets/yixuanli97/finding-the-right-fit.

Yixuan Li, Yiyun Zhou, Yao Long Teng +6

College of Computing and Data Science, Nanyang Technological University, Singapore

Agent Harness Agentic Benchmarks Openai

Ask PDF

Sep 29, 2026cs.CL

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy--cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model--harness combinations: the best reaches 68% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.

Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1

University of Alberta · Alberta Machine Intelligence Institute (Amii)

Language Model Computes

Ask PDF

May 12, 2026cs.SE

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

This paper experimentally analyzes how the level of harness engineering affects the operational performance of small language models (SLMs, 2-3B parameters). Three harness conditions - model-only (raw prompt), minimal-shell (wrapper tags), and a 4-stage pipeline (plan->execute->verify->recover) - are applied to three models (Gemma4 E2B, Qwen3.5:2B, LLaMA 3.2 3B) across 24 tasks, comparing Task Success Rate (TSR) and Valid TSR (VTSR). The pipeline harness achieves TSR=0.952 and VTSR=1.000 on Gemma4 E2B (T1-T5, 21 tasks). A non-monotonic phenomenon - minimal-shell TSR < model-only TSR - is observed in two models. In LLaMA 3.2 3B model-only, seven format violations yield TSR=0.429, revealing scaffold collapse: the model abandons JSON structure under complex format requirements without harness support. Ablation shows planning and recovery each contribute approximately 24.7% of total gain. VCR (Verification Catch Rate)=0.625 across all pipeline runs.

Yong-eun Cho

KailosLab, Seoul, Republic of Korea

Small Large Language Models Task Success Rate

Ask PDF