HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

Mingyu Lu; Timothy Ossowski; Tristan Naumann; Hoifung Poon; Peniel Argaw

cs.CLOct 13, 2026

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

Authors: Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw

Organizations: University of Washington · Microsoft Research · Recursion

Abstract

Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce HERO Harnessing Evidence for Refinement and Optimization), which combines offline and test-time harness optimization. At the first stage, HERO learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, improving pass@1 by 7.6 points over the initial prompt, while test-time refinement adds another 2.0 points and improves execution efficiency.

Figures & tables

Figure 1: Overview of HERO. (a) HERO-O: Offline prompt optimization learns a shared prompt across training tasks. (b) HERO-T: Test-time refinement refines the generated artifact using environment feedback.

	Baseline	Offline Prompt Optimization					Test-Time Adaptation				Harness Optimization
Dataset	Seed	HERO-O	SkillOpt	GEPA	MIPROv2	ACE	DC-Cu	HERO-O+T	HERO-O + ReAct	HERO-O + DC-Cu	Meta- Harness
BIRD DEV	59.39 (0.32)	63.55 (0.21)	61.31 (1.19)	62.59 (0.59)	60.87 (0.57)	60.87 (0.10)	60.22 (0.79)	64.81 (0.21)	63.12 (1.03)	62.44 (1.43)	58.24 (2.18)
BigCodeBench	47.93 (0.74)	52.59 (0.31)	51.81 (0.47)	50.13 (1.00)	50.65 (0.59)	51.96 (1.25)	44.79 (0.83)	53.79 (0.33)	52.64 (0.24)	51.70 (0.89)	45.26 (2.52)
DSCodeBench	44.74 (0.24)	54.84 (1.87)	51.87 (1.50)	52.14 (1.99)	51.73 (1.76)	51.31 (0.43)	42.81 (0.24)	56.04 (1.87)	56.22 (2.87)	54.70 (0.87)	45.99 (0.52)
CypherBench	74.00 (0.44)	85.50 (1.40)	84.10 (1.30)	78.73 (1.53)	78.27 (3.04)	77.13 (1.74)	77.13 (1.74)	89.93 (1.46)	87.00 (0.21)	81.77 (2.98)	77.03 (2.59)

Table 1: Performance across four code-generation benchmarks. Results (pass@1) are reported as mean (standard deviation) over three runs. Bold highlights the best mean within each category.

Figure 2: Cumulative fix-to-break ratio. The y-axis reports the cumulative ratio across optimization iterations on the x-axis. The dashed line indicates an equal number of fixes and regressions.

Figure 3: BIRD-SQL DEV pass@1 by question difficulty (simple

n{=}923

, moderate

n{=}463

, challenging

n{=}144

scored instances). Bars are the mean over three seeds; error bars are the sample standard deviation.

Method	Tokens/call (K)	Prompt length (tokens)
ACE	31.89 (1.17)	35,881 (7,563)
MIPROv2	3.47 (0.02)	2,704 (70)
SkillOpt	2.87 (0.10)	2,021 (628)
GEPA	2.47 (0.24)	2,003 (541)
HERO-O (Ours)	2.79 (0.18)	1,949 (389)

Table 2: Optimization context and token cost. Input plus output tokens per LLM call during optimization and the length of the final prompt. Averaged over four benchmarks per seed; mean (SD) over three seeds.

Method	BIRD	BigCodeBench	DSCodeBench	CypherBench
Seed prompt	79.1 (5.0)	61.2 (3.6)	48.7 (0.5)	85.6 (1.5)
Stage Offline Harness Optimization
GEPA	73.1 (3.6)	69.1 (3.9)	56.0 (2.2)	86.7 (3.8)
MIPROv2	77.9 (7.1)	69.7 (2.7)	55.4 (1.3)	86.5 (3.2)
ACE	72.3 (2.2)	71.3 (2.5)	55.3 (0.5)	103.8 (15.2)
SkillOpt	75.9 (4.0)	69.7 (1.0)	54.8 (1.0)	137.7 (50.8)

Table 3: Valid Efficiency Score (VES) of generated programs. Results are mean (SD) over three seeds; higher is better. Bold denotes the best mean.

Figure 4: Sensitivity to the number of rendered diagnostic cases. Pass@1 as the number of cases rendered varies. Results are the mean

\pm

SD over three seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	$\mathcal{D}_{\mathrm{diag}}$	$\mathcal{D}_{\mathrm{upd}}$	$\mathcal{D}_{\mathrm{sel}}$	$\mathcal{D}_{\mathrm{test}}$	Rollout budget
BIRD-SQL	Text-to-SQL	500	250	1,400	1,530	5,400
CypherBench	Text-to-Cypher	250	250	250	1,000	3,700
BigCodeBench	Python	250	250	250	637	5,400
DSCodeBench	Python	250	250	250	482	3,600

Appendix

Table 4: Benchmarks, data splits, and optimization budgets. Rollout budgets cap task executions during offline optimization.

Method	BIRD-SQL	BigCodeBench	CypherBench	DSCodeBench
Tokens per call (K)
ACE	38.46 $\pm$ 1.06	31.51 $\pm$ 6.16	20.84 $\pm$ 1.66	36.63 $\pm$ 1.97
MIPROv2	2.17 $\pm$ 0.11	2.41 $\pm$ 0.33	1.82 $\pm$ 0.10	7.47 $\pm$ 0.12
SkillOpt	3.17 $\pm$ 0.43	2.52 $\pm$ 0.19	2.36 $\pm$ 0.43	3.41 $\pm$ 0.40
GEPA	2.42 $\pm$ 0.29	1.86 $\pm$ 0.12	2.24 $\pm$ 0.22	3.36 $\pm$ 0.49
Ours	3.10 $\pm$ 0.48	2.10 $\pm$ 0.18	2.57 $\pm$ 0.09	3.36 $\pm$ 0.18

Appendix

Table 5: Optimization cost across datasets. Input plus output tokens per LLM call during optimization (top) and token length of the final prompt (bottom); mean

\pm

SD over three seeds. Bold marks the lowest value per dataset.

Figure 5: Pass@1 versus cost. Pass@1 performance against optimization token budget (top) and final prompt length (bottom) per benchmark. Points show 3-seed means with standard deviation (log-scale x-axes).

Hyperparameter	Value	BIRD-SQL	BigCodeBench
Candidates per round $K$	$4^{\dagger}$	63.55 (0.21)	51.96 (0.00)
	6	61.90 (1.25)	52.07 (0.66)
	8	62.39 (1.52)	51.38 (0.55)
	10	61.74 (1.61)	51.65 (1.41)
	12	62.83 (0.81)	51.81 (1.34)
Search rounds $T$	3	61.66 (1.52)	51.86 (0.81)

Appendix

Table 6: Offline search hyperparameter sensitivity. Pass@1 (mean (SD), three seeds) on BIRD-SQL DEV and BigCodeBench. Each section varies one setting;

\dagger

marks defaults (

K=4

T=5

, 32 rendered cases per outcome). Bold denotes the best mean per section and benchmark.

Setting	BIRD-SQL	BigCodeBench	DSCodeBench	CypherBench	Mean
Full	3.01	1.61	1.70	2.06	2.10
Failure-only	2.61	1.71	1.49	1.41	1.81

Appendix

Table 7: Effect of successful diagnostic examples. Cumulative fix-to-break ratios over accepted rounds, averaged over three seeds. Full: 32 failed + 32 passing cases; failure-only: 32 failed cases. Mean averages the four benchmarks.

		Refinement budget $B$						ReAct
Dataset	Metric	0	2	$\mathbf{3}^{\dagger}$	4	6	8	$\mathbf{B=3}$
BIRD-SQL	Pass@1 (%)	62.53 (1.82)	63.38 (1.83)	63.59 (1.43)	63.53 (1.53)	63.77 (1.47)	64.05 (1.57)	63.12 (1.02)
	Env. actions	0	1.35	1.59	1.66	1.76	1.78	0.79
	LLM calls	0	3.15	3.33	3.40	3.46	3.43	1.93
BigCodeBench	Pass@1 (%)	52.59 (0.31)	53.38 (0.87)	53.79 (0.33)	53.95 (0.74)	54.06 (0.33)	53.85 (0.31)	52.64 (0.24)
	Env. actions	0	1.88	2.31	2.40	2.49	2.50	1.91

Appendix

Table 8: Test-time refinement budget sensitivity.

B

caps environment actions per query;

\dagger

marks the default.

B=0

denotes no refinement. Pass@1: mean (SD) over three seeds. Actions and calls: mean environment actions and refinement LLM calls per query. Bold denotes the highest mean pass@1 per benchmark.

Figure 6: Refinement resource usage on BigCodeBench. Mean environment actions (left) and LLM calls (right) per query versus budget

B

. Hollow points: optimized prompt only; dashed lines: full budget utilization; gray points: plain ReAct (

B=3

Figure 7: Refinement resource usage on BIRD-SQL. Mean environment actions (left) and LLM calls (right) per query versus budget

B

. Hollow points: optimized prompt only; dashed lines: full budget utilization; gray points: plain ReAct (

B=3

	Baseline	Offline Prompt Optimization					Test-Time Adaptation				Harness Optimization
Dataset	Seed	Ours	SkillOpt	GEPA	MIPROv2	ACE	DC-Cu	Ours + Refine	Ours + ReAct	Ours + DC-Cu	Meta- Harness
BIRD DEV	58.91 (0.44)	63.66 (1.18)	64.38 (1.21)	63.01 (1.30)	63.03 (0.84)	63.36 (1.57)	62.14 (0.27)	65.38 (1.13)	63.57 (1.10)	64.12 (0.69)	58.52 (0.72)
BigCodeBench	52.22 (0.77)	52.43 (0.57)	52.96 (0.36)	52.69 (0.48)	52.54 (1.94)	52.49 (0.36)	50.71 (1.77)	52.59 (0.57)	53.48 (0.50)	53.38 (1.13)	51.49 (0.28)
DSCodeBench	50.83 (0.55)	55.39 (1.70)	54.56 (0.72)	53.94 (0.55)	54.84 (0.94)	55.26 (1.22)	45.09 (1.69)	54.63 (0.32)	54.08 (0.43)	54.49 (0.21)	50.14 (2.80)
CypherBench	73.13 (0.45)	83.67 (1.10)	79.00 (1.37)	76.87 (2.05)	79.37 (2.36)	82.17 (2.10)	73.83 (1.01)	88.63 (2.02)	87.87 (1.42)	81.13 (3.20)	82.47 (1.96)

Appendix

Table 9: Performance across four code-generation benchmarks using GPT-5.4. Results (pass@1) are reported as mean (standard deviation) over three runs. Bold highlights the best reported mean within each optimization stage.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback
Apr 6, 2026Samira Hajizadeh, Suman JanaInference-Time Optimization Code Optimization
Department of Electrical Engineering, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
VERSE: Verified Self-Evolving Optimizer for Agent Harnesses
Oct 2, 2026Zekai Wang, Yingqiang Ge, Zekun Wang +6Agent Harness Optimization Self-Evolving Agents
MIT · Amazon
TTHE: Test-Time Harness Evolution
Jul 9, 2026Jun Nie, Yonggang Zhang, Jun Song +5Test-Time Adaptation Agent Harness Optimization
Hong Kong Baptist University · University of Science and Technology of China · Hong Kong Generative AI Research & Development Center, The Hong Kong University of Science and Technology +1

Apr 6, 2026J/K move · Enter open · S save

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

Samira Hajizadeh, Suman Jana

Department of Electrical Engineering, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA

Inference-Time Optimization Code Optimization LLM Code Generation Execution-Guided Code Generation Differential Testing

Open paper AskPDF arXiv

Large language models (LLMs) can generate functionally correct programs that differ substantially in execution efficiency. Existing inference-time optimization methods typically refine each candidate using pointwise runtime or profiling feedback, which identifies how costly an implementation is or where the cost arises, but offers limited guidance on how the computation should change. We introduce DIFFERENTIAL EXECUTION FEEDBACK (DEF), which compares implementations that are nearby in structural program space but separated in performance, turning their execution and implementation differences into directional optimization evidence. We demonstrate DEF in EffiPair, a training-free, test-time framework that pairs structurally similar programs with different efficiencies, distills their relative execution behavior into compact feedback, and iteratively refines a candidate pool. Across EvalPerf, Mercury, and ENAMEL, using GPT-4o mini, DeepSeek-V4.1 Flash, and GPT-5 mini, EffiPair achieves the highest value on each benchmark's official efficiency metric in all nine model-benchmark settings under matched evaluation conditions. Moreover, two contrastive refinement rounds improve efficiency over the selected initial draft in every setting while preserving or improving Pass@1. These results demonstrate the effectiveness of relational execution feedback as a lightweight signal for test-time code optimization.

Abstract

Automatic Prompt Optimization

Explore similar work

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

TTHE: Test-Time Harness Evolution