EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

Samira Hajizadeh; Suman Jana

cs.PLApr 6, 2026

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

Authors: Samira Hajizadeh, Suman Jana

Organizations: Department of Electrical Engineering, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA

Abstract

Large language models (LLMs) can generate functionally correct programs that differ substantially in execution efficiency. Existing inference-time optimization methods typically refine each candidate using pointwise runtime or profiling feedback, which identifies how costly an implementation is or where the cost arises, but offers limited guidance on how the computation should change. We introduce DIFFERENTIAL EXECUTION FEEDBACK (DEF), which compares implementations that are nearby in structural program space but separated in performance, turning their execution and implementation differences into directional optimization evidence. We demonstrate DEF in EffiPair, a training-free, test-time framework that pairs structurally similar programs with different efficiencies, distills their relative execution behavior into compact feedback, and iteratively refines a candidate pool. Across EvalPerf, Mercury, and ENAMEL, using GPT-4o mini, DeepSeek-V4.1 Flash, and GPT-5 mini, EffiPair achieves the highest value on each benchmark's official efficiency metric in all nine model-benchmark settings under matched evaluation conditions. Moreover, two contrastive refinement rounds improve efficiency over the selected initial draft in every setting while preserving or improving Pass@1. These results demonstrate the effectiveness of relational execution feedback as a lightweight signal for test-time code optimization.

Figures & tables

Figure 1: Overview of EffiPair . Given a coding task, the LLM samples $N$ candidate programs. Candidates are checked for correctness and profiled for efficiency, then stored in a candidate pool. EffiPair selects a pair consisting of an efficient reference program $p^{+}$ and a similar but less efficient candidate $p^{-}$ , summarizes their relative execution differences into compact Differential Execution Feedback (DEF) , and uses this signal to refine the program. The refined program is added back to the pool for subsequent rounds.

		EvalPerf			Mercury			ENAMEL
Model	Method	DPS norm	Pass@1	Tok.	Beyond@1	Pass@1	Tok.	eff@1	Pass@1	Tok.
GPT-4o mini	Baseline (instruct)	75.50	89.83	0.2	81.06	92.19	0.5	49.78	83.80	0.3
	EffiLearner-Gen	80.28	88.98	0.6	79.94	90.62	0.5	48.51	83.10	0.6
	ECCO nl2code	79.49	82.20	0.3	79.68	89.45	0.5	53.99	83.80	0.7
	ECCO exec-refine	74.88	67.80	0.7	80.32	89.84	1.2	47.99	75.35	1.3
	ECCO self-refine	74.59	61.02	2.1	74.49	85.16	2.6	36.22	60.56	2.9

Table 1: EffiPair versus prior efficiency-oriented methods, all reproduced with the same models, decoding settings and evaluation harnesses. Each benchmark reports its official efficiency metric and Pass@1 (%).

r_{0}

is EffiPair ’s selected draft before refinement and

r_{2}

its output after two contrastive rounds; both are means over three independent runs, while other rows are single runs. Bold marks the best efficiency and Pass@1 within each model block. Tok. is thousands of tokens per task (prompt

+

completion) for that run; for EffiPair it covers the three sampled drafts and, in the

r_{2}

row, the two refinement rounds.

Generation
Method	DeepSeek-V4.1 Flash	Gemini 3.8 Flash	GPT-5.6 Luna	Claude Sonnet 5
	eff@1 / Pass@1	eff@1 / Pass@1	eff@1 / Pass@1	eff@1 / Pass@1
Greedy Draft	55.73 / 93.66	60.88 / 97.18	55.29 / 90.14	56.01 / 94.37
Single Sampled Draft	65.30 ${\scriptstyle\pm 2.34}$ / 93.66	78.99 ${\scriptstyle\pm 1.34}$ / 96.71	64.36 ${\scriptstyle\pm 0.71}$ / 89.20	62.76 ${\scriptstyle\pm 0.96}$ / 93.66
Best-of-5	72.61 ${\scriptstyle\pm 1.99}$ / 93.19	85.23 ${\scriptstyle\pm 0.96}$ / 98.12	71.58 ${\scriptstyle\pm 1.23}$ / 94.37	69.09 ${\scriptstyle\pm 1.74}$ / 96.24
Pool Selection ( $r_{0}$ )	71.52 ${\scriptstyle\pm 1.32}$ / 94.84	83.70 ${\scriptstyle\pm 0.81}$ / 97.42	69.91 ${\scriptstyle\pm 1.06}$ / 93.66	67.38 ${\scriptstyle\pm 1.20}$ / 94.60

Table 2: Component ablation on ENAMEL (142 tasks), reporting the official eff@1 / Pass@1 (%) metrics across four model families. We compare generation-only baselines with isolated refinement, random-pair refinement, and matched-pair refinement, with and without the corresponding constructed execution feedback (EF). In the last row, DEF is implemented by combining matched-pair refinement with execution feedback to create EffiPair . Best-of-5 is included as a generation-only control. All refinement methods operate on the same initial candidate pool and use the same correctness gate. Bold denotes the best reported value for each metric within each model.

Figure 2: Iterative refinement behavior of EffiPair on EvalPerf across evaluated models.

Arm	Contrast source	$n$	Mean gain (%)
Best seed (no refinement)	none	125	1.21
Solo (profile only)	none	125	1.19
Same-site pair	resampled variant	125	1.24
History pair	own prior version	107	1.16
Sibling pair	similar function	79	0.85
History $+$ sibling	both	119	1.06

Table 3: EffiPair at repository scale on SWE-Perf (125 of 140 instances; exclusions in Appendix A.5 ). Each arm commits one patch per instance, the fastest correct member of its pool on EffiPair ’s own internal clock, which is not the metric reported here, and all arms and the expert patch are then measured together in one scoring session; Mean gain is the significance-gated improvement under SWE-Perf’s unmodified harness over that row’s

n

instances.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: DPS, $\text{DPS}_{\text{norm}}$ , and Pass@1 for various similarity thresholds on the EvalPerf benchmark.

Panel of Figure 4 — Figure 4: Mercury #552 (DeepSeek-V4.1 Flash): the round-1 pair and the final refined program, each with its statement-level AST. Up to cosmetic edits, the pair differs only in how the table is stored (shaded) and the indexing this forces (blue); the refined program keeps the six states in scalars. ... marks omitted lines. Full case in Appendix A.4 .

draw	style	gate	wall (s)	workload (ms)	profile (ms)
1d7bfd0f	medium	ok	1.156	1085.6	1255.0
eb72593a	fast	ok	0.797	728.4	899.6
09cc3033	naive	timeout	30.031	–	–

Appendix

Table 4: Gate and clocks of the three draws (iteration 0).

run	iter.	source	Cand. B	gate	wall (s)	workload (ms)	profile (ms)
09cc3033	0	naive draw	–	timeout	30.031	–	–
1d7bfd0f	0	medium draw	–	ok	1.156	1085.6	1255.0
eb72593a	0	fast draw	–	ok	0.797	728.4	899.6
49b12e17	1	refines eb72593a	1d7bfd0f	ok	0.593	523.4	561.8
a4522979	2	refines 49b12e17	1d7bfd0f	ok	0.157	88.5	100.3

Appendix

Table 5: Every entry of the pool with its clocks.

pick	run	passes	runtime (s, median of 3)	Beyond@1
$r_{0}$	eb72593a	yes	0.3756	54.0
$r_{1}$	49b12e17	yes	0.2760	66.2
$r_{2}$	a4522979	yes	0.0508	94.0

Appendix

Table 6: Committed picks and their official Mercury scores.

rep.	best draw ( $r_{0}$ )	after round 1	after round 2	partner	naive (s)
1	1cefd2c5 (medium) 1258.7	bd58c82f 999.2	6b512ed5 162.3	aa9818e2	0.071
2	e543525f (fast) 957.3	31af93ac 313.2	c8225cda 195.4	0e89ba0d	0.064
3 ∗	eb72593a (fast) 899.6	49b12e17 561.8	a4522979 100.3	1d7bfd0f	30.031

Appendix

Table 7: Profile time (ms) of the best correct program after each stage in the three repetitions, the partner (Candidate B) of both rounds, and the wall time (s) of the naive draw’s gate run. ∗ The repetition shown above.

stage	prompt	completion
generation, three draws ( $r_{0}$ )	1139	799
refinement round 1	1276	606
refinement round 2	1528	414
total ( $r_{2}$ )	3943	1819

Appendix

Table 8: Tokens for this task in repetition 3.

Inference-Time Optimization

Inference-time optimization adapts model computation, outputs, or runtime settings to improve quality under deployment constraints.

See all 96 papers

EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
May 19, 2025Yuhao Qing, Boyu Zhu, Mingzhe Du +9Code Generation Benchmarks LLM Code Generation
HKU · UCL · NTU +6
EffiSkill: Agent Skill Based Automated Code Efficiency Optimization
Mar 29, 2026Zimu Wang, Yuling Shi, Mengfan Li +4Code Optimization Large Language Model-Guided Optimization
Shanghai Jiao Tong University China · University of California, Berkeley USA · Columbia University USA +2
HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
Oct 13, 2026Mingyu Lu, Timothy Ossowski, Tristan Naumann +2Automatic Prompt Optimization LLM Code Generation
University of Washington · Microsoft Research · Recursion

May 19, 2025J/K move · Enter open · S save

EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo+8

HKU · UCL · NTU+6

Code Generation Benchmarks LLM Code Generation Multilingual Code Generation

Open paper AskPDF arXiv

Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X, the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around 62% of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1's Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x.

Abstract

Inference-Time Optimization

Explore similar work

EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

EffiSkill: Agent Skill Based Automated Code Efficiency Optimization

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation