Function-Level Execution Feedback for Code Preference Optimization

Idris Nechnech; Sehwan Kim; Jimin Seo; Yeongoon Kim; Minhae Oh; Sangwoo Hong; Jungwoo Lee

cs.AIAug 23, 2026

Function-Level Execution Feedback for Code Preference Optimization

Authors: Idris Nechnech, Sehwan Kim, Jimin Seo, Yeongoon Kim, Minhae Oh, Sangwoo Hong, Jungwoo Lee

Organizations: Department of Electrical and Computer Engineering, Seoul National University · Department of Computer Science and Engineering, Konkuk University

Abstract

Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize. We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program. We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: https://github.com/inechnech/STEP-KTODER.

Figures & tables

Figure 1: Overview of the Step-KTOder framework. A reference solution $y^{\star}$ is decomposed into module-level functions $s_{1},\ldots,s_{n}$ , and function-level unit tests are generated for each function. Candidate completions are evaluated against these tests to obtain step labels $z_{i}$ indicating local correctness, alongside an outcome label $o$ from the dataset-provided test suite. The resulting labeled data is used to train the model with a joint objective that combines outcome-level KTO ( $\mathcal{L}_{\mathrm{out}}$ ) and function-level supervision ( $\mathcal{L}_{\mathrm{step}}$ ), reinforcing locally correct functions while penalizing locally incorrect ones.

Figure 2: Training signal comparison for a failing candidate. The wrong candidate differs from the decomposed reference solution only in the highlighted line, yielding locally correct functions

z_{1},z_{2},z_{3}=1

, a faulty final function

z_{4}=0

, and outcome label

o=0

. Outcome-only KTO penalizes all function spans, whereas Step-KTOder preserves the locally correct functions and penalizes only the faulty one.

Figure 3: Distribution of per-problem validated-test availability. Most problems achieve near-full test availability.

Dataset composition
	Qwen-1.5B	Qwen-3B	DeepSeek
Total rows	11,397	11,440	6,573
GT rows	5,018	5,018	0
Generated rows	6,379	6,422	6,573
Step supervision
Step-supervised	9,376	9,455	5,157

Table 1: Training dataset statistics. Step-supervised rows contain at least two supervised function-level labels and activate the step-level loss; fallback KTO rows use outcome-only KTO. Avg. steps / active row is computed over step-supervised rows. AST alignment measures exact agreement with the decomposition skeleton.

Method	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
Qwen2.5-Coder-1.5B-Instruct	0.689	0.604	0.664	0.556	0.240	0.054	0.052
w/ DPO	0.695	0.610	0.664	0.563	0.245	0.054	0.056
w/ Target-DPO	0.683	0.628	0.664	0.558	0.245	0.068	0.019
w/ KTO	0.701	0.610	0.648	0.542	0.245	0.061	0.052
w/ Step-KTOder	0.701	0.616	0.656	0.548	0.248	0.068	0.056

Table 2: Pass rate across model families and scales. Relative improvements are computed over outcome-only KTO within each model; gains are largest on harder benchmarks. The best results are highlighted in bold.

Conflict-handling policy
Method	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
KTO (no step loss)	0.878	0.835	0.741	0.624	0.367	0.115	0.116
Step-KTOder + mask conflicts	0.878	0.835	0.741	0.624	0.367	0.115	0.119
Step-KTOder + keep conflicts (default)	0.878	0.835	0.746	0.630	0.367	0.128	0.127
Step loss weight $\lambda_{\mathrm{step}}$

Table 3: Core ablations on Qwen2.5-Coder-3B-Instruct. We vary conflict handling, the step-loss weight

\lambda_{\mathrm{step}}

, and the minimum number of supervised steps required to activate the step loss.

Replacement target	Repair rate
Local-negative ( $z_{i}=0$ )	74.9%
Random observed	32.9%
Local-positive ( $z_{i}=1$ )	1.2%

Table 4: Repair intervention on failing candidates.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
KTO	$\mathbf{0.872_{\pm 0.006}}$	$0.827_{\pm 0.007}$	$\mathbf{0.743_{\pm 0.003}}$	$\mathbf{0.628_{\pm 0.003}}$	$0.364_{\pm 0.004}$	$0.113_{\pm 0.004}$	$0.109_{\pm 0.006}$
Step-KTOder	$\mathbf{0.872_{\pm 0.006}}$	$\mathbf{0.829_{\pm 0.006}}$	$\mathbf{0.743_{\pm 0.004}}$	$\mathbf{0.628_{\pm 0.003}}$	$\mathbf{0.365_{\pm 0.003}}$	$\mathbf{0.128_{\pm 0.000}}$	$\mathbf{0.123_{\pm 0.006}}$

Appendix

Table 5: Mean pass rate

\pm

standard deviation across three training seeds. Bold denotes the best mean within each benchmark.

Method	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
Qwen2.5-7B-Instruct	0.811	0.768	0.839	0.704	0.366	0.135	0.134
w/ DPO	0.817	0.768	0.844	0.706	0.362	0.149	0.142
w/ Target-DPO	0.799	0.744	0.796	0.677	0.376	0.108	0.138
w/ KTO	0.823	0.780	0.847	0.709	0.361	0.155	0.138
w/ Step-KTOder	0.829	0.787	0.844	0.717	0.362	0.162	0.142

Appendix

Table 6: Pass rates for the general-purpose Qwen2.5-7B-Instruct model.

Source	Difficulty	# Problems	%
TACO	easy	8,904	35.0
	medium	3,244	12.7
	medium_hard	2,745	10.8
	hard	3,162	12.4
	very_hard	2,374	9.3
	unknown_difficulty	5,014	19.7

Appendix

Table 7: Difficulty distribution of the raw TACO and APPS training splits before any filtering.

Source	Difficulty	Total	Validated	Pass rate
TACO	easy	4,112	2,141	52.1%
	medium	1,200	225	18.8%
	medium_hard	520	193	37.1%
	Total	5,832	2,559	43.9%
APPS	introductory	2,325	2,069	89.0%
APPS	interview	442	390	88.2%

Appendix

Table 8: Ground-truth decomposition validation rates by difficulty.

		Qwen2.5-Coder family			DeepSeek-Coder
Source	Difficulty	Ground truth anchors	1.5B gen.	3B gen.	6.7B gen.
TACO	easy	2,141	2,671	2,695	2,748
	medium	225	271	278	301
	medium_hard	193	245	237	260
APPS	introductory	2,069	2,643	2,657	2,661
APPS	interview	390	549	555	603

Appendix

Table 9: Difficulty distribution of the final training sets for each target model. Qwen2.5-Coder-1.5B-Instruct and Qwen2.5-Coder-3B-Instruct use the same validated decomposed reference solutions as positive anchors, but differ in their on-policy generated candidates. DeepSeek-Coder-6.7B-Instruct uses only on-policy generated candidates.

Decomposition scale
	TACO	APPS
Validated problems	2,559	2,459
Total functions	7,199	6,926
Testable functions	7,144	6,920
Unit-test generation
First-pass parse success	98.8%	99.6%

Appendix

Table 10: Function-level unit test generation quality. Validated-test availability is over testable functions.

Configuration	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
$\lambda_{U}=\lambda_{U,\mathrm{step}}=1.0$ (default)	0.878	0.835	0.746	0.630	0.367	0.128	0.127
$\lambda_{U,\mathrm{step}}=2.5$	0.878	0.835	0.746	0.624	0.365	0.122	0.127
$\lambda_{U}=2.5$	0.878	0.835	0.746	0.624	0.365	0.122	0.049

Appendix

Table 11: Effect of rebalancing outcome- and step-level

\lambda_{U}

on Qwen2.5-Coder-3B-Instruct.

Method	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
KTO only	0.878	0.835	0.741	0.624	0.367	0.115	0.116
Step only	0.866	0.823	0.741	0.630	0.360	0.100	0.112
Step-KTOder (default)	0.878	0.835	0.746	0.630	0.367	0.128	0.127

Appendix

Table 12: Step-only objective ablation on Qwen2.5-Coder-3B-Instruct.

Training data	Method	HumanEval		MBPP		BigCodeBench		LiveCodeBench
		Base	Plus	Base	Plus	Full	Hard
On-policy (default)	KTO	0.793	0.726	0.749	0.640	0.347	0.122	0.127
On-policy (default)	Step-KTOder	0.799	0.732	0.754	0.648	0.347	0.128	0.131
Off-policy (Qwen 3B-generated)	KTO	0.805	0.735	0.749	0.640	0.345	0.115	0.112
Off-policy (Qwen 3B-generated)	Step-KTOder	0.805	0.735	0.751	0.648	0.347	0.122	0.127

Appendix

Table 13: Effect of training with off-policy data on DeepSeek-Coder-6.7B-Instruct.

Panel of Figure 4 — Figure 4: LLM-as-a-judge agreement with execution-based labels is high for failing functions ( $z{=}0$ ) but much lower for passing functions ( $z{=}1$ ), revealing a bias toward predicting failure.

Step labels	HumanEval		MBPP		BigCodeBench		LiveCodeBench
	Base	Plus	Base	Plus	Full	Hard
KTO baseline	0.878	0.835	0.741	0.624	0.367	0.115	0.116
Step-KTOder + execution (default)	0.878	0.835	0.746	0.630	0.367	0.128	0.127
Step-KTOder + LLM-as-a-judge (GPT-5.4 mini)	0.860	0.817	0.746	0.635	0.354	0.095	0.112
Relative change vs. execution	-2.1%	-2.2%	+0.0%	+0.8%	-3.5%	-25.8%	-11.8%

Appendix

Table 14: Effect of replacing execution-based step labels with GPT-5.4 mini judgments on Qwen2.5-Coder-3B-Instruct.

Panel of Figure 5 — Figure 5: System and user prompts used to decompose reference solutions into behavior-preserving multi-function programs.

Panel of Figure 6 — Figure 6: User prompt schema and constraints for generating function-level unit tests. The model returns only test inputs; expected outputs are derived by executing the reference implementation.

Panel of Figure 7 — Figure 7: User prompt for candidate generation. The target model fills the function skeleton (with pass as each function body) constructed in Stage 3 of our pipeline.

Panel of Figure 8 — Figure 8: System prompt for the GPT-5.4 mini judge in the LLM-as-a-judge experiment (Appendix N ). The judge sees the function’s source code and its generated unit tests, and predicts whether the function passes them.

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 316 papers

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning
May 12, 2026Hao Wang, Rui Li, Lei Sha +1RL for Language Model Reasoning Execution-Guided Code Generation
Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China +1
EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback
Apr 6, 2026Samira Hajizadeh, Suman JanaInference-Time Optimization Code Optimization
Department of Electrical Engineering, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation
Aug 5, 2026Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova +3Large Language Model-Guided Optimization Code Generation
King’s College London, University of London · National University of Singapore · University College London, University of London

May 12, 2026J/K move · Enter open · S save

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

Hao Wang, Rui Li, Lei Sha, Jie M. Zhang

Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China+1

RL for Language Model Reasoning Execution-Guided Code Generation Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment

Open paper AskPDF arXiv

Existing code reasoning methods primarily supervise final code outputs, ignoring intermediate states, often leading to reward hacking where correct answers are obtained through inconsistent reasoning. We propose StepCodeReasoner, a framework that introduces explicit intermediate execution-state supervision. By automatically inserting structured print-based execution-trace anchors into code, the model is trained to predict runtime states at each step, transforming code reasoning into a verifiable, stepwise execution modeling problem. Building on this execution-aware method, we introduce Bi-Level GRPO, a reinforcement learning algorithm for structured credit assignment at two levels: inter-trajectory, comparing alternative execution paths, and intra-trajectory, rewarding intermediate accuracy based on its impact on downstream correctness. Extensive experiments demonstrate that StepCodeReasoner achieves SOTA performance in code reasoning. In particular, our 7B model achieves 91.1% on CRUXEval and 86.5% on LiveCodeBench, outperforming the CodeReasoner-7B baseline (86.0% and 77.7%) and GPT-4o (85.6% and 75.1%). Furthermore, on the execution-trace benchmark REval, our model scores 82.9%, outperforming baseline CodeReasoner-7B (72.3%), its 14B counterpart (81.1%), and GPT-4o (77.3%). Additionally, our approach also improves code generation performance, demonstrating that explicit execution modeling enhances both code reasoning and code generation.

Abstract

Code Generation

Explore similar work

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation