Execution-Grounded Credit Assignment for GRPO in Code Generation

Abhijit Kumar; Natalya Kumar; Shikhar Gupta

cs.LGMar 17, 2026

Execution-Grounded Credit Assignment for GRPO in Code Generation

Authors: Abhijit Kumar, Natalya Kumar, Shikhar Gupta

Abstract

Critic-free reinforcement learning with verifiable rewards (RLVR) improves code generation by optimizing unit-test pass rates, but GRPO-style updates suffer from coarse credit assignment: a single outcome signal is spread uniformly across long programs even when failure stems from a localized semantic error. We propose Execution-Grounded Credit Assignment (EGCA), which localizes GRPO updates using execution traces. For programs that satisfy algorithmic constraints but fail tests, EGCA executes the candidate and a canonical reference solution (curated once offline; used for analysis, not supervision) under identical instrumentation, identifies the earliest semantic divergence, and assigns advantage only to the corresponding token span while masking downstream tokens. EGCA is a drop-in modification requiring no critic, auxiliary loss, or learned verifier, yielding 82.1% pass@1 on HumanEval (+3.1 over GRPO) and 68.9% on MBPP (+1.5) with 18% wall-clock overhead.

Figures & tables

Figure 1: Motivation: credit smear vs. localized updates. Sequence-level RLVR objectives apply unit-test outcomes uniformly across long programs, penalizing large spans of correct code for a localized semantic bug. EGCA concentrates gradient mass on the earliest semantically divergent span (identified via execution) while masking downstream tokens, improving credit assignment in the near-correct regime.

Figure 2: EGCA pipeline. We extract constraints from a canonical reference, sample and execute a group of programs, route each into syntax / constraint / logic / correct via deterministic gates, and apply token-level GRPO by localizing advantage (compiler span for syntax , earliest reference-trace divergence for logic ) while masking downstream tokens.

Method	HumanEval	MBPP
DeepSeek-Coder (6.7B) base	78.6	65.4
SFT	71.9	60.3
Vanilla PPO	78.0	65.6
GRPO	79.0	67.4
RLTF	77.9	64.5
StepCoder-mask	78.7	67.0

Table 1: Main performance (pass@1, %) on HumanEval and MBPP.

Model	HumanEval	MBPP
Qwen2.5-Coder-1.5B-Instruct	70.7	69.2
Qwen2.5-Coder-7B-Instruct	84.8	79.2

Table 2: Code-generation capability of debugger LLMs (pass@1, %).

Debugger model	HumanEval	MBPP
Qwen2.5-Coder-1.5B-Instruct	78.9	66.1
Qwen2.5-Coder-7B-Instruct	82.1	68.9
Claude 4.5 Sonnet	83.7	67.8

Table 3: EGCA performance when varying the debugger/localizer LLM (pass@1, %).

Method	HumanEval	MBPP
Teacher SFT	60.9	58.1
Teacher-critique RL	76.3	66.1
EGCA (Ours)	82.1	68.9

Table 4: Distillation controls (pass@1, %).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Fraction in syntax mode	$\sim 8\%$
Fraction in constraint mode	$\sim 12\%$
Fraction in logic (near-correct) mode	$\sim 35\%$
Fraction in correct mode	$\sim 45\%$
Average boundaries $K$ per sample	12.4
Wall-clock overhead vs. vanilla GRPO	$+18\%$

Appendix

Table 5: Overhead accounting (averages across APPS+ training).

Target	HumanEval	MBPP
Earliest divergence (EGCA default)	82.1	68.9
Last divergence	80.2	67.5
Random executed boundary	79.6	67.1
No localization (uniform)	79.0	67.4

Appendix

Table 6: Ablations on localization target under identical training budgets.

Step	$i$	$j$	nums[i]	nums[j]	Divergence?
1	1	0	3	1	No
2	2	0	3	1	No
3	2	1	3	3	Yes

Appendix

Table 7: Trace comparison for [1,3,3,5] . The first divergence occurs when the candidate uses

\geq

while the reference uses

>

Hyperparameter	Value
SFT learning rate	$2\times 10^{-5}$
SFT epochs	3
SFT warmup	0.3 epochs
SFT LR schedule	Linear decay to zero
GRPO policy learning rate	$5\times 10^{-7}$
GRPO optimizer	AdamW

Appendix

Table 8: Hyperparameters used in our experiments.

RL for Code Generation

Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.

See all 149 papers

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning
May 12, 2026Hao Wang, Rui Li, Lei Sha +1RL for Language Model Reasoning Execution-Guided Code Generation
Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China +1
Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL
Sep 26, 2026Jinhao Dong, Liang Zhao, Zihao Yue +7RL for Code Generation LLM Agent Training
LLM Core, Xiaomi · Renmin University of China · Peking University +1
Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
May 1, 2026Xin-Ye Li, Ren-Biao Liu, Yun-Ji Zhang +3RL for Code Generation Reinforcement Learning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China

May 12, 2026J/K move · Enter open · S save

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

Hao Wang, Rui Li, Lei Sha, Jie M. Zhang

Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China+1

RL for Language Model Reasoning Execution-Guided Code Generation Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment

Open paper AskPDF arXiv

Existing code reasoning methods primarily supervise final code outputs, ignoring intermediate states, often leading to reward hacking where correct answers are obtained through inconsistent reasoning. We propose StepCodeReasoner, a framework that introduces explicit intermediate execution-state supervision. By automatically inserting structured print-based execution-trace anchors into code, the model is trained to predict runtime states at each step, transforming code reasoning into a verifiable, stepwise execution modeling problem. Building on this execution-aware method, we introduce Bi-Level GRPO, a reinforcement learning algorithm for structured credit assignment at two levels: inter-trajectory, comparing alternative execution paths, and intra-trajectory, rewarding intermediate accuracy based on its impact on downstream correctness. Extensive experiments demonstrate that StepCodeReasoner achieves SOTA performance in code reasoning. In particular, our 7B model achieves 91.1% on CRUXEval and 86.5% on LiveCodeBench, outperforming the CodeReasoner-7B baseline (86.0% and 77.7%) and GPT-4o (85.6% and 75.1%). Furthermore, on the execution-trace benchmark REval, our model scores 82.9%, outperforming baseline CodeReasoner-7B (72.3%), its 14B counterpart (81.1%), and GPT-4o (77.3%). Additionally, our approach also improves code generation performance, demonstrating that explicit execution modeling enhances both code reasoning and code generation.

Abstract

RL for Code Generation

Explore similar work

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL

Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation