cs.CLOct 19, 2026

Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift

Authors: Amarsaikhan Batjargal, Noriko Tomuro

Organizations: Jarvis College of Computing and Digital Media, DePaul University, Chicago, IL, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution. Execution also provides behavioral information that can be reused as additional supervision. The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them. We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement. We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct. Our results do not support either hypothesis under the configuration tested. Although co-evolution improves training performance, these gains do not generalize to held-out problems. Compared at the same stage of training, rollout recycling produces roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Over training, trained tokens per GRPO sequence fall from 185 to 86 as training pass rate increases, while held-out performance declines or fails to sustain early gains. We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization. The results show that higher training reward alone is not evidence of generalizable code self-improvement and highlight the importance of held-out evaluation and drift diagnostics.

Figures & tables

ObjectiveWeightInputSupervised target
Pointwise0.00one candidate solutioncorrectness label (not used)
Pairwise0.05two candidate solutionsthe better candidate
Reflection1.00failing code + execution tracecorrect rollout, else canonical solution
TABLE I: The three SPARK-style auxiliary objectives in our final co-evolution configuration. Pointwise judgment is not used, to avoid short-target interference with full code generation.
Fig. 1: Execution-only GRPO vs. single-adapter rollout recycling. The blue path shows the shared execution-grounded GRPO loop used by both conditions: MBPP problems are sampled, the policy generates candidate programs, programs are executed in a sandbox, execution rewards are assigned, group-relative advantages are computed, and the GRPO policy update is applied. Condition A stops after the GRPO update and discards rollouts. Condition C adds the orange auxiliary branch, where execution-labeled rollouts are recycled into pointwise, pairwise, and reflection supervision. Both the GRPO policy update and the auxiliary SFT step update the same shared LoRA adapter. The right-hand boxes show per-iteration evaluation, diagnostics, and artifacts.
ConditionHE@1HE@5MBPP@1MBPP@5Train PassMean RewardKLRefl. Fix
Baseline0.79630.85370.63400.6800–––0.1176
A: Exec-only GRPO0.80490.85370.63600.69000.63850.67130.0011–
C-light: Naive Co-Evolve0.77320.82320.65800.68000.65770.69300.09410.1389
C-reg: Regularized Co-Evolve0.80000.85370.63000.72000.62880.65780.01360.0968
TABLE II: Preliminary hyperparameter study. HE = HumanEval; MBPP = held-out MBPP; @1 and @5 denote pass@1 and pass@5; Train Pass = training pass rate; Refl. Fix = reflection fix rate. Iteration 0 is the shared pretrained baseline; metrics shown are final-iteration values. Bold marks the best non-baseline value in each column.
Model / CheckpointHE@1HE@5MBPP@1MBPP@5Train PassMean RewardKLRefl. Fix
Baseline0.79630.85370.63400.6800–––0.1176
A-long best (iter 4)0.81590.87200.63800.71000.64860.68440.0023–
A-long matched (iter 5)0.79630.85370.63600.69000.67230.70860.0024–
C-reg2 best HE@1 (iter 1)0.79880.84760.63800.70000.59330.62370.00030.0606
C-reg2 matched (iter 5)0.77070.82320.62800.67000.65740.68570.08220.0811
C-reg2 final (iter 6)0.77440.82320.63200.68000.69620.72740.09570.0833
TABLE III: Main experiment summary, with columns as in Table II . The table includes best-checkpoint rows, matched iteration 5 rows, and the final C-reg2 checkpoint. Bold marks the best value in each column.
Fig. 2: Held-out benchmark trajectories. Condition A achieves the strongest HumanEval pass@1, while C-reg2 does not improve cross-benchmark generalization. Dashed lines mark the pretrained iteration-0 baseline. Comparison is across iterations 0–5 where both conditions have completed checkpoints. In all figures, the runs are labeled A-stage2 and C-reg2-stage2 in the plots, from the internal naming of the second experiment stage; these are A-long and C-reg2-long.
Fig. 3: Mechanism diagnostics. Both conditions improve training pass rate, A slightly more than C-reg2, but C-reg2 shows much larger KL drift and a sharp decline in tokens per trained sequence. In the train-minus-HumanEval panel, both gaps narrow because training pass rate rises; the difference is that A’s HumanEval holds near baseline while C-reg2’s falls.
More in this topic

RL for Code Generation

Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.

See all 150 papers

Explore similar work

CardsList
May 28, 2026

Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback

Reinforcement learning with verifiable rewards (RLVR) trains language models using programmatically checkable signals such as unit-test outcomes, enabling direct optimization for functional correctness in code generation. We conduct an empirical study of RLVR for Python code generation on the MBPP benchmark using two small models (Qwen3-0.6B and Llama3.2-1B) with LoRA fine-tuning. Across multiple reward formulations such as: unit-test-only rewards, static-analysis-only shaping via the Ruff linter, and a combined reward, we compare group-based policy optimization variants (GRPO and GSPO) and evaluate both functional correctness and behavioral diagnostics. In our experimental setting, RLVR improves pass@1 on MBPP test by up to 13 percentage points under proposed combined reward configuration. However, we find that reward shaping can induce systematic behavioral shifts: using only static-analysis penalties may bias the policy toward shorter completions that reduce lint errors without reliably improving functional correctness. In contrast, combined rewards mitigate this degeneration and yield more stable trade-offs between correctness and style constraints. Overall, our results highlight that RLVR effectiveness for code generation is highly sensitive to reward design and optimization granularity, and that diagnostics beyond pass@1, including generation length, Ruff severity profiles, and execution error types are useful for identifying failure modes.

Egor Skopin, Evgeny Kotelnikov

RL for Code GenerationReward ShapingReinforcement Learning with Verifiable RewardsCode Generation EvaluationVerified Code Generation
AskPDF
Sep 18, 2026

Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation

Self-play methods that co-train a single language model as both coder and test author promise to move code-generation RL beyond fixed test suites, but they suffer from two coupled pathologies: permissiveness collapse, where pass-rate rewards are maximised by trivial, non-discriminative tests, and concentration bias, where i.i.d. sampled tests cluster on modal inputs and inflate estimator variance. We introduce CoVer (Co-trained Coder and Verifier), a single-policy GRPO framework that addresses both failure modes. First, an information-gain (IG) reward scores each self-generated test by the mutual information between its pass/fail vector and a graded, ground-truth-anchored correctness signal y [0, 1] m, gated by the sign of their covariance so that only positively discriminative tests receive reward. Second, a three-stage diversity-aware selection step prunes a candidate pool to a behaviourally non-redundant suite (invalidity, input-string, execution-profile filtering), raising the effective sample size of the IG estimator at fixed execution budget. On five benchmarks (LiveBench, MBPP, LiveCodeBench, CodeContests, Code-Forces), CoVer raises one-shot pass@1 by +5.8 points at 7B and +7.1 points at 14B over the Qwen2.5-Instruct backbone, and achieves the highest macro-average among all compared methods at both scales. As a drop-in backbone inside the CodeT ranking pipeline, CoVer-7B adds +3.5 points, demonstrating the dual benefit of co-training for both generation and selection.

Ana Nunez, Peyman Najafirad

Secure AI and Autonomy Lab University of Texas at San Antonio

RL for Code GenerationCode GenerationReinforcement Learning with Verifiable RewardsAutomated Test Generation
AskPDF
Sep 8, 2026

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.

Jiacheng Xu, Feng Chen, Xiuneng Xu +1

Nanyang Technological University, Singapore

RL for Code GenerationPolicy OptimizationTest-Time RLReward Design for RL
AskPDF