cs.LGMay 27, 2026

Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL

Authors: Kunhao Zheng, Pierre Chambon, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Negrevergne, Gabriel Synnaeve

Organizations: Meta Superintelligence Labs - FAIR · Université Paris-Dauphine PSL

Abstract

Linear interpolation between fine-tuned checkpoints has been shown to trace the Pareto front between competing objectives, but whether extrapolative weight averaging can extend such frontiers to new checkpoints useful at inference time, without additional RL training, remains unclear. We study this question in RL for competitive programming, where hidden unit tests under time and memory limits enforce both functional correctness and computational efficiency. Starting from a shared initialization, we train checkpoints under nested unit-test coverage: low-coverage rewards require passing smaller-input tests, while high-coverage rewards require passing progressively larger tests up to the full suite. This sweep reveals the emergence of a correctness-efficiency frontier: on hard problems, higher-coverage reward reduces optimization failures but increases correctness failures, leaving solve rate nearly unchanged. Interpolation between low- and high-coverage checkpoints recovers this frontier, while extrapolation extends it beyond the trained endpoints. Both the frontier and its extrapolative continuation appear across three inference settings, pure reasoning, tool use, and agentic coding, and across two model scales, 32B and 7B. At the problem level, moving along the frontier changes which problems are solved, making extrapolated checkpoints complementary policies in inference-time scaling. Ensembles with extrapolative weight averaging broaden coverage and improve pass@250 on LCB/hard by 3.3% over the best single checkpoint at matched sample budget. These results show that nested unit-test coverage in code RL induces a frontier that extrapolative weight averaging can navigate, extend, and exploit.

Explore similar work

CardsList
Jul 29, 2026cs.LG

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can pass the same tests while differing greatly in runtime. We study how to train code agents to prefer faster correct implementations, rather than treating efficiency only as an evaluation metric. The key difficulty is that runtime is a fragile reward. It is meaningful only after a program is correct, varies across tasks, and gives little guidance when most sampled programs fail to compile or run. We propose \textbf{RLPF}, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward. Failed programs are ordered by execution progress, while correct programs are ranked by their relative improvement from the baseline toward the expert reference. This gives useful feedback before correctness and performance-sensitive feedback after correctness. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from 11.1%11.1\% to 54.6%54.6\% and improves relative efficiency from 8.1%8.1\% to 38.6%38.6\%. The trained model becomes competitive with stronger open-weight systems, and its optimization behavior transfers modestly to EffiBench-X. Additional studies show that model-generated references provide useful but weaker supervision, and that the full composite reward is more reliable than correctness-only or runtime-only baselines. These results suggest that code agents can be trained not only to pass tests, but also to optimize the programs they write.
Huihao Jing, Haozhe Cui, Wenbin Hu +9
Code GenerationFeedback LearningPrompt OptimizationOutcome
AskPDF
Jul 28, 2026cs.LG

Reinforcement Learning for Code Optimization

RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).
Pierre Chambon, Kunhao Zheng, Juliette Decugis +2
Code OptimizationReinforcement Learning With Verifiable RewardCode Generation
AskPDF
May 26, 2026cs.CL

Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning

Repeated sampling with a verifier is the standard way to allocate test-time compute for code generation, with pass@KK as the canonical metric. Yet the standard policy class draws KK independent samples from a single answer distribution, so attempts often collapse onto near-duplicate reasoning paths and waste the budget on redundant rollouts. This failure is costly in competitive programming, where many problems admit multiple distinct algorithmic strategies and pass@KK requires only one correct attempt. We propose Coordinated Pass@KK Policy Optimization (CPPO), which turns pass@KK generation into joint exploration over strategies: a planner emits a tuple of K=4K{=}4 alternative high-level methods, and a shared solver attempts one solution per method. CPPO trains this joint policy with a multiplicative planner reward, Rplan=Jψ⋅RoutR_{\mathrm{plan}} = J_ψ\cdot R_{\mathrm{out}}, assigning credit only to valid strategy tuples that lead to verifier-confirmed pass@KK success. Across APPS, CodeContests, and LiveCodeBench-v6, CPPO improves pass@44 over direct sampling, planning baselines, planner-only SFT, and pass@KK-oriented RL under the same K=4K{=}4 solver-attempt budget, with statistically significant gains on six of nine model--benchmark cells. The largest single gain is +0.16+0.16 on Qwen3.5-9B LiveCodeBench-v6 over the strongest baseline, PKPO (0.588→0.7480.588 \rightarrow 0.748; paired bootstrap, p<0.05p < 0.05).
Yilong Li, Suman Banerjee, Tong Che
Code GenerationMulti-Objective Policy OptimizationCodebooksTest Time
AskPDF