The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning

Daniel Thi Graviet; Lovre Pesut; Ivan Dagelic; Vedran Jukic; Ivan Burazin

cs.LGJul 1, 2026

The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning

Authors: Daniel Thi Graviet, Lovre Pesut, Ivan Dagelic, Vedran Jukic, Ivan Burazin

Organizations: Daytona United States

Abstract

Coding-agent reinforcement learning treats execution infrastructure as a background implementation detail, despite relying on large numbers of interactive software rollouts. This is a missed opportunity: measuring infrastructure overhead can reveal practical efficiency gains for RL post-training, where small per-rollout savings compound at scale. We present a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. We find up to $110\times$ variation in cold-start latency and a $1.8\times$ spread in projected worker-hours for one million 150-step trajectories. Our results suggest that future coding-agent RL systems should optimize execution substrates as part of the training system itself, not merely as deployment plumbing.

Explore similar work

RLPF: Reinforcement Learning from Performance Feedback for Code Generation
Jul 29, 2026Huihao Jing, Haozhe Cui, Wenbin Hu +9Code Generation Feedback Learning
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Jul 12, 2026Hong Yang, Qi Yu, Travis DesellCoding Agents Ablation
CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
Jul 21, 2026Qijia He, Jiayi Cheng, Chenqian Le +8Coding Agents Retrying

Jul 29, 2026 · cs.LGJ/K move · Enter open · S save

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

Huihao Jing, Haozhe Cui, Wenbin Hu, Shaojin Chen+8

Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can pass the same tests while differing greatly in runtime. We study how to train code agents to prefer faster correct implementations, rather than treating efficiency only as an evaluation metric. The key difficulty is that runtime is a fragile reward. It is meaningful only after a program is correct, varies across tasks, and gives little guidance when most sampled programs fail to compile or run. We propose \textbf{RLPF}, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward. Failed programs are ordered by execution progress, while correct programs are ranked by their relative improvement from the baseline toward the expert reference. This gives useful feedback before correctness and performance-sensitive feedback after correctness. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from

11.1\%

54.6\%

and improves relative efficiency from

8.1\%

38.6\%

. The trained model becomes competitive with stronger open-weight systems, and its optimization behavior transfers modestly to EffiBench-X. Additional studies show that model-generated references provide useful but weaker supervision, and that the full composite reward is more reliable than correctness-only or runtime-only baselines. These results suggest that code agents can be trained not only to pass tests, but also to optimize the programs they write.

Code Generation Feedback Learning Prompt Optimization Outcome

Open paper Ask about it PDF

Abstract

Explore similar work

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

When Does Restricting a Coding Agent to execute_code Help? A Regime ×\times× Agent-Design Ablation

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation