ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents

Chengyi Yang; Renxu Bao; Jue Hou; Zongpei Teng; Ziheng Wang; Yujie Lin; Chengsong Huang; Junnan Zhu

cs.SEOct 9, 2026

ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents

Authors: Chengyi Yang, Renxu Bao, Jue Hou, Zongpei Teng, Ziheng Wang, Yujie Lin, Chengsong Huang, Junnan Zhu,

Organizations: Xiamen University · Washington University in St. Louis · Institute of Automation, Chinese Academy of Sciences · Unisound

Abstract

Coding agents are increasingly used to build programs from scratch as software engineers do rather than to resolve issues within a repository or to generate a repository from a given requirements document. However, scalable verifiable training environments that force agents to discover what is required and to decide how the system should be designed remain scarce. Moreover, existing agent workflows also cannot sustain long-horizon development in which the agent explores the reference program before implementation. To address these challenges, we introduce ProgramGym, a framework consisting of Program-Env and Program-Agent for scalable training environments of long-horizon program reproduction agents. Specifically, Program-Env constructs verifiable training environments from repositories through a feedback-driven loop that repairs runtime sandbox builds and the quality of black-box tests. Using this novel pipeline, we obtain 1,648 training environments spanning C, C++, Go, and Rust. Program-Agent combines a role-isolated workflow of analysis, implementation, and review with revision management for compilation checks and failure recovery. Experiments on ProgramBench show that both Program-Env and Program-Agent improve performance on program reproduction. Fine-tuning on trajectories with these constructed environments raises the performance of Qwen3.8-27B from 34.87% to 42.84%. Program-Agent further raises the base and SFT models to 55.71% and 56.25%, both exceeding some flagship models. We will release the ProgramGym to support further research on coding agents that build programs from scratch.

Figures & tables

Figure 2: Overview of the Program-Env. Stage 1 collects repositories from GitHub by language and stars. Stage 2 builds the gold executable $B_{i}^{\star}$ and runtime sandbox $\mathcal{S}_{i}$ . Bridging the two stages, the dispatcher extracts documentation $D_{i}$ and splits the source into modules. In Stage 3, each module is assigned to a subagent that iteratively writes the test suite $\mathcal{T}_{i}$ under feedback.

Figure 3: Overview of Program-Agent. The Analysis stage explores the usage documentation and reference program to identify behavioral requirements. The Implement and Review stages then alternate to refine the candidate implementation for a fixed loop budget. Handoff carries behavioral evidence and delivery reports across isolated sessions, Workspace maintains the evolving implementation, and Revisions preserves independently buildable source code for validation and recovery.

Flagship Models ( Yang et al., 2026b )
Models	Agent Scaffolds	Tests Passed		Almost Resolved		Resolved
Models	Agent Scaffolds	(%, $\uparrow$ )	$\Delta$	(%, $\uparrow$ )	$\Delta$	(%, $\uparrow$ )	$\Delta$
GPT 5 mini	mini-SWE-agent	16.00	–	0.00	–	0.00	–
GPT 5.4 mini	mini-SWE-agent	16.40	–	0.00	–	0.00	–
Claude Haiku 4.5	mini-SWE-agent	30.00	–	0.00	–	0.00	–
Gemini 3 Flash	mini-SWE-agent	31.80	–	0.00	–	0.00	–

Table 2: Results on ProgramBench across models and agent scaffolds. Program-Agent uses

K{=}5

implementation rounds.

\Delta

is relative to the base model with Claude Code .

Figure 5: Distributions of code lines, coverage, and test cases in Program-Env.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Repositories	Excluded	Cumulative Retention	C	C++	Go	Rust
Comprehensive Search	41,833	–	100.00%	9,736	13,147	11,730	7,220
Metadata Validation	35,710	6,123	85.36%	7,604	10,720	10,777	6,609
Commit Locking	35,710	0	85.36%	7,604	10,720	10,777	6,609
Build Config Detection	30,459	5,251	72.81%	5,695	8,229	9,944	6,591

Appendix

Table 3: Repository statistics across the four-stage collection pipeline.

Language	Source of incompatibility	Representative failure
[.25em]C++	Build tools remove support for older policy versions.	CMake 4.0 rejects cmake_minimum_required with VERSION 3.4 by default, because policy compatibility below 3.5 is no longer supported.
[.25em]C++	Standard-library availability depends on the selected C++ language mode and library implementation.	In C++17 mode, libc++ omits std::auto_ptr and std::random_shuffle by default, causing compilation failure.
[.25em]Rust	Unstable features can change or be removed, making some projects dependent on a dated nightly toolchain.	Code enabling box_syntax fails with error E0557 on toolchains where that feature has been removed.

Appendix

Table 4: Selected examples of toolchain-version incompatibility.

Check	Scope	Pattern flagged	Level
Failure masking	Both	Expressions that override a failed exit status, such as \|\| true .	Error
Error suppression	Both	Redirections that discard standard error, such as 2>/dev/null , except in simple tool-availability or version checks.	Error
Stream separation	Both	Redirections that combine stdout and stderr, such as 2>&1 or &>log , except in simple tool-availability or version checks.	Error
Placeholder content	Both	Standalone placeholder lines such as ... , <script> , or [truncated] .	Error
Package installation	Compilation	Package-manager installation, such as apt , pip , npm , or cargo install <crate> . Project install targets such as make install are allowed.	Error
Target artifact reference	Compilation	The script omits a reference to the designated output artifact, named executable .	Error

Appendix

Table 5: Selected static checks on generated scripts. Preparation denotes download.sh ; compilation denotes compile.sh .

Finding	Suggested fix
Linter check
Failure masking	Remove \|\| true . Handle commands that may legitimately fail with an explicit if or case branch.
Error suppression	Remove 2>/dev/null . Handle acceptable failures explicitly so that other errors remain visible.
Stream separation	Keep standard output and standard error separate: leave them unredirected, or write them to separate logs (e.g., >build.log 2>build.err ).
Placeholder content	Replace the placeholder with the actual commands.
Package installation	Move package-manager installation to download.sh . Keep only project build and install targets in compile.sh .

Appendix

Table 6: Suggested fixes attached to each finding of the linter check and the execution check.

Figure 10: A case of executable and sandbox generation guided by feedback. A missing package breaks offline compilation; the agent reads the feedback, adds libopenexr-dev to download.sh , and passes all checks on the next attempt.

Figure 11: Overview of the dispatcher bridging Stage 2 and Stage 3. It decomposes the repository into modules, instruments the executable for coverage, extends

\mathcal{S}_{i}

into

\mathcal{S}_{i}^{\mathrm{test}}

, and extracts documentation

D_{i}

. Each module is then handed to a subagent in its own workspace, with

k

workspaces running in parallel in Stage 3.

Type	Behavior to explore
Args	Flag syntax, required and optional arguments, positional values, and argument type checks.
Config	Settings read from environment variables and configuration files, including precedence when values conflict.
Help	Output of --help and -h , usage summaries, and help for individual subcommands.
I/O	Reading from standard input and files, separating standard output from standard error, exit status, and output formatting.
Subcommand	Routing to the requested subcommand, resolving aliases, and applying global and subcommand-specific flags.
TUI	Keyboard navigation, key bindings, screen updates, and transitions between interaction modes.

Appendix

Table 7: Interaction categories used in test generation. Each category directs a subagent to a specific behavioral aspect of the black-box executable.

Language	Instrumentation method	Cases beyond rule-based instrumentation
[.25em] C	Add --coverage to compilation and linking through build variables, compiler wrappers, and recognized Makefiles or .mak files. Read .gcno/.gcda with GNU gcov or llvm-cov gcov , and summarize coverage with gcovr .	Build scripts may overwrite injected CFLAGS / LDFLAGS or use custom .mak variables that the language-specific rule does not reliably update. Applying the same flags to every step can also break the build. In kafel, for example, the injected --coverage option is passed directly to ld , which does not accept it.
[.25em] C++	Apply the same approach to C++ compilation and linking, including CXXFLAGS and compiler wrappers. Support both GNU gcov and llvm-cov gcov , with gcovr for reporting.	Project build settings can overwrite injected flags, while direct compiler calls can bypass the wrappers. Global injection can also interfere with build-system checks. In fast-wfc, for example, a CMake compiler test is instrumented but linked without the coverage runtime, causing configuration to fail before the project builds.
[.25em] Rust	Set RUSTFLAGS to -C instrument-coverage , then merge .profraw with llvm-profdata and export coverage with llvm-cov .	Replacing RUSTFLAGS can remove conditional-compilation or CPU-feature options required by the project, breaking its build, so coverage flags must be merged with the existing options. Targets without profiler_builtins also need a toolchain that provides the coverage runtime.
[.25em] Go	Build with -cover and -coverpkg . Collect profiles under GOCOVERDIR and convert them with go tool covdata textfmt .	Matching a fixed set of command forms can miss calls such as $GOROOT/bin/go , so the build receives no coverage flags. Building individual .go files can also produce a package name that differs from the repository import path, so the generated -coverpkg pattern selects no linked package.

Appendix

Table 8: Coverage instrumentation methods and cases beyond rule-based instrumentation.

Rule	Detection pattern	Level
Test structure
test_missing_run_sh	Missing eval/run.sh.	Error
test_no_discoverable_pytest	No test files, or a test file cannot be parsed.	Error
no_assertions	No assert statement in the test function.	Error
pass_body	The test body contains only pass statements.	Error
trivially_true	A recognized always-true assertion.	Error

Appendix

Table 9: Linter rules for rejecting weak oracles (1/2).

Rule	Detection pattern	Level
Black-box access
test_imports_source_module	The test file imports source or a source submodule.	Error
test_runtime_source_tree_dependency	The launcher or test references /workspace/source at runtime.	Error
test_executable_path_contract	The test uses a recognized fragile relative executable path.	Error
Output comparison
float_exact_match	A pattern suggests exact comparison of floating-point output.	Warning

Appendix

Table 10: Linter rules for rejecting weak oracles (2/2).

Language	Collection and export	Counting unit
C / C++	GNU gcov or llvm-cov gcov reads .gcno / .gcda ; gcovr exports coverage reports.	Executable source lines.
Rust	LLVM source-based coverage; llvm-profdata merges .profraw , and llvm-cov exports line counts in LCOV format.	Executable source lines.
Go	Runtime counters in GOCOVERDIR are converted by go tool covdata textfmt to a coverage profile.	Basic blocks weighted by statement count.

Appendix

Table 11: Language-specific backends for computing core coverage.

Figure 12: Program analysis guiding test writing on COLMAP. Top: coverage feedback (dynamic) steers the agent toward uncovered code in ply.cc . Bottom: the call graph (static analysis) reveals a lightweight entry to a target the agent had skipped.

Panel of Figure 13 — Figure 13: Iteration-wise coverage, test counts, and validation metrics for two representative source modules. (a) paullouisageneau/violet shows the agentic loop in Program-Env repairing weak assertions and dummy false passes. (b) Ramilito/kubesess shows it steadily raising coverage under feedback.

Parameter	Setting
Functionality coverage threshold	80%
Iteration limit	5–20 rounds
Earliest early-stopping round	Round 5
Early-stopping window	3 rounds

Appendix

Table 12: Termination settings for test generation.

Category	Description	Repos
Developer Tools	Build, test, lint and format, version control, documentation, web frameworks, and general-purpose libraries.	441
CLI & System Utilities	File and disk management, process and device control, container and cluster management, networking, text processing, compression, and terminal tools.	372
Data & Scientific Computing	Databases, structured data processing, numerical computation, simulation, bioinformatics, machine learning, and NLP.	243
Security & Forensics	Vulnerability scanning, authentication, cryptography and key management, auditing, reverse engineering, and digital forensics.	227
Media & Graphics	Image, audio, and video processing, 3D rendering, plotting, and GUI or TUI components.	115
Games & Education	Games and game engines, programming tutorials, exercises, terminal animations, and demos.	110

Appendix

Table 13: Functional categories of the 1,648 Program-Env repositories.

Figure 14: Distributions of community activity, development history, and codebase size across the 1,648 collected repositories.

Figure 15: Kernel density estimates of stars, commits, code files, and code lines across the 1,648 collected repositories, on a log scale. Means are annotated on the right.

Tool	Repositories	Total time	Avg. per repo
Joern	1,345	7.00 h	18.73 s
CodeQL	303	4.74 h	56.29 s
Total	1,648	11.74 h	25.64 s

Appendix

Table 14: Cost of static program analysis.

Figure 16: Tests harvested from existing repository test suites, for the 272 repositories that ship native tests. Each point is one repository, plotted against its total test count.

Figure 17: Token cost and generation quality of Program-Env. Dispatcher stage covers the LLM calls for coverage instrumentation, task decomposition and docs extraction.

Figure 18: Coverage, total tokens across the three stages, and test cases as a function of reference code lines. Each point is one repository, colored by language. Token and test counts grow with repository size.

Resource	Analysis	Implement	Review
Executable access
Reference executable	Execute only	Unavailable	Execute only
Candidate executable	Unavailable	Working candidate	Immutable copy
Source workspace
Product source	Disposable copy	Shared Workspace; read/write	Disposable copy
Handoff artifacts

Appendix

Table 15: Resource access across Program-Agent stages.

Stage	Completion conditions
Analysis	analysis.md exists and is non-empty.
Implement	delivery.md exists and is non-empty; tracked files contain no staged or unstaged changes; compile.sh is committed; and a clean source snapshot independently produces an executable.
Review	review.md exists and is non-empty.

Appendix

Table 16: Machine-checked stage completion conditions.

Setting	Value
Base model	Qwen3.8-27B
Training framework	ms-swift
Training epochs	2
Precision	BF16
Maximum sequence length	262,144 tokens
Sequence packing	Enabled (bin packing)

Appendix

Table 17: Main training settings for Qwen3.8-27B . The epoch count reflects the actual training duration. The warmup fraction follows the configured training schedule.

Figure 19: Performance distribution of large language models on ProgramBench with respect to output token length and test pass rate.

Figure 20: Test pass rate versus output tokens across models and agents.

Figure 21: Cumulative distribution of test pass rates across models and agent scaffolds on ProgramBench.

Figure 22: Reference vs. model code lines on a log–log scale. The

y=x

diagonal indicates parity. Only solutions passing

\geq

75% of tests are shown.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 329 papers

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
Sep 18, 2026Bowen Ye, Lei Li, Shicheng Li +16LLM Agent Training RL for Code Generation
LLM Core, Xiaomi · Peking University · University of Hong Kong +1
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
Feb 18, 2026Yiqing Xie, Emmy Liu, Gaokai Zhang +7AI Coding Agents Software Engineering Agents
Carnegie Mellon University · All Hands AI
ProgramBench: Can Language Models Rebuild Programs From Scratch?
May 5, 2026John Yang, Kilian Lieret, Jeffrey Ma +9Software Engineering Agents Software Engineering Benchmarks
1Meta FAIR · 4Harvard University · 2Meta TBD +1

Sep 18, 2026J/K move · Enter open · S save

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

Bowen Ye, Lei Li, Shicheng Li, Zihao Yue+15

LLM Core, Xiaomi · Peking University · University of Hong Kong+1

LLM Agent Training RL for Code Generation

Open paper AskPDF arXiv

Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: agents explore implemented functionality to formulate behavioral specifications, construct tests grounded in execution of the original code, and validate and filter candidate tasks through execution checks and repeated solution rollouts. The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains. Training MiMo-V2.5 on these tasks with GRPO improves performance on all five diverse benchmarks, covering issue repair (DeepSWE + 11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing the number of high-quality training tasks improves performance. Trajectory analysis shows the RL-trained agent demonstrates better behaviors like increasing codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.

Abstract

AI Coding Agents

Explore similar work

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

Hybrid-Gym: Training Coding Agents to Generalize Across Tasks

ProgramBench: Can Language Models Rebuild Programs From Scratch?