Coding agents are increasingly used to build programs from scratch as software engineers do rather than to resolve issues within a repository or to generate a repository from a given requirements document. However, scalable verifiable training environments that force agents to discover what is required and to decide how the system should be designed remain scarce. Moreover, existing agent workflows also cannot sustain long-horizon development in which the agent explores the reference program before implementation. To address these challenges, we introduce ProgramGym, a framework consisting of Program-Env and Program-Agent for scalable training environments of long-horizon program reproduction agents. Specifically, Program-Env constructs verifiable training environments from repositories through a feedback-driven loop that repairs runtime sandbox builds and the quality of black-box tests. Using this novel pipeline, we obtain 1,648 training environments spanning C, C++, Go, and Rust. Program-Agent combines a role-isolated workflow of analysis, implementation, and review with revision management for compilation checks and failure recovery. Experiments on ProgramBench show that both Program-Env and Program-Agent improve performance on program reproduction. Fine-tuning on trajectories with these constructed environments raises the performance of Qwen3.8-27B from 34.87% to 42.84%. Program-Agent further raises the base and SFT models to 55.71% and 56.25%, both exceeding some flagship models. We will release the ProgramGym to support further research on coding agents that build programs from scratch.
Figures & tables
Figure 1: Overview of ProgramGym .
Figure 2: Overview of the Program-Env. Stage 1 collects repositories from GitHub by language and stars. Stage 2 builds the gold executable Bi⋆ and runtime sandbox Si . Bridging the two stages, the dispatcher extracts documentation Di and splits the source into modules. In Stage 3, each module is assigned to a subagent that iteratively writes the test suite Ti under feedback.
Figure 3: Overview of Program-Agent. The Analysis stage explores the usage documentation and reference program to identify behavioral requirements. The Implement and Review stages then alternate to refine the candidate implementation for a fixed loop budget. Handoff carries behavioral evidence and delivery reports across isolated sessions, Workspace maintains the evolving implementation, and Revisions preserves independently buildable source code for validation and recovery.
Figure 4
Models
Agent Scaffolds
Tests Passed
Almost Resolved
Resolved
(%, ↑ )
Δ
(%, ↑ )
Δ
(%, ↑ )
Δ
Flagship Models ( Yang et al., 2026b )
GPT 5 mini
mini-SWE-agent
16.00
–
0.00
–
0.00
–
GPT 5.4 mini
mini-SWE-agent
16.40
–
0.00
–
0.00
–
Claude Haiku 4.5
mini-SWE-agent
30.00
–
0.00
–
0.00
–
Gemini 3 Flash
mini-SWE-agent
31.80
–
0.00
–
0.00
–
Table 2: Results on ProgramBench across models and agent scaffolds. Program-Agent uses K=5 implementation rounds. Δ is relative to the base model with Claude Code .
Figure 5: Distributions of code lines, coverage, and test cases in Program-Env.
Figure 7
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Repositories
Excluded
Cumulative Retention
C
C++
Go
Rust
Comprehensive Search
41,833
–
100.00%
9,736
13,147
11,730
7,220
Metadata Validation
35,710
6,123
85.36%
7,604
10,720
10,777
6,609
Commit Locking
35,710
0
85.36%
7,604
10,720
10,777
6,609
Build Config Detection
30,459
5,251
72.81%
5,695
8,229
9,944
6,591
Appendix
Table 3: Repository statistics across the four-stage collection pipeline.
Language
Source of incompatibility
Representative failure
[.25em]C++
Build tools remove support for older policy versions.
CMake 4.0 rejects cmake_minimum_required with VERSION 3.4 by default, because policy compatibility below 3.5 is no longer supported.
[.25em]C++
Standard-library availability depends on the selected C++ language mode and library implementation.
In C++17 mode, libc++ omits std::auto_ptr and std::random_shuffle by default, causing compilation failure.
[.25em]Rust
Unstable features can change or be removed, making some projects dependent on a dated nightly toolchain.
Code enabling box_syntax fails with error E0557 on toolchains where that feature has been removed.
Appendix
Table 4: Selected examples of toolchain-version incompatibility.
Check
Scope
Pattern flagged
Level
Failure masking
Both
Expressions that override a failed exit status, such as || true .
Error
Error suppression
Both
Redirections that discard standard error, such as 2>/dev/null , except in simple tool-availability or version checks.
Error
Stream separation
Both
Redirections that combine stdout and stderr, such as 2>&1 or &>log , except in simple tool-availability or version checks.
Error
Placeholder content
Both
Standalone placeholder lines such as ... , <script> , or [truncated] .
Error
Package installation
Compilation
Package-manager installation, such as apt , pip , npm , or cargo install <crate> . Project install targets such as make install are allowed.
Error
Target artifact reference
Compilation
The script omits a reference to the designated output artifact, named executable .
Remove || true . Handle commands that may legitimately fail with an explicit if or case branch.
Error suppression
Remove 2>/dev/null . Handle acceptable failures explicitly so that other errors remain visible.
Stream separation
Keep standard output and standard error separate: leave them unredirected, or write them to separate logs (e.g., >build.log 2>build.err ).
Placeholder content
Replace the placeholder with the actual commands.
Package installation
Move package-manager installation to download.sh . Keep only project build and install targets in compile.sh .
Appendix
Table 6: Suggested fixes attached to each finding of the linter check and the execution check.
Figure 10: A case of executable and sandbox generation guided by feedback. A missing package breaks offline compilation; the agent reads the feedback, adds libopenexr-dev to download.sh , and passes all checks on the next attempt.
Figure 11: Overview of the dispatcher bridging Stage 2 and Stage 3. It decomposes the repository into modules, instruments the executable for coverage, extends Si into Sitest , and extracts documentation Di . Each module is then handed to a subagent in its own workspace, with k workspaces running in parallel in Stage 3.
Type
Behavior to explore
Args
Flag syntax, required and optional arguments, positional values, and argument type checks.
Config
Settings read from environment variables and configuration files, including precedence when values conflict.
Help
Output of --help and -h , usage summaries, and help for individual subcommands.
I/O
Reading from standard input and files, separating standard output from standard error, exit status, and output formatting.
Subcommand
Routing to the requested subcommand, resolving aliases, and applying global and subcommand-specific flags.
TUI
Keyboard navigation, key bindings, screen updates, and transitions between interaction modes.
Appendix
Table 7: Interaction categories used in test generation. Each category directs a subagent to a specific behavioral aspect of the black-box executable.
Language
Instrumentation method
Cases beyond rule-based instrumentation
[.25em] C
Add --coverage to compilation and linking through build variables, compiler wrappers, and recognized Makefiles or .mak files. Read .gcno/.gcda with GNU gcov or llvm-cov gcov , and summarize coverage with gcovr .
Build scripts may overwrite injected CFLAGS / LDFLAGS or use custom .mak variables that the language-specific rule does not reliably update. Applying the same flags to every step can also break the build. In kafel, for example, the injected --coverage option is passed directly to ld , which does not accept it.
[.25em] C++
Apply the same approach to C++ compilation and linking, including CXXFLAGS and compiler wrappers. Support both GNU gcov and llvm-cov gcov , with gcovr for reporting.
Project build settings can overwrite injected flags, while direct compiler calls can bypass the wrappers. Global injection can also interfere with build-system checks. In fast-wfc, for example, a CMake compiler test is instrumented but linked without the coverage runtime, causing configuration to fail before the project builds.
[.25em] Rust
Set RUSTFLAGS to -C instrument-coverage , then merge .profraw with llvm-profdata and export coverage with llvm-cov .
Replacing RUSTFLAGS can remove conditional-compilation or CPU-feature options required by the project, breaking its build, so coverage flags must be merged with the existing options. Targets without profiler_builtins also need a toolchain that provides the coverage runtime.
[.25em] Go
Build with -cover and -coverpkg . Collect profiles under GOCOVERDIR and convert them with go tool covdata textfmt .
Matching a fixed set of command forms can miss calls such as $GOROOT/bin/go , so the build receives no coverage flags. Building individual .go files can also produce a package name that differs from the repository import path, so the generated -coverpkg pattern selects no linked package.
Appendix
Table 8: Coverage instrumentation methods and cases beyond rule-based instrumentation.
Rule
Detection pattern
Level
Test structure
test_missing_run_sh
Missing eval/run.sh.
Error
test_no_discoverable_pytest
No test files, or a test file cannot be parsed.
Error
no_assertions
No assert statement in the test function.
Error
pass_body
The test body contains only pass statements.
Error
trivially_true
A recognized always-true assertion.
Error
Appendix
Table 9: Linter rules for rejecting weak oracles (1/2).
Rule
Detection pattern
Level
Black-box access
test_imports_source_module
The test file imports source or a source submodule.
Error
test_runtime_source_tree_dependency
The launcher or test references /workspace/source at runtime.
Error
test_executable_path_contract
The test uses a recognized fragile relative executable path.
Error
Output comparison
float_exact_match
A pattern suggests exact comparison of floating-point output.
Warning
Appendix
Table 10: Linter rules for rejecting weak oracles (2/2).
Language
Collection and export
Counting unit
C / C++
GNU gcov or llvm-cov gcov reads .gcno / .gcda ; gcovr exports coverage reports.
Executable source lines.
Rust
LLVM source-based coverage; llvm-profdata merges .profraw , and llvm-cov exports line counts in LCOV format.
Executable source lines.
Go
Runtime counters in GOCOVERDIR are converted by go tool covdata textfmt to a coverage profile.
Basic blocks weighted by statement count.
Appendix
Table 11: Language-specific backends for computing core coverage.
Figure 12: Program analysis guiding test writing on COLMAP. Top: coverage feedback (dynamic) steers the agent toward uncovered code in ply.cc . Bottom: the call graph (static analysis) reveals a lightweight entry to a target the agent had skipped.
Figure 13: Iteration-wise coverage, test counts, and validation metrics for two representative source modules. (a) paullouisageneau/violet shows the agentic loop in Program-Env repairing weak assertions and dummy false passes. (b) Ramilito/kubesess shows it steadily raising coverage under feedback.
Parameter
Setting
Functionality coverage threshold
80%
Iteration limit
5–20 rounds
Earliest early-stopping round
Round 5
Early-stopping window
3 rounds
Appendix
Table 12: Termination settings for test generation.
Category
Description
Repos
Developer Tools
Build, test, lint and format, version control, documentation, web frameworks, and general-purpose libraries.
441
CLI & System Utilities
File and disk management, process and device control, container and cluster management, networking, text processing, compression, and terminal tools.
372
Data & Scientific Computing
Databases, structured data processing, numerical computation, simulation, bioinformatics, machine learning, and NLP.
243
Security & Forensics
Vulnerability scanning, authentication, cryptography and key management, auditing, reverse engineering, and digital forensics.
227
Media & Graphics
Image, audio, and video processing, 3D rendering, plotting, and GUI or TUI components.
115
Games & Education
Games and game engines, programming tutorials, exercises, terminal animations, and demos.
110
Appendix
Table 13: Functional categories of the 1,648 Program-Env repositories.
Figure 14: Distributions of community activity, development history, and codebase size across the 1,648 collected repositories.
Figure 15: Kernel density estimates of stars, commits, code files, and code lines across the 1,648 collected repositories, on a log scale. Means are annotated on the right.
Tool
Repositories
Total time
Avg. per repo
Joern
1,345
7.00 h
18.73 s
CodeQL
303
4.74 h
56.29 s
Total
1,648
11.74 h
25.64 s
Appendix
Table 14: Cost of static program analysis.
Figure 16: Tests harvested from existing repository test suites, for the 272 repositories that ship native tests. Each point is one repository, plotted against its total test count.
Figure 17: Token cost and generation quality of Program-Env. Dispatcher stage covers the LLM calls for coverage instrumentation, task decomposition and docs extraction.
Figure 18: Coverage, total tokens across the three stages, and test cases as a function of reference code lines. Each point is one repository, colored by language. Token and test counts grow with repository size.
Resource
Analysis
Implement
Review
Executable access
Reference executable
Execute only
Unavailable
Execute only
Candidate executable
Unavailable
Working candidate
Immutable copy
Source workspace
Product source
Disposable copy
Shared Workspace; read/write
Disposable copy
Handoff artifacts
Appendix
Table 15: Resource access across Program-Agent stages.
Stage
Completion conditions
Analysis
analysis.md exists and is non-empty.
Implement
delivery.md exists and is non-empty; tracked files contain no staged or unstaged changes; compile.sh is committed; and a clean source snapshot independently produces an executable.
Table 17: Main training settings for Qwen3.8-27B . The epoch count reflects the actual training duration. The warmup fraction follows the configured training schedule.
Figure 19: Performance distribution of large language models on ProgramBench with respect to output token length and test pass rate.
Figure 20: Test pass rate versus output tokens across models and agents.
Figure 21: Cumulative distribution of test pass rates across models and agent scaffolds on ProgramBench.
Figure 22: Reference vs. model code lines on a log–log scale. The y=x diagonal indicates parity. Only solutions passing ≥ 75% of tests are shown.
Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: agents explore implemented functionality to formulate behavioral specifications, construct tests grounded in execution of the original code, and validate and filter candidate tasks through execution checks and repeated solution rollouts. The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains. Training MiMo-V2.5 on these tasks with GRPO improves performance on all five diverse benchmarks, covering issue repair (DeepSWE + 11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing the number of high-quality training tasks improves performance. Trajectory analysis shows the RL-trained agent demonstrates better behaviors like increasing codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.
Bowen Ye, Lei Li, Shicheng Li +16
LLM Core, Xiaomi · Peking University · University of Hong Kong +1
When assessing the quality of coding agents, predominant benchmarks focus on solving single issues on GitHub, such as SWE-Bench. In contrast, in real use, these agents solve more various and complex tasks that involve other skills such as exploring codebases, testing software, and designing architecture. In this paper, we first characterize some transferable skills that are shared across diverse tasks by decomposing trajectories into fine-grained components, and derive a set of principles for designing auxiliary training tasks to teach language models these skills. Guided by these principles, we propose a training environment, Hybrid-Gym, consisting of a set of scalable synthetic tasks, such as function localization and dependency search. Experiments show that agents trained on our synthetic tasks effectively generalize to diverse real-world tasks that are not present in training, improving a base model by 25.4% absolute gain on SWE-Bench Verified, 7.9% on SWT-Bench Verified, and 5.1% on Commit-0 Lite. Hybrid-Gym also complements datasets built for the downstream tasks (e.g., improving SWE-Play by 4.9% on SWT-Bench Verified). Code available at: https://github.com/yiqingxyq/Hybrid-Gym.
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.