Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

Xiaojun Wu; Cehao Yang; Honghao Liu; Xueyuan Lin; Zhichao Shi; Hao Zhou; Xuhui Jiang; Chengjin Xu

cs.CLAug 14, 2026

Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

Authors: Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Zhichao Shi, Hao Zhou, Xuhui Jiang, Chengjin Xu,

Organizations: IDEA Research · The Hong Kong University of Science and Technology (Guangzhou) · DataArcTech Ltd.

Abstract

Reinforcement learning (RL) for terminal agents needs executable training environments with reliable rewards and useful difficulty. Fixed recipes such as few-shot, Self-Instruct, and Evol-Instruct apply the same prompting policy to every seed, even when the current policy would benefit from a harder, easier, or simply different task. We present Envs-FORGE, a prompting policy that converts verifier rewards into per-seed environment-synthesis actions. Envs-FORGE estimates seed pass rates, scores six projection--direction actions around a target learning frontier, and solves a per-seed mixed-integer linear program (MILP) to choose the action that conditions generation. The selected action drives synchronized rewriting of the instruction, fixtures, oracle solution, tests, and Docker environment; only gold-verified bundles enter RL training. The indexed MILP form also supports optional soft skill coverage for portfolio planning. On Qwen 3.5 35B, Envs-FORGE improves Pass@1 over Base by 9.2 percentage points on tb-core (40.0% to 49.2%) and 6.4 points on tb-2.0 (23.0% to 29.4%), exceeding the strongest fixed-recipe baseline by 2.4 and 2.1 points. It reaches 77.1% on SWE-bench Verified versus 73.4% for Base, and improves tb-core by 6.8--9.2 points across the evaluated 4B--35B models. All synthesis methods export 100 verified environments and use 2.27M--2.88M synthesis tokens, placing the comparison at the same downstream training-set size and the same operational scale. The source code is available at https://github.com/DataArcTech/DataArc-SynData-Toolkit/.

Figures & tables

Figure 1: Motivation contrast. Fixed prompt recipes apply the same rewrite strategy to every seed, while Envs-FORGE estimates the seed’s pass rate first and then chooses among increase, reduce, and diversify projections to land near the learning frontier.

Figure 2: Envs-FORGE pipeline. Seed tasks are scored by a frontier-aware pass-rate estimator, and the MILP jointly chooses a projection and evolution direction before the synthesis model rewrites the full environment bundle. Only tasks that pass static checks and gold verification enter GRPO training.

Training data / model	Method	Records	Accepted	Task dirs	Attempt sum	Prompt tokens	Final completion tokens	Total synthesis tokens	tb-core Pass@1	tb-2.0 Pass@1
Base	–	–	–	–	–	–	–	–	40.0%	23.0%
few-shot	few_shot	121	100	210	226	1,835,473	607,592	2,443,065	43.2%	24.1%
Self-Instruct	self_instruct	113	100	194	190	1,745,028	527,247	2,272,275	45.6%	27.3%
Evol-Instruct	evol_instruct_in_depth + evol_instruct_in_breadth	112	100	195	207	1,974,723	543,023	2,517,746	46.8%	25.6%
Envs-FORGE	MILP	120	100	203	291	1,862,146	618,910	2,881,056	49.2%	29.4%

Table 1: Training-data synthesis statistics and downstream Pass@1. Each synthesis run exports exactly 100 gold-verified environment bundles (the Accepted column) to downstream RL, with task directories and token use remaining in the same broad range across methods. Records, task directories, attempts, and token counts are synthesis-stage totals used to reach that fixed endpoint; they do not increase the downstream training-set cardinality. The Base row uses no synthesized training data.

Figure 3: Downstream performance and synthesis cost. (a) Pass@1 on tb-core and tb-2.0; the Envs-FORGE run is highest on both benchmarks. (b) Synthesis-stage prompt and final-completion token totals for the four methods; labels above bars show total tokens and attempt counts. Totals occupy a comparable 2.27M–2.88M range, and each method exports exactly 100 gold-verified environments to RL. The bars measure the cost of reaching a fixed training-set size, not extra downstream training examples.

Figure 4: Benchmark and model ablations. (a) Pass@1 for the five training conditions across tb-core, tb-2.0, and SWE-bench Verified; labels show FORGE values and gains over Base. Marker shape redundantly encodes the training condition. (b) Base and FORGE tb-core Pass@1 across Qwen 3.5 model sizes; labels show the paired gain at each size.

Benchmark	Base	few-shot	Self-Inst.	Evol-Inst.	FORGE
tb-core	40.0%	43.2%	45.6%	46.8%	49.2%
tb-2.0	23.0%	24.1%	27.3%	25.6%	29.4%
SWE-bench Verified	73.4%	74.6%	75.2%	75.8%	77.1%

Table 2: Benchmark and model ablations. In (a), bold marks the best training condition for each benchmark. Panel (b) reports tb-core, where

\Delta

is FORGE minus Base in percentage points.

Training data	Attempts / accepted	Tokens / accepted	Tokens / attempt
few-shot	2.26	24,431	10,810
Self-Instruct	1.90	22,723	11,959
Evol-Instruct	2.07	25,177	12,163
Envs-FORGE	2.91	28,811	9,901

Table 3: Derived synthesis-cost statistics. Each synthesis run accepts 100 outputs, and all normalized costs remain within the same overall scale. Values are computed from the aggregate counts in Table 1 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Projection	Agent-relative regime	Permitted movement	Invariants
increase	Seed is too easy	Add one or two constraints, edge cases, larger fixtures, or stricter deterministic outputs.	Preserve the core skill subgraph; update the solution and tests to cover every added requirement.
reduce	Seed is too hard	Remove secondary systems or replace brittle infrastructure with a smaller bridge task.	Preserve the target skill path and its essential artifacts; simplification must not collapse the task into an unrelated problem.
diversify	Seed is near the frontier	Change fixtures, neighboring requirements, or scenario while keeping similar difficulty.	Avoid unrelated breadth jumps; synchronize expected outputs and verifier logic with the new fixture.

Appendix

Table 4: Operational semantics of the Envs-FORGE action space and its restricted baseline views. The baseline correspondences describe approximate prompt semantics, not exact equivalence between generation procedures.

Category	Skill nodes
Domain	system_admin , data_processing , testing , security , ml_ops , game_reasoning
Tool	bash , python , systemd , docker , sql , node
Artifact	logs , csv , json , jsonl , source_code , config
Operation	merge , parse , repair , configure , generate_tests , rank , validate
Constraint	concurrency , scheduling , atomicity
FORGE-added nodes	edge_cases , structured_output , deterministic_sorting , alternate_fixture , neighboring_constraint , file_level_simulation , deterministic_fixture

Appendix

Table 5: Skill-node taxonomy used for candidate metadata. These nodes define the required set

R_{i,a,d}

and optional addable or removable set

O_{i,a,d}

attached to each action candidate; they are metadata tags for the action contract rather than separate generation artifacts.

Symbol	Meaning	Symbol	Meaning
$s_{i}$	Seed environment bundle	$\hat{p}_{i}$	Current policy pass-rate estimate on seed $i$
$a,d$	Projection and evolution direction	$\tilde{p}_{i,a,d}$	Predicted pass rate after applying $(a,d)$
$F_{i,a,d}$	Frontier utility of a candidate	$x_{i,a,d}$	Binary candidate-selection variable
$R_{i,a,d}$	Required skill nodes	$O_{i,a,d}$	Optional addable or removable skill nodes
$u_{i,a,d,v}$	Binary activation of skill node $v$	$\xi_{v}$	Continuous shortfall for target coverage $m_{v}$ ; capped by $\bar{\xi}=0.2$ in reported experiments
$N$	Selected seed–action count in indexed portfolio form; one per local task	$\rho_{b,a,d}$	Constant action mask for restricted baseline $b$

Appendix

Table 6: MILP notation and implementation boundary. Candidate scores and metadata are constants when the solver is invoked.

Panel of Figure 5 — Figure 5: Detailed agentic case study of policy-relative environment synthesis. Each row follows a seed contract through the MILP decision and synchronized materialization to its preserved learning signal and verification result. The capability estimate $\hat{p}$ , projected rate $\hat{p}^{\prime}$ , and frontier score motivate each action: red panels denote complexity increases for easy seeds, blue panels denote deterministic reductions for seeds unsolved in the recorded rollouts, and the green panel denotes frontier-preserving diversification. All five materializations co-edit the required task artifacts, pass static validation, and obtain oracle reward $1.0$ . These traces illustrate selection and consistency semantics; downstream effects are evaluated by the benchmark results.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 608 papers

Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis
May 14, 2026Yucheng Shi, Zhenwen Liang, Kishan Panaganti +3Reinforcement Learning RL for Language Model Reasoning
Tencent HY LLM
EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
May 18, 2026Minrui Xu, Zilin Wang, Mengyi DENG +12Tool-Augmented Language Model Agents Agentic RL
LARK, HKUST (GZ) · University of Cambridge · UCL +2
Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning
Apr 10, 2026Siyuan Xu, Shiyang Li, Xin Liu +9Tool Use RL for Tool Use
The Pennsylvania State University · Amazon · Georgia Institute of Technology

May 14, 2026J/K move · Enter open · S save

Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis

Yucheng Shi, Zhenwen Liang, Kishan Panaganti, Dian Yu+2

Tencent HY LLM

Reinforcement Learning RL for Language Model Reasoning LLM-Based Program Synthesis Reinforcement Learning with Verifiable Rewards Language Model Self-Improvement

Open paper AskPDF arXiv

We pursue a vision for self-improving language models in which the model does not merely generate problems or traces to imitate, but constructs the environments that train it. In zero-data reasoning RL, this reframes self-improvement from a data-generation loop into an environment-construction loop, where each artifact is a reusable executable object that samples instances, computes references, and scores responses. Whether this vision sustains improvement hinges on a single property: the environments must exhibit stable solve--verify asymmetry, the model must be able to write an oracle once that it cannot reliably execute in natural language on fresh instances. This asymmetry takes two complementary forms. Some tasks are algorithmically hard to reason through but trivial as code: a dynamic program or graph traversal, compiled once, yields unboundedly many calibrated instances. Others are intrinsically hard to solve but easy to verify, like planted subset-sum or constraint satisfaction. Both create a durable gap between proposing and solving that the policy cannot close by gaming the verifier, and it is this gap that keeps reward informative as the learner improves. We instantiate this view in EvoEnv, a single-policy generator, solver method that synthesizes Python environments from ten seeds and admits them only after staged validation, semantic self-review, solver-relative difficulty calibration, and novelty checks. The strongest evidence comes from the already-strong regime: on Qwen3-4B-Thinking, fixed public-data RLVR and fixed hand-crafted environment RLVR reduce the average, while EvoEnv improves it from 72.4 to 74.8, a relative gain of 3.3%. Stable self-improvement, we suggest, depends not on producing more synthetic data, but on models learning to construct worlds whose difficulty stays structurally beyond their own reach.

Abstract

RL for LLM Agents

Explore similar work

Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning