EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning

Sihan Yang; Weichen Bai; Ruichuan An; Yiming Cheng; Zhiyuan Feng; Tianyi Jiang; Renye Yan; Ning Ding

cs.LGOct 27, 2026

EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning

Authors: Sihan Yang, Weichen Bai, Ruichuan An, Yiming Cheng, Zhiyuan Feng, Tianyi Jiang, Renye Yan, Ning Ding,

Organizations: PKU · XJTU · THU · Samsung Research ·

Abstract

As executable environments for language agents scale, generating and verifying their rollouts becomes increasingly expensive. Under finite rollout budgets, this raises an important question: can data management at the environment level make agentic reinforcement learning (RL) more efficient by organizing, evaluating, and scheduling task instances together with the environments that determine their tools, states, transitions, and verification? To address this question, we introduce EnvSched (Environment-aware Scheduling), an online scheduler that learns from rollout history. EnvSched assigns scheduling value through two complementary modeling choices: exploitation prioritizes tasks likely to provide informative reward differences, while coverage spreads each batch across underobserved environments and interaction patterns. EnvSched organizes environment--task pairs as nodes in a dual-relation graph, with within-environment edges capturing task semantics and shared tool, state, and verifier structure, and cross-environment edges capturing abstract execution and reward topology. By propagating historical rollout statistics along both edge types, it predicts within-task reward variance. These predictions guide a scheduler that allocates each batch to tasks with high predicted reward variance and underobserved environments and tasks. Environment quotas and a similarity penalty further limit concentration and redundancy, yielding informative and diverse batches without additional pilot rollouts. In RL on Qwen3-8B, EnvSched achieves an average score of 48.81 across BFCL-v3, tau-Bench, and ACEBench-Agent after 40 training steps, exceeding the highest score attained by random task selection after 120 steps. It also improves downstream performance across model sizes at the same training budget.

Figures & tables

Figure 1: The left panel summarizes the growth of environments and RL resources in recent agent systems. The right panel shows EnvSched ’s training-efficiency gain on Qwen3-8B under matched rollout budgets. Scores are averaged across BFCL-v3, $\tau$ -Bench, and ACEBench-Agent.

Figure 2: EnvSched training loop. Circles with the same color denote task–environment pairs from the same environment.

Figure 3: Dual-relation graph and informativeness prediction. (a) Within-environment and cross-environment edges propagate historical statistics from observed task–environment pairs to the target. (b) Relation-specific estimates are fused with the target’s history to produce the final score.

Model	Steps	BFCL-v3 Multi-Turn					$\tau$ -Bench			ACEBench-Agent
Model	Steps	Base	Miss- Func	Miss- Param	Long- Context	Overall	Retail	Airline	Overall	Multi- Step	Multi- Turn	Overall
Qwen3-1.7B	-	13.50	6.00	12.50	7.00	9.75	8.99	16.00	11.11	35.00	27.78	30.67
+ Random	200	13.17	10.50	10.50	7.67	10.46	15.36	16.00	15.56	35.00	24.44	28.67
+ DPS	200	12.50	10.83	9.33	7.00	$\text{9.92}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-0.54}}}$	16.81	17.33	$\text{16.97}_{{\color[rgb]{0,0.2734,0}\text{+1.41}}}$	33.33	26.67	$\text{29.33}_{{\color[rgb]{0,0.2734,0}\text{+0.66}}}$
+ LEEPS	200	13.83	11.67	11.00	8.50	$\text{11.25}_{{\color[rgb]{0,0.2734,0}\text{+0.79}}}$	16.52	17.33	$\text{16.77}_{{\color[rgb]{0,0.2734,0}\text{+1.21}}}$	35.00	27.78	$\text{30.67}_{{\color[rgb]{0,0.2734,0}\text{+2.00}}}$
+ EnvSched	40	13.50	13.00	11.50	9.00	$\text{11.75}_{{\color[rgb]{0,0.2734,0}\text{+1.29}}}$	17.10	16.67	$\text{16.97}_{{\color[rgb]{0,0.2734,0}\text{+1.41}}}$	40.00	24.44	$\text{30.67}_{{\color[rgb]{0,0.2734,0}\text{+2.00}}}$

Table 1: Main results. “Overall” refers to the weighted score. Green and red subscripts denote variations relative to random scheduling.

Figure 4: Selected-task reward variance and environment coverage over training steps, with log-transformed vertical-axis values.

	Mean reward variance	Median reward variance	Median reward	Environments covered per batch	Tools used per batch	BFCL-v3 Multi-Turn
Random	0.03208	0.01468	0.595	36.53	413.41	28.83
EnvSched	0.04290	0.02108	0.592	45.96	478.39	31.88
Difference	+34%	+44%	-0.5%	+26%	+16%	+3.05

Table 2: Training statistics for random scheduling and EnvSched over 200 GRPO training steps. BFCL-v3 Multi-Turn reports the overall score.

Configuration	BFCL-v3 Multi-Turn	$\tau$ -Bench	ACEBench Agent	Mean reward variance	Environments per batch	Tools used per batch
Random	28.83	37.78	56.00	0.0321	36.53	413.41
(a) Component contributions
Full EnvSched	31.88	41.01	58.00	0.0429	45.96	478.39
Within-environment edges only	$\text{30.42}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-1.46}}}$	$\text{40.40}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-0.61}}}$	58.00	$0.0391_{{\color[rgb]{0.7148,0.125,0.0859}\text{-0.0038}}}$	$46.70_{{\color[rgb]{0,0.2734,0}\text{+0.74}}}$	$498.55_{{\color[rgb]{0,0.2734,0}\text{+20.16}}}$
Cross-environment edges only	$\text{30.63}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-1.25}}}$	41.01	$\text{57.33}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-0.67}}}$	$0.0383_{{\color[rgb]{0.7148,0.125,0.0859}\text{-0.0046}}}$	$46.18_{{\color[rgb]{0,0.2734,0}\text{+0.22}}}$	$466.50_{{\color[rgb]{0.7148,0.125,0.0859}\text{-11.89}}}$
Coverage only ( $\rho=1$ )	$\text{29.71}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-2.17}}}$	$\text{39.39}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-1.62}}}$	$\text{56.67}_{{\color[rgb]{0.7148,0.125,0.0859}\text{-1.33}}}$	$0.0357_{{\color[rgb]{0.7148,0.125,0.0859}\text{-0.0072}}}$	$51.00_{{\color[rgb]{0,0.2734,0}\text{+5.04}}}$	$502.10_{{\color[rgb]{0,0.2734,0}\text{+23.71}}}$

Table 3: Component ablations and neighborhood-size sensitivity. Panel (a) varies one component of the full scheduler; panel (b) varies one neighbor count at a time in complete 200-step training runs. Benchmark columns report weighted scores. Green and red subscripts denote changes relative to the full EnvSched or the default neighborhood size.

(a) Selection quality across random initializations
Metric	Step 1	Step 7	Step 7 range
Mean batch reward variance	$0.0596\;[0.0584,0.0609]$	$0.0813\;[0.0803,0.0823]$	$0.0726$ – $0.0894$
Cumulative mean reward variance	$0.0534\;[0.0524,0.0543]$	$0.0679\;[0.0673,0.0684]$	$0.0639$ – $0.0723$
Exploitation hit rate (%)	$57.2\;[55.5,58.9]$	$92.1\;[91.1,93.0]$	$84.4$ – $100.0$

Table 4: Initialization sensitivity over 100 seven-step runs. Panel (a) reports means with 95% confidence intervals and 5th–95th percentile ranges; panel (b) compares the lowest and highest initial-variance quartiles. Hit rate measures exploitation tasks in the highest-variance quartile.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 232 papers

Jul 19, 2026

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

Long-horizon rollout generation has become the dominant systems bottleneck in agentic reinforcement learning (RL). As agents interact with environments over many turns, trajectories rapidly grow to tens of thousands of tokens, making synchronous RL training increasingly constrained by rollout. We propose WAR, a workload-aware rollout system that substantially accelerates synchronous agentic RL by jointly optimizing decoding and scheduling. WAR is built on a key observation: the optimal rollout optimization strategy depends on runtime load: (1) Under low load, WAR enables model-free speculative decoding with SuffixDecoding, which reuses suffix patterns from previously completed trajectories as speculative drafts for future rollouts. Unlike model-based drafters, SuffixDecoding introduces no additional draft model and avoids GPU contention with rollout generation. (2) Under high load, where saturated batched decoding leaves limited room for speculative speedup, WAR shifts the optimization focus to cache-aware scheduling. A global scheduler places requests across rollout replicas based on cache locality, trajectory progress and server load, reducing redundant KV-cache recomputation and mitigating load imbalance. By combining decoding-level suffix reuse with system-level rollout scheduling, WAR delivers robust throughput improvements across workload regimes without changing the underlying RL algorithm. WAR improves long-context agentic rollout throughput by 1.4x under low load and up to 1.6x under high load. These results show that WAR removes a major rollout bottleneck in synchronous agentic RL and provides a practical path toward scalable long-context agent training.

Ryan Xu, Atlas Zhao, David Bao +1

Agentic RL Reinforcement Learning LLM Inference Scheduling LLM Inference Acceleration RL for LLM Agents

Ask PDF

May 18, 2026

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

Equipping LLMs with tool-use capabilities via Agentic Reinforcement Learning (Agentic RL) is bottlenecked by two challenges: the lack of scalable, robust execution environments and the scarcity of realistic training data that captures implicit human reasoning. Existing approaches depend on costly real-world APIs, hallucination-prone LLM simulators, or synthetic environments that are often single-turn or depend on pre-collected documents. Moreover, synthetic trajectories are frequently over-specified, resembling instruction sequences rather than natural human intents, reducing their effectiveness for RL training. We introduce EnvFactory, a fully automated framework that addresses both challenges. EnvFactory autonomously explores and verifies stateful, executable tool environments from authentic resources, and synthesizes natural multi-turn trajectories through topology-aware sampling and calibrated refinement, producing grounded queries with implicit intents. Using only 85 verified environments across 7 domains, EnvFactory generates 2,575 SFT and RL trajectories. Despite using significantly fewer environments than prior work, which are often 5 times more, EnvFactory achieves superior training efficiency and downstream performance, improving Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks including

τ^2

-Bench and VitaBench. By fully automating both environment construction and trajectory synthesis, EnvFactory provides a scalable, extensible, and robust foundation for Agentic RL.

Minrui Xu, Zilin Wang, Mengyi DENG +12

LARK, HKUST (GZ) · University of Cambridge · UCL +2

Tool-Augmented Language Model Agents Agentic RL LLM Agent Trajectory Synthesis LLM Agents LLM Agent Training RL for Tool Use

Ask PDF

Oct 9, 2026

AutoEnvScaling: Automating the Data Flywheel with Terminal Agents

Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.

Simon Yu, Baolin Peng, Bo Liu +8

Northeastern University · Microsoft · Stanford University +2

LLM Agent Training Self-Evolving Agents Terminal Agents Reinforcement Learning

Ask PDF

Abstract

Agentic RL

Explore similar work

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

AutoEnvScaling: Automating the Data Flywheel with Terminal Agents