Co-Evolving Acting and Forecasting for Language Agents

Haofei Yu; Yixing Chen; Yining Zhao; Xuyan Ye; Yuxiang Liu; Zhigang Hua; Yan Xie; Shuang Yang

cs.AIOct 14, 2026

Co-Evolving Acting and Forecasting for Language Agents

Authors: Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu, Zhigang Hua, Yan Xie, Shuang Yang,

Organizations: University of Illinois Urbana-Champaign · University of California San Diego · Meta AI

Abstract

Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose CoPE, which adds a third kind of multi-step action targets. On successful on-policy trajectories, CoPE trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $τ^2$ -bench, and AppWorld), CoPE (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at https://github.com/ulab-uiuc/CoPE.

Figures & tables

Figure 1: Motivation of CoPE for sample-efficient RL. ( a ) A terminal observation contains many task-irrelevant tokens, whereas the corresponding action is short and decision-relevant. ( b ) Three kinds of targets at the same history

h_{t}

. Value targets are shared across steps: every step of a trajectory is labeled with the same final outcome

Y

. The observation target

\hat{o}_{t+1}

is noisy and reaches only one step ahead. CoPE ’s action targets condition on

Y=1

and are specific to each step:

h_{t}

is labeled with the actions that actually followed it,

a_{t:t+K}

(

K=3

here), and the next step with

a_{t+1:t+K+1}

. ( c ) On AppWorld, CoPE reaches a training task score of 0.5 with

1.6\times

fewer RL steps than GRPO.

Figure 2: Overview of CoPE . (i) Joint training of acting and forecasting. One LLM backbone is shared by both, and

\mathcal{L}_{\mathrm{AF}}

fits the successful action chunk

a_{t:t+K}

. (ii) Co-evolution of acting and forecasting. Acting helps forecasting via trajectory collection, and forecasting helps acting via implicit temporal ensembling.

Method	Telec.	Retail	Air	Overall
Agentic Prompting
Qwen2.5	$14.4$	$13.1$	$7.5$	$11.7$
ReAct	$13.1$	$13.8$	$10.0$	$12.3$
Published RL results
RAGEN	$17.5$	$17.5$	$15.0$	$16.7$
Search-R1	$15.6$	$15.0$	$10.0$	$13.5$

Table 1: Results on

\tau^{2}

-bench . All results are Avg@4 for task success rate with Qwen2.5-7B-Instruct.

Method	ALFWorld							WebShop		ScienceWorld
Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.	Score	Succ.
Agentic prompting
Gemini-2.5-flash-lite	$82.61$	$36.84$	$37.84$	$20.00$	$21.43$	$48.89$	$46.00$	$7.83$	$6.50$	$13.85$	$15.00$
GPT-5-mini	$82.61$	$47.37$	$27.03$	$36.00$	$14.29$	$75.56$	$52.00$	$11.74$	$7.00$	$40.13$	$43.50$
GPT-5.5	97.83	100.00	$91.89$	$88.00$	89.29	93.33	$93.50$	$66.10$	$56.50$	66.77	79.00
Qwen2.5-7B	$69.57$	$31.58$	$29.73$	$36.00$	$17.86$	$11.11$	$34.00$	$25.28$	$2.50$	$-13.34$	$2.50$

Table 3: Evaluation results on simulated agent tasks. For ALFWorld, we report the success rate (%) on each subtask and overall. For WebShop and ScienceWorld, we report the average score and the success rate (%). All RL methods use Qwen2.5-7B-Instruct. Dark gray and light gray mark the best and second-best results in each column across all methods. More details are available in Appendix B .

Table 5

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Train	Test	Pool	Unit of a task
ALFWorld	$2{,}420$	$200$	$2{,}620$	TextWorld game file
WebShop	$3{,}930$	$200$	$6{,}910$	shopping instruction (goal)
SciWorld	$2{,}120$	$200$	$4{,}639$	(task type, variation) pair
$\tau^{2}$	$178$	$100$	$278$	customer-service task
AppWorld	$90$	$168/417$	$732$	task in a scenario

Appendix

Table 7: Splits used in this work. “Pool” is the number of distinct tasks the environment can instantiate at all, from which the two splits are drawn; the splits do not have to exhaust it. Counts are measured, not quoted.

Split	All	Pick	Look	Clean	Heat	Cool	Pick2
Train	$2{,}420$	$672$	$196$	$448$	$321$	$394$	$389$
Test	$200$	$46$	$19$	$37$	$25$	$28$	$45$

Appendix

Table 8: ALFWorld subtask families. Derived from the task type encoded in each game file’s name, so the two splits are counted the same way.

GRPO

ECHO

RWML

EMPG

Hyperparameter

ALFWorld

WebShop

SciWorld

ALFWorld

WebShop

SciWorld

ALFWorld

WebShop

SciWorld

ALFWorld

WebShop

SciWorld

Learning rate

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

KL coefficient

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

Entropy coefficient

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

ECHO coefficient

–

0.01

–

Training batch size

Appendix

Table 9: Details of training hyperparameters . We include experiments for GRPO, ECHO, RWML, and EMPG. The table is organized first by method and then by dataset. CoPE settings are identical to GRPO with a coefficient of

0.01

for Action Forecasting loss.

Setting	AppWorld value
Batch construction
Training batch size	$16$
Rollouts per task	$8$
Trajectories per step	$128$
PPO mini-batch size	$16$
Micro-batch size per GPU	$1$

Appendix

Table 10: AppWorld training settings. GRPO (ours) and CoPE use the same settings except for the additional forecasting objective in CoPE .

Setting	$\tau^{2}$ value
Length limits
Training max rounds	$50$
Evaluation max rounds	$50$
Maximum prompt length (tokens)	$8{,}192$
Maximum response length (tokens)	$16{,}384$
Maximum model length (tokens)	$24{,}576$

Appendix

Table 11:

\tau^{2}

-bench training settings. Only settings with direct counterparts in the other environment summaries are included.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 612 papers

Oct 1, 2026

Do Your Own Research: Learning to Forecast by Learning to Search

Outcome-based reinforcement learning can train language models to forecast real-world events, but prior forecasting work either freezes research context before training or deploys agentic research only at test time, so the skill of gathering evidence is never shaped by the reward. We introduce an agentic forecasting environment, dataset, and harness built from 2,100+ resolved Polymarket questions; the agent acquires its own context at rollout time (web search, page reading, and financial time series, all restricted by layered leak filtering to information published before each question's cutoff), and we train Qwen3.5-35B-A3B (3B active parameters) on it with single-epoch GRPO under a Brier-score reward. Training changes how the agent interacts with information: calibration improves 30-40%, and search attempts fall from 3.8 to 2.25 per rollout as evidence discipline is learned. Evaluated in an identical harness against four frontier models, the trained policy also finishes ahead of every frontier model tested at evidence-based forecasting, including Claude Opus 4.5 (soft-Brier 0.254 vs. 0.256, n=265), at about 5% of the inference cost, and its margin is widest on the hardest questions, the ones the crowd itself had not decided. We release the environment, dataset, and per-rollout records as a reusable harness for temporal forecasting agents.

Yusuf Afifi, Artur Kiulian, Anton Polishko +3

Future Principle

Agentic RL Forecasting Benchmarks Probabilistic Forecasting RL for LLM Agents

Ask PDF

May 7, 2026

Milestone-Guided Policy Learning for Long-Horizon Language Agents

While long-horizon agentic tasks require language agents to perform dozens of sequential decisions, training such agents with reinforcement learning remains challenging. We identify two root causes: credit misattribution, where correct early actions are penalized due to terminal failures, and sample inefficiency, where scarce successful trajectories result in near-total loss of learning signal. We introduce a milestone-guided policy learning framework, BEACON, that leverages the compositional structure of long-horizon tasks to ensure precise credit assignment. BEACON partitions trajectories at milestone boundaries, applies temporal reward shaping within segments to credit partial progress, and estimates advantages at dual scales to prevent distant failures from corrupting the evaluation of local actions. On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO. Notably, on long-horizon ALFWorld tasks, BEACON achieves 92.9% success rate, nearly doubling GRPO's 53.5%, while improving effective sample utilization from 23.7% to 82.0%. These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents. Code is available at https://github.com/ZJU-REAL/BEACON.

Zixuan Wang, Yuchen Yan, Hongxing Li +7

Zhejiang University · Baidu Inc.

Credit Assignment in RL Long-Horizon Agent Tasks Long-Horizon LLM Agents Policy Learning RL for LLM Agents Long-Horizon RL

Ask PDF

Jun 1, 2026

Policy and World Modeling Co-Training for Language Agents

Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training.

Ning Lu, Baijiong Lin, Shengcai Liu +9

Southern University of Science and Technology · Hong Kong University of Science and Technology · Hong Kong University of Science and Technology (Guangzhou) +2

Agentic RL World Model Learning LLM Agent Training RL for LLM Agents

Ask PDF

Abstract

RL for LLM Agents

Explore similar work

Do Your Own Research: Learning to Forecast by Learning to Search

Milestone-Guided Policy Learning for Long-Horizon Language Agents

Policy and World Modeling Co-Training for Language Agents