Co-Evolving Acting and Forecasting for Language Agents

Haofei Yu; Yixing Chen; Yining Zhao; Xuyan Ye; Yuxiang Liu; Zhigang Hua; Yan Xie; Shuang Yang

cs.AIOct 14, 2026

Co-Evolving Acting and Forecasting for Language Agents

Authors: Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu, Zhigang Hua, Yan Xie, Shuang Yang,

Organizations: University of Illinois Urbana-Champaign · University of California San Diego · Meta AI

Abstract

Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose CoPE, which adds a third kind of multi-step action targets. On successful on-policy trajectories, CoPE trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $τ^2$ -bench, and AppWorld), CoPE (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at https://github.com/ulab-uiuc/CoPE.

Figures & tables

Figure 1: Motivation of CoPE for sample-efficient RL. ( a ) A terminal observation contains many task-irrelevant tokens, whereas the corresponding action is short and decision-relevant. ( b ) Three kinds of targets at the same history

h_{t}

. Value targets are shared across steps: every step of a trajectory is labeled with the same final outcome

Y

. The observation target

\hat{o}_{t+1}

is noisy and reaches only one step ahead. CoPE ’s action targets condition on

Y=1

and are specific to each step:

h_{t}

is labeled with the actions that actually followed it,

a_{t:t+K}

(

K=3

here), and the next step with

a_{t+1:t+K+1}

. ( c ) On AppWorld, CoPE reaches a training task score of 0.5 with

1.6\times

fewer RL steps than GRPO.

Figure 2: Overview of CoPE . (i) Joint training of acting and forecasting. One LLM backbone is shared by both, and

\mathcal{L}_{\mathrm{AF}}

fits the successful action chunk

a_{t:t+K}

. (ii) Co-evolution of acting and forecasting. Acting helps forecasting via trajectory collection, and forecasting helps acting via implicit temporal ensembling.

Method	Telec.	Retail	Air	Overall
Agentic Prompting
Qwen2.5	$14.4$	$13.1$	$7.5$	$11.7$
ReAct	$13.1$	$13.8$	$10.0$	$12.3$
Published RL results
RAGEN	$17.5$	$17.5$	$15.0$	$16.7$
Search-R1	$15.6$	$15.0$	$10.0$	$13.5$

Table 1: Results on

\tau^{2}

-bench . All results are Avg@4 for task success rate with Qwen2.5-7B-Instruct.

Method	ALFWorld							WebShop		ScienceWorld
Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.	Score	Succ.
Agentic prompting
Gemini-2.5-flash-lite	$82.61$	$36.84$	$37.84$	$20.00$	$21.43$	$48.89$	$46.00$	$7.83$	$6.50$	$13.85$	$15.00$
GPT-5-mini	$82.61$	$47.37$	$27.03$	$36.00$	$14.29$	$75.56$	$52.00$	$11.74$	$7.00$	$40.13$	$43.50$
GPT-5.5	97.83	100.00	$91.89$	$88.00$	89.29	93.33	$93.50$	$66.10$	$56.50$	66.77	79.00
Qwen2.5-7B	$69.57$	$31.58$	$29.73$	$36.00$	$17.86$	$11.11$	$34.00$	$25.28$	$2.50$	$-13.34$	$2.50$

Table 3: Evaluation results on simulated agent tasks. For ALFWorld, we report the success rate (%) on each subtask and overall. For WebShop and ScienceWorld, we report the average score and the success rate (%). All RL methods use Qwen2.5-7B-Instruct. Dark gray and light gray mark the best and second-best results in each column across all methods. More details are available in Appendix B .

Table 5

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Train	Test	Pool	Unit of a task
ALFWorld	$2{,}420$	$200$	$2{,}620$	TextWorld game file
WebShop	$3{,}930$	$200$	$6{,}910$	shopping instruction (goal)
SciWorld	$2{,}120$	$200$	$4{,}639$	(task type, variation) pair
$\tau^{2}$	$178$	$100$	$278$	customer-service task
AppWorld	$90$	$168/417$	$732$	task in a scenario

Appendix

Table 7: Splits used in this work. “Pool” is the number of distinct tasks the environment can instantiate at all, from which the two splits are drawn; the splits do not have to exhaust it. Counts are measured, not quoted.

Split	All	Pick	Look	Clean	Heat	Cool	Pick2
Train	$2{,}420$	$672$	$196$	$448$	$321$	$394$	$389$
Test	$200$	$46$	$19$	$37$	$25$	$28$	$45$

Appendix

Table 8: ALFWorld subtask families. Derived from the task type encoded in each game file’s name, so the two splits are counted the same way.

GRPO

ECHO

RWML

EMPG

Hyperparameter

ALFWorld

WebShop

SciWorld

ALFWorld

WebShop

SciWorld

ALFWorld

WebShop

SciWorld

ALFWorld

WebShop

SciWorld

Learning rate

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

1\times 10^{-6}

KL coefficient

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

Entropy coefficient

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

ECHO coefficient

–

0.01

–

Training batch size

Appendix

Table 9: Details of training hyperparameters . We include experiments for GRPO, ECHO, RWML, and EMPG. The table is organized first by method and then by dataset. CoPE settings are identical to GRPO with a coefficient of

0.01

for Action Forecasting loss.

Setting	AppWorld value
Batch construction
Training batch size	$16$
Rollouts per task	$8$
Trajectories per step	$128$
PPO mini-batch size	$16$
Micro-batch size per GPU	$1$

Appendix

Table 10: AppWorld training settings. GRPO (ours) and CoPE use the same settings except for the additional forecasting objective in CoPE .

Setting	$\tau^{2}$ value
Length limits
Training max rounds	$50$
Evaluation max rounds	$50$
Maximum prompt length (tokens)	$8{,}192$
Maximum response length (tokens)	$16{,}384$
Maximum model length (tokens)	$24{,}576$

Appendix

Table 11:

\tau^{2}

-bench training settings. Only settings with direct counterparts in the other environment summaries are included.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 612 papers

Do Your Own Research: Learning to Forecast by Learning to Search
Oct 1, 2026Yusuf Afifi, Artur Kiulian, Anton Polishko +3Agentic RL Forecasting Benchmarks
Future Principle
Milestone-Guided Policy Learning for Long-Horizon Language Agents
May 7, 2026Zixuan Wang, Yuchen Yan, Hongxing Li +7Credit Assignment in RL Long-Horizon Agent Tasks
Zhejiang University · Baidu Inc.
Policy and World Modeling Co-Training for Language Agents
Jun 1, 2026Ning Lu, Baijiong Lin, Shengcai Liu +9Agentic RL World Model Learning
Southern University of Science and Technology · Hong Kong University of Science and Technology · Hong Kong University of Science and Technology (Guangzhou) +2

Oct 1, 2026J/K move · Enter open · S save

Do Your Own Research: Learning to Forecast by Learning to Search

Yusuf Afifi, Artur Kiulian, Anton Polishko, Mykola Khandoga+2

Future Principle

Agentic RL Forecasting Benchmarks Probabilistic Forecasting RL for LLM Agents

Open paper AskPDF arXiv

Outcome-based reinforcement learning can train language models to forecast real-world events, but prior forecasting work either freezes research context before training or deploys agentic research only at test time, so the skill of gathering evidence is never shaped by the reward. We introduce an agentic forecasting environment, dataset, and harness built from 2,100+ resolved Polymarket questions; the agent acquires its own context at rollout time (web search, page reading, and financial time series, all restricted by layered leak filtering to information published before each question's cutoff), and we train Qwen3.5-35B-A3B (3B active parameters) on it with single-epoch GRPO under a Brier-score reward. Training changes how the agent interacts with information: calibration improves 30-40%, and search attempts fall from 3.8 to 2.25 per rollout as evidence discipline is learned. Evaluated in an identical harness against four frontier models, the trained policy also finishes ahead of every frontier model tested at evidence-based forecasting, including Claude Opus 4.5 (soft-Brier 0.254 vs. 0.256, n=265), at about 5% of the inference cost, and its margin is widest on the hardest questions, the ones the crowd itself had not decided. We release the environment, dataset, and per-rollout records as a reusable harness for temporal forecasting agents.

Abstract

RL for LLM Agents

Explore similar work

Do Your Own Research: Learning to Forecast by Learning to Search

Milestone-Guided Policy Learning for Long-Horizon Language Agents

Policy and World Modeling Co-Training for Language Agents