BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Xuanren Chen; Chongyang Tao; Wenpeng Hu; Yong Wang; Bo Deng; Xinlei Zheng; Shuai Ma

cs.AINov 2, 2026

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Authors: Xuanren Chen, Chongyang Tao, Wenpeng Hu, Yong Wang, Bo Deng, Xinlei Zheng, Shuai Ma

Organizations: Beihang University · Alibaba Group

Abstract

Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.

Figures & tables

Figure 1: Overview of BLPO . (1) Trajectory histories and concrete actions from different tasks are mapped to shared task-progress states and action functions. (2) Within each concrete context, returns are centered and averaged per abstract state–action pair to update a global memory across training iterations. (3) Action contrasts estimated from this memory are scaled by effective support and uncertainty. The resulting task-progress advantages are combined with trajectory-relative advantages and local advantages over normalized actions to optimize the policy.

Model	Type	Method	ALFWorld		WebShop
Model	Type	Method	In-Success	Out-Success	Task Score	Task Success
Closed	Prompting	GPT-4o	48.0	46.0	31.8	23.7
Closed	Prompting	Gemini-2.5-Pro	60.3	50.5	42.5	35.9
Qwen2.5-1.5B-Instruct	Prompting	Qwen2.5	4.1	–	23.1	5.2
	Prompting	ReAct	12.8	–	40.1	11.3
	Prompting	Reflexion	21.8	–	55.8	21.9

Table 1: Performance on ALFWorld and WebShop. In-Success and Out-Success measure in-distribution and out-of-distribution success. WebShop Task Score and Task Success measure partial and complete request satisfaction. RL results are mean

\pm

standard deviation over three seeds. Bold marks the best result per backbone; dashes indicate unavailable results.

Variant	In-Suc.	Out-Suc.	Score	Success
	ALFWorld		WebShop
BLPO	94.01	91.54	87.43	74.02
Local credit
w/o Act. pool.	91.34 $\blacktriangledown$ 2.67	83.14 $\blacktriangledown$ 8.40	82.79 $\blacktriangledown$ 4.64	66.21 $\blacktriangledown$ 7.81
Task-progress groups
w/o $A^{\mathrm{prog}}$	92.25 $\blacktriangledown$ 1.76	83.53 $\blacktriangledown$ 8.01	82.75 $\blacktriangledown$ 4.68	68.49 $\blacktriangledown$ 5.53

Table 2: Component ablations with Qwen2.5-1.5B. Gray values show absolute drops from full BLPO .

Figure 2: ALFWorld learning dynamics.

Method	Adv. (s)	Update (s)	Share	Total (h)
GiGPO	0.89	335.2	0.26%	14.90
HGPO	0.96	362.6	0.26%	16.11
BLPO	2.12	349.7	0.61%	15.54

Table 3: Training cost on ALFWorld (4 A100s, 160 updates). Adv. and Update are mean times per update; Share is their ratio. Evaluation time is excluded.

Figure 3: Coverage and consistency of global credit. (a) Fractions of decisions with nonzero local credit (gray) and additional coverage from task-progress credit (blue). (b) Sign agreement across confidence bins and held-out task folds in ALFWorld.

Figure 5: Examples of cross-task credit in ALFWorld and WebShop.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Task-relative decision	Action function
ALFWorld	In Pick2, take a target from outside the destination.	Acquire an unplaced target
ALFWorld	In Pick2, take a target back out of the destination.	Retake a placed target
WebShop	Select a requested option that is not yet satisfied.	Satisfy an option
WebShop	Select a conflicting option or repeat an already satisfied choice.	Violate or repeat an option

Appendix

Table 5: Examples of task-relative distinctions preserved by action abstraction.

Figure 6: Local and task-progress credit across training phases. Bars partition all decisions by nonzero credit availability; hatching marks shared coverage with opposing advantage signs.

Figure 7: Success by ALFWorld task type with Qwen2.5-1.5B at update 160, evaluated on the same 140 tasks as Figure 2 . Axes scale linearly from 0% at the center to 100% at the boundary.

Figure 8: Training diagnostics on ALFWorld with Qwen2.5-1.5B, using one training run per method. Pale lines show per-update measurements; solid lines show 10-update trailing means.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 637 papers

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks
Jul 22, 2026Kaibing Yang, Guangfeng Cai, Shengtian Yang +6Group Relative Policy Optimization RL Exploration
Southeast University · Kuaishou Technology · Meituan
Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
Feb 26, 2026Shuo He, Lang Feng, Qi Wei +3Agentic RL RL for LLM Agents
Nanyang Technological University · Southeast University
Multiscale Policy Optimization for Long-Horizon LLM Agents
Oct 26, 2026Tao Ren, Hui Yang, Jia Deng +16Policy Optimization RL for LLM Agents
Peking University · IQuest Research · Tsinghua University +3

Jul 22, 2026J/K move · Enter open · S save

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks

Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He+5

Southeast University · Kuaishou Technology · Meituan

Group Relative Policy Optimization RL Exploration Sparse-Reward RL Long-Horizon Agent Tasks Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon agentic tasks, we found this comparison suffers from a sampling imbalance: repeated or low-effect actions dominate the high-probability region of the policy, leaving useful state-changing actions under-sampled. This imbalance produces many all-failed rollout groups, where the relative advantages within such a group provide no direction for updating the policy. Consequently, the sampling imbalance and the all-failed groups collapse into a self-perpetuating credit trap: failure-dominated sampling yields no learning signal, allowing repeated low-effect actions to persist. To break this trap, we propose Coverage-conditioned Group Policy Optimization (CovGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward. Specifically, within such groups, CovGPO assigns higher relative advantages to state-exploring actions without corrupting the original reward signal, as reaching new observations is a prerequisite for task success. Experiments on two challenging agentic benchmarks, ALFWorld and WebShop, show that CovGPO consistently improves over group-based baselines, with particularly large gains on hard tasks.

Abstract

RL for LLM Agents

Explore similar work

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks

Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

Multiscale Policy Optimization for Long-Horizon LLM Agents