One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

Shaojin Ma; Ruitao Wang; Pengyi Li; Jie Yang; Liangyu Chen; Gangyi Zhao; Hongyao Tang; Yan Zheng

cs.LGOct 21, 2026

One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

Authors: Shaojin Ma, Ruitao Wang, Pengyi Li, Jie Yang, Liangyu Chen, Gangyi Zhao, Hongyao Tang, Yan Zheng,

Organizations: Tianjin University · Fudan University · Renmin University of China

Abstract

Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks. However, despite progress in fine-grained credit assignment, policy optimization still applies these learning signals through response-specific probability ratios. Different textual responses can induce the same executable effect, yet credit differences arising from subsequent outcomes remain tied to their individual textual realizations. We identify this mismatch as credit-update misalignment. Our analysis reveals that credit-text pairing directly shapes policy-update directions, even when the policy context and executable effect are held fixed. Motivated by this finding, we propose Effect-Marginal Policy Optimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations. EMPO preserves the original credit signals, integrates with both trajectory-level and step-level credit estimators, and requires no additional critics or rollouts. Experiments on ALFWorld, WebShop, and Sokoban show that EMPO outperforms strong baselines in both task performance and learning efficiency.

Figures & tables

Figure 1: Execution-equivalent responses may receive different credit and undergo divergent response-wise policy updates despite inducing the same immediate effect. Such within-effect variation can propagate into inconsistent ratio changes and unstable policy updates over training.

Figure 2: Overview of Effect-Marginal Policy Optimization (EMPO). EMPO groups locally comparable responses with the same canonical execution effect and aggregates their response-level policy ratios into a shared effect ratio. The shared ratio is combined with the original advantages to coordinate policy updates across execution-equivalent responses.

Closed-source models
Method	ALFWorld							WebShop
	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Success
Prompting
GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Qwen2.5-1.5B-Instruct

Table 1: Performance comparison on ALFWorld and WebShop benchmarks. Results are reported as mean

\pm

std over 3 random seeds.

\Delta

denotes the change from the corresponding baseline.

Method	ALFWorld SR	WebShop Score	WebShop SR
EMPO-G	83.3	83.1	70.3
w/o Length Normalization	61.7	43.3	27.3
w/o Ratio Sharing	79.7	65.3	55.5
w/o Effect-Based Grouping	78.9	77.5	64.1
EMPO-GiG	94.0	87.9	77.3
w/o Length Normalization	60.2	75.2	43.8

Table 4: Ablation results using Qwen2.5-1.5B-Instruct.

Figure 3: Policy-update analysis on ALFWorld with Qwen2.5-1.5B-Instruct. Panels (a) and (b) show pairwise cosine similarities of response-level objective gradients under GiGPO and EMPO-GiG. Black lines mark effect-cell boundaries and the top strips indicate advantage signs. Panel (c) reports the within-cell dispersion $D_{K}$ of log-probability changes after controlled one-step updates.

Figure 4: Effect-cell size distributions of EMPO-GiG at different training stages. Bars show the proportion of eligible cells in each size range, while response coverage measures the fraction of sampled responses assigned to eligible cells with

|K|\geq 2

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	ALFWorld	WebShop	Sokoban
Max. prompt length	2,048	4,096	1,024
Max. environment steps	50	15	15
Groups $\times$ trajectories	$16\times 8$	$16\times 8$	$32\times 8$
Actor mini-batch size	256	64	64

Appendix

Table 5: Environment-specific training configurations.

Figure 5: Training curves on ALFWorld, WebShop, and Sokoban.

Figure 6: Training dynamics on ALFWorld with Qwen2.5-1.5B-Instruct. We report token entropy, KL divergence to the reference policy, and gradient norm.

Figure 7: Training dynamics on WebShop with Qwen2.5-1.5B-Instruct. We report token entropy, KL divergence to the reference policy, and gradient norm.

Figure 8: Relative magnitude of the within-effect covariance term on six effect cells from ALFWorld. For each cell

K

, we report

\|\widehat{\operatorname{Cov}}_{K}(\hat{A},s)\|/\|G_{K}^{\mathrm{ind}}\|

under GRPO and GiGPO.

Base	Method	ALFWorld SR	WebShop Score	WebShop SR
GRPO	Advantage Aggregation	80.47	82.1	70.2
	EMPO-G	83.3	83.1	70.3
GiGPO	Advantage Aggregation	92.97	83.6	71.1
	EMPO-GiG	94.0	87.9	77.3

Appendix

Table 6: Comparison between EMPO and advantage-side aggregation.

Figure 9: Prompt template used for ALFWorld.

Figure 10: Prompt template used for WebShop.

Figure 11: Prompt template used for Sokoban.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
Jun 4, 2026Yuanfan Li, Qi Zhou, Wenjing Duan +1Credit Assignment in RL Step-Level Credit Assignment in RL
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University
Contextual Credit Policy Optimization
Oct 13, 2026Huiwen Han, Ziyao Zeng, Zihao Zhu +6RL for LLM Agents Step-Level Credit Assignment in RL
Stanford University · Texas A&M University · Yale University +3
IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
Aug 25, 2026Bo Ren, Yirong Mao, Yi Yang +1LLM Agent Training Credit Assignment in RL
WeChat, Tencent Inc.

Jun 4, 2026J/K move · Enter open · S save

When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training

Yuanfan Li, Qi Zhou, Wenjing Duan, Lu Chen

X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University

Credit Assignment in RL Step-Level Credit Assignment in RL Policy Gradient Methods Sparse-Reward RL Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards. Recent group-based methods such as GiGPO improve over GRPO by constructing step-level advantages at repeated anchor states. However, we show that such dense credit can be statistically unreliable: under limited rollouts, rare but lucky actions may receive overly large advantages, producing divergent anchor bias and late-stage training oscillation. We propose Evidence-Calibrated Policy Optimization (ECPO), a critic-free policy optimization algorithm that calibrates step-level credit before policy updates. ECPO combines Evidence-Calibrated Action Advantage, which groups rollouts by canonical actions and shrinks low-count estimates, with Variance-Gated Credit Weighting, which suppresses anchor states dominated by within-action noise. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B show that ECPO consistently outperforms strong baselines, improving GiGPO by +5.2/+7.3 success points on ALFWorld/WebShop with Qwen2.5-1.5B while adding only 0.1% additional advantage-computation overhead.

Abstract

Policy Optimization

Explore similar work

When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training

Contextual Credit Policy Optimization

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents