Long-Horizon RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Latest papers

32 papers12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Dec 28, 2025Yingru Li, Jiacai Liu, Jiawei Xu +4RL for Language Models Reinforcement Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen
Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
Oct 31, 2025Kristina Levina, Nikolaos Pappas, Athanasios Karapantelakis +2Reinforcement Learning Q-Learning
Linköping University, Linköping, Sweden · Ericsson Research, Stockholm, Sweden

Dec 28, 2025J/K move · Enter open · S save

Trust Region Masking for Long-Horizon LLM Reinforcement Learning

Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong+3

Fudan University · The Chinese University of Hong Kong, Shenzhen

RL for Language Models Reinforcement Learning Long-Horizon RL Policy Gradient Methods Trust Region Policy Optimization Policy Learning+1

Open paper AskPDF arXiv

Policy gradient methods for Large Language Models optimize a policy

π_θ

via a surrogate objective computed from samples of a rollout policy

π_{\text{roll}}

. However, modern LLM-RL pipelines suffer from unavoidable implementation divergences -- backend discrepancies, Mixture-of-Experts routing discontinuities, and distributed training staleness -- causing off-policy mismatch (

π_{\text{roll}} \neq π_θ

) and approximation errors between the surrogate and the true objective. We demonstrate that classical trust region bounds on this error scale as

O(T^2)

with sequence length

T

, rendering them vacuous for long-horizon tasks. To address this, we derive a family of bounds -- both KL-based and TV-based -- including a Pinsker-Marginal bound (

O(T^{3/2})

), a Mixed bound (

O(T)

), and an Adaptive bound that strictly generalizes the Pinsker-Marginal bound via per-position importance-ratio decomposition. Taking the minimum over all bounds yields the tightest known guarantee across all divergence regimes. Crucially, all bounds depend on the maximum token-level divergence

D_{\mathrm{KL}}^{\mathrm{tok,max}}

(or

D_{\mathrm{TV}}^{\mathrm{tok,max}}

), a sequence-level quantity that cannot be controlled by token-independent methods like PPO clipping. We propose Trust Region Masking (TRM), which masks entire sequences violating the trust region, enabling the first non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.

Latest papers

Trust Region Masking for Long-Horizon LLM Reinforcement Learning

Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines