cs.AINov 18, 2026

Depth Before Breadth: Reverse-to-Forward Curriculum with Advantage Decomposition for LLM Agent

Authors: Siyuan Du, Yuxin Liu, Meng Tang, Haoran Li, Yechao Bai, Feiyang Pan, Jia He

Organizations: Nanjing University · Peking University · University of Wales, Aberystwyth

Abstract

Reinforcement learning (RL) has become a common approach for training large language model (LLM) agents on long-horizon tasks. However, learning from sparse terminal rewards makes credit assignment and exploration difficult. We approach these challenges through a policy-induced capability region, the set of states from which the current policy can reliably complete a task. Near its boundary, an agent may need only a few actions to reach a more solvable state, even when the goal remains distant. This reduces long-horizon learning to two coupled subproblems of reaching that state and completing the task from there. We address them with paired advantages and a progressive reverse-to-forward curriculum. The advantages provide separate signals for reaching more solvable states and completing the task from them, while the curriculum expands the capability region backward along successful reference trajectories and then into off-reference states through on-policy exploration. On ALFWorld and WebShop, our method outperforms competitive agentic RL methods at both model scales, achieving success rates of 90.1%/96.1% and 80.2%/82.6% with Qwen2.5-1.5B/7B-Instruct, respectively.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

    Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM AgentsHierarchical RL

  2. LLM Value Functions for Credit Assignment in Agentic RL

    Oct 11, 2026Kaijie Zhu, Weixiang Yan, Omar Attia +3RL for LLM AgentsAgentic RL

  3. Hindsight Credit Assignment over Trees for Agentic RL

    Nov 2, 2026Penghong Zhao, Yihao Chen, Maoheng Yuan +8Reinforcement LearningRL for LLM Agents