cs.LGNov 2, 2026

Hindsight Credit Assignment over Trees for Agentic RL

Authors: Penghong Zhao, Yihao Chen, Maoheng Yuan, Yunjian Zhang, Abudukelimu Wuerkaixi, Haoran Xu, Man Lv, Yaolong Wang, 

Organizations: Alibaba Cloud Computing · Fudan University

Abstract

Reinforcement learning for long-horizon LLM agents faces a fundamental dilemma: successful experiences are essential for policy improvement, yet limited policy capabilities make such experiences difficult to discover. Sparse outcome rewards compound this challenge, leaving sampled trajectories with little relative feedback for learning. In this paper, we propose TriGRPO, a tree-based reinforcement learning framework that turns scarce successes into exploration and learning signals through hindsight credit assignment within and across rollout trees. TriGRPO constructs rollout trees through hindsight-guided branching: when an ongoing trajectory reaches a matched decision state, a successful action discovered during rollout initiates a sibling branch that continues autonomously, enabling further exploration and parent-sibling comparisons. Three complementary groups organize policy optimization over these trees: root groups capture within-prompt relative returns, node groups assign credit from parent-sibling outcome differences, and cross groups supplement sparse feedback with historical performance on related tasks. Experimental results demonstrate consistent gains in task performance and rollout efficiency across model scales, highlighting structured hindsight credit assignment as an effective approach to long-horizon agent learning.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents

    Oct 9, 2026Debajoy Mukherjee, Jeremy Carleton, Sushil Vemuri +2LLM Agent TrainingRL for LLM Agents

  2. Hindsight Credit Assignment for Long-Horizon LLM Agents

    Mar 7, 2026Hui-Ze Tan, Xiao-Wen Yang, Hao Chen +7Credit Assignment in RLLong-Horizon LLM Agents

  3. GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents

    Sep 28, 2026Haodong Zhu, Yangyang Ren, Changbai Li +4Credit Assignment in RLStep-Level Credit Assignment in RL