Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Heming Zou; Qi Wang; Yixiu Mao; Yun Qu; Yuyi Guo; Zeyuan Liu; Chenglin Li; Weijie Liu

cs.LGOct 16, 2026

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Authors: Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu,

Organizations: Tsinghua University · Foundation Model Department, Tencent

Abstract

Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.

Figures & tables

Figure 1: Target construction: terminal rewards at the blooming (successful) and withered (failed) branch tips are backed up through shared prefixes to define a frozen turn-level teacher along the luminous green path. Distribution matching: starting from the behavior-policy rollout (purple trail), the student policy progressively aligns with this outcome-derived target through posterior distillation.

Figure 2: TreeOPSD framework. The behavior policy

\pi_{\theta_{\mathrm{old}}}

generates a shared-prefix rollout tree, and terminal verifier rewards

R_{k}

are backed up into turn credits

V_{i}

. Each credit defines a frozen Bernoulli teacher

q_{i}=\tfrac{1}{2}+V_{i}/\tau

, and each bar shows one turn against the behavior level

\tfrac{1}{2}

(dashed). In each of the

E

sequential mini-batches, the student posterior

p_{\theta,i}=\sigma(s_{\theta,i})

is recomputed under the current policy

\pi_{\theta}

, and the spring-like ascent weight

\Gamma_{i}=\tau(q_{i}-p_{\theta,i})

pulls it toward

q_{i}

, vanishing at the teacher and reversing sign after overshoot.

Figure 3: From backed-up credit to a finite, self-correcting target. (a) The affine credit-to-target map commutes with likelihood-weighted tree backup, so the teacher

q_{u}

at a shared turn equals the weighted average of its child teachers. (b) In log-ratio score space

s_{\theta,i}

, TreeOPSD and fixed tree policy gradient share the same tangent at the behavior policy (

s=0

), but only TreeOPSD has a finite minimizer

s^{\star}=\log r^{\star}

. (c) The student probability

p_{\theta,i}=\sigma(s_{\theta,i})

converges toward the target

q^{\star}

from either side because the residual reverses sign after overshoot, whereas fixed credit keeps pushing in the initial direction.

Method	Multi-Hop QA					Deep Research
Method	Hotpot	2Wiki	MuSiQue	Bamboogle	Avg	BC+	BC	GAIA	xbench	Avg
Qwen3-4B-Thinking-2507
ReAct	31.61	29.63	9.68	25.60	24.13	4.70	1.97	15.53	13.00	8.80
GRPO	46.93	48.62	21.93	56.00	43.37	10.00	2.29	18.45	18.00	12.18
GSPO	46.55	49.10	21.76	55.20	43.15	9.88	2.37	18.45	18.00	12.17
TreePO	47.25	49.90	22.05	54.40	43.40	11.08	2.76	19.42	19.00	13.07

Table 1: Performance comparison on Multi-Hop QA and Deep Research. We train on HotpotQA and OpenResearcher, and evaluate Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507 on four test sets for each scenario. TreeOPSD consistently outperforms competitive baselines on average. The best results are indicated in bold .

Figure 4: Test accuracy during training. The four panels report TreeOPSD and GRPO with Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507 on the Multi-Hop QA average (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle) and on BrowseComp-Plus. Higher curves indicate stronger final policies under identical sampling budgets.

Figure 5: Pointwise credit and the posterior map. (a) Two useful children share a parent backup. Pointwise credit is

V_{c}

, and listwise credit is the group-relative

V_{c}-V_{i}

. (b) BrowseComp-Plus with Qwen3-4B-Thinking-2507. The pointwise curve is the TreeOPSD run from Figure 4 (c). (c) The odds map and a scaled logistic map agree at the first update, and the student then stops at the odds teacher. (d) The odds map against the logistic map on that split. In (b) and (d), higher curves indicate stronger final policies under identical sampling budgets.

(a) Rollout allocation			(b) Target radius
$(M,N)$	Multi-Hop QA	Deep Research	$\rho$	Multi-Hop QA	Deep Research
$(8,0)$	43.66	11.28	$0.10$	43.79	14.62
$(4,0)$	42.98	10.05	$0.20$	44.01	15.06
$(4,2)$	44.13	15.45	$0.25$	44.13	15.45
$(2,6)$	44.07	14.97	$0.30$	43.95	14.88
$(2,2)$	43.25	11.33	$0.40$	43.81	14.69

Table 2: Rollout allocation and target-radius sensitivity. Unweighted four-benchmark averages on Multi-Hop QA and Deep Research with Qwen3-4B-Thinking-2507. Panel (a) varies the initialize-then-expand split at

\rho=0.25

, and panel (b) varies the target radius at

(M,N)=(4,2)

. The gray row is the default setting used in Table 1 and shared by both panels.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Qwen3-1.7B					Llama-3.2-3B-Instruct
Method	Hotpot	2Wiki	MuSiQue	Bamboogle	Avg	Hotpot	2Wiki	MuSiQue	Bamboogle	Avg
ReAct	23.04	26.11	6.37	19.20	18.68	12.18	7.38	4.22	20.00	10.95
GRPO	29.10	31.48	8.61	33.60	25.70	36.80	36.35	13.78	40.00	31.73
GSPO	29.09	31.85	8.52	32.80	25.57	36.48	36.73	13.61	39.20	31.50
TreePO	29.86	32.18	8.90	33.60	26.13	37.56	37.43	14.23	40.00	32.30
TreeOPSD	31.30	33.81	9.60	32.80	26.88	39.59	41.15	15.97	45.60	35.58

Appendix

Table 3: Performance comparison on Multi-Hop QA with smaller backbones. We train on HotpotQA and evaluate Qwen3-1.7B and Llama-3.2-3B-Instruct on four test sets. TreeOPSD consistently outperforms competitive baselines on average. The best results are indicated in bold .

Figure 6: Test accuracy during training. The two panels report TreeOPSD and GRPO with Qwen3-1.7B and Llama-3.2-3B-Instruct on the Multi-Hop QA average (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle). Higher curves indicate stronger final policies under identical sampling budgets.

Figure 7: Numerical studies of turn credit and the posterior map. (a) Repeated updates share one scalar policy score. TreeOPSD stops at the teacher, while TreePO keeps the collection-time coefficient. (b) Two useful children share a credit

m

. Pointwise credit is

V_{c}

, and listwise credit is the group-relative

V_{c}-V_{i}

. (c) The odds map with Forward KL preserves the parent gradient after tree backup. Reverse KL and a logistic map do not.

Figure 8: Interaction length and training efficiency. Panels (a) and (b) report the average number of assistant turns during evaluation and training on BrowseComp-Plus with Qwen3-30B-A3B-Thinking-2507. Panel (c) plots test accuracy against cumulative training time and shows TreeOPSD only within the total time used by GRPO. Higher curves in (c) indicate stronger final policies under identical training-time budgets.

Method	Multi-Hop QA	Deep Research
ReAct	$24.13\pm 0.05$	$8.80\pm 0.26$
GRPO	$43.37\pm 0.19$	$12.18\pm 0.27$
GSPO	$43.15\pm 0.19$	$12.17\pm 0.32$
TreePO	$43.40\pm 0.17$	$13.07\pm 0.37$
TreeOPSD	$\mathbf{44.13\pm 0.16}$	$\mathbf{15.45\pm 0.39}$

Appendix

Table 4: Mean

\pm

std over five training seeds with Qwen3-4B-Thinking-2507. Averages use the same Multi-Hop QA and Deep Research protocol as Table 1 . TreeOPSD consistently outperforms competitive baselines on average. The best results are indicated in bold .

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 608 papers

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Aug 6, 2026Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10Agentic RL Reinforcement Learning
Tsinghua University · Meituan · Zhejiang University
Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RL Self-Distillation
Zhejiang University · Meituan · Tsinghua University
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Jul 30, 2026Xingjian Wu, Junlin Liu, Xingchen Liu +6Agentic RL RL for Language Model Reasoning
Meituan

Aug 6, 2026J/K move · Enter open · S save

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu+9

Tsinghua University · Meituan · Zhejiang University

Agentic RL Reinforcement Learning Credit Assignment in RL Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Policy Learning

Open paper AskPDF arXiv

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

Abstract

RL for LLM Agents

Explore similar work

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Self-Distilled Agentic Reinforcement Learning

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation