Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.
Figures & tables
Figure 1: Target construction: terminal rewards at the blooming (successful) and withered (failed) branch tips are backed up through shared prefixes to define a frozen turn-level teacher along the luminous green path. Distribution matching: starting from the behavior-policy rollout (purple trail), the student policy progressively aligns with this outcome-derived target through posterior distillation.
Figure 2: TreeOPSD framework. The behavior policy πθold generates a shared-prefix rollout tree, and terminal verifier rewards Rk are backed up into turn credits Vi . Each credit defines a frozen Bernoulli teacher qi=21+Vi/τ , and each bar shows one turn against the behavior level 21 (dashed). In each of the E sequential mini-batches, the student posterior pθ,i=σ(sθ,i) is recomputed under the current policy πθ , and the spring-like ascent weight Γi=τ(qi−pθ,i) pulls it toward qi , vanishing at the teacher and reversing sign after overshoot.
Figure 3: From backed-up credit to a finite, self-correcting target. (a) The affine credit-to-target map commutes with likelihood-weighted tree backup, so the teacher qu at a shared turn equals the weighted average of its child teachers. (b) In log-ratio score space sθ,i , TreeOPSD and fixed tree policy gradient share the same tangent at the behavior policy ( s=0 ), but only TreeOPSD has a finite minimizer s⋆=logr⋆ . (c) The student probability pθ,i=σ(sθ,i) converges toward the target q⋆ from either side because the residual reverses sign after overshoot, whereas fixed credit keeps pushing in the initial direction.
Method
Multi-Hop QA
Deep Research
Hotpot
2Wiki
MuSiQue
Bamboogle
Avg
BC+
BC
GAIA
xbench
Avg
Qwen3-4B-Thinking-2507
ReAct
31.61
29.63
9.68
25.60
24.13
4.70
1.97
15.53
13.00
8.80
GRPO
46.93
48.62
21.93
56.00
43.37
10.00
2.29
18.45
18.00
12.18
GSPO
46.55
49.10
21.76
55.20
43.15
9.88
2.37
18.45
18.00
12.17
TreePO
47.25
49.90
22.05
54.40
43.40
11.08
2.76
19.42
19.00
13.07
Table 1: Performance comparison on Multi-Hop QA and Deep Research. We train on HotpotQA and OpenResearcher, and evaluate Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507 on four test sets for each scenario. TreeOPSD consistently outperforms competitive baselines on average. The best results are indicated in bold .
Figure 4: Test accuracy during training. The four panels report TreeOPSD and GRPO with Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507 on the Multi-Hop QA average (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle) and on BrowseComp-Plus. Higher curves indicate stronger final policies under identical sampling budgets.
Figure 5: Pointwise credit and the posterior map. (a) Two useful children share a parent backup. Pointwise credit is Vc , and listwise credit is the group-relative Vc−Vi . (b) BrowseComp-Plus with Qwen3-4B-Thinking-2507. The pointwise curve is the TreeOPSD run from Figure 4 (c). (c) The odds map and a scaled logistic map agree at the first update, and the student then stops at the odds teacher. (d) The odds map against the logistic map on that split. In (b) and (d), higher curves indicate stronger final policies under identical sampling budgets.
(a) Rollout allocation
(b) Target radius
(M,N)
Multi-Hop QA
Deep Research
ρ
Multi-Hop QA
Deep Research
(8,0)
43.66
11.28
0.10
43.79
14.62
(4,0)
42.98
10.05
0.20
44.01
15.06
(4,2)
44.13
15.45
0.25
44.13
15.45
(2,6)
44.07
14.97
0.30
43.95
14.88
(2,2)
43.25
11.33
0.40
43.81
14.69
Table 2: Rollout allocation and target-radius sensitivity. Unweighted four-benchmark averages on Multi-Hop QA and Deep Research with Qwen3-4B-Thinking-2507. Panel (a) varies the initialize-then-expand split at ρ=0.25 , and panel (b) varies the target radius at (M,N)=(4,2) . The gray row is the default setting used in Table 1 and shared by both panels.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Qwen3-1.7B
Llama-3.2-3B-Instruct
Hotpot
2Wiki
MuSiQue
Bamboogle
Avg
Hotpot
2Wiki
MuSiQue
Bamboogle
Avg
ReAct
23.04
26.11
6.37
19.20
18.68
12.18
7.38
4.22
20.00
10.95
GRPO
29.10
31.48
8.61
33.60
25.70
36.80
36.35
13.78
40.00
31.73
GSPO
29.09
31.85
8.52
32.80
25.57
36.48
36.73
13.61
39.20
31.50
TreePO
29.86
32.18
8.90
33.60
26.13
37.56
37.43
14.23
40.00
32.30
TreeOPSD
31.30
33.81
9.60
32.80
26.88
39.59
41.15
15.97
45.60
35.58
Appendix
Table 3: Performance comparison on Multi-Hop QA with smaller backbones. We train on HotpotQA and evaluate Qwen3-1.7B and Llama-3.2-3B-Instruct on four test sets. TreeOPSD consistently outperforms competitive baselines on average. The best results are indicated in bold .
Figure 6: Test accuracy during training. The two panels report TreeOPSD and GRPO with Qwen3-1.7B and Llama-3.2-3B-Instruct on the Multi-Hop QA average (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle). Higher curves indicate stronger final policies under identical sampling budgets.
Figure 7: Numerical studies of turn credit and the posterior map. (a) Repeated updates share one scalar policy score. TreeOPSD stops at the teacher, while TreePO keeps the collection-time coefficient. (b) Two useful children share a credit m . Pointwise credit is Vc , and listwise credit is the group-relative Vc−Vi . (c) The odds map with Forward KL preserves the parent gradient after tree backup. Reverse KL and a logistic map do not.
Figure 8: Interaction length and training efficiency. Panels (a) and (b) report the average number of assistant turns during evaluation and training on BrowseComp-Plus with Qwen3-30B-A3B-Thinking-2507. Panel (c) plots test accuracy against cumulative training time and shows TreeOPSD only within the total time used by GRPO. Higher curves in (c) indicate stronger final policies under identical training-time budgets.
Method
Multi-Hop QA
Deep Research
ReAct
24.13±0.05
8.80±0.26
GRPO
43.37±0.19
12.18±0.27
GSPO
43.15±0.19
12.17±0.32
TreePO
43.40±0.17
13.07±0.37
TreeOPSD
44.13±0.16
15.45±0.39
Appendix
Table 4: Mean ± std over five training seeds with Qwen3-4B-Thinking-2507. Averages use the same Multi-Hop QA and Deep Research protocol as Table 1 . TreeOPSD consistently outperforms competitive baselines on average. The best results are indicated in bold .
Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.
Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10
Tsinghua University · Meituan · Zhejiang University
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.
Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8
Zhejiang University · Meituan · Tsinghua University
Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.