Reinforcement learning improves problem solving in large language model multi-agent systems. Existing approaches overlook communication knowledge or compress it into scalar rewards, leaving coarse credit with little guidance on which decisions each agent should reinforce or revise. OPSD4MAS converts future communication into fine-grained token-level credit through On Policy Self Distillation (OPSD). An external model extracts facts and assesses their trajectory relations for coarse-grained filtering, followed by fine-grained leave-one-out filtering of target-token influence against agent-wise outcome advantages. The fact-conditioned policy supplies token-level credit, combined with GRPO through a linearly annealed joint objective. Our method improves mathematical pass@4 over GRPO by up to 20.00 percentage points. Ablations support complete filtering, while frozen-history injection diagnostics support fact utility. Together, these results support combining agent-level outcome guidance with communication-derived token-level credit. Code is available at https://github.com/Camel-Prince/OPSD4MAS.
Figures & tables
Figure 1: (a) Two scalar-reward RL routes versus our RLVR–OPSD combination. (b) Test pass@1/pass@4 for CoMAS, DrMAS (GRPO/DAPO), and OPSD4MAS. FW2/FW4 denote OPSD4MAS with future windows of two/four responses, both trained with OPSD and GRPO.
Figure 2: OPSD4MAS combines agent-wise GRPO credit with token-level OPSD credit. Future communication is converted into facts, ranked by association with query-normalized trajectory advantage A(τj) , and filtered by LOO against DrMAS agent-wise advantage Air , also used by GRPO. The selected set Fi conditions each role’s privileged teacher. Probability displays and numerical examples are illustrative.
Method
AMC23
AIME24
AIME25
MATH-500
Olympiad
Minerva
p1
p4
p1
p4
p1
p4
p1
p4
p1
p4
p1
p4
Base Model
60.24
74.70
25.83
43.33
19.17
23.33
82.40
90.40
41.04
53.04
24.72
31.99
Frozen-weight MAS: no LLM parameter updates
Solver–Verifier MAS
56.63
71.08
22.50
36.67
16.67
26.67
83.50
91.80
46.81
59.26
33.27
41.18
AFlow
61.44
77.11
27.50
46.67
22.50
26.67
79.15
88.60
45.44
58.07
30.61
39.34
GPTSwarm
61.44
74.70
20.83
36.67
20.83
30.00
80.75
89.60
44.33
54.22
24.72
34.19
Table 1: Test performance (%): p1 / p4 denote pass@1/pass@4. Bold underlined and italic underlined values rank first and second in each column. All nine methods are evaluated on six datasets. Olympiad denotes English text-only OlympiadBench mathematics. Evaluation protocols and scoring qualifications appear in Section 5.1 and Appendix B.1 .
Condition
E
A
L
p1
p4
No OPSD
–
–
–
49.63
66.60
Raw Commun.
–
–
–
33.24
50.46
Unfiltered Facts
✓
–
–
42.57
63.97
Coarse-filtered Facts
✓
✓
–
42.99
65.18
Full
✓
✓
✓
64.91
82.74
Table 2: Training ablations: mean pass@1/pass@4 ( p1/p4 , %) on AMC23, AIME24, and AIME25, with equal dataset weights. Raw Commun. abbreviates Raw future communication. E/A/L : extraction/Association/LOO. No OPSD and Full reuse GRPO and FW4 from Table 1 .
Figure 3: Quantity-matched control design: Relaxed has the largest matched–shuffled gains. Nonzero y-axis ranges differ across panels, so compare numerical values across checkpoints. Protocol: Appendix B.3 .
Figure 4: Filtering-strength trend (60–90% axis): Relaxed is best at all checkpoints. None is the no-injection reference, and Strict through All relax LOO. Category spacing does not encode fact quantity.
Figure 5: Selected cases connect filtering decisions to fact content: (a) no actionable repair, (b) a falsifiable error, (c) a concrete correction. Queries and facts are abridged. Boxed facts are proposed privileged contexts for the named responses. DROP/REJECT denotes non-retention, not injection. A/E/N/P denote association, RMS effect, net alignment, and directional precision. Content checks are independent of logged selection, with sources and scores in Appendix D .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
AMC23
AIME24
AIME25
p1
p4
p1
p4
p1
p4
No OPSD
68.07
83.13
48.33
63.33
32.50
53.33
Raw future communication
57.22
74.70
25.00
46.67
17.50
30.00
Unfiltered Facts
63.55
81.92
40.83
66.67
23.33
43.33
Coarse-filtered Facts
65.63
85.54
34.17
63.33
29.17
46.67
Full
82.23
91.57
62.50
83.33
50.00
73.33
Appendix
Table 3: Per-dataset ablation results (%). Displayed values are rounded, whereas Table 2 averages the original supplied precision, not these rounded cells.
Panel / ID
Step
Fact ID
Fact / traj. line
Source / target
Decision
(a) / B3
130
3a722f49767d87d137954ec5
500 / 230
2 / 1
Association rejected
(b) / C1
130
4c0f1ebe82fad73ec9e52a59
652 / 248
3 / 2
LOO rejected
(c) / A1
130
3fa5b26b4d135ef12d49acb0
558 / 15
5 / 4
LOO aligned
Appendix
Table 4: Sources of the cases in Figure 5 . Line numbers index the fact-card and trajectory exports from one. Source/target gives the response positions within each trajectory.
Reinforcement learning with verifiable rewards ordinarily spreads a single advantage over a whole trajectory, a granularity too coarse to isolate the few decisions on which success in a long, multi-turn interaction actually turns. Privileged self-distillation supplies denser supervision, but it leaves unanswered how a set of local teacher-student discrepancies should be converted into credit along a sequence. We introduce AgentOPSD, a critic-free scheme that assigns credit recursively, one interaction turn at a time. Inside a turn, the token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed, and that sum acts as evidence in a recursive Bayesian update carried out in log-odds space; the resulting revisions reallocate the outcome-derived advantage across turns through a multiplier that is bounded and preserves its sign. With Qwen2.5-3B and Qwen2.5-7B as policy backbones on ALFWorld, the method improves consistently on GRPO and on self-distillation baselines, and attains 89.1% success at the 7B scale. Ablations attribute the gain to aggregating evidence at turn boundaries and to making the belief revision depend on history, rather than to the local gap by itself.
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.
Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8
Zhejiang University · Meituan · Tsinghua University
Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.
Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10
Tsinghua University · Meituan · Zhejiang University