cs.AINov 12, 2026

Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation

Authors: Zixu Wang, Yu Zhao, Xing Zheng, Bingbing Xu, Tian Xueyun, JiaTang Luo, Weiqiang Wang, Bowen Song, 

Organizations: Institute of Computing Technology, Chinese Academy of Sciences · Ant Group · University of Chinese Academy of Sciences

Abstract

Reinforcement learning improves problem solving in large language model multi-agent systems. Existing approaches overlook communication knowledge or compress it into scalar rewards, leaving coarse credit with little guidance on which decisions each agent should reinforce or revise. OPSD4MAS converts future communication into fine-grained token-level credit through On Policy Self Distillation (OPSD). An external model extracts facts and assesses their trajectory relations for coarse-grained filtering, followed by fine-grained leave-one-out filtering of target-token influence against agent-wise outcome advantages. The fact-conditioned policy supplies token-level credit, combined with GRPO through a linearly annealed joint objective. Our method improves mathematical pass@4 over GRPO by up to 20.00 percentage points. Ablations support complete filtering, while frozen-history injection diagnostics support fact utility. Together, these results support combining agent-level outcome guidance with communication-derived token-level credit. Code is available at https://github.com/Camel-Prince/OPSD4MAS.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Self-Distilled Agentic Reinforcement Learning

    May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RLSelf-Distillation

  2. AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

    Aug 6, 2026Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10Agentic RLReinforcement Learning