cs.LGSep 26, 2025

Implicit Credit Learning for Sparse-Reward Multi-Agent Reinforcement Learning

Authors: The Viet Bui, Tien Mai, Hong Thanh Nguyen

Organizations: School of Computing and Information Systems Singapore Management University, Singapore · University of Oregon Eugene, Oregon United States

Abstract

When a cooperative team receives feedback only at the end of an episode, a single outcome must guide many agents' earlier actions. Learning a dense team reward helps, but its decomposition does not automatically give each agent a useful policy update. We introduce IMAP, an online method that fits a shared Q/V reward model by comparing completed episodes according to their observed terminal outcomes. Local Q/V residuals provide agent-specific rewards; the same normalized weights combine them for a centralized critic and scale local advantages for decentralized PPO actors. We characterize when these local updates follow the learned team objective. With full returns and frozen reward heads, a sharp bound relates their gradient error to variation in future Q/V compatibility; an exact identity identifies additional continuation terms for general GAE. In experiments with terminal-only feedback, IMAP's local-actor configuration attains the highest reported mean among the compared methods on all nine SMACv2 scenarios and four MAMuJoCo tasks, and on two of three SMACLite scenarios. Controlled relay interventions confirm the predicted changes in actor credit while holding the learned team reward fixed. The gradient result concerns the fitted objective; benchmark returns measure performance on the original task outcomes.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning

    Jan 29, 2026Bang Giang Le, Viet Cuong TaCredit Assignment in RLCredit Assignment

  2. ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

    Jun 29, 2026Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara +4Multi-Agent System OptimizationMulti-Agent Coordination

  3. HySTAR: Anchored Hypergraphs for Stable Credit Assignment in Cooperative Multi-Agent Reinforcement Learning

    Sep 25, 2026Xinglong Luo, Yuding Zhang, Yuheng Kuang +5Cooperative MARLMulti-Agent Reinforcement Learning