Implicit Credit Learning for Sparse-Reward Multi-Agent Reinforcement Learning
Organizations: School of Computing and Information Systems Singapore Management University, Singapore · University of Oregon Eugene, Oregon United States
Abstract
When a cooperative team receives feedback only at the end of an episode, a single outcome must guide many agents' earlier actions. Learning a dense team reward helps, but its decomposition does not automatically give each agent a useful policy update. We introduce IMAP, an online method that fits a shared Q/V reward model by comparing completed episodes according to their observed terminal outcomes. Local Q/V residuals provide agent-specific rewards; the same normalized weights combine them for a centralized critic and scale local advantages for decentralized PPO actors. We characterize when these local updates follow the learned team objective. With full returns and frozen reward heads, a sharp bound relates their gradient error to variation in future Q/V compatibility; an exact identity identifies additional continuation terms for general GAE. In experiments with terminal-only feedback, IMAP's local-actor configuration attains the highest reported mean among the compared methods on all nine SMACv2 scenarios and four MAMuJoCo tasks, and on two of three SMACLite scenarios. Controlled relay interventions confirm the predicted changes in actor credit while holding the learned team reward fixed. The gradient result concerns the fitted objective; benchmark returns measure performance on the original task outcomes.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | SMACv2 | MAMuJoCo |
| Actor / critic learning rate | ||
| Discount / GAE | / | / |
| PPO clipping / epochs | / 10 | / 10 |
| PPO minibatch size | 256 | 512 |
| Entropy coefficient | ||
| Parallel environments | 8 | 8 |
| Output shift | Mean | Sample SD | Seed range |
|---|---|---|---|
| Joint Q/V | 0.00605 | 0.00275 | |
| Q-only | 0.11496 | 0.05218 |
| Positive alignment | Negative alignment | Oracle-scale certified | |
| 60 | 0 | 0 | |
| 60 | 0 | 0 | |
| 60 | 0 | 0 | |
| 60 | 0 | 6 | |
| 60 | 0 | 0 | |
| 57 | 3 | 0 |
Cooperative MARL
Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.
See all 267 papers