Cooperative Multi-Agent Reinforcement Learning via Epigraph-Form Guided Exploration
Organizations: Seoul National University
Abstract
Discovering cooperative behavior in multi-agent reinforcement learning (MARL) is challenging due to the combinatorial complexity of joint state-action spaces, which hinders the emergence of coordinated behaviors from trial-and-error alone. Intrinsic rewards are often used to aid discovery, but naively combining them with team objectives can distort the learning signal, compromising task performance. In this paper, we propose EFXPLORER, a constrained exploration framework that maximizes exploration objectives subject to a constraint that preserves established task performance. We solve the resulting problem using an epigraph reformulation that introduces adaptive exploration budgets. This approach separates intrinsic rewards from task objectives and regulates exploration through task feasibility. To further encourage diverse and temporally extended exploration, we incorporate a successor distance-based intrinsic reward that captures long-horizon dependencies. Empirically, our method outperforms strong baselines and induces novel cooperative strategies across SMAX, VMAS, and MPE benchmark suites.
Figures & tables
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Episode Horizon | [ ] |
|---|---|---|
| MPE-corridor | 256 | [-512, 0] |
| MPE-tag | 256 | [-768, 640] |
| VMAS-balance | 250 | [0, 130] |
| VMAS-give way | 600 | [0, 400] |
| VMAS-wheel | 600 | [0, 500] |
| VMAS-transport | 600 | [0, 1500] |
| Parameter | MPE | VMAS | SMAX |
| discount factor | 0.99 | 0.99 | 0.99 |
| GAE coefficient | 0.95 | 0.9 | 0.95 |
| gradient clip norm | 0.6 | 10 | 0.6 |
| epsilon clip | 0.3 | 0.15 | 0.1 |
| coefficient of entropy | 0.015 | - | 0.01 |
| extrinsic value loss coefficient | 0.5 | - | 0.5 |
| Parameter | MPE | VMAS | SMAX |
|---|---|---|---|
| Successor distance network | |||
| latent dimension size | 64 | 128 | 256 |
| batch size | 512 | 512 | 512 |
| learning rate | |||
| number of epoch | 25 | 16 | 10 |
| number of residual blocks | 10 | 10 | 4 |
| Parameter | MPE | VMAS | SMAX |
|---|---|---|---|
| Successor distance network | |||
| latent dimension size | 64 | 128 | 126 |
| batch size | 512 | 512 | 512 |
| learning rate | |||
| number of epoch | 25 | 16 | 10 |
| number of residual blocks | 10 | 10 | 4 |
| Parameter | MPE | VMAS | SMAX |
|---|---|---|---|
| influence hidden dimension size | 64 | 64 | 128 |
| influence learning rate | |||
| curiosity hidden dimension size | 64 | 64 | 128 |
| curiosity learning rate | |||
| influence scale | 0.0005 (0.01 for tag) | 0.0005 | 0.0005 |
| curiosity scale | 0.0005 (0.01 for tag) | 0.0005 | 0.0005 |
| Parameter | MPE | VMAS | SMAX |
|---|---|---|---|
| number of parallel environments | 32 (16 for tag) | 60 | 64 |
| number of epoch | 6 | 1000 | 300 |
| RNN hidden state dim | 256 (64 for tag) | - | 128 |
| initial entropy temperature | 0.1 | 0.1 | |
| target entropy ratio | 0.5 | 0.2 | 0.05 |
| value loss | plain MSE loss | smooth | plain MSE loss |
| Parameter | MPE | VMAS | SMAX |
|---|---|---|---|
| desired SND | 0.1 | 0.1 (balance, transport) 1.0 (wheel, give way) | 0.1 |
| influence learning rate | |||
| learning rate | 0.0005 | 0.0005 | 0.0005 |
| bootstrap from desired SND | true | true | true |
| Environment | ||||
|---|---|---|---|---|
| VMAS-balance | ||||
| VMAS-transport | ||||
| VMAS-wheel |
| Environment | Method | HP-avg. return | Steps to (M) | Seed variance | HP variance |
|---|---|---|---|---|---|
| MPE-tag | MAPPO-Linear | 0.99 | 11.70 | 0.0098 | 0.0011 |
| MAPPO-Lagrangian | 0.69 | N/A | 0.0007 | 0.0031 | |
| EFXPLORER | 1.00 | 11.25 | 0.0069 | 0.0009 | |
| VMAS-wheel | MAPPO-Linear | 1.00 | 2.61 | 0.0401 | 0.0365 |
| MAPPO-Lagrangian | 0.42 | 5.50 | 0.1452 | 0.5431 | |
| EFXPLORER | 1.00 | 2.25 | 0.0309 | 0.0178 |
| Env. steps | Mean budget, | Mean budget, | CV, | CV, |
|---|---|---|---|---|
| 1M | 0.94 | 0.98 | 7.4% | 8.0% |
| 10M | 1.92 | 2.01 | 5.0% | 9.1% |
| 20M | 2.58 | 1.74 | 5.3% | 12.5% |
| Environment | Steps (M) | Variant | Normalized return | ||
|---|---|---|---|---|---|
| VMAS-transport | 3 | EFXPLORER | 0.315 | 0.288 | 0.018 |
| 0.250 | 0.025 | 0.004 | |||
| 6 | EFXPLORER | 0.563 | 0.801 | 0.762 | |
| 0.295 | 0.207 | 0.164 | |||
| 9 | EFXPLORER | 0.392 | 0.764 | 1.065 | |
| 0.409 | 0.445 | 0.797 |
| Env. steps | Method | Normalized return | Episode length | |
|---|---|---|---|---|
| 5M | EFXPLORER | 75% | 0.23 | 40.0 |
| MAPPO-Linear ( ) | 33% | 0.20 | 14.5 | |
| MAPPO-Linear ( ) | 28% | 0.23 | 13.9 | |
| 10M | EFXPLORER | 67% | 0.34 | 45.3 |
| MAPPO-Linear ( ) | 31% | 0.28 | 14.7 | |
| MAPPO-Linear ( ) | 25% | 0.28 | 14.1 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,835 papers