Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization

Seongmin Kim; Giseung Park; Woojun Kim; Jiwon Jeon; Seungyul Han; Youngchul Sung

cs.MAMar 3, 2026

Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization

Authors: Seongmin Kim, Giseung Park, Woojun Kim, Jiwon Jeon, Seungyul Han, Youngchul Sung

Organizations: KAIST Daejeon, Korea · University of Toronto Toronto, Ontario, Canada · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · UNIST Ulsan, Korea

Abstract

In this paper, we propose a novel framework for multi-agent reinforcement learning that enhances sample efficiency and coordination through accurate per-agent advantage estimation. The core of our approach is Generalized Per-Agent Advantage Estimator (GPAE), which employs a per-agent value iteration operator to compute precise per-agent advantages. This operator enables stable off-policy learning by indirectly estimating values via action probabilities, eliminating the need for direct Q-function estimation. To further refine estimation, we introduce a double-truncated importance sampling ratio scheme. This scheme improves credit assignment for off-policy trajectories by balancing sensitivity to the agent's own policy changes with robustness to non-stationarity from other agents. Experiments on benchmarks demonstrate that our approach outperforms existing approaches, excelling in coordination and sample efficiency for complex scenarios.

Figures & tables

Advantage	Credit	Off-Policy	Additional
Advantage	Assignment	Off-Policy	Bias( $\lambda=1$ )
GAE	No	No	No
COMA	1-step	No	No
DAE	$n$ -step	No	Yes
GPAE (Ours)	$\boldsymbol{n}$ -step	Yes	No

Table 1. Comparison of MAPG advantage estimators

Panel of Figure 1 — Figure 1. 1(a) Advantage gap $\Delta A$ indicating how strongly each method penalizes the anomalous agent. Higher $\Delta A$ means more effective credit assignment against the “stop” action. 1(b) Average win rates, illustrating overall learning stability and performance. (a) Advantage gap $\DeltaA$ indicating how strongly each method penalizes the anomalous agent. Higher $\DeltaA$ means more effective credit assignment against the “stop” action. (b) Average win rates, illustrating overall learning stability and performance.

Panel of Figure 2 — Figure 2. Off-policy correction comparisons in the SMAX-1s1z task. 2(a) Distance from true $\rho^{i}$ , 2(b) Distance from joint true $\boldsymbol{\rho}$ , 2(c) Gap between (a) and (b), 2(d) Final performance. Legend is shared across all plots. DT-ISR demonstrates the lowest gap $\Delta c^{i}$ and the highest performance. Off-policy correction comparisons in the SMAX-1s1z task. \subref{subfig:dt_diff1} Distance from true $\rho^i$ , \subref{subfig:dt_diff2} Distance from joint true $\boldsymbol{\rho}$ , \subref{subfig:dt_diff_diff} Gap between (a) and (b), \subref{subfig:dt_performance} Final performance. Legend is shared across all plots. DT-ISR demonstrates the lowest gap $\Deltac^i$ and the highest performance.

Panel of Figure 3 — Figure 3. Illustration of the two evaluation environments. Illustration of the two evaluation environments.

Domain	Task	GPAE (off)	GPAE (on)	MAPPO	DAE	COMA	QMIX	VDN
SMAX	3s5z_vs_3s6z	87.3 $\pm$ 3.9	6.1 $\pm$ 2.0	2.6 $\pm$ 0.7	6.5 $\pm$ 2.0	0.0 $\pm$ 0.0	1.3 $\pm$ 0.8	0.5 $\pm$ 0.4
	5m_vs_6m	93.7 $\pm$ 1.0	8.8 $\pm$ 0.9	3.1 $\pm$ 1.8	4.2 $\pm$ 2.5	0.3 $\pm$ 0.6	3.9 $\pm$ 0.6	3.2 $\pm$ 1.9
	10m_vs_11m	98.5 $\pm$ 2.7	47.5 $\pm$ 2.8	36.4 $\pm$ 2.4	25.0 $\pm$ 8.2	1.3 $\pm$ 0.3	32.8 $\pm$ 0.9	22.6 $\pm$ 8.0
	6h_vs_8z	99.5 $\pm$ 0.1	99.8 $\pm$ 0.2	99.0 $\pm$ 0.2	98.9 $\pm$ 0.1	80.4 $\pm$ 4.9	75.3 $\pm$ 5.3	96.1 $\pm$ 2.6
	smacv2_5_units	81.0 $\pm$ 1.3	80.0 $\pm$ 1.7	75.2 $\pm$ 0.9	77.1 $\pm$ 1.2	54.0 $\pm$ 1.9	59.5 $\pm$ 3.3	39.4 $\pm$ 2.9
	smacv2_10_units	75.0 $\pm$ 1.3	69.0 $\pm$ 1.7	63.5 $\pm$ 2.5	65.7 $\pm$ 0.6	47.4 $\pm$ 0.4	62.1 $\pm$ 1.6	29.5 $\pm$ 2.4

Table 2. Performance comparison after learning

1\times 10^{7}

timesteps. The upper section shows the average win rate performances in SMAX tasks, with values presented as mean (standard deviation). The lower section shows the episode returns in MABrax tasks, with values presented as mean

\pm

standard deviation. For each task, we conducted experiments with 5 random seeds. The bold numerals denote the highest performance for each task. Note that QMIX and VDN are not applicable to continuous tasks, and therefore, no results are available for MABrax tasks.

Panel of Figure 4 — Figure 4. (a) and (b) represent the learning curve of the aggregated performance for all tasks on SMAX and MABrax, respectively. (a) and (b) represent the learning curve of the aggregated performance for all tasks on SMAX and MABrax, respectively.

method	5m_vs_6m	3s5z_vs_3s6z
DT-ISR	93.7	87.3
ST-ISR	44.4	80.8
IT-ISR	58.6	83.7
No Correction	34.5	74.9

Table 3. Ablation study on DT-ISR.

$\eta$	5m_vs_6m		3s5z_vs_3s6z
$\eta$	win rate	$\Delta c^{i}$	win rate	$\Delta c^{i}$
1.0	95.2	-0.00154	84.5	-0.00105
1.056	93.7	-0.00409	87.3	-0.0067
1.1	93.1	-0.00553	83.2	-0.00918
1.15	39.4	-0.00561	80.3	-0.00938

Table 4. Impact of

\eta

on DT-ISR weights.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Brief overview of GPAE framework Brief overview of GPAE framework

Task Name	Ally Units	Enemy Units
2s3z	2 Stalkers, 3 Zealots	2 Stalkers, 3 Zealots
3s5z	3 Stalkers, 5 Zealots	3 Stalkers, 5 Zealots
5m_vs_6m	5 Marines	6 Marines
10m_vs_11m	10 Marines	11 Marines
3s5z_vs_3s6z	3 Stalkers, 5 Zealots	3 Stalkers, 6 Zealots
6h_vs_8z	6 Hydralisks	8 Zealots

Appendix

Table 5. Evaluated tasks in SMAX. These scenarios involve discrete actions and cooperative combat against adversarial teams.

Panel of Figure 6 — Figure 6. Figures represent the learning curve of the average win rate performance for 6 tasks on SMAX, and the learning curve of the average episode return performance for 6 tasks on MABrax. Figures represent the learning curve of the average win rate performance for 6 tasks on SMAX, and the learning curve of the average episode return performance for 6 tasks on MABrax.

method	GPAE	GAE	DAE
DT-ISR	93.7	35.1	29.4
ST-ISR	44.4	10.9	9.7
IT-ISR	58.6	9.6	14.5
No Correction	34.5	10.6	10.8

Appendix

Table 6. Ablation study on DT-ISR.

Parameter	SMAX	MABrax
# training timesteps	$1\times 10^{7}$	$1\times 10^{7}$
# parallel environments	$128$	$64$
# rollout steps	$128$	$128$
Adam learning rate	$5\times 10^{-4}$	$1\times 10^{-3}$
Anneal learning rate	True	True
Update epochs	$5$	$5$

Appendix

Table 7. Hyperparameter settings of PPO and Off-Policy PPO in the SMAX and MABrax environment

Parameter	SMAX
# training timesteps	$1\times 10^{7}$
# parallel environments	$16$
# rollout steps	$128$
Adam learning rate	$5\times 10^{-5}$
Anneal learning rate	True
Buffer size	$5\times 10^{3}$

Appendix

Table 8. Hyperparameter settings of QMIX and VDN in the SMAX environment

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning
Apr 9, 2026Teng Pang, Zhiqiang Dong, Yan Zhang +3Offline MARL Cooperative MARL
School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China
Adaptive Robust Estimator for Multi-Agent Reinforcement Learning
Mar 23, 2026Zhongyi Li, Wan Tian, Jingyu Chen +8Multi-Agent Reinforcement Learning RL for Language Model Reasoning
Beihang University · Peking University · Chinese Academy of Sciences +1
Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Flow Matching Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design

Apr 9, 2026J/K move · Enter open · S save

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Teng Pang, Zhiqiang Dong, Yan Zhang, Rongjian Xu+2

School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China

Offline MARL Cooperative MARL Flow Matching

Open paper AskPDF arXiv

Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets, requiring a trade-off between maximizing global returns and mitigating distribution shift from offline data. Recent studies use diffusion or flow generative models to capture complex joint policy behaviors among agents; however, they typically rely on multi-step iterative sampling, thereby reducing training and inference efficiency. Although further research improves sampling efficiency through methods like distillation, it remains sensitive to the behavior regularization coefficient. To address the above-mentioned issues, we propose Value Guidance Multi-agent MeanFlow Policy (VGM

^2

P), a simple yet effective flow-based policy learning framework that enables efficient action generation with coefficient-insensitive conditional behavior cloning. Specifically, VGM

^2

P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning. Additionally, to improve policy expressiveness and inference efficiency in multi-agent scenarios, it leverages classifier-free guidance MeanFlow for both policy training and execution. Experiments on tasks with both discrete and continuous action spaces demonstrate that, even when trained solely via conditional behavior cloning, VGM

^2

P efficiently achieves performance comparable to state-of-the-art methods.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Adaptive Robust Estimator for Multi-Agent Reinforcement Learning

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies