GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization

Sihan Zhou; Tiantian He; Yifan Lu; Yaqing Hou; Yew-Soon Ong

cs.MAApr 1, 2026

GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization

Authors: Sihan Zhou, Tiantian He, Yifan Lu, Yaqing Hou, Yew-Soon Ong

Organizations: College of Computer Science and Technology, and Key Laboratory of Social Computing and Cognitive Intelligence (Ministry of Education), Dalian University of Technology, China · Center for Frontier AI Research, Institute of High Performance Computing, Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research (A*STAR), Singapore · School of Computer Science and Engineering, Nanyang Technological University, Singapore

Abstract

Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations. Existing approaches like Centralized Training with Decentralized Execution (CTDE) and sequential update schemes mitigate this issue. However, since the perception of the policies of other agents remains dependent on sampling environmental interaction data, the agent essentially operates in a passive perception state. This inevitably triggers equilibrium oscillations and significantly slows the convergence speed of the system. To address this issue, we propose Gradient Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution. The core mechanism of GRASP involves utilizing the independent gradients of agents to derive a defined consensus gradient, enabling agents to actively perceive policy updates and optimize team collaboration. Theoretically, we leverage the Kakutani Fixed-Point Theorem to prove that the consensus direction $u^*$ guarantees the existence and attainability of this equilibrium. Extensive experiments on StarCraft II Multi-Agent Challenge (SMAC) and Google Research Football (GRF) demonstrate the scalability and promising performance of the framework.

Figures & tables

Figure 1: Comparison of win rate learning curves between GRASP and baseline algorithms (MAPPO, HAPPO, and $\text{MA}^{2}\text{E}$ ) across six scenarios in SMAC and SMACv2. Solid lines represent the average win rate from multiple runs, while shaded areas indicate standard deviation.

Scenario	GRASP (Ours)	MAPPO	HAPPO	$\text{MA}^{2}\text{E}$
MMM2	0.967 $\pm$ 0.031	0.717 $\pm$ 0.218	0.509 $\pm$ 0.168	0.839 $\pm$ 0.021
3s5z vs 3s6z	0.783 $\pm$ 0.360	0.619 $\pm$ 0.345	0.461 $\pm$ 0.232	0.027 $\pm$ 0.010
5m vs 6m	0.909 $\pm$ 0.044	0.815 $\pm$ 0.075	0.350 $\pm$ 0.321	0.372 $\pm$ 0.114
Terran (10 agents)	0.467 $\pm$ 0.103	0.571 $\pm$ 0.000	0.233 $\pm$ 0.074	0.583 $\pm$ 0.057
Protoss (10 agents)	0.563 $\pm$ 0.045	0.491 $\pm$ 0.094	0.250 $\pm$ 0.069	0.391 $\pm$ 0.054
Zerg (5 agents)	0.491 $\pm$ 0.084	0.418 $\pm$ 0.083	0.363 $\pm$ 0.020	0.396 $\pm$ 0.065

Table 1: Win rate comparison (Mean

\pm

Std) on different scenarios. The best results are highlighted in bold .

Figure 2: Comparison of win rate learning curves between GRASP and baseline algorithms (MAPPO and $\text{MA}^{2}\text{E}$ ) across three scenarios in GRF. Solid lines represent the average win rate from multiple runs, while shaded areas indicate standard deviation.

Figure 3: Comparison of win rate learning curves between GRASP and MAPPO in the Simple Spread scenario of MPE. Solid lines represent the average win rate from multiple runs, while shaded areas indicate standard deviation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Comparison of win rate learning curves between GRASP-QMIX and QMIX across three scenarios in SMAC. Solid lines represent the average win rate from multiple runs, while shaded areas indicate standard deviation.

Figure 5: Comparison of win rate learning curves between GRASP and MAPPO in 27m_vs_30m scenario of MPE. Solid lines represent the average win rate from multiple runs, while shaded areas indicate standard deviation.

Hyperparameter	GRASP	MAPPO	HAPPO	$\text{MA}^{2}\text{E}$	GRASP+QMIX	QMIX
Learning Rate	$5\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Optimizer	Adam	Adam	Adam	Adam	Adam	Adam
PPO Epochs	5 – 10	5 – 10	5	N/A	N/A	N/A
Mini-batch Size	1 – 2	1 – 2	1	32	32	32
Gamma ( $\gamma$ )	0.99	0.99	0.99	0.99	0.99	0.99
Buffer size	64	64	64	5000	5000	5000

Appendix

Table 2: Hyperparameter settings for GRASP, MAPPO, HAPPO, and

\text{MA}^{2}\text{E}

Component	Layer Type	Hidden Units	Activation
Input Encoder	Full Connect Linear	64	-
Recurrent Layer	GRU	64	ReLU
Output Head	Full Connect Linear	$A_{\text{dim}}$	-

Appendix

Table 3: Agent Network Architecture Hyperparameters.

Layer	Type	Hidden Units	Activation
Input	Full Connect Linear	64	ReLU
Hidden 1	Full Connect Linear	64	ReLU
Hidden 2	Full Connect Linear	64	ReLU
Output	Full Connect Linear	1	-

Appendix

Table 4: Centralized Critic Network Architecture.

Component	Layer Type	Output Dim	Activation
Mixing Network (Inputs: $N$ Agent Q-values)
Mixing Layer 1	Full Connect Linear	32	ELU
Mixing Output	Full Connect Linear	1	-
Hypernetworks (Input: Global State $S$ )
Hyper-Input	Full Connect Linear	64	ReLU
Hyper-Weight 1	Full Connect Linear	$N\times 32$	Abs

Appendix

Table 5: Mixing Network and Hypernetwork Architecture.

Hyperparameters	Value
Batch size	32
Input embedding	24
The number of heads	4
The number of encoder layers	3
The number of decoder layers	2
Steps for fine-tuning	500

Appendix

Table 6: The hyperparameter settings for

\text{MA}^{2}\text{E}

Scenario Name	Scale	Description	Key Challenges	Testing Focus
Academy Counterattack Hard	Small (4 Agents)	Rapid transition from defense to offense at high speed.	High Dynamics, Split-second Decision Making	Responsiveness, Dynamic Control
Academy 3 Vs 1 With Keeper	Small (3 Agents)	Exploiting numerical advantage to dismantle defense.	Local Coordination, Spatial Reasoning	Tactical Cooperation, Precise Passing
Academy Corner	Large (11 Agents)	Full-team corner kick situation with crowded penalty area.	Massive State Space, Occlusion & Collisions	Scalability, Consensus in Noise

Appendix

Table 7: Characteristics of the Selected Google Research Football Scenarios. These scenarios were chosen to evaluate the GRASP framework across varying levels of difficulty, agent scale, and tactical requirements.

Scenario	Ally Units	Enemy Units	Difficulty	Step Limit
SMAC
5m_vs_6m	5 Marines	6 Marines	Hard	70
MMM2	10 Units 1 1 1 1 Medivac, 2 Marauders, 7 Marines	12 Units 2 2 2 1 Medivac, 3 Marauders, 8 Marines	Super Hard	180
3s5z_vs_3s6z	8 Units 3 3 3 3 Stalkers, 5 Zealots	9 Units 4 4 4 3 Stalkers, 6 Zealots	Hard	170
27m_vs_30m	27 Marines	30 Marines	Super Hard	180
8m_vs_9m	8 Marines	9 Marines	Hard	120

Appendix

Table 8: Detailed configurations for SMAC and SMACv2 scenarios.

Scenario	Ally Agents	Step Limit	Reward Type
Google Football
Counterattack Hard	4	400	Team collaboration provides
3 Vs 1 with Keeper	3	400	Team collaboration provides
Academy Corner	10	400	Team collaboration provides
MPE
Simple Spread	3	25	Individual rewards are consolidated into team rewards

Appendix

Table 9: Configurations for Google Football and MPE scenarios.

Component	Configuration Detail
CPU Model	13th Gen Intel(R) Core(TM) i5-13400
CPU Architecture	x86_64 (10 Cores, 16 Threads)
CPU Max Frequency	4.60 GHz
GPU Model	NVIDIA GeForce RTX 4090
GPU Memory	24564 MiB (24 GB GDDR6X)
Driver Version	535.183.01

Appendix

Table 10: Hardware and Software Configurations for the Training Environment.

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 244 papers

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
Jun 29, 2026Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara +4Multi-Agent System Optimization Multi-Agent Coordination
1KAIST · 2KRAFTON · University of Toronto +2
Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning
Feb 23, 2026Shan Yang, Yang LiuCooperative MARL Multi-Agent Reinforcement Learning
National University of Singapore
Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization
Mar 3, 2026Seongmin Kim, Giseung Park, Woojun Kim +3Multi-Agent Reinforcement Learning Sample-Efficient RL
KAIST Daejeon, Korea · University of Toronto Toronto, Ontario, Canada · Carnegie Mellon University Pittsburgh, Pennsylvania, USA +1

Jun 29, 2026J/K move · Enter open · S save

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara, Scott Sanner+3

1KAIST · 2KRAFTON · University of Toronto+2

Multi-Agent System Optimization Multi-Agent Coordination Decentralized MARL Policy Learning Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.

Abstract

Cooperative MARL

Explore similar work

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning

Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization