SpecMAGIC: Specification-Guided Multi-Task Generalization in Multi-Agent Reinforcement Learning

Zijian Guo; İlker Işık; H. M. Sabbir Ahmad; Wenchao Li

cs.AIOct 21, 2026

SpecMAGIC: Specification-Guided Multi-Task Generalization in Multi-Agent Reinforcement Learning

Authors: Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li

Organizations: Division of Systems Engineering, Boston University · Department of Electrical and Computer Engineering, Boston University

Abstract

Multi-task generalization is an important yet underexplored problem in multi-agent reinforcement learning (MARL). Many real-world applications require multi-agent systems to generalize across complex, temporally extended tasks with diverse inter-agent coordination requirements. We introduce a specification language that combines temporal-logic task specifications with operators for expressing such requirements. We propose SpecMAGIC, a hierarchical framework that leverages the automaton structure of task specifications for subtask decomposition. A centralized high-level planner identifies and allocates compatible subtask sequences across agents, while decentralized low-level policies execute individual subtasks. This enables zero-shot generalization to new tasks by composing reusable subtask policies according to new specifications, without retraining. We establish soundness and probabilistic completeness of the high-level planner. Experiments on multi-agent navigation tasks show that SpecMAGIC substantially outperforms existing approaches on unseen task specifications and maintains strong performance as the number of agents increases.

Figures & tables

Figure 1: Overview of the proposed hierarchical framework. Specifications are decomposed into reach-avoid subtasks, from which a centralized sampling-based planner constructs a feasible subtask sequence for execution by decentralized low-level policies. Solid and dotted blocks and arrows indicate components used at test time and during training, respectively.

Figure 2

$N$	$\|\mathcal{B}_{\varphi_{\texttt{global}}}\|$	$\|\mathcal{T}\|$	Iterations	$t_{\texttt{path}}$
3	$98.7_{\pm 42.5}$	$7.9_{\pm 3.3}$	$64.7_{\pm 100.5}$	$0.02_{\pm 0.02}$
5	$1789.3_{\pm 1267.3}$	$10.4_{\pm 5.2}$	$320.4_{\pm 511.7}$	$0.17_{\pm 0.24}$
10	$>118814.3$	$12.6_{\pm 5.2}$	$477.4_{\pm 530.2}$	$8.64_{\pm 0.77}$

Table 2: High-level planner scalability. We report the product-automaton size

|\mathcal{B}_{\varphi_{\texttt{global}}}|

, the constructed tree size

|\mathcal{T}|

, sampling iterations, and planning time

t_{\texttt{path}}

, averaged over 10 seeds.

Figure 3: Performance under different numbers of agents during training and evaluation. We report the average success rate $\eta_{s}$ , violation rate $\eta_{v}$ , and number of steps $\mu$ to specification satisfaction for $N_{\texttt{train}},N_{\texttt{eval}}\in\{3,5,10\}$ . Error bars indicate one standard deviation.

Figure 4: Average success rate $\eta_{s}$ , violation rate $\eta_{v}$ , and average number of steps $\mu$ to satisfy the IND specifications $\Phi_{\texttt{ind}}$ , $\Phi_{\texttt{ind/dep}}$ , $\Phi_{\texttt{ind/cnt}}$ , and $\Phi_{\texttt{ind/dep/cnt}}$ listed in Table 3 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Visualizations of the Zone environments with $N=3$ , $5$ , and $10$ agents, from left to right. The positions of agents and zones are randomly sampled each episode.

Specification types		IND		OOD
Finite- horizon (3 agents)	sa	$\varphi_{1}$	$(\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{b}\lor\mathsf{g}))\land(\ \mathsf{F}\ \mathsf{m})$	$\varphi_{16}$	$\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{b}\land\ \mathsf{F}\ (\neg\mathsf{y}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})))$
	sa	$\varphi_{2}$	$(\ \mathsf{F}\ \mathsf{b})\land(\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{y}))$	$\varphi_{17}$	$\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ (\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{y}\land\ \mathsf{F}\ \mathsf{b})))$
	sa	$\varphi_{3}$	$\neg(\mathsf{m}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{b}\land\ \mathsf{F}\ \mathsf{g})$	$\varphi_{18}$	$\neg(\mathsf{m}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{b}\land\ \mathsf{F}\ (\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})))$
	sa	$\varphi_{4}$	$\neg\mathsf{m}\ \mathsf{U}\ ((\mathsf{b}\lor\mathsf{y})\land\ \mathsf{F}\ \mathsf{g})$	$\varphi_{19}$	$\neg\mathsf{m}\ \mathsf{U}\ ((\mathsf{b}\lor\mathsf{y})\land\ \mathsf{F}\ (\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})))$
	sa	$\varphi_{5}$	$\neg(\mathsf{b}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})$	$\varphi_{20}$	$\neg(\mathsf{b}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ (\neg\mathsf{g}\ \mathsf{U}\ (\mathsf{m}\land\ \mathsf{F}\ \mathsf{y})))$
	sa	$\varphi_{6}$	$(\ \mathsf{F}\ \mathsf{m})\land(\neg\mathsf{y}\ \mathsf{U}\ (\mathsf{b}\lor\mathsf{g}))$	$\varphi_{21}$	$\neg\mathsf{y}\ \mathsf{U}\ ((\mathsf{b}\lor\mathsf{g})\land\ \mathsf{F}\ (\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{y}\land\ \mathsf{F}\ \mathsf{b})))$

Appendix

Table 3: Specifications used for evaluation.

\mathsf{b}

\mathsf{g}

\mathsf{m}

, and

\mathsf{y}

denote

\mathsf{blue}

\mathsf{green}

\mathsf{magenta}

, and

\mathsf{yellow}

, respectively.

\mathsf{sa}

and

\mathsf{ma}

denote specifications for single-agent and multi-agent behaviors, respectively. Top: finite-horizon specifications with 3 agents, with results in Table 1 . Bottom left: finite-horizon specifications with 5 and 10 agents for the scalability study, with results in Figure 3 . Bottom right: infinite-horizon specifications with 3 agents, with results in Table 5 .

Method	Specifications	$\eta_{s}\uparrow$	$\eta_{v}\downarrow$	$\mu\downarrow$
ACC-MARL	Reach-Only	$0.77_{\pm 0.08}$	$0.19_{\pm 0.07}$	$261.27_{\pm 39.58}$
ACC-MARL	Reach-Avoid	$0.60_{\pm 0.09}$	$0.37_{\pm 0.08}$	$226.06_{\pm 38.71}$
Ours	Reach-Only	$0.97_{\pm 0.02}$	$0.02_{\pm 0.02}$	$139.92_{\pm 14.16}$
Ours	Reach-Avoid	$0.98_{\pm 0.02}$	$0.01_{\pm 0.01}$	$148.40_{\pm 15.47}$

Appendix

Table 4: Evaluation results on simple specifications: success rate

\eta_{s}

, violation rate

\eta_{v}

, and average number of steps

\mu

\uparrow

: higher is better;

\downarrow

: lower is better. Each value is averaged over 5 seeds, with 100 trajectories per seed.

Method	Metrics	$\Phi_{\mathrm{indep}}$	$\Phi_{\mathrm{indep/req}}$	$\Phi_{\mathrm{indep/cnt}}$	$\Phi_{\mathrm{indep/req/cnt}}$	Average
Ours	$\mu_{\texttt{acc}}\uparrow$	$10.71_{\pm 1.86}$	$5.98_{\pm 0.87}$	$10.48_{\pm 1.72}$	$5.55_{\pm 0.84}$	$8.18_{\pm 2.80}$
Ours	$\eta_{v}\downarrow$	$0.05_{\pm 0.03}$	$0.15_{\pm 0.03}$	$0.05_{\pm 0.02}$	$0.16_{\pm 0.04}$	$0.10_{\pm 0.06}$

Appendix

Table 5: Evaluation results of infinite-horizon tasks: violation rate

\eta_{v}

and average number of visits to accepting states

\mu_{\text{acc}}

on infinite-horizon tasks listed in Table 3 .

\uparrow

: higher is better;

\downarrow

: lower is better. Each value is averaged over 5 seeds, with 100 trajectories per seed.

Figure 6: Performance as the complexity of inter-agent coordination specifications increases. The top row varies the number of dependency operators $n=|\texttt{Req}|$ in $\varphi_{\texttt{dep}}$ , while the bottom row varies the number of count operators $n=|\texttt{Sync}|+|\texttt{Async}|$ in $\varphi_{\texttt{cnt}}$ . We report the success rate $\eta_{s}$ , violation rate $\eta_{v}$ , average number of steps $\mu$ to satisfy a specification, and planning time $t_{\texttt{path}}$ . Solid lines show the mean, and shaded regions indicate one standard deviation.

Panel of Figure 7 — Figure 7: Visualization of the trajectory of specification $\{\varphi_{4},\varphi_{5},\varphi_{6},\varphi_{11},\varphi_{14}\}$ . For each subplot, $t$ denotes the timestep, and the set denotes the propositions that are true at the current timestep.

Panel of Figure 8 — Figure 8: Visualization of the planning process of the high-level sampling-based planner for specification $\{\varphi_{4},\varphi_{5},\varphi_{6},\varphi_{11},\varphi_{14}\}$ . Each subplot shows the planning state at timestep $t$ . Yellow blocks denote the selected subtask sequence, while gray blocks denote sampled nodes that are not part of the selected sequence.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 371 papers

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning
May 3, 2026Haohan Yu, Jinmiao Cong, Shengzhi Wang +2Multi-Agent Coordination Intrinsic Reward Methods
1Dalian University of Technology · 2Microsoft, Beijing, China
MagicAgent: Towards Generalized Agent Planning
Feb 22, 2026Xuhui Ren, Shaokang Dong, Chen Yang +21LLM Agent Planning LLM Agent Training
Honor Device Co., Ltd · Fudan University
Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
Sep 3, 2026Oussama Hidaoui, Omer Ebead, Ulrich Armel Mbou Sob +14Zero-Shot Learning Reinforcement Learning
InstaDeep · AIMS · Stellenbosch University

May 3, 2026J/K move · Enter open · S save

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning

Haohan Yu, Jinmiao Cong, Shengzhi Wang, Lu Wang+1

1Dalian University of Technology · 2Microsoft, Beijing, China

Multi-Agent Coordination Intrinsic Reward Methods Reinforcement Learning Cooperative MARL Causal RL Reward Design for RL+1

Open paper AskPDF arXiv

A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals requires estimating how one agent's current action affects its teammates over future interaction steps. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that estimates multi-step action effects between agents and selectively converts them into intrinsic rewards. MAGIC uses counterfactual action interventions to compare teammate futures under factual and counterfactual branches, and introduces a gate based on advantage to direct exploration toward beneficial behaviors aligned with the task goal. Experiments on Multi-Agent Particle Environments (MPE) and StarCraft micromanagement benchmarks (SMAC and SMACv2) show that MAGIC consistently outperforms leading prior methods, with average relative final performance improvements of 26.9% and 10.1%, respectively.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning

MagicAgent: Towards Generalized Agent Planning

Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning