SpecMAGIC: Specification-Guided Multi-Task Generalization in Multi-Agent Reinforcement Learning

Zijian Guo; İlker Işık; H. M. Sabbir Ahmad; Wenchao Li

cs.AIOct 21, 2026

SpecMAGIC: Specification-Guided Multi-Task Generalization in Multi-Agent Reinforcement Learning

Authors: Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li

Organizations: Division of Systems Engineering, Boston University · Department of Electrical and Computer Engineering, Boston University

Abstract

Multi-task generalization is an important yet underexplored problem in multi-agent reinforcement learning (MARL). Many real-world applications require multi-agent systems to generalize across complex, temporally extended tasks with diverse inter-agent coordination requirements. We introduce a specification language that combines temporal-logic task specifications with operators for expressing such requirements. We propose SpecMAGIC, a hierarchical framework that leverages the automaton structure of task specifications for subtask decomposition. A centralized high-level planner identifies and allocates compatible subtask sequences across agents, while decentralized low-level policies execute individual subtasks. This enables zero-shot generalization to new tasks by composing reusable subtask policies according to new specifications, without retraining. We establish soundness and probabilistic completeness of the high-level planner. Experiments on multi-agent navigation tasks show that SpecMAGIC substantially outperforms existing approaches on unseen task specifications and maintains strong performance as the number of agents increases.

Figures & tables

Figure 1: Overview of the proposed hierarchical framework. Specifications are decomposed into reach-avoid subtasks, from which a centralized sampling-based planner constructs a feasible subtask sequence for execution by decentralized low-level policies. Solid and dotted blocks and arrows indicate components used at test time and during training, respectively.

Figure 2

$N$	$\|\mathcal{B}_{\varphi_{\texttt{global}}}\|$	$\|\mathcal{T}\|$	Iterations	$t_{\texttt{path}}$
3	$98.7_{\pm 42.5}$	$7.9_{\pm 3.3}$	$64.7_{\pm 100.5}$	$0.02_{\pm 0.02}$
5	$1789.3_{\pm 1267.3}$	$10.4_{\pm 5.2}$	$320.4_{\pm 511.7}$	$0.17_{\pm 0.24}$
10	$>118814.3$	$12.6_{\pm 5.2}$	$477.4_{\pm 530.2}$	$8.64_{\pm 0.77}$

Table 2: High-level planner scalability. We report the product-automaton size

|\mathcal{B}_{\varphi_{\texttt{global}}}|

, the constructed tree size

|\mathcal{T}|

, sampling iterations, and planning time

t_{\texttt{path}}

, averaged over 10 seeds.

Figure 3: Performance under different numbers of agents during training and evaluation. We report the average success rate $\eta_{s}$ , violation rate $\eta_{v}$ , and number of steps $\mu$ to specification satisfaction for $N_{\texttt{train}},N_{\texttt{eval}}\in\{3,5,10\}$ . Error bars indicate one standard deviation.

Figure 4: Average success rate $\eta_{s}$ , violation rate $\eta_{v}$ , and average number of steps $\mu$ to satisfy the IND specifications $\Phi_{\texttt{ind}}$ , $\Phi_{\texttt{ind/dep}}$ , $\Phi_{\texttt{ind/cnt}}$ , and $\Phi_{\texttt{ind/dep/cnt}}$ listed in Table 3 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Visualizations of the Zone environments with $N=3$ , $5$ , and $10$ agents, from left to right. The positions of agents and zones are randomly sampled each episode.

Specification types		IND		OOD
Finite- horizon (3 agents)	sa	$\varphi_{1}$	$(\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{b}\lor\mathsf{g}))\land(\ \mathsf{F}\ \mathsf{m})$	$\varphi_{16}$	$\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{b}\land\ \mathsf{F}\ (\neg\mathsf{y}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})))$
	sa	$\varphi_{2}$	$(\ \mathsf{F}\ \mathsf{b})\land(\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{y}))$	$\varphi_{17}$	$\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ (\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{y}\land\ \mathsf{F}\ \mathsf{b})))$
	sa	$\varphi_{3}$	$\neg(\mathsf{m}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{b}\land\ \mathsf{F}\ \mathsf{g})$	$\varphi_{18}$	$\neg(\mathsf{m}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{b}\land\ \mathsf{F}\ (\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})))$
	sa	$\varphi_{4}$	$\neg\mathsf{m}\ \mathsf{U}\ ((\mathsf{b}\lor\mathsf{y})\land\ \mathsf{F}\ \mathsf{g})$	$\varphi_{19}$	$\neg\mathsf{m}\ \mathsf{U}\ ((\mathsf{b}\lor\mathsf{y})\land\ \mathsf{F}\ (\neg\mathsf{b}\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})))$
	sa	$\varphi_{5}$	$\neg(\mathsf{b}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ \mathsf{m})$	$\varphi_{20}$	$\neg(\mathsf{b}\lor\mathsf{y})\ \mathsf{U}\ (\mathsf{g}\land\ \mathsf{F}\ (\neg\mathsf{g}\ \mathsf{U}\ (\mathsf{m}\land\ \mathsf{F}\ \mathsf{y})))$
	sa	$\varphi_{6}$	$(\ \mathsf{F}\ \mathsf{m})\land(\neg\mathsf{y}\ \mathsf{U}\ (\mathsf{b}\lor\mathsf{g}))$	$\varphi_{21}$	$\neg\mathsf{y}\ \mathsf{U}\ ((\mathsf{b}\lor\mathsf{g})\land\ \mathsf{F}\ (\neg\mathsf{m}\ \mathsf{U}\ (\mathsf{y}\land\ \mathsf{F}\ \mathsf{b})))$

Appendix

Table 3: Specifications used for evaluation.

\mathsf{b}

\mathsf{g}

\mathsf{m}

, and

\mathsf{y}

denote

\mathsf{blue}

\mathsf{green}

\mathsf{magenta}

, and

\mathsf{yellow}

, respectively.

\mathsf{sa}

and

\mathsf{ma}

denote specifications for single-agent and multi-agent behaviors, respectively. Top: finite-horizon specifications with 3 agents, with results in Table 1 . Bottom left: finite-horizon specifications with 5 and 10 agents for the scalability study, with results in Figure 3 . Bottom right: infinite-horizon specifications with 3 agents, with results in Table 5 .

Method	Specifications	$\eta_{s}\uparrow$	$\eta_{v}\downarrow$	$\mu\downarrow$
ACC-MARL	Reach-Only	$0.77_{\pm 0.08}$	$0.19_{\pm 0.07}$	$261.27_{\pm 39.58}$
ACC-MARL	Reach-Avoid	$0.60_{\pm 0.09}$	$0.37_{\pm 0.08}$	$226.06_{\pm 38.71}$
Ours	Reach-Only	$0.97_{\pm 0.02}$	$0.02_{\pm 0.02}$	$139.92_{\pm 14.16}$
Ours	Reach-Avoid	$0.98_{\pm 0.02}$	$0.01_{\pm 0.01}$	$148.40_{\pm 15.47}$

Appendix

Table 4: Evaluation results on simple specifications: success rate

\eta_{s}

, violation rate

\eta_{v}

, and average number of steps

\mu

\uparrow

: higher is better;

\downarrow

: lower is better. Each value is averaged over 5 seeds, with 100 trajectories per seed.

Method	Metrics	$\Phi_{\mathrm{indep}}$	$\Phi_{\mathrm{indep/req}}$	$\Phi_{\mathrm{indep/cnt}}$	$\Phi_{\mathrm{indep/req/cnt}}$	Average
Ours	$\mu_{\texttt{acc}}\uparrow$	$10.71_{\pm 1.86}$	$5.98_{\pm 0.87}$	$10.48_{\pm 1.72}$	$5.55_{\pm 0.84}$	$8.18_{\pm 2.80}$
Ours	$\eta_{v}\downarrow$	$0.05_{\pm 0.03}$	$0.15_{\pm 0.03}$	$0.05_{\pm 0.02}$	$0.16_{\pm 0.04}$	$0.10_{\pm 0.06}$

Appendix

Table 5: Evaluation results of infinite-horizon tasks: violation rate

\eta_{v}

and average number of visits to accepting states

\mu_{\text{acc}}

on infinite-horizon tasks listed in Table 3 .

\uparrow

: higher is better;

\downarrow

: lower is better. Each value is averaged over 5 seeds, with 100 trajectories per seed.

Figure 6: Performance as the complexity of inter-agent coordination specifications increases. The top row varies the number of dependency operators $n=|\texttt{Req}|$ in $\varphi_{\texttt{dep}}$ , while the bottom row varies the number of count operators $n=|\texttt{Sync}|+|\texttt{Async}|$ in $\varphi_{\texttt{cnt}}$ . We report the success rate $\eta_{s}$ , violation rate $\eta_{v}$ , average number of steps $\mu$ to satisfy a specification, and planning time $t_{\texttt{path}}$ . Solid lines show the mean, and shaded regions indicate one standard deviation.

Panel of Figure 7 — Figure 7: Visualization of the trajectory of specification $\{\varphi_{4},\varphi_{5},\varphi_{6},\varphi_{11},\varphi_{14}\}$ . For each subplot, $t$ denotes the timestep, and the set denotes the propositions that are true at the current timestep.

Panel of Figure 8 — Figure 8: Visualization of the planning process of the high-level sampling-based planner for specification $\{\varphi_{4},\varphi_{5},\varphi_{6},\varphi_{11},\varphi_{14}\}$ . Each subplot shows the planning state at timestep $t$ . Yellow blocks denote the selected subtask sequence, while gray blocks denote sampled nodes that are not part of the selected sequence.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 371 papers

May 3, 2026

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning

A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals requires estimating how one agent's current action affects its teammates over future interaction steps. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that estimates multi-step action effects between agents and selectively converts them into intrinsic rewards. MAGIC uses counterfactual action interventions to compare teammate futures under factual and counterfactual branches, and introduces a gate based on advantage to direct exploration toward beneficial behaviors aligned with the task goal. Experiments on Multi-Agent Particle Environments (MPE) and StarCraft micromanagement benchmarks (SMAC and SMACv2) show that MAGIC consistently outperforms leading prior methods, with average relative final performance improvements of 26.9% and 10.1%, respectively.

Haohan Yu, Jinmiao Cong, Shengzhi Wang +2

1Dalian University of Technology · 2Microsoft, Beijing, China

Multi-Agent Coordination Intrinsic Reward Methods Reinforcement Learning Cooperative MARL Causal RL Reward Design for RL+1

Ask PDF

Feb 22, 2026

MagicAgent: Towards Generalized Agent Planning

The evolution of Large Language Models (LLMs) from passive text processors to autonomous agents has established planning as a core component of modern intelligence. However, achieving generalized planning remains elusive, not only by the scarcity of high-quality interaction data but also by inherent conflicts across heterogeneous planning tasks. These challenges result in models that excel at isolated tasks yet struggle to generalize, while existing multi-task training attempts suffer from gradient interference. In this paper, we present MagicAgent, a series of foundation models specifically designed for generalized agent planning. We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks, including hierarchical task decomposition, tool-augmented planning, multi-constraint scheduling, procedural logic orchestration, and long-horizon tool execution. To mitigate training conflicts, we propose a two-stage training paradigm comprising supervised fine-tuning followed by multi-objective reinforcement learning over both static datasets and dynamic environments. Empirical results show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks (e.g.,

75.1\%

on Worfbench and

86.9\%

on BFCL-v3), as well as strong results on our in-house MagicEval benchmarks, substantially outperforming existing sub-100B models and surpassing leading ultra-scale models, including GPT-5.2, Kimi-K2 and GLM-4.7.

Xuhui Ren, Shaokang Dong, Chen Yang +21

Honor Device Co., Ltd · Fudan University

LLM Agent Planning LLM Agent Training LLM Agent Trajectory Synthesis RL for LLM Agents

Ask PDF

Sep 3, 2026

Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning

Generalising to unseen tasks remains a fundamental challenge in offline multi-agent reinforcement learning (MARL). In this work, we present a principled analysis of zero-shot task generalisation in the offline setting and conduct an extensive empirical investigation into the scaling behaviour governing task diversity, dataset size, and network capacity. To facilitate this study, we extend offline sequence modelling architectures to handle multi-task observation and action spaces alongside variable agent counts across tasks. Our primary finding is that scaling task diversity---rather than sheer dataset size is the dominant factor in achieving robust zero-shot transfer. Through large-scale experiments across four challenging environments (Connector, RWARE, SMAX, and LBF), we demonstrate that our multi-task approach achieves a mean improvement of 3.2x on held-out test tasks compared to single-task models and consistently outperforms strong behaviour cloning baselines. These results suggest that the development of generalisable MARL agents should prioritise the diversity of the training distribution with varying numbers of agents, providing a roadmap for scaling offline MARL effectively.

Oussama Hidaoui, Omer Ebead, Ulrich Armel Mbou Sob +14

InstaDeep · AIMS · Stellenbosch University

Zero-Shot Learning Reinforcement Learning OOD Generalization Multi-Task RL Offline MARL

Ask PDF

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning

MagicAgent: Towards Generalized Agent Planning

Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning