Multi-Agent Reinforcement Learning

Also known as MARL

Ask about this topic

Overview

Multi-agent reinforcement learning studies how multiple decision-makers learn when each agent’s actions affect both the environment and other agents, across cooperative, competitive, and mixed-motive settings. Common approaches include centralized training with decentralized execution, independent or shared actor-critic policies, value factorization such as QMIX, self-play and game-solving methods, and graph-based communication or hierarchical policies. Researchers test these methods in benchmarks such as SMAC, MPE, Overcooked, and multi-agent MuJoCo, and apply them to robotics, traffic, wireless networks, energy systems, and language-agent workflows. A central challenge is scaling coordination and credit assignment while keeping learning stable under non-stationarity and policies robust, safe, and fair with unfamiliar partners and conditions.

Latest papers

CardsList

373 papers47 papers in the last four weeks, up 81% on the four weeks before. 0.3% of all new papers.

Feb 24, 2026

Federated Multi Agent Deep Learning and Neural Networks for Advanced Distributed Sensing in Wireless Networks

Multi-agent deep learning (MADL), including multi-agent deep reinforcement learning (MADRL), distributed/federated training, and graph-structured neural networks, is becoming a unifying framework for decision-making and inference in wireless systems where sensing, communication, and computing are tightly coupled. Recent 5G-Advanced and 6G visions strengthen this coupling through integrated sensing and communication, edge intelligence, open programmable RAN, and non-terrestrial/UAV networking, which create decentralized, partially observed, time-varying, and resource-constrained control problems. This survey synthesizes the state of the art, with emphasis on 2021-2025 research, on MADL for distributed sensing and wireless communications. We present a task-driven taxonomy across (i) learning formulations (Markov games, Dec-POMDPs, CTDE), (ii) neural architectures (GNN-based radio resource management, attention-based policies, hierarchical learning, and over-the-air aggregation), (iii) advanced techniques (federated reinforcement learning, communication-efficient federated deep RL, and serverless edge learning orchestration), and (iv) application domains (MEC offloading with slicing, UAV-enabled heterogeneous networks with power-domain NOMA, intrusion detection in sensor networks, and ISAC-driven perceptive mobile networks). We also provide comparative tables of algorithms, training topologies, and system-level trade-offs in latency, spectral efficiency, energy, privacy, and robustness. Finally, we identify open issues including scalability, non-stationarity, security against poisoning and backdoors, communication overhead, and real-time safety, and outline research directions toward 6G-native sense-communicate-compute-learn systems.

Nadine Muller, Stefano DeRosa, Su Zhang +1

Faculty of Information Technology and Electrical Engineering, University of Oulu

Multi-Agent Reinforcement LearningMulti-Agent SystemsFederated RLIntegrated Sensing and Communication
AskPDF
Feb 23, 2026

Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning

Scaling cooperative multi-agent reinforcement learning (MARL) is fundamentally limited by cross-agent noise. When agents share a common reward, each agent's learning signal is computed from a shared return that depends on all agents, so the stochasticity of the other agents enters the signal as cross-agent noise that grows with NN. Many engineering systems, such as cloud computing and power systems, have differentiable analytical models that prescribe efficient system states, providing a new reference beyond noisy shared returns. In this work, we propose Descent-Guided Policy Gradient (DG-PG), a framework that augments policy-gradient updates with a low-noise descent signal toward this reference. We prove that DG-PG reduces policy-gradient estimator variance from O(N)\mathcal{O}(N) to O(1)\mathcal{O}(1), preserves the stationary points of the cooperative objective, and achieves agent-independent sample complexity O(1/ε3)\mathcal{O}(1/ε^3). We evaluate DG-PG on cloud resource scheduling and power dispatch tasks, which involve discrete and continuous decisions, respectively. In both tasks, DG-PG converges within 10 episodes on average at every scale up to 1500 heterogeneous agents.

Shan Yang, Yang Liu

National University of Singapore

Cooperative MARLMulti-Agent Reinforcement LearningPolicy Gradient Methods
AskPDF
Feb 23, 2026

Safe Multi-Agent Deep Reinforcement Learning for Privacy-Aware Edge-Device Collaborative DNN Inference

As Deep Neural Network (DNN) inference becomes increasingly prevalent on edge and mobile platforms, critical challenges emerge in privacy protection, resource constraints, and dynamic model deployment. This paper proposes a privacy-aware collaborative inference framework, in which adaptive model partitioning is performed across edge devices and servers. To jointly optimize inference delay, energy consumption, and privacy cost under dynamic service demands and resource constraints, we formulate the joint problem as a Constrained Markov Decision Process (CMDP) that integrates model deployment, user-server association, model partitioning, and resource allocation. We propose a Hierarchical Constrained Multi-Agent Proximal Policy Optimization with Lagrangian relaxation (HC-MAPPO-L) algorithm, a safe reinforcement learning-based framework that enhances Multi-Agent Proximal Policy Optimization (MAPPO) with adaptive Lagrangian dual updates to enforce long-term delay constraints. To ensure tractability while maintaining coordination, we decompose the CMDP into three hierarchically structured policy layers: an auto-regressive based model deployment policy, a Lagrangian-enhanced user association and model partitioning policy, and an attention-based resource allocation policy. Extensive experimental results demonstrate that HC-MAPPO-L consistently satisfies stringent delay constraints while achieving a superior balance among energy consumption and privacy cost, outperforming representative baseline algorithms across varying problem scales and resource configurations.

Hong Wang, Xuwei Fan, Zhipeng Cheng +4

School of Future Science and Engineering, Soochow University, Suzhou 215006, China · College of Computer and Information Sciences, Fujian Agriculture and Forestry University, Fuzhou 350002, China · School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China +2

Constrained MARLEfficient Neural Network InferenceMulti-Agent Reinforcement LearningPrivacy-Preserving ML
AskPDF
Feb 22, 2026

Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environment

The optimization of urban energy systems is crucial for the advancement of sustainable and resilient smart cities, which are becoming increasingly complex with multiple decision-making units. To address scalability and coordination concerns, Multi-Agent Reinforcement Learning (MARL) is a promising solution. This paper addresses the imperative need for comprehensive and reliable benchmarking of MARL algorithms on energy management tasks. CityLearn is used as a case study environment because it realistically simulates urban energy systems, incorporates multiple storage systems, and utilizes renewable energy sources. By doing so, our work sets a new standard for evaluation, conducting a comparative study across multiple key performance indicators (KPIs). This approach illuminates the key strengths and weaknesses of various algorithms, moving beyond traditional KPI averaging which often masks critical insights. Our experiments utilize widely accepted baselines such as Proximal Policy Optimization (PPO) and Soft Actor Critic (SAC), and encompass diverse training schemes including Decentralized Training with Decentralized Execution (DTDE) and Centralized Training with Decentralized Execution (CTDE) approaches and different neural network architectures. Our work also proposes novel KPIs that tackle real world implementation challenges such as individual building contribution and battery storage lifetime. Our findings show that DTDE consistently outperforms CTDE in both average and worst-case performance. Additionally, temporal dependency learning improved control on memory dependent KPIs such as ramping and battery usage, contributing to more sustainable battery operation. Results also reveal robustness to agent or resource removal, highlighting both the resilience and decentralizability of the learned policies.

Aymen Khouja, Imen Jendoubi, Oumayma Mahjoub +4

InstaDeep

Multi-Agent Reinforcement LearningDecentralized MARLRL Benchmarks
AskPDF
Feb 21, 2026

Carbon-aware decentralized dynamic task offloading in MIMO-MEC networks via multi-agent reinforcement learning

Massive internet of things microservices require integrating renewable energy harvesting into mobile edge computing (MEC) for sustainable eScience infrastructures. Spatiotemporal mismatches between stochastic task arrivals and intermittent green energy along with complex inter-user interference in multi-antenna (MIMO) uplinks complicate real-time resource management. Traditional centralized optimization and off-policy reinforcement learning struggle with scalability and signaling overhead in dense networks. This paper proposes CADDTO-PPO, a carbon-aware decentralized dynamic task offloading framework based on multi-agent proximal policy optimization. The multi-user MIMO-MEC system is modeled as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) to jointly minimize carbon emissions and buffer latency and energy wastage. A scalable architecture utilizes decentralized execution with parameter sharing (DEPS), which enables autonomous IoT agents to make fine-grained power control and offloading decisions based solely on local observations. Additionally, a carbon-first reward structure adaptively prioritizes green time slots for data transmission to decouple system throughput from grid-dependent carbon footprints. Finally, experimental results demonstrate CADDTO-PPO outperforms deep deterministic policy gradient (DDPG) and lyapunov-based baselines. The framework achieves the lowest carbon intensity and maintains near-zero packet overflow rates under extreme traffic loads. Architectural profiling validates the framework to demonstrate a constant O(1)O(1) inference complexity and theoretical lightweight feasibility for future generation sustainable IoT deployments.

Mubshra Zulfiqar, Muhammad Ayzed Mirza, Basit Qureshi

School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, 430000, Hubei, China · School of Computer Science and Information Engineering, Qilu Institute of Technology, Jinan, 250200, Shandong, China · Department of Computer Science, Prince Sultan University, Riyadh, 12435, Riyadh Province, Saudi Arabia

Computation OffloadingCarbon-Aware ComputingEdge ComputingWireless Resource AllocationEdge-Cloud ComputingDecentralized MARL+1
AskPDF
Feb 20, 2026

Toward AI Autonomous Navigation for Mechanical Thrombectomy using Hierarchical Modular Multi-agent Reinforcement Learning (HM-MARL)

Mechanical thrombectomy (MT) is typically the optimal treatment for acute ischemic stroke involving large vessel occlusions, but access is limited due to geographic and logistical barriers. Reinforcement learning (RL) shows promise in autonomous endovascular navigation, but generalization across 'long' navigation tasks remains challenging. We propose a Hierarchical Modular Multi-Agent Reinforcement Learning (HM-MARL) framework for autonomous two-device navigation in vitro, enabling efficient and generalizable navigation. HM-MARL was developed to autonomously navigate a guide catheter and guidewire from the femoral artery to the internal carotid artery (ICA). A modular multi-agent approach was used to decompose the complex navigation task into specialized subtasks, each trained using Soft Actor-Critic RL. The framework was validated in both in silico and in vitro testbeds to assess generalization and real-world feasibility. In silico, a single-vasculature model achieved 92-100% success rates on individual anatomies, while a multi-vasculature model achieved 56-80% across multiple patient anatomies. In vitro, both HM-MARL models successfully navigated 100% of trials from the femoral artery to the right common carotid artery and 80% to the right ICA but failed on the left-side vessel superhuman challenge due to the anatomy and catheter type used in navigation. This study presents the first demonstration of in vitro autonomous navigation in MT vasculature. While HM-MARL enables generalization across anatomies, the simulation-to-real transition introduces challenges. Future work will refine RL strategies using world models and validate performance on unseen in vitro data, advancing autonomous MT towards clinical translation.

Harry Robertshaw, Nikola Fischer, Lennart Karstensen +6

School of Biomedical Engineering & Imaging Sciences, Kings College London, London, UK · School of Engineering & Materials Science, Queen Mary University of London, London, UK

Robotic Endovascular InterventionMulti-Agent Reinforcement LearningHierarchical RL
AskPDF
Feb 20, 2026

Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first Online off-policy MARL framework using Diffusion policies (OMAD) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across 1010 diverse tasks, demonstrating a remarkable 2.5×2.5\times to 5×5\times improvement in sample efficiency.

Zhuoran Li, Hai Zhong, Xun Wang +3

Institute for Interdisciplinary Information Sciences, Tsinghua University · ByteDance

Multi-Agent Reinforcement LearningMulti-Agent CoordinationDiffusion PolicyOff-Policy RLMaximum Entropy RL
AskPDF
Feb 20, 2026

Mean-Field Reinforcement Learning without Synchrony

Mean-field reinforcement learning (MF-RL) scales multi-agent RL to large populations by reducing each agent's dependence on others to a single summary statistic -- the mean action. However, this reduction requires every agent to act at every time step; when some agents are idle, the mean action is simply undefined. Addressing asynchrony therefore requires a different summary statistic -- one that remains defined regardless of which agents act. The population distribution μ∈Δ(O)μ\in Δ(\mathcal{O}) -- the fraction of agents at each observation -- satisfies this requirement: its dimension is independent of NN, and under exchangeability it fully determines each agent's reward and transition. Existing MF-RL theory, however, is built on the mean action and does not extend to μμ. We therefore construct the Temporal Mean Field (TMF) framework around the population distribution μμ from scratch, covering the full spectrum from fully synchronous to purely sequential decision-making within a single theory. We prove existence and uniqueness of TMF equilibria, establish an O(1/N)O(1/\sqrt{N}) finite-population approximation bound that holds regardless of how many agents act per step, and prove convergence of a policy gradient algorithm (TMF-PG) to the unique equilibrium. Experiments on a resource selection game and a dynamic queueing game confirm that TMF-PG achieves near-identical performance whether one agent or all NN act per step, with approximation error decaying at the predicted O(1/N)O(1/\sqrt{N}) rate.

Shan Yang

Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.

Mean-Field MARLReinforcement LearningMulti-Agent Reinforcement LearningAsynchronous RLSequential MARL
AskPDF
Feb 19, 2026

Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form

Multi-agent reinforcement learning (MARL) has made significant progress in recent years, but most algorithms still rely on a discrete-time Markov Decision Process (MDP) with fixed decision intervals. This formulation is often ill-suited for complex multi-agent dynamics, particularly in high-frequency or irregular time-interval settings, leading to degraded performance and motivating the development of continuous-time MARL (CT-MARL). Existing CT-MARL methods are mainly built on Hamilton-Jacobi-Bellman (HJB) equations. However, they rarely account for safety constraints such as collision penalties, since these introduce discontinuities that make HJB-based learning difficult. To address this challenge, we propose a continuous-time constrained MDP (CT-CMDP) formulation and a novel MARL framework that transforms discrete MDPs into CT-CMDPs via an epigraph-based reformulation. We then solve this by proposing a novel physics-informed neural network (PINN)-based actor-critic method that enables stable and efficient optimization in continuous time. We evaluate our approach on continuous-time safe multi-particle environments (MPE) and safe multi-agent MuJoCo benchmarks. Results demonstrate smoother value approximations, more stable training, and improved performance over safe MARL baselines, validating the effectiveness and robustness of our method.

Xuefeng Wang, Lei Zhang, Henglin Pu +2

Purdue University

Multi-Agent Reinforcement LearningContinuous-Time RLSafe RLConstrained RL
AskPDF
Feb 19, 2026

Spatio-temporal dual-stage hypergraph MARL for human-centric multimodal corridor traffic signal control

Human-centric traffic signal control in corridor networks must increasingly account for multimodal travelers, particularly high-occupancy public transportation, rather than focusing solely on vehicle-centric performance. This paper proposes STDSH-MARL (Spatio-Temporal Dual-Stage Hypergraph based Multi-Agent Reinforcement Learning), a multi-agent deep reinforcement learning framework that follows a centralized training and decentralized execution paradigm. The proposed method captures spatio-temporal dependencies through a novel dual-stage hypergraph attention mechanism that models interactions across both spatial and temporal hyperedges. In addition, a hybrid discrete action space is introduced to jointly determine the next signal phase configuration and its corresponding green duration, enabling more adaptive signal timing decisions. Experiments conducted on a corridor network under five traffic scenarios demonstrate that STDSH-MARL achieves strong overall multimodal performance, with substantial and relatively consistent reductions in tram waiting time, while improvements in bus waiting time are more variable across traffic scenarios. These results highlight the trade-off among overall network efficiency, tram priority, and bus service quality. Compared with state-of-the-art baseline methods, the proposed approach achieves superior overall performance. Further ablation studies confirm the contribution of each component of STDSH-MARL, with temporal hyperedges identified as the most influential factor driving the observed performance gains.

Xiaocai Zhang, Neema Nassir, Milad Haghani

Department of Infrastructure Engineering, Faculty of Engineering and Information Technology, The University of Melbourne, VIC 3010, Australia

Traffic Signal ControlMulti-Agent Reinforcement LearningHypergraph Neural Networks
AskPDF
Feb 19, 2026

Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables QtotQ^{\text{tot}} to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.

Yonghyeon Jo, Sunwoo Lee, Seungyul Han

Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea 44919

Cooperative MARLMulti-Agent Reinforcement LearningRL Exploration
AskPDF
Feb 19, 2026

A Unified Framework for Locality in Scalable MARL

Scalable methods for networked multi-agent reinforcement learning let each agent plan using only a small neighborhood of the agent graph. This works only when the system is value-local, meaning a perturbation at one agent affects the long-run value at another agent weakly when the two are far apart. In the average-reward setting, the standard way to certify locality is the Dobrushin row-sum bound on a single matrix CπC^π that captures how each agent's next state depends on each other agent's current state. To make this matrix easy to work with, prior work bounds it by a supremum over joint actions. The resulting bound is independent of the policy, but it is loose whenever the policy never picks the worst-case action. We split CπC^π into pieces that separately track environment sensitivity and policy sensitivity, Cπ⪯Es+EaΠ(π)C^π\preceq E^{\mathrm s}+E^{\mathrm a}Π(π), where EsE^{\mathrm s} measures how the next state moves with the current state, EaE^{\mathrm a} measures how it moves with the current action, and Π(π)Π(π) measures how reactive the policy is to changes in state. The spectral radius of Hπ:=Es+EaΠ(π)H^π:= E^{\mathrm s}+E^{\mathrm a}Π(π) then controls the decay of the average-reward Poisson solution, and the spectral certificate ρ(Hπ)<1ρ(H^π)<1 is strictly weaker than the row-sum condition ∥Hπ∥∞<1\|H^π\|_\infty<1 on the same matrix and applies in regimes where policy-independent action-supremum bounds used in prior Dobrushin-style work cannot. For temperature-ττ softmax policies we get Π(π)≤L/(2τ)Π(π)\le L/(2τ), so the softmax temperature directly controls locality. We use this decay result to give a deterministic oracle guarantee for a block-coordinate KL-proximal policy-improvement template whose truncation bias decays exponentially in the message-passing radius κκ.

Sourav Chakraborty, Amit Kiran Rege, Claire Monteleoni +1

University of Colorado, Boulder, USA. · INRIA Paris, France.

Multi-Agent Reinforcement LearningAverage-Reward RL
AskPDF
Feb 18, 2026

Discovering Multiagent Learning Algorithms with Large Language Models

Much of the advancement in Multi-Agent Reinforcement Learning (MARL) for imperfect-information games has historically depended on the manual, iterative refinement of algorithmic baselines. Recently, evolutionary coding agents powered by Large Language Models (LLMs) have emerged as powerful tools to automate this discovery process. In this work, we deploy one of such agentic frameworks, AlphaEvolve, to navigate the design spaces of two distinct game-theoretic paradigms: counterfactual regret minimization (CFR) and policy-space response oracles (PSRO). This automated search yielded two algorithms: Volatility-Adaptive Discounted (VAD-) CFR and Smoothed Hybrid Optimistic Regret (SHOR-) PSRO, which are consistently competitive with state-of-the-art human-designed baselines across an 18-game evaluation suite spanning Poker, Goofspiel, Liar's Dice, Blotto, and Battleship variants. However, because the LLM optimizes for fitness on a specific training set, it often constructs highly synergistic, complex mechanisms tailored to those environments. Through systematic ablation studies, we demonstrate that while these mechanisms are tightly coupled, the true driver of generalization lies in a minimal algorithmic core. By distilling the LLM's discoveries down to their most fundamental principles, we produce two minimal solvers: Warm-started Optimistic Predictive (WOP-)CFR and Projection Matching (PM-)PSRO. These distilled versions achieve superior performance on generalization with greatly reduced structural complexity, providing a clear methodology for using LLMs in algorithmic discovery.

Zun Li, John Schultz, Daniel Hennes +1

Google DeepMind

Counterfactual Regret MinimizationMulti-Agent Reinforcement LearningRegret MinimizationGeneralization in RLLarge Language Model-Guided Optimization
AskPDF
Feb 18, 2026

A Scalable Approach to Solving Simulation-Based Network Security Games

We introduce MetaDOAR, a lightweight meta-controller that augments the Double Oracle / PSRO paradigm with a learned, partition-aware filtering layer and Q-value caching to enable scalable multi-agent reinforcement learning on very large cyber-network environments. MetaDOAR learns a compact state projection from per node structural embeddings to rapidly score and select a small subset of devices (a top-k partition) on which a conventional low-level actor performs focused beam search utilizing a critic agent. Selected candidate actions are evaluated with batched critic forwards and stored in an LRU cache keyed by a quantized state projection and local action identifiers, dramatically reducing redundant critic computation while preserving decision quality via conservative k-hop cache invalidation. Empirically, MetaDOAR attains higher player payoffs than SOTA baselines on large network topologies, without significant scaling issues in terms of memory usage or training time. This contribution provide a practical, theoretically motivated path to efficient hierarchical policy learning for large-scale networked decision problems.

Michael Lanier, Yevgeniy Vorobeychik

Multi-Agent Reinforcement LearningHierarchical RLCybersecurity
AskPDF
Feb 18, 2026

Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking

Radio resource allocation (RRA) is a critical function in cellular vehicle-to-everything (C-V2X) networks, where vehicles must share limited wireless resources to support safety-critical communications. Multi-agent reinforcement learning (MARL) has emerged as a promising approach for this problem. However, key MARL challenges, including non-stationarity, coordination difficulty, large action space, partial observability, and limited robustness and generalization, are often intertwined, making it difficult to assess their individual impact on performance in vehicular environments. Moreover, existing studies primarily focus on developing new algorithms, while systematic benchmarking and comparative analyses remain limited. To address this gap, we formulate C-V2X RRA as a hierarchy of multi-agent interference games that progressively introduce key MARL challenges. Based on this framework, we develop a suite of benchmark learning tasks and construct training and testing datasets from SUMO-generated highway traces with diverse vehicular topologies and interference conditions. Using the proposed benchmark, we evaluate representative MARL algorithms spanning value-based, actor-critic, Independent Learning (IL), and Centralized Training with Decentralized Execution (CTDE) paradigms. The results identify robustness and generalization across diverse vehicular topologies as the dominant challenge among those considered in this work, reducing average normalized return by up to 59 percentage points, and show that, on the most challenging task, the best actor-critic method outperforms the best value-based method by 42%. By revealing the relative strengths and limitations of different MARL paradigms and open-sourcing the code, datasets, and benchmark suite, this work provides a systematic and reproducible foundation for evaluating and advancing MARL algorithms in vehicular networks.

Siyuan Wang, Lei Lei, Pranav Maheshwari +2

College of Engineering, University of Guelph, Guelph, Ontario, Canada · College of Electrical Engineering and Computer Sciences, Ningbo University, Ningbo, 315211, China

RL BenchmarksReinforcement LearningV2X CommunicationWireless Resource AllocationRobust RLGeneralization in RL+1
AskPDF
Feb 18, 2026

CAFE: Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning

Automated feature engineering (AFE) enables AI systems to autonomously construct high-utility representations from raw tabular data. However, existing AFE methods rely on statistical heuristics, yielding brittle features that fail under distribution shift. We introduce CAFE, a framework that reformulates AFE as a causally-guided sequential decision process, bridging causal discovery with reinforcement learning-driven feature construction. Phase I learns a sparse directed acyclic graph over features and the target to obtain soft causal priors, grouping features as direct, indirect, or other based on their causal influence with respect to the target. Phase II uses a cascading multi-agent deep Q-learning architecture to select causal groups and transformation operators, with hierarchical reward shaping and causal group-level exploration strategies that favor causally plausible transformations while controlling feature complexity. Across 15 public benchmarks (classification with macro-F1; regression with inverse relative absolute error), CAFE achieves statistically significant gains over the strongest AFE baseline (Wilcoxon p<0.001, large effect), reduces episodes-to-convergence, and delivers competitive time-to-target. Under both controlled covariate shifts and naturally occurring distribution shifts (Folktables), CAFE reduces out-of-distribution performance drop by up to 4x relative to a non-causal multi-agent baseline, and produces more compact feature sets with more stable post-hoc attributions. These findings underscore that causal structure, used as a soft inductive prior rather than a rigid constraint, can substantially improve the robustness and efficiency of automated feature engineering.

Arun Vignesh Malarkkan, Wangyang Ying, Yanjie Fu

School of Computing, Arizona State University

Automated Feature EngineeringDistribution Shift RobustnessMulti-Agent Reinforcement Learning
AskPDF
Feb 17, 2026

Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation

We propose MARL-Rad, a multi-modal multi-agent reinforcement learning framework for radiology report generation that trains the entire agentic system on policy within its deployed radiology workflow. MARL-Rad addresses the limitation of post-hoc agentization, where fixed LLMs are organized into hand-designed agentic workflows without being optimized for their assigned roles. Our framework decomposes chest X-ray interpretation into region-specific agents and a global integrating agent, and jointly optimizes them using clinically verifiable rewards. Experiments on the MIMIC-CXR and IU X-ray datasets show that MARL-Rad consistently improves clinical efficacy metrics such as RadGraph, CheXbert, and GREEN scores, achieving state-of-the-art clinical efficacy performance. Further analyses show that MARL-Rad improves laterality consistency and produces more accurate and detailed reports. A blinded clinician evaluation further suggests that MARL-Rad produces reports clinically comparable to ground-truth reports.

Kaito Baba, Risa Kishikawa, Satoshi Kodera

Department of Cardiovascular Medicine The University of Tokyo Hospital, Tokyo, Japan

Radiology Report GenerationMultimodal AgentsRL for LLM AgentsLLM Agent Workflow OptimizationMulti-Agent Reinforcement Learning
AskPDF
Feb 17, 2026

Fairness over Equality: Correcting Social Incentives in Asymmetric Sequential Social Dilemmas

Sequential Social Dilemmas (SSDs) provide a key framework for studying how cooperation emerges when individual incentives conflict with collective welfare. In Multi-Agent Reinforcement Learning, these problems are often addressed by incorporating intrinsic drives that encourage prosocial or fair behavior. However, most existing methods assume that agents face identical incentives in the dilemma and require continuous access to global information about other agents to assess fairness. In this work, we introduce asymmetric variants of well-known SSD environments and examine how natural differences between agents influence cooperation dynamics. Our findings reveal that existing fairness-based methods struggle to adapt under asymmetric conditions by enforcing raw equality that wrongfully incentivize defection. To address this, we propose three modifications: (i) redefining fairness by accounting for agents' reward ranges, (ii) introducing an agent-based weighting mechanism to better handle inherent asymmetries, and (iii) localizing social feedback to make the methods effective under partial observability without requiring global information sharing. Experimental results show that in asymmetric scenarios, our method fosters faster emergence of cooperative policies compared to existing approaches, without sacrificing scalability or practicality.

Alper Demir, Hüseyin Aydın, Kale-ab Abebe Tessera +2

Izmir University of Economics & University of Edinburgh, Izmir, Turkey · Middle East Technical University & Utrecht University, Ankara, Turkey · University of Edinburgh, Edinburgh, United Kingdom +1

Cooperative MARLMulti-Agent Reinforcement LearningReward Design for RLAlgorithmic Fairness
AskPDF
Feb 16, 2026

Fluid-Agent Reinforcement Learning

The primary focus of multi-agent reinforcement learning (MARL) has been to study interactions among a fixed number of agents embedded in an environment. However, in the real world, the number of agents is neither fixed nor known a priori. Moreover, an agent can decide to create other agents (for example, a cell may divide, or a company may spin off a division). In this paper, we propose a framework that allows agents to create other agents; we call this a fluid-agent environment. We present game-theoretic solution concepts for fluid-agent games and empirically evaluate the performance of several MARL algorithms within this framework. Our experiments include fluid variants of established benchmarks such as Predator-Prey and Level-Based Foraging, where agents can dynamically spawn, as well as a new environment we introduce that highlights how fluidity can unlock novel solution strategies beyond those observed in fixed-population settings. We demonstrate that this framework yields agent teams that adjust their size dynamically to match environmental demands.

Shishir Sharma, Doina Precup, Theodore J. Perkins

Mila – Quebec Artificial Intelligence Institute, Montreal, Canada · McGill University, Montreal, Canada · Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada

Multi-Agent Reinforcement LearningMulti-Agent CoordinationGame Theory
AskPDF
Feb 9, 2026

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6% avg@16 and +4.6% pass@16 on math, and +15.2% avg@16 and +13.1% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.

Lang Feng, Longtao Zheng, Shuo He +2

Nanyang Technological University, Singapore

Multi-Agent LLM SystemsMulti-Agent System OptimizationGroup Relative Policy OptimizationRL for LLM AgentsMulti-Agent Reinforcement Learning
AskPDF
Feb 5, 2026

Flow Matching for Offline Reinforcement Learning with Discrete Actions

Generative policies based on diffusion models and flow matching have shown strong promise for offline reinforcement learning (RL), but their applicability remains largely confined to continuous action spaces. To address a broader range of offline RL settings, we extend flow matching to a general framework that supports discrete action spaces with multiple objectives. Specifically, we replace continuous flows with continuous-time Markov chains, trained using a Q-weighted flow matching objective. We then extend our design to multi-agent settings, mitigating the exponential growth of joint action spaces via a factorized conditional path. We theoretically show that, under idealized conditions, optimizing this objective recovers the optimal policy. Extensive experiments further demonstrate that our method performs robustly across diverse settings and benchmarks, including high-dimensional control, multi-agent games, and dynamically changing preferences over multiple objectives, while outperforming traditional offline RL methods in practical multi-modal decision-making scenarios. Our discrete framework can also be applied to continuous-control problems through action quantization, providing a flexible trade-off between representational complexity and performance.

Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang +2

Department of Computer Science, University of Kentucky · Department of Computing and Information Sciences, Rochester Institute of Technology

Markov ChainsReinforcement LearningOffline RLFlow MatchingMulti-Agent Reinforcement LearningMulti-Objective Reinforcement Learning
AskPDF
Jan 29, 2026

Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning

To promote cooperation in Multi-Agent Reinforcement Learning, the reward signals of all agents can be aggregated together, forming global rewards that are commonly known as the fully cooperative setting. However, global rewards are usually noisy because they contain the contributions of all agents, which have to be resolved in the credit assignment process. On the other hand, using local reward benefits from faster learning due to the separation of agents' contributions, but can be suboptimal as agents myopically optimize their own reward while disregarding the global optimality. In this work, we propose a method that combines the merits of both approaches. By using a graph of interaction between agents, our method discerns the individual agent contribution in a more fine-grained manner than a global reward, while alleviating the cooperation problem with agents' local reward. We also introduce a practical approach for approximating such a graph. Our experiments demonstrate the flexibility of the approach, enabling improvements over the traditional local and global reward settings.

Bang Giang Le, Viet Cuong Ta

Human-Machine Interaction Laboratory VNU University of Engineering and Technology, Hanoi, Vietnam

Credit Assignment in RLCredit AssignmentCooperative MARLFine-Grained Credit AssignmentMulti-Agent Reinforcement Learning
AskPDF
Jan 24, 2026

Does Centralized Learning Help Embodied Agents Coordinate, and When Can a Tabular Study Tell?

Tabular studies are a common way to compare independent and centralized multi-agent learners, because they avoid function approximation. A natural expectation is that centralized learning, which learns joint actions, helps agents coordinate, especially when their bodies make solo success hard. Earlier work tests this only in parts: it measures coordination for one learning method, compares methods with the agents' bodies fixed, or varies speed but reports only how well the team did. We test it in a predator-prey environment, where teammates cooperatively hunt together while two teams compete. We compare independent Q-learning (IQL) with central Q-learning (CQL) in all four predator and prey pairings, under limits on speed and stamina, on a 5x5 and an 8x8 grid trained for 200,000 episodes each, and check that the agents learn on both. We measure coordination directly, with predictions for the main measurement fixed in advance. In this task, where one predator can capture prey alone, we find three things. First, CQL costs its agents in both roles on both grids, and its predators coordinate no more than IQL's. Second, its one benefit comes from acting on the best joint action: this cut capture time by 42% on the small grid with faster prey and unlimited stamina, but helped nowhere on the larger grid, where the best joint action had mostly never been tried. Third, on the larger grid a tie-breaking default decided which method won, even though the agents had passed a learning check. A tabular comparison can therefore be trusted only if it also checks how much of the state space the agents covered and implements the methods alike. We propose simple checks for both.

Muhammad Ahmed Atif, Nehal Naeem Haji, Muhammad Ebad Atif +1

Dhanani School of Science and Technology, Habib University, Karachi, Pakistan

Multi-Agent Reinforcement LearningCooperative MARLMulti-Agent Coordination
AskPDF
Jan 10, 2026

HiDVFS: Hierarchical Multi-Agent DVFS for Real-Time OpenMP DAG Workloads

Leakage power in multicore embedded systems now rivals dynamic power, so DVFS schedulers must respect deadlines and thermal limits, not just average makespan. Existing heuristics lack per-core, temperature-aware control and overlook the irregular execution of OpenMP DAGs. We propose HiDVFS, a general, extensible hierarchical multi-agent DVFS scheduler: a profiler agent selects cores and frequencies, a thermal agent groups cores by temperature, and a priority agent orders tasks under contention, all trained with a makespan-focused reward using short-horizon future-state shaping for sample efficiency. Deadlines are soft, derived from a measured reference cost; a federated schedulability gate keeps operating points feasible, and a calibrated split-conformal shield bounds each action's predicted response time. On Jetson TX2 with multi-seed validation, HiDVFS attains a 4.16+/-0.58 s L10 makespan, a 2.83x speedup and 32.9% energy reduction over a fairness-corrected GearDVFS port, and a 4.62x average speedup with 55.7% energy reduction across all 12 BOTS benchmarks. Cross-platform results on TX2, Orin NX, and RubikPi show deadline-aware DVFS cuts energy 15 to 18% versus pinning the maximum frequency, and a measured mixed-criticality study shows cluster-aware reservation is required to keep a high-criticality task's deadline-miss ratio at zero.

Mohammad Pivezhandi, Abusayeed Saifullah, Ali Jannesari

Department of Computer Science Wayne State University, Detroit, MI, USA · Department of Computer Science University of Texas at Dallas, Richardson, TX, USA · Department of Computer Science Iowa State University, Ames, IA, USA

Real-Time SchedulingMulti-Agent Reinforcement Learning
AskPDF
Dec 11, 2025

UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors

This paper introduces a reinforcement learning framework that enables controllable and diverse player behaviors without relying on human gameplay data. Existing approaches often require large-scale player trajectories, train separate models for different player types, or provide no direct mapping between interpretable behavioral parameters and the learned policy, limiting their scalability and controllability. We define player behavior in an N-dimensional continuous space and uniformly sample target behavior vectors from a region that encompasses the subset representing real human styles. During training, each agent receives both its current and target behavior vectors as input, and the reward is based on the normalized reduction in distance between them. This allows the policy to learn how actions influence behavioral statistics, enabling smooth control over attributes such as aggressiveness, mobility, and cooperativeness. A single PPO-based multi-agent policy can reproduce new or unseen play styles without retraining. Experiments conducted in a custom multi-player Unity game show that the proposed framework produces significantly greater behavioral diversity than a win-only baseline and reliably matches specified behavior vectors across diverse targets. The method offers a scalable solution for automated playtesting, game balancing, human-like behavior simulation, and replacing disconnected players in online games.

Atahan Cilan, Atay Özgövde

Turkish Aerospace, Istanbul, Turkey · Department of Computer Engineering, Boğaziçi University, Istanbul, Turkey

Reinforcement LearningHuman Behavior SimulationGoal-Conditioned RLMulti-Agent Reinforcement Learning
AskPDF
Nov 17, 2025

Transformer-Based Multi-Agent Reinforcement Learning for Networked Systems with Long-Range Interactions

Multi-agent reinforcement learning (MARL) has shown promise for large-scale network control, yet existing methods face two major limitations. First, they typically rely on an exponential decay property of agent interactions on far-away nodes, which can be exploited to develop more efficient and tractable MARL algorithms. When this exponential decay property does not hold, these algorithms do not account for long-range interactions such as epidemic outbreaks or cascading power failures. Second, existing approaches lack network generalizability, or the ability to generalize to networks of different topological structure and scale than those seen during training. In this work, we first present a mean-field stability analysis and empirical study investigating the conditions for long-range network interactions. These results motivate our primary contribution: STACCA (Shared Transformer Actor-Critic with Counterfactual Advantage), a transformer-based MARL framework that addresses both long-range interactions and network generalizability. STACCA employs a centralized Graph Transformer Critic to model long-range dependencies and provide system-level feedback, while its shared Graph Transformer Actor learns a generalizable policy capable of adapting across diverse network topologies. To improve credit assignment during training, STACCA integrates a novel counterfactual advantage estimator that is compatible with state-value critic estimates. We evaluate STACCA on epidemic containment and rumor-spreading network control tasks, demonstrating improved performance and network generalizability. These results highlight the potential of transformer-based MARL architectures to achieve generalizable control in large-scale networked systems.

Vidur Sinha, Muhammed Ustaomeroglu, Guannan Qu

Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, USA

Counterfactual Credit AssignmentGraph TransformerMulti-Agent Reinforcement Learning
AskPDF
Nov 4, 2025

A Quantitative Comparison of Centralised and Distributed Reinforcement Learning-Based Control for Soft Robotic Arms

This paper presents a quantitative comparison between centralised and distributed multi-agent reinforcement learning (MARL) architectures for controlling a soft robotic arm modelled as a Cosserat rod in simulation. Using PyElastica and the OpenAI Gym interface, we train both a global Proximal Policy Optimisation (PPO) controller and a Multi-Agent PPO (MAPPO) under identical budgets. Both approaches are based on the arm having nn number of controlled sections. The study systematically varies nn and evaluates the performance of the arm to reach a fixed target in three scenarios: default baseline condition, recovery from external disturbance, and adaptation to actuator failure. Quantitative metrics used for the evaluation are mean action magnitude, mean final distance, mean episode length, and success rate. The results show that there are no significant benefits of the distributed policy when the number of controlled sections n≤4n\le4. In very simple systems, when n≤2n\le2, the centralised policy outperforms the distributed one. When nn increases to 4<n≤124< n\le 12, the distributed policy shows a high sample efficiency. In these systems, distributed policy promotes a stronger success rate, resilience, and robustness under local observability and yields faster convergence given the same sample size. However, centralised policies achieve much higher time efficiency during training as it takes much less time to train the same size of samples. These findings highlight the trade-offs between centralised and distributed policy in reinforcement learning-based control for soft robotic systems and provide actionable design guidance for future sim-to-real transfer in soft rod-like manipulators.

Linxin Hou, Qirui Wu, Zhihang Qin +3

Department of Electrical and Computer Engineering, National University of Singapore · Department of Computer Science, National University of Singapore · Department of Mechanical Engineering, National University of Singapore

RL ControlRobotic Policy EvaluationDecentralized MARLSoft RoboticsMulti-Agent Reinforcement Learning
AskPDF
Oct 13, 2025

LLM-Empowered Agentic MAC Protocols: A Dynamic Stackelberg Game Approach

Medium Access Control (MAC) protocols, essential for wireless networks, are typically manually configured. While deep reinforcement learning (DRL)-based protocols enhance task-specified network performance, they suffer from poor generalizability and resilience, demanding costly retraining to adapt to dynamic environments. To overcome this limitation, we introduce a game-theoretic LLM-empowered multi-agent DRL (MARL) framework, in which the uplink transmission between a base station and a varying number of user equipments is modeled as a dynamic multi-follower Stackelberg game (MFSG), capturing the network's natural hierarchical structure. Within this game, LLM-driven agents, coordinated through proximal policy optimization (PPO), synthesize adaptive, semantic MAC protocols in response to network dynamics. Protocol action grammar (PAG) is employed to ensure the reliability and efficiency of this process. Under this system, we further analyze the existence and convergence behavior in terms of a Stackelberg equilibrium by studying the learning dynamics of LLM-empowered unified policies in response to changing followers. Simulations corroborate that our framework achieves a 77.6% greater throughput and a 65.2% fairness improvement over conventional baselines. Besides, our framework generalizes excellently to a fluctuating number of users without requiring retraining or architectural changes.

Renxuan Tan, Rongpeng Li, Fei Wang +4

College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China · Huawei Technologies Company Ltd., Shanghai 210026, China · Zhejiang Lab as well as Zhejiang University, Hangzhou 310027, China +1

Agentic RLStackelberg GamesWireless CommunicationsWireless Resource AllocationMulti-Agent Reinforcement Learning
AskPDF
Oct 9, 2025

Climate Surrogates for Scalable Multi-Agent Reinforcement Learning: A Case Study with CICERO-SCM

Climate policy analysis requires models that capture multi-gas climate effects, but such models are too slow to embed in reinforcement learning loops at scale. In collaboration with the European Environment Agency, we develop a multi-agent reinforcement learning (MARL) framework that integrates a higher-fidelity climate surrogate as the environment transition, enabling regional agents to learn policies under multi-gas dynamics. We train a recurrent surrogate on 20,00020{,}000 multi-gas emission pathways to emulate CICERO-SCM. The surrogate achieves near-simulator accuracy (global-mean temperature RMSE ≈ ⁣4 ⁣× ⁣10−4 K\approx\!4\!\times \!10^{-4}\,\mathrm{K}) with ∼ ⁣1000×\sim\!1000\times faster one-step inference and yields > ⁣100×>\!100\times end-to-end MARL training speed-up. We show policy agreement with the simulator in tractable settings and propose a replay- and rank-consistency test (Kendall's ττ) for assessing policy fidelity when simulator-in-the-loop training is infeasible. This enables large-scale multi-agent policy experiments while retaining high-fidelity multi-gas climate response.

Oskar Bohn Lassen, Serio Angelo Maria Agriesti, Filipe Rodrigues +2

Technical University of Denmark Kongens Lyngby, Denmark

Neural Surrogate ModelingSurrogate ModelingClimate ScienceClimate ModelingMulti-Agent Reinforcement Learning
AskPDF
Sep 27, 2025

Game-Theoretic Understandings of Multi-Agent Systems with Multiple Objectives

Agents in many multi-agent systems must balance several, possibly conflicting, objectives, and the trade-offs an agent can achieve depend on how the other agents behave. We study this setting through the Multi-Objective Markov Game (MOMG), a Markov game in which each agent receives a vector of rewards, and adopt the Pareto-Nash Equilibrium (PNE) as the solution concept: at a PNE, no agent can unilaterally improve one of its objectives without sacrificing another. We show that PNEs always exist and coincide with the Nash equilibria of the linearly scalarized games with strictly positive weights, so standard single-objective Markov-game solvers suffice to compute them. We further define the Weak Pareto-Nash Equilibrium (WPNE), which excludes only unilateral deviations that strictly improve all objectives at once. The associated gap (WPNG) is zero exactly at WPNEs and never exceeds the Nash gap of any linearly scalarized game; hence an approximate Nash equilibrium of a scalarized game is also an approximate WPNE. When the preference profile is known in advance, learning in an MOMG therefore reduces to learning in a single scalarized Markov game, for which we provide a sample-efficient algorithm. When preferences are revealed only after data collection, we design a two-phase algorithm that first explores without any preference information; with high probability, every Nash equilibrium of the learned scalarized game is then an approximate WPNE for every queried preference profile, and the WPNG of any policy can be estimated accurately from the learned model. Experiments on tabular games and four multi-objective multi-agent environments illustrate the theory and show the effect of preference-free exploration.

Zixin Lin, Qianyi Tan, Sihong He +1

Department of Electrical and Computer Engineering, Department of Computer Science, University of Central Florida, Orlando, FL 32816

Game TheoryMulti-Objective Reinforcement LearningMulti-Agent Reinforcement Learning
AskPDF