SEER: A Scalable Event-Driven Runtime for Parallel Execution of Multi-Agent Reinforcement Learning Environments

Jiming Su; Feng Zhu; Yuhao Xiao; Miao Zhang

cs.DCNov 5, 2026

SEER: A Scalable Event-Driven Runtime for Parallel Execution of Multi-Agent Reinforcement Learning Environments

Authors: Jiming Su, Feng Zhu, Yuhao Xiao, Miao Zhang

Organizations: NUDT

Abstract

Multi-agent reinforcement learning (MARL) is increasingly used in systems where many entities interact in a shared environment. In these workloads, policy inference can be batched on GPUs, but the CPU-side environment path still updates entity states, resolves local interactions, and builds observations inside each $\mathrm{env.step}$ . This path can dominate latency in interaction-intensive workloads after policy batching. Existing vectorized environment interfaces and general event-driven runtimes do not directly address the fine-grained cross-entity work inside one bounded MARL step while preserving the step-level semantics expected by learning systems. We present SEER (Scalable Event-Driven Environment Runtime), a runtime for MARL environment execution under an explicit event-locality contract. SEER models each environment step as a bounded stream of timestamped events. If direct state updates are entity-local and cross-entity effects are represented as ordered future events, SEER can run independent events in parallel while preserving serial-equivalent boundary outputs. To make this contract practical, SEER uses a CAS-based dual-layer event queue: local event queues preserve per-entity causal order, while a parallel executive list exposes ready work to global worker threads. SEER further uses STPA, a preflight-guided thread-pool selection mechanism that combines short runtime traces with static workload and hardware features. We evaluate SEER on two parameterized MARL workloads: compute-intensive UAV swarm control (Aircraft) and communication-intensive wireless management (PCS). On Aircraft, SEER reduces inference latency by 16.2% over the baseline. On PCS, SEER reduces environment-execution latency by 46.9%, 19.5%, and 64.9% over ROSS-T, NBSE, and ROOT-Sim, respectively.

Figures & tables

Figure 1: MARL environment-execution workflow.

Figure 2: Single-step inference-time distribution in representative MARL environment scenarios. The policy network is MAPPO ( Yu et al. 2022 ) , and one agent corresponds to one environment entity.

Figure 3: End-to-end MARL inference-throughput ratio between ROSS-T and MAgent2 in the motivation-study workloads. The ratio is computed as ROSS-T throughput divided by MAgent2 throughput under the same motivation-study configuration. Event time denotes the average execution time of events within one environment step. These workloads differ from those used in the main evaluation.

Figure 4: SEER CAS-based dual-layer event queue for MARL environment execution.

Figure 5: STPA thread-pool prediction for event-driven MARL environment execution.

Figure 6: Architecture of the SEER event-driven environment-execution runtime.

Figure 7: Absolute environment-execution throughput of ROSS-T, SEER w/o STPA, and SEER under different event granularities in the Aircraft workload on Platform A. Throughput is measured in environment steps per second; the reported values are not speedup ratios.

Figure 8: Environment-execution latency of ROSS-T, SEER w/o STPA, and SEER under different entity scales in the Aircraft workload on Platform A.

Figure 9: MARL environment-execution performance for Aircraft: (a–b) single-thread-normalized speedup versus average event time, and (c–d) single-step environment-execution latency versus entity scale. Speedup is computed relative to the corresponding single-thread throughput under the same workload and platform configuration.

Figure 10: End-to-end MARL inference throughput of PCS on Platform B under varying event granularities and entity scales. Throughput is measured in inference steps per second.

Figure 11: Per-thread CPU-time distribution of ROSS-T, ROOT-Sim, and SEER in the evaluated PCS configuration. A more uniform distribution across worker threads indicates better load balance.

Model	Training Set		Test Set
	MAE	MSE	MAE	MSE
LSTM	8.9976	155.5296	9.3542	161.8647
XGBoost	0.5525	0.6449	1.2832	5.1508
TimesURL	1.8344	13.7815	2.2386	16.6097
TR+GP	2.6535	18.3720	3.6187	31.2561
STPA	0.0108	0.0109	0.3394	4.5944

Table 1: Thread-count prediction errors of the evaluated thread-pool selection methods.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 392 papers

Mar 24, 2026

Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

While Multi-Agent Reinforcement Learning (MARL) algorithms achieve unprecedented successes across complex continuous domains, their standard deployment strictly adheres to a synchronous operational paradigm. Under this paradigm, agents are universally forced to execute deep neural network inferences at every micro-frame, regardless of immediate necessity. This dense throughput acts as a fundamental barrier to physical deployment on edge-devices where thermal and metabolic budgets are highly constrained. We propose Epistemic Time-Dilation MAPPO (ETD-MAPPO), augmented with a Dual-Gated Epistemic Trigger. Instead of depending on rigid frame-skipping (macro-actions), agents autonomously modulate their execution frequency by interpreting aleatoric uncertainty (via Shannon entropy of their policy) and epistemic uncertainty (via state-value divergence in a Twin-Critic architecture). To format this, we structure the environment as a Semi-Markov Decision Process (SMDP) and build the SMDP-Aligned Asynchronous Gradient Masking Critic to ensure proper credit assignment. Empirical findings demonstrate massive improvements (> 60% relative baseline acquisition leaps) over current temporal models. By assessing LBF, MPE, and the 115-dimensional state space of Google Research Football (GRF), ETD correctly prevented premature policy collapse. Remarkably, this unconstrained approach leads to emergent Temporal Role Specialization, reducing computational overhead by a statistically dominant 73.6% entirely during off-ball execution without deteriorating centralized task dominance.

Igor Jankowski

Semi-Markov Decision Process Reinforcement Learning Asynchronous RL Cooperative MARL Sequential Decision Making

Ask PDF

Feb 10, 2026

FlexMARL: Graph-Driven Rollout-Training Co-Design for LLM-Based Multi-Agent Reinforcement Learning

Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying infrastructure for large-scale MARL remains underexplored. Existing training frameworks primarily optimize for single-agent scenarios and fail to capture the graph-structured nature of MARL. This fundamental mismatch leads to three MARL-specific challenges: topology-induced rollout skew, critical path synchronization barriers, and dynamic training demand. To bridge this gap, we propose FlexMARL, a graph-driven rollout-training co-designed framework for large-scale LLM-based MARL. FlexMARL introduces the multi-agent execution graph (MEG) and materializes the intermediate representation as the MEG Store, which provides decentralized multi-table storage and serves as the data hub for system components. Built around the MEG Store, the Rollout Engine develops dependency parallel sampling with hierarchical load balancing, which reduces inference latency and balances intra-agent/inter-agent loads. The Joint Orchestrator exploits an agent-level asynchronous pipeline to mitigate the critical path synchronization barriers while maintaining consistency. The Training Engine achieves on-demand resource allocation by collectively managing training processes for individual agents. The training states of different agents are swapped via location-agnostic communication APIs. Empirical results on a production cluster demonstrate that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.

Zhida Jiang, Zhaolong Xing, Jiawei Lu +13

Huawei · Huazhong University of Science and Technology

Multi-Agent Reinforcement Learning Distributed RL RL for LLM Agents

Ask PDF

Nov 16, 2023

JaxMARL: Multi-Agent RL Environments and Algorithms in JAX

Benchmarks are crucial in the development of machine learning algorithms, with available environments significantly influencing reinforcement learning (RL) research. Traditionally, RL environments run on the CPU, which limits their scalability with typical academic compute. However, recent advancements in JAX have enabled the wider use of hardware acceleration, enabling massively parallel RL training pipelines and environments. While this has been successfully applied to single-agent RL, it has not yet been widely adopted for multi-agent scenarios. In this paper, we present JaxMARL, the first open-source, Python-based library that combines GPU-enabled efficiency with support for a large number of commonly used MARL environments and popular baseline algorithms. Our experiments show that, in terms of wall clock time, our JAX-based training pipeline is around 14 times faster than existing approaches, and up to 12500x when multiple training runs are vectorized. This enables efficient and thorough evaluations, potentially alleviating the evaluation crisis in the field. We also introduce and benchmark SMAX, a JAX-based approximate reimplementation of the popular StarCraft Multi-Agent Challenge, which removes the need to run the StarCraft II game engine. This not only enables GPU acceleration, but also provides a more flexible MARL environment, unlocking the potential for self-play, meta-learning, and other future applications in MARL. The code is available at https://github.com/flairox/jaxmarl.

Alexander Rutherford, Benjamin Ellis, Matteo Gallici +18

University of Oxford · Universitat Politècnica de Catalunya · University College London +2

RL Benchmarks Reinforcement Learning Game-Playing Agents GPU-Accelerated RL Multi-Agent Reinforcement Learning

Ask PDF

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

FlexMARL: Graph-Driven Rollout-Training Co-Design for LLM-Based Multi-Agent Reinforcement Learning

JaxMARL: Multi-Agent RL Environments and Algorithms in JAX