Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

Igor Jankowski

cs.MAMar 24, 2026

Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

Authors: Igor Jankowski

Abstract

While Multi-Agent Reinforcement Learning (MARL) algorithms achieve unprecedented successes across complex continuous domains, their standard deployment strictly adheres to a synchronous operational paradigm. Under this paradigm, agents are universally forced to execute deep neural network inferences at every micro-frame, regardless of immediate necessity. This dense throughput acts as a fundamental barrier to physical deployment on edge-devices where thermal and metabolic budgets are highly constrained. We propose Epistemic Time-Dilation MAPPO (ETD-MAPPO), augmented with a Dual-Gated Epistemic Trigger. Instead of depending on rigid frame-skipping (macro-actions), agents autonomously modulate their execution frequency by interpreting aleatoric uncertainty (via Shannon entropy of their policy) and epistemic uncertainty (via state-value divergence in a Twin-Critic architecture). To format this, we structure the environment as a Semi-Markov Decision Process (SMDP) and build the SMDP-Aligned Asynchronous Gradient Masking Critic to ensure proper credit assignment. Empirical findings demonstrate massive improvements (> 60% relative baseline acquisition leaps) over current temporal models. By assessing LBF, MPE, and the 115-dimensional state space of Google Research Football (GRF), ETD correctly prevented premature policy collapse. Remarkably, this unconstrained approach leads to emergent Temporal Role Specialization, reducing computational overhead by a statistically dominant 73.6% entirely during off-ball execution without deteriorating centralized task dominance.

Figures & tables

Panel of Figure 1 — Figure 1: Algorithmic Logic Flowchart. The environment loop executes exclusively against the strictly sequential evaluations of both Aleatoric inference outputs and True Epistemic variation.

Panel of Figure 2 — Figure 2: Detailed Neural Architecture depicting exact hidden state preservation boundaries ( $m_{t}$ ).

Component	Parameter	Value
Architecture	Actor Network	3-Layer MLP (64) $\to$ GRU (128)
Architecture	Twin-Critic Network	3-Layer MLP (64) $\to$ GRU (128)
PPO Optimization	Optimizer	Adam ( $lr=5\times 10^{-4}$ )
	Discount Factor ( $\gamma$ ) / GAE ( $\lambda$ )	0.99 / 0.95
	PPO Clip Coef ( $\epsilon$ )	0.20
	Value Loss Coef ( $c_{2}$ )	0.50

Table 1: ETD-MAPPO Architecture and Hyperparameter Specifications

Method	Win Rate	FLOP Reduction	Status
Vanilla MAPPO	$40.0\%$	$0.0\%$	Baseline
Fixed-Skip ( $N=3$ )	$20.0\%$	$17.7\%$	Critical Decay
ETD-MAPPO (Ours)	$60.0\%\pm 3.2\%$	$0.0\%$	Safe Computation

Table 2: Level-Based Foraging (LBF) Peak Performance Metrics

Figure 3: Empirical training progression across Level-Based Foraging. The Dual-Gated ETD model statistically outperforms fixed-parameter models natively driving explicit success logic.

Method	Goal Rate	FLOP Reduction	Policy Status
Vanilla MAPPO	$100.0\%$	$0.0\%$	Converged
Static Entropy (No Critics)	$0.0\%$	$78.0\%$	Total Collapse
ETD-MAPPO (Twin-Critics)	$100.0\%\pm 0.0\%$	$5.2\%$	Converged & Optimized

Table 3: Google Research Football (3vs1) Final Performance

Figure 4: Recorded 20,000 algorithmic updates modeling GRF acquisition states.

Entity Role	Strategic Objective	Average Inference Skip Rate
Adversary 0	Pursuit / Tracking	$3.5\%$
Adversary 1	Pursuit / Tracking	$3.6\%$
Adversary 2	Pursuit / Tracking	$3.4\%$
Agent 0 (Prey)	Linear Evasion	$73.6\%\pm 2.1\%$

Table 4: MPE Simple Tag: Empirical Temporal Role Specialization

Figure 5: Mapping explicit continuous allocation ranges mapping MPE environments accurately targeting structural differences across agents.

Semi-Markov Decision Process

See all 10 papers

KD-MARL: Resource-Aware Knowledge Distillation in Multi-Agent Reinforcement Learning
Apr 8, 2026Monirul Islam Pavel, Siyi Hu, Muhammad Anwar Masum +3Policy Distillation Knowledge Distillation
School of Computer Science and Information Technology, Adelaide University, Australia · School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Australia · Systems Research Institute, Polish Academy of Sciences, Poland
Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning
Jun 3, 2026Zicheng Zhao, Yu Lan, Chengzhengxu Li +2Reinforcement Learning Game-Playing Agents
Xi’an Jiaotong University · Queen Mary University of London
Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering
May 25, 2026Maxim Mednikov, Oren GalRobust RL Multi-Agent Reinforcement Learning
University of Haifa, Israel

Apr 8, 2026J/K move · Enter open · S save

KD-MARL: Resource-Aware Knowledge Distillation in Multi-Agent Reinforcement Learning

Monirul Islam Pavel, Siyi Hu, Muhammad Anwar Masum, Mahardhika Pratama+2

School of Computer Science and Information Technology, Adelaide University, Australia · School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Australia · Systems Research Institute, Polish Academy of Sciences, Poland

Policy Distillation Knowledge Distillation Cooperative MARL Model Compression Decentralized MARL

Open paper AskPDF arXiv

Real world deployment of multi agent reinforcement learning MARL systems is fundamentally constrained by limited compute memory and inference time. While expert policies achieve high performance they rely on costly decision cycles and large scale models that are impractical for edge devices or embedded platforms. Knowledge distillation KD offers a promising path toward resource aware execution but existing KD methods in MARL focus narrowly on action imitation often neglecting coordination structure and assuming uniform agent capabilities. We propose resource aware Knowledge Distillation for Multi Agent Reinforcement Learning KD MARL a two stage framework that transfers coordinated behavior from a centralized expert to lightweight decentralized student agents. The student policies are trained without a critic relying instead on distilled advantage signals and structured policy supervision to preserve coordination under heterogeneous and limited observations. Our approach transfers both action level behavior and structural coordination patterns from expert policies while supporting heterogeneous student architectures allowing each agent model capacity to match its observation complexity which is crucial for efficient execution under partial or limited observability and limited onboard resources. Extensive experiments on SMAC and MPE benchmarks demonstrate that KD MARL achieves high performance retention while substantially reducing computational cost. Across standard multi agent benchmarks KD MARL retains over 90 percent of expert performance while reducing computational cost by up to 28.6 times FLOPs. The proposed approach achieves expert level coordination and preserves it through structured distillation enabling practical MARL deployment across resource constrained onboard platforms.

Abstract

Semi-Markov Decision Process

Explore similar work

KD-MARL: Resource-Aware Knowledge Distillation in Multi-Agent Reinforcement Learning

Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering