cs.LGSep 26, 2025

HARL-A: An Extensible Benchmark Framework for Heterogeneous Multi-Agent Adversarial Reinforcement Learning in IsaacLab

Authors: Isaac Peterson, Christopher Allred, Jacob Morrey, Mario Harper

Organizations: Utah State University · US DEVCOM Army Research Laboratory

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Progress in adversarial multi-agent reinforcement learning (MARL) for robotics has been hampered by a lack of shared, extensible infrastructure that supports heterogeneous agent morphologies in high-fidelity physics simulation. Existing frameworks either focus on cooperative tasks, rely on simplified physics engines, or provide isolated implementations that are difficult to extend. We present HARL-A, an open-source, actively maintained framework built on IsaacLab that enables scalable training and benchmarking of adversarial policies across morphologically diverse robot teams with any number of teams and any mix of robot morphologies per team. HARL-A extends the HARL algorithm library and IsaacLab with adversarial multi-agent support and contributes three components: (1) a modular software architecture that reduces the engineering overhead of defining new heterogeneous adversarial environments, (2) a suite of three benchmark environments---Sumo, Soccer, and 3D Galaga---spanning contact-rich pushing, ball-skill competition, and pursuit/evasion, (3) over ten pretrained policies spanning homogeneous and heterogeneous team configurations, released publicly on Hugging Face to enable immediate exploration of adversarial learning dynamics without retraining from scratch. We demonstrate the framework across multiple competitive scenarios, showing that it reliably produces learned adversarial policies and emergent role specialization. All code environments, trained policies, and documentation are openly available at https://github.com/DIRECTLab/IsaacLab-HARL.

Figures & tables

Fig. 1 : HARL-A environments in IsaacLab. (Top) Two heterogeneous teams—each comprising one Anymal C quadruped and one Leatherback rover—compete in an adversarial Sumo task. (Bottom) Multi-team configurations with three distinct agent morphologies illustrate the framework’s support for arbitrary team composition and scale.
Author (Year)EnvironmentAdversarialHigh-Fidelity PhysicsMulti-AgentHeterogeneousExtensible Framework
Bansal et al. (2018) [ 8 ]MuJoCo✓    
Gleave et al. (2020) [ 12 ]MuJoCo✓    
Lowe et al. (2017) [ 13 ]Particle Env.✓ ✓  
Baker et al. (2020) [ 9 ]MuJoCo✓ ✓  
Schwarting et al. (2021) [ 14 ]Visual RL✓ ✓  
Liu et al. (2021) [ 15 ]MuJoCo✓ ✓  
TABLE I : Comparison of adversarial MARL works. HARL-A is the only framework that simultaneously satisfies all five properties.
Fig. 2 : Actor-critic training paradigms in adversarial reinforcement learning. Prior cooperative frameworks supported paradigm C (heterogeneous coordination with shared critic). HARL-A adds support for paradigms B (heterogeneous single-agent adversarial) and D (heterogeneous multi-team adversarial), enabling the full spectrum of competitive training configurations.
Fig. 3 : Evolution of the observation space across Sumo curriculum stages for the Anymal C robot. Zero-padded entries (shown as 0 ) are replaced with meaningful features as task complexity increases, following the zero-buffer strategy of [ 21 ] . This maintains a fixed observation dimension across all stages, enabling seamless policy transfer. This strategy was used for all benchmark environments presented in this paper.
Fig. 4 : Competitive behaviors in the heterogeneous Soccer environment (Anymal C vs. Leatherback). Top: Leatherback maintains possession and scores. Bottom: Leatherback possession followed by a turnover and an Anymal goal, illustrating the competitive dynamics that emerge from morphological asymmetry.
Fig. 5 : 3D Galaga: Anti-Aircraft Defense. MiniTanks fire arm-aligned laser-tag rays (2, 3) each control step. A drone (path shown in 1) is knocked out when it enters the radius of an active ray (4). The environment demonstrates HARL-A’s ability to compose pretrained locomotion primitives into novel adversarial scenarios.
Fig. 6 : Win rates of adversarially trained policies vs. a fixed baseline policy across the three benchmark environments (top: 3D Galaga, middle: Sumo, bottom: Soccer). Blue shows the agents trained adversarially, while orange represents agents trained against a fixed policy. For each saved checkpoint, we evaluate 1000 parallel environments against the initial baseline policy and average the win rate. Increasing win rate over training confirms that meaningful adversarial learning occurs across all three environments and task types.
Fig. 7 : Emergent role specialization in the heterogeneous Sumo environment (one Anymal C and one Leatherback per team). Without any explicit role assignment, the Leatherback learned to destabilize the opposing Anymal by targeting its legs (Top), while the Anymal learned to drag the opposing Leatherback out of the arena using its limbs (Bottom). These complementary strategies emerged consistently from morphological asymmetry.
Fig. 8 : FPS training distribution based on the number of agents in each environment. As can be seen in the figure, increased agent count decreases the FPS of the training.
EnvironmentStartingTrained
Leatherback-Stage1-Soccer-v0NOYES
Leatherback-Stage2-Soccer-v0YESYES
AnymalC_Soccer_Hetero_By_Team-v0YESNO
Sumo-Stage2-Hetero-By-Team-v0YESNO
Sumo-Stage2-Hetero-Same-Critic-v0YESNO
Sumo-Stage2-Hetero-Same-Critic-No-Neg-v0YESNO
TABLE II : Pretrained policies available on Hugging Face. Environments span all three benchmark task categories, multiple curriculum stages, and both homogeneous and heterogeneous team configurations.

Explore similar work

CardsList

  1. RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

    Jun 29, 2026Adithya Mohan, Daniel Kriegl, Torsten SchönDeep Q-Networks

    AImotion Bavaria · Technische Hochschule Ingolstadt · Ingolstadt, Germany

  2. Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning

    May 18, 2026Sunwoo Lee, Mingu Kang, Yonghyeon Jo +1Multi-Agent Reinforcement LearningAdversarial Training

    Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea

  3. HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning

    Jun 7, 2026Zechu Li, Yufeng Jin, Xiaoyang Liu +4Scalable Robot LearningSimulation-Based Reinforcement Learning

    TU Darmstadt 1 · Hessian.AI 7 · Honda Research Institute Europe 2 +4