Learning Communication Between Heterogeneous Agents in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence

Alex Popa; Adrian Taylor; Ranwa Al Mallah

cs.CRMar 17, 2026

Learning Communication Between Heterogeneous Agents in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence

Authors: Alex Popa, Adrian Taylor, Ranwa Al Mallah

Organizations: Department of Electrical and Computer Engineering, Royal Military College of Canada, Kingston Ontario, Canada · Defence Research and Development Canada, Ottawa Ontario, Canada · Department of Computer Engineering and Software Engineering, Polytechnique Montr´eal, Montr´eal Qu´ebec, Canada

Abstract

Reinforcement learning techniques are being explored as solutions to the threat of cyber attacks on enterprise networks. Recent research in the field of AI in cyber security has investigated the ability of homogeneous multi-agent reinforcement learning agents, capable of inter-agent communication, to respond to cyberattacks. This paper advances the study of learned communication in multi-agent systems by examining heterogeneous agent capabilities within a simulated network environment. To this end, we leverage CommFormer, a publicly available state-of-the-art communication algorithm, to train and evaluate agents within the Cyber Operations Research Gym (CybORG). Our results show that CommFormer agents with heterogeneous capabilities can outperform other algorithms deployed in the CybORG environment, by converging to an optimal policy up to four times faster while improving standard error by up 38%. The agents implemented in this project provide an additional avenue for exploration in the field of AI for cyber security, enabling further research involving realistic networks.

Figures & tables

FIGURE 1: Homogeneous agent scenario training curve. The inset plot shows best case scenario performance of approximately -4 average reward.

FIGURE 2: Heterogeneous agent scenario training curve.

FIGURE 3: Homogeneous agent scenario training curve. The inset plot provides a closer view of peak performance across training and evaluation episodes.

Autonomous Cyber Defense

Autonomous cyber defense uses AI agents to detect, investigate, and contain attacks while balancing operational safety, latency, and human oversight.

See all 72 papers

No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents
Sep 30, 2026Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian +1LLM Red Teaming AI Agent Evaluation
Indiana University Bloomington, IN, USA · Rutgers University Piscataway, NJ, USA · Johns Hopkins University Baltimore, MD, USA
Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations
Jul 30, 2026Konur Tholl, François Rivest, Mariam El Mezouar +2Autonomous Cyber Defense LLM-Guided RL
Department of Electrical and Computer Engineering, Royal Military College of Canada, Kingston, Canada · Department of Mathematics and Computer Science, Royal Military College of Canada, Kingston, Canada · Defence Research and Development Canada, Ottawa, Canada +1
NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions
Apr 10, 2026Igor JankowskiAutonomous Cyber Defense Cybersecurity

Sep 30, 2026J/K move · Enter open · S save

No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian, Ankit Shah

Indiana University Bloomington, IN, USA · Rutgers University Piscataway, NJ, USA · Johns Hopkins University Baltimore, MD, USA

LLM Red Teaming AI Agent Evaluation AI Agent Security Benchmarks LLMs for Cybersecurity Automated Red Teaming

Open paper AskPDF arXiv

Autonomous red team agents increasingly stress-test AI-enabled cyber defenses by planning strategy and executing multistage attacks. Reinforcement learning (RL) and large language models (LLMs) offer complementary mechanisms for the planning and execution such agents require, and prior work has combined them in hybrid hierarchies. Yet a given architecture is typically developed and evaluated within a single environment, leaving open whether an observed advantage reflects a generally stronger decision mechanism or merely alignment with a particular setting. We address this gap with a controlled cross-environment comparison of two homogeneous hierarchical red team architectures: an RL planner with an RL executor (RL+RL) and an LLM planner with an LLM executor (LLM+LLM). We evaluate both against expert autonomous defenders in CybORG CAGE-4 and in Cyberwheel at two network scales, across 18 configurations under one unified disruption metric. We find a pronounced environment-dependent inversion. RL+RL wins the compact, densely rewarded CAGE-4 (78.5% disruption success versus 18.0% for the strongest LLM configuration) and the 100-host Cyberwheel network (81.0% versus 50.5%), while a pretrained cybersecurity LLM agent wins the larger, escalation-gated 1010-host Cyberwheel network (55.0% versus 0.0% for RL). A kill-chain analysis explains the inversion through architecture-specific bottlenecks that aggregate success rates conceal.In the 1010-host Cyberwheel network, RL discovers and compromises hosts but stalls at privilege escalation, whereas in CAGE-4, LLM agents obtain privileged access but rarely convert it into operational impact. These results indicate that conclusions drawn in a single environment may not generalize, and that hybrid planner-executor designs should be motivated by specific failure modes rather than the assumption that one architecture is universally preferable.

Abstract

Autonomous Cyber Defense

Explore similar work

No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions