Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning

Yiyao Zhang; Diksha Goel; Hussain Ahmad

cs.CRApr 6, 2026

Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning

Authors: Yiyao Zhang, Diksha Goel, Hussain Ahmad

Organizations: Adelaide University, Australia · CSIRO’s Data61, Australia

Abstract

Autonomous agents are increasingly deployed in both offensive and defensive cyber operations, creating high-speed, closed-loop interactions in critical infrastructure environments. Advanced Persistent Threat (APT) actors exploit "Living off the Land" techniques and targeted telemetry perturbations to induce ambiguity in monitoring systems, causing automated defenses to overreact or misclassify benign behavior as malicious activity. Existing monolithic and multi-agent defense pipelines largely operate on correlation-based signals, lack structural constraints on response actions, and are vulnerable to reasoning drift under ambiguous or adversarial inputs. We present the Causal Multi-Agent Decision Framework (C-MADF), a structurally constrained architecture for autonomous cyber defense that integrates causal modeling with adversarial dual-policy control. C-MADF first learns a Structural Causal Model (SCM) from historical telemetry and compiles it into an investigation-level Directed Acyclic Graph (DAG) that defines admissible response transitions. This roadmap is formalized as a Markov Decision Process (MDP) whose action space is explicitly restricted to causally consistent transitions. Decision-making within this constrained space is performed by a dual-agent reinforcement learning system in which a threat-optimizing Blue-Team policy is counterbalanced by a conservatively shaped Red-Team policy. Inter-policy disagreement is quantified through a Policy Divergence Score and exposed via a human-in-the-loop interface equipped with an Explainability-Transparency Score that serves as an escalation signal under uncertainty. On the real-world CICIoT2023 dataset, C-MADF reduces the false-positive rate from 11.2%, 9.7%, and 8.4% in three cutting-edge literature baselines to 1.8%, while achieving 0.997 precision, 0.961 recall, and 0.979 F1-score.

Figures & tables

Acronym/Term	Definition used in this paper
C-MADF	Causal Multi-Agent Decision Framework
APT	Advanced Persistent Threat
SCM	Structural Causal Model
DAG	Directed Acyclic Graph
MDP	Markov Decision Process
MDP-DAG roadmap	Causally constrained investigation roadmap compiled from SCM into executable MDP transitions

TABLE I: Acronyms and standardized terminology used in this paper

Approach	Formally Structured	Causal	Multi-Agent	Integrated	Decision-Level
	Decision Model	Grounding	Deliberation	XAI	Robustness
Causal Security Analytics [ 26 , 27 ]	$\triangle$ (offline)	$\checkmark$	$\times$	$\triangle$	$\triangle$
ML for Intrusion Detection [ 38 , 39 , 40 ]	$\times$	$\times$	$\times$	$\times$	$\triangle$ (data-level)
AICA-style Agents [ 19 ]	$\triangle$	$\times$	$\times$	$\times$	$\triangle$
XAI for Cybersecurity [ 29 , 23 ]	$\times$	$\times$	$\times$	$\checkmark$ (post-hoc)	$\times$
MAS in Cybersecurity [ 31 , 32 ]	$\times$	$\times$	$\checkmark$	$\times$	$\triangle$

TABLE II: Comparison of C-MADF with Representative Autonomous Cyber Defense Approaches

Symbol	Description
$\mathcal{S},\mathcal{A}$	State space and action space of the MDP.
$P(s^{\prime}\mid s,a),R(s,a),\gamma$	Transition kernel, reward function, and discount factor.
$\pi,V^{\pi},Q^{\pi}$	Policy, state-value function, and action-value function under policy $\pi$ .
$\mathcal{G}=(\mathcal{V},E)$	DAG induced by the learned SCM.
$\mathcal{M}_{C}=\langle U,\mathcal{V},F\rangle$	SCM with exogenous variables $U$ , endogenous variables $\mathcal{V}$ , and structural equations $F$ .
$\pi_{B},\pi_{R}$	Blue-Team and Red-Team policies in the dual-agent architecture.

TABLE III: Summary of Core Notation

Capability	Formal Description
Initial Compromise	Obtains unauthorized access to a subset $\mathcal{N}_{c}\subset\mathcal{N}$ via credential theft, phishing, or exploitation of vulnerabilities.
Lateral Movement	Moves within $\mathcal{N}$ using legitimate administrative tools and credentials (e.g., LotL techniques) to minimize detection signals.
Selective Telemetry Manipulation	Drops, delays, replays, injects, or perturbs subsets of host-, network-, or control-layer telemetry streams to induce ambiguity in state estimation.
Partial Historical Poisoning	Introduces bounded perturbations into a fraction of historical logs used for causal discovery, without full rewriting of all historical data.
Evasion Strategies	Mimics benign operational patterns and exploits monitoring blind spots to delay detection or trigger false mitigation decisions.

TABLE IV: Adversary Capabilities under the Partial-Compromise Model

Constraint	Scenario Restriction
Node-Level Compromise Bound	Adversary control is limited to a strict subset $\mathcal{N}_{c}\subset\mathcal{N}$ ; complete network takeover is excluded.
Telemetry Survivability	The adversary cannot suppress all sensing channels simultaneously; at least one trustworthy telemetry path remains observable.
Defender Runtime Integrity	The C-MADF execution environment, source code, and runtime parameters are not directly compromiseable by the adversary.
Secure-Channel and Crypto Integrity	The adversary cannot subvert secure communication channels or break underlying cryptographic primitives.
Out-of-Scope Full-Compromise Attacks	Full compromise of the defense stack, arbitrary code execution inside C-MADF, and complete rewriting of all historical data are out of scope.

TABLE V: Scenario Constraints under the Partial-Compromise Model

Fig. 1: Illustration of a Shadow-Jitter telemetry manipulation scenario. Controlled perturbations in host and network logs distort apparent event correlations, leading an unconstrained correlation-based defense to misclassify benign activity as malicious. In contrast, C-MADF applies causal filtering and adversarial dual-policy validation within a constrained MDP-DAG structure, reducing spurious mitigation actions under ambiguous observations.

ID	Objective Definition	Mechanisms in C-MADF
G1	Maintain high detection recall while limiting false-positive mitigation actions under bounded telemetry manipulation and partial historical poisoning.	Causally constrained action space derived from learned DAG $\mathcal{G}$ ; dual-policy adversarial deliberation; divergence-based escalation under high policy disagreement $\mathcal{D}(s_{t})$ .
G2	Ensure every executable mitigation action corresponds to an admissible transition in a formally defined investigation structure.	MDP-DAG structure restricting admissible state transitions; complete logging of state–action trajectories; compatibility with temporal-logic and model-checking analysis.
G3	Provide structured evidence traces and uncertainty signals for each proposed or executed action to support human oversight.	Human-in-the-loop interface exposing evidence traces and alternative hypotheses; ETS aggregating clarity, coverage, and policy uncertainty; explicit visualization of policy divergence $\mathcal{D}(s_{t})$ .
G4	Scale to large networks and high-volume telemetry streams without exponential growth in online decision complexity.	Modular architecture separating offline causal discovery from online decision-making; compact dual-policy networks $\pi_{B},\pi_{R}$ ; reduced branching via constrained action subsets $\mathcal{A}_{C}(s)$ .
G5	Update policies and causal estimates over time while preserving structural constraints on admissible action sequences.	Periodic or drift-triggered SCM re-estimation; continual reinforcement learning updates; recalibration of escalation thresholds under distributional shift.

TABLE VI: Mapping Design Objectives to C-MADF Mechanisms

Module	Inputs / Outputs	Formal Guarantee
Causal Discovery Engine	In: historical telemetry and event logs $\mathcal{D}$ ; Out: weighted causal DAG $\mathcal{G}=(V,E)$ with edge confidence scores.	Causal identifiability: filters spurious statistical dependencies and enables intervention- and counterfactual-consistent reasoning under structural constraints.
Causally-Constrained MDP (MDP-DAG)	In: $\mathcal{G}$ and investigation ontology; Out: constrained MDP $\mathcal{M}=(S,A,P,R)$ over admissible roadmap states and actions.	Semantic soundness: restricts policy exploration to transitions consistent with causal structure and operational semantics, ensuring verifiable action admissibility.
Council of Rivals Deliberation	In: current state $s_{t}$ and roadmap constraints; Out: adversarially validated action $\hat{a}_{t}$ and policy divergence score $\mathcal{D}(s_{t})$ .	Robust decision validation: dual-policy asymmetric self-play with objective divergence reduces reasoning drift, mitigates overconfidence, and exposes epistemic uncertainty via calibrated disagreement.
Explainable Human-in-the-Loop (HITL) Interface	In: deliberation artifacts (evidence traces, counterarguments, $\mathcal{D}(s_{t})$ ); Out: structured explanations, Explanation Transparency Score (ETS $(s_{t})$ ), and escalation triggers.	Operational transparency: produces operator-aligned rationales with quantifiable reliability calibration and principled escalation under high uncertainty.

TABLE VII: C-MADF Architectural Modules and Formal Guarantees

Fig. 2: The architecture of the Causal Multi-Agent Decision Framework (C-MADF). The process begins with the Causal Discovery Module learning a causal model from data. This model informs the MDP-DAG Roadmap, which provides a verifiable structure for investigations. The Council of Rivals, consisting of a Blue-Team and a Red-Team agent, deliberates on the best course of action within this roadmap. Their debate and the resulting policy divergence are fed into the Explainable Human-in-the-Loop Interface, which computes the ETS and presents a clear recommendation to the supervisory adjudication.

Component	Specification and Rationale
Discovery paradigm	Constraint-based causal structure learning (PC/FCI family), selected for its ability to produce an explicit, interpretable DAG consistent with observed conditional-independence relations in telemetry data. FCI is employed when latent confounding cannot be excluded.
Conditional independence testing	Hybrid CI testing pipeline: partial correlation tests for approximately Gaussian continuous variables; kernel-based or mutual-information–based tests for mixed or non-Gaussian variables. Tests are selected based on feature-type metadata and validated via hold-out consistency checks.
Significance threshold	Statistical significance level $\alpha\in[0.01,0.05]$ , selected through validation to balance false edge removal (Type I error) and spurious dependency retention (Type II error). Sensitivity analysis is performed across this range.
Maximum conditioning set size	Conditioning set size bounded (e.g., $\|Z\|\leq 2$ – $4$ ) to control computational complexity and reduce instability in high-dimensional telemetry. This bounds worst-case complexity of independence testing and mitigates overfitting.
Edge orientation constraints	Standard v-structure orientation and acyclicity enforcement, optionally augmented with domain priors (e.g., host process creation precedes outbound network flow). Prior constraints are encoded as forbidden or required edges.
Concept drift handling	Periodic offline re-estimation or incremental updates triggered when distributional shift exceeds a predefined divergence threshold (e.g., population stability index or KL divergence). Drift detection prevents structural staleness.

TABLE VIII: Causal Discovery Configuration and Design Choices

Fig. 3: The MDP-DAG Investigation Roadmap showing valid state transitions (orange arrows with reward values) and causally inconsistent blocked transitions (red X). The roadmap constrains the action space to ensure verifiable investigation flows, preventing logically inconsistent paths such as jumping directly from Initial Alert to Threat Mitigated without sufficient evidence collection.

Reward Component	Blue-Team Policy ( $\pi_{B}$ )	Red-Team Policy ( $\pi_{R}$ )
Verified mitigation	Large positive reward for reaching causally justified terminal states (e.g., “Threat Mitigated”) after traversing required evidence states.	Small positive reward only if mitigation is supported by sufficient evidence; no reward for premature containment.
False-positive intervention	Penalty proportional to operational disruption cost (e.g., unnecessary isolation, service interruption).	Large positive reward for preventing or opposing unjustified disruptive actions; penalty if false positives are not challenged.
Evidence acquisition	Positive reward for actions that increase information gain or reduce causal uncertainty along the learned DAG.	Positive reward for demanding additional evidence under high epistemic uncertainty; penalty for accepting unsupported mitigation claims.
Investigation efficiency	Small step penalty to discourage unnecessary actions and encourage timely resolution.	Smaller penalty weight relative to Blue-Team, prioritizing verification over speed in ambiguous states.
Roadmap compliance	Penalty for proposing actions inconsistent with MDP-DAG constraints.	Equivalent penalty for violating causal or roadmap admissibility constraints.
Human escalation	Neutral or small penalty when escalation is triggered under high divergence $\mathcal{D}(s_{t})$ .	Positive reward when escalation is correctly triggered under high epistemic uncertainty.

TABLE IX: Reward Shaping Design for the Council of Rivals

Fig. 4: Council of Rivals adversarial deliberation architecture. At each investigation state $s_{t}$ in the causally constrained MDP-DAG, the Blue-Team policy proposes a mitigation action based on its learned value estimates, while the Red-Team policy evaluates and challenges the proposal under a conservatively shaped objective. The interaction proceeds through iterative hypothesis evaluation, evidential justification, and action arbitration within the constrained action space. Inter-policy disagreement is quantified via the Policy Divergence Score $\mathcal{D}(s_{t})$ , which serves as an epistemic uncertainty signal and triggers human escalation when $\mathcal{D}(s_{t})>\tau_{\text{div}}$ . The policies are trained via self-play in a two-player stochastic game defined over the MDP-DAG, yielding stabilized adversarial decision policies.

a_{t}\leftarrow\mathrm{Arbitrate}(a_{B},a_{R},s_{t})

Algorithm 3 Council of Rivals Self-Play Training

Parameter	Value	Specification
State dimension $d$	256	Resulting dimensionality after concatenation and linear projection of host, network, and investigation-context features.
Hidden layer sizes	[256, 256, 128]	Shared multilayer perceptron backbone with ReLU activations and Layer Normalization.
Optimizer	Adam	$\beta_{1}=0.9$ , $\beta_{2}=0.999$ , default $\epsilon=10^{-8}$ .
Learning rate	$3\times 10^{-4}$	Selected via validation-based hyperparameter search.
Batch size	128	Mini-batches sampled from prioritized replay buffer.
Replay buffer capacity	$5\times 10^{5}$ transitions	Per-agent buffer with proportional prioritized sampling.

TABLE X: Council of Rivals Training Configuration

Component	Operational Definition
Clarity	Concentration and stability of explanatory signals, quantified via (i) entropy of feature-level attention distributions, (ii) local linear fidelity of the Blue-Team value function $Q_{B}$ , and (iii) stability of action rankings under bounded state perturbations.
Completeness	Evidential sufficiency and coverage, measured by (i) replay-buffer state density in a neighborhood of $s_{t}$ , (ii) entropy of empirically observed action distributions in similar historical states, and (iii) diversity of feature-attribution contributions.
Confidence	Epistemic certainty of the recommended action $a^{\star}$ , estimated through (i) inverse effective policy temperature $\tau_{\pi_{B}}$ , (ii) bootstrap variance of $Q_{B}(s_{t},a^{\star})$ , and (iii) a monotonic transformation of the inter-agent Policy Divergence Score $\mathcal{D}(s_{t})$ .
Aggregation	Weighted combination $\text{ETS}(s_{t})=w_{1}\cdot\text{Clarity}+w_{2}\cdot\text{Completeness}+w_{3}\cdot\text{Confidence}$ , followed by bounded normalization to $[0,1]$ for use in escalation gating.

TABLE XI: Decomposition of the ETS

Fig. 5: Illustrative decomposition of the ETS into its three primary components, Clarity, Completeness, and Confidence, and their respective sub-metrics. The figure visualizes the aggregation structure used to compute ETS $(s_{t})$ and demonstrates how the resulting score is compared against the escalation threshold $\tau_{\text{escalate}}$ . In this example, ETS $(s_{t})=0.82$ exceeds the calibrated threshold, and the system proceeds without mandatory human escalation. When ETS $(s_{t})$ falls below the threshold, structured explanation artifacts and human oversight are triggered.

\displaystyle\mathrm{Clarity}(s_{t})=\ \alpha_{1}(1-H(\alpha_{t}))+\alpha_{2}\mathrm{Linearity}(Q_{B})+\alpha_{3}\mathrm{Consistency}(\pi_{B}).

Algorithm 4 ETS Computation

Fig. 6: Illustration of the Shadow-Jitter attack scenario. Adversarial timing perturbations and log injections distort telemetry observations, inducing misleading correlations in conventional monolithic detection pipelines. In contrast, C-MADF leverages causal filtering and adversarial multi-agent validation to distinguish injected noise from genuine causal signals, thereby reducing false-positive mitigation actions under partial-compromise assumptions.

Framework	Precision	Recall	F1-Score	FPR
DUGAT-LSTM [ 40 ]	0.979	0.905	0.941	11.2%
BiLSTM IDS [ 39 ]	0.982	0.915	0.947	9.7%
Hybrid Weighted-XGBoost [ 45 ]	0.985	0.941	0.962	8.4%
C-MADF	0.997	0.961	0.979	1.8%

TABLE XII: Performance Comparison of C-MADF and Baseline Models (mean over 3 random seeds; standard deviations

<0.02

for all metrics)

Fig. 7: Defensive response under Shadow-Jitter injection: C-MADF maintains high detection with low false positives, structured human review, and strong explainability, while the baseline AI fails catastrophically due to hallucinated threat confirmation and lack of verification.

Configuration	Precision	Recall	F1-Score	FPR
C-MADF (Full Framework)	0.997	0.961	0.979	1.8%
w/o Causal SCM	0.903	0.959	0.930	59.4%
w/o Red Team	0.963	0.963	0.963	21.2%
w/o Blue Team	0.957	0.972	0.964	25.1%

TABLE XIII: Ablation Study: Impact of Individual Components on CICIoT2023 (mean over 3 random seeds; standard deviations

<0.02

for all entries)

Adversarial MARL

See all 17 papers

Mar 22, 2026

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Advanced Persistent Threats (APTs) are stealthy, multi-stage attacks that require adaptive and timely defense. While deep reinforcement learning (DRL) enables autonomous cyber defense, its decisions are often opaque and difficult to trust in operational environments. This paper presents DeepXplain, an explainable DRL framework for stage-aware APT defense. Building on our prior DeepStage model, DeepXplain integrates provenance-based graph learning, temporal stage estimation, and a unified XAI pipeline that provides structural, temporal, and policy-level explanations. Unlike post-hoc methods, explanation signals are incorporated directly into policy optimization through evidence alignment and confidence-aware reward shaping. To the best of our knowledge, DeepXplain is the first framework to integrate explanation signals into reinforcement learning for APT defense. Experiments in a realistic enterprise testbed show improvements in stage-weighted F1-score (0.887 to 0.915) and success rate (84.7% to 89.6%), along with higher explanation confidence (0.86), improved fidelity (0.79), and more compact explanations (0.31). These results demonstrate enhanced effectiveness and trustworthiness of autonomous cyber defense.

Trung V. Phan, Thomas Bauschert

Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany

Autonomous Cyber Defense Reinforcement Learning Interpretability in RL Explainable Artificial Intelligence

Ask PDF

Jul 30, 2026

Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

Autonomous Cyber Operations (ACO) are increasingly important for defending enterprise networks as cyber threats continue to evolve in sophistication. ACO applications commonly employ Reinforcement Learning (RL) agents to learn defensive behaviors through interaction with environments. However, RL agents typically require extensive exploration during training, often resulting in unstable behavior and poor initial decision-making before converging toward effective defense strategies. In this work, we investigate the use of a Large Language Model (LLM) to improve autonomous defensive decision-making within an ACO environment. Through prompt engineering rather than fine-tuning, we demonstrate that an 8-billion parameter LLM pretrained on cybersecurity data can outperform a baseline RL agent in a modified CybORG CAGE Challenge 2 environment. We then propose an online policy distillation framework that transfers the LLM's defensive policy into a lightweight RL agent containing only 64,910 parameters, reducing model size by several orders of magnitude while maintaining effective defensive capabilities. This provides a pathway toward operationalizing frontier cybersecurity models within lightweight, deployable agents. To evaluate transferability, we construct CybORG scenarios ranging from 4 to 12 hosts and assess the approach across varying network configurations. We also evaluate teacher-guided RL stabilization strategies and observe that none consistently surpass the optimized teacher policy, suggesting policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies. Our results demonstrate the potential of cybersecurity-focused LLMs as sources of expertise for autonomous cyber defense, while policy distillation provides a practical path toward operationalizing frontier cybersecurity models within efficient, scalable agents.

Konur Tholl, François Rivest, Mariam El Mezouar +2

Department of Electrical and Computer Engineering, Royal Military College of Canada, Kingston, Canada · Department of Mathematics and Computer Science, Royal Military College of Canada, Kingston, Canada · Defence Research and Development Canada, Ottawa, Canada +1

Autonomous Cyber Defense LLM-Guided RL LLMs for Cybersecurity Language Model Distillation

Ask PDF

Aug 5, 2026

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied. Meanwhile, recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have improved LLM reasoning, but their integration into cybersecurity remains elusive due to the absence of suitable benchmark environments and interaction datasets. To bridge this gap, we introduce Trident, an agentic LLM red teaming framework comprising three components: a dynamic benchmark with isolated sandbox servers spanning CybORG CAGE 4 and CyberWheel, a dataset comprises over 13,000 high-fidelity red-blue interaction trajectories for RLVR, and a ``Code-as-Policy'' RLVR agentic architecture Trident Agentic). The latter reformulates red agent training as a contextual bandit via a tripartite Log Summarizer--Planner--Coder design, where a trainable Planner generates complete attack strategies from compressed execution logs, which a frozen Coder translates into executable Python policies deployed against live DRL defenders. Empirical evaluations reveal a fundamental brittleness in existing defenses: with a single trainable 7B planner, Trident reduces blue agent defensive performance by an average of 522% compared to static red agent baselines while autonomously discovering emergent behaviors such as decoy avoidance and adaptive state prioritization that static heuristics entirely fail to uncover.

Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6

University of California, Irvine · Northeastern University · Johns Hopkins University

Autonomous Cyber Defense LLM Red Teaming AI Agent Security Benchmarks Reinforcement Learning with Verifiable Rewards Automated Red Teaming

Ask PDF

Abstract

Adversarial MARL

Explore similar work

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)