Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning

Yiyao Zhang; Diksha Goel; Hussain Ahmad

cs.CRApr 6, 2026

Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning

Authors: Yiyao Zhang, Diksha Goel, Hussain Ahmad

Organizations: Adelaide University, Australia · CSIRO’s Data61, Australia

Abstract

Autonomous agents are increasingly deployed in both offensive and defensive cyber operations, creating high-speed, closed-loop interactions in critical infrastructure environments. Advanced Persistent Threat (APT) actors exploit "Living off the Land" techniques and targeted telemetry perturbations to induce ambiguity in monitoring systems, causing automated defenses to overreact or misclassify benign behavior as malicious activity. Existing monolithic and multi-agent defense pipelines largely operate on correlation-based signals, lack structural constraints on response actions, and are vulnerable to reasoning drift under ambiguous or adversarial inputs. We present the Causal Multi-Agent Decision Framework (C-MADF), a structurally constrained architecture for autonomous cyber defense that integrates causal modeling with adversarial dual-policy control. C-MADF first learns a Structural Causal Model (SCM) from historical telemetry and compiles it into an investigation-level Directed Acyclic Graph (DAG) that defines admissible response transitions. This roadmap is formalized as a Markov Decision Process (MDP) whose action space is explicitly restricted to causally consistent transitions. Decision-making within this constrained space is performed by a dual-agent reinforcement learning system in which a threat-optimizing Blue-Team policy is counterbalanced by a conservatively shaped Red-Team policy. Inter-policy disagreement is quantified through a Policy Divergence Score and exposed via a human-in-the-loop interface equipped with an Explainability-Transparency Score that serves as an escalation signal under uncertainty. On the real-world CICIoT2023 dataset, C-MADF reduces the false-positive rate from 11.2%, 9.7%, and 8.4% in three cutting-edge literature baselines to 1.8%, while achieving 0.997 precision, 0.961 recall, and 0.979 F1-score.

Figures & tables

Acronym/Term	Definition used in this paper
C-MADF	Causal Multi-Agent Decision Framework
APT	Advanced Persistent Threat
SCM	Structural Causal Model
DAG	Directed Acyclic Graph
MDP	Markov Decision Process
MDP-DAG roadmap	Causally constrained investigation roadmap compiled from SCM into executable MDP transitions

TABLE I: Acronyms and standardized terminology used in this paper

Approach	Formally Structured	Causal	Multi-Agent	Integrated	Decision-Level
	Decision Model	Grounding	Deliberation	XAI	Robustness
Causal Security Analytics [ 26 , 27 ]	$\triangle$ (offline)	$\checkmark$	$\times$	$\triangle$	$\triangle$
ML for Intrusion Detection [ 38 , 39 , 40 ]	$\times$	$\times$	$\times$	$\times$	$\triangle$ (data-level)
AICA-style Agents [ 19 ]	$\triangle$	$\times$	$\times$	$\times$	$\triangle$
XAI for Cybersecurity [ 29 , 23 ]	$\times$	$\times$	$\times$	$\checkmark$ (post-hoc)	$\times$
MAS in Cybersecurity [ 31 , 32 ]	$\times$	$\times$	$\checkmark$	$\times$	$\triangle$

TABLE II: Comparison of C-MADF with Representative Autonomous Cyber Defense Approaches

Symbol	Description
$\mathcal{S},\mathcal{A}$	State space and action space of the MDP.
$P(s^{\prime}\mid s,a),R(s,a),\gamma$	Transition kernel, reward function, and discount factor.
$\pi,V^{\pi},Q^{\pi}$	Policy, state-value function, and action-value function under policy $\pi$ .
$\mathcal{G}=(\mathcal{V},E)$	DAG induced by the learned SCM.
$\mathcal{M}_{C}=\langle U,\mathcal{V},F\rangle$	SCM with exogenous variables $U$ , endogenous variables $\mathcal{V}$ , and structural equations $F$ .
$\pi_{B},\pi_{R}$	Blue-Team and Red-Team policies in the dual-agent architecture.

TABLE III: Summary of Core Notation

Capability	Formal Description
Initial Compromise	Obtains unauthorized access to a subset $\mathcal{N}_{c}\subset\mathcal{N}$ via credential theft, phishing, or exploitation of vulnerabilities.
Lateral Movement	Moves within $\mathcal{N}$ using legitimate administrative tools and credentials (e.g., LotL techniques) to minimize detection signals.
Selective Telemetry Manipulation	Drops, delays, replays, injects, or perturbs subsets of host-, network-, or control-layer telemetry streams to induce ambiguity in state estimation.
Partial Historical Poisoning	Introduces bounded perturbations into a fraction of historical logs used for causal discovery, without full rewriting of all historical data.
Evasion Strategies	Mimics benign operational patterns and exploits monitoring blind spots to delay detection or trigger false mitigation decisions.

TABLE IV: Adversary Capabilities under the Partial-Compromise Model

Constraint	Scenario Restriction
Node-Level Compromise Bound	Adversary control is limited to a strict subset $\mathcal{N}_{c}\subset\mathcal{N}$ ; complete network takeover is excluded.
Telemetry Survivability	The adversary cannot suppress all sensing channels simultaneously; at least one trustworthy telemetry path remains observable.
Defender Runtime Integrity	The C-MADF execution environment, source code, and runtime parameters are not directly compromiseable by the adversary.
Secure-Channel and Crypto Integrity	The adversary cannot subvert secure communication channels or break underlying cryptographic primitives.
Out-of-Scope Full-Compromise Attacks	Full compromise of the defense stack, arbitrary code execution inside C-MADF, and complete rewriting of all historical data are out of scope.

TABLE V: Scenario Constraints under the Partial-Compromise Model

Fig. 1: Illustration of a Shadow-Jitter telemetry manipulation scenario. Controlled perturbations in host and network logs distort apparent event correlations, leading an unconstrained correlation-based defense to misclassify benign activity as malicious. In contrast, C-MADF applies causal filtering and adversarial dual-policy validation within a constrained MDP-DAG structure, reducing spurious mitigation actions under ambiguous observations.

ID	Objective Definition	Mechanisms in C-MADF
G1	Maintain high detection recall while limiting false-positive mitigation actions under bounded telemetry manipulation and partial historical poisoning.	Causally constrained action space derived from learned DAG $\mathcal{G}$ ; dual-policy adversarial deliberation; divergence-based escalation under high policy disagreement $\mathcal{D}(s_{t})$ .
G2	Ensure every executable mitigation action corresponds to an admissible transition in a formally defined investigation structure.	MDP-DAG structure restricting admissible state transitions; complete logging of state–action trajectories; compatibility with temporal-logic and model-checking analysis.
G3	Provide structured evidence traces and uncertainty signals for each proposed or executed action to support human oversight.	Human-in-the-loop interface exposing evidence traces and alternative hypotheses; ETS aggregating clarity, coverage, and policy uncertainty; explicit visualization of policy divergence $\mathcal{D}(s_{t})$ .
G4	Scale to large networks and high-volume telemetry streams without exponential growth in online decision complexity.	Modular architecture separating offline causal discovery from online decision-making; compact dual-policy networks $\pi_{B},\pi_{R}$ ; reduced branching via constrained action subsets $\mathcal{A}_{C}(s)$ .
G5	Update policies and causal estimates over time while preserving structural constraints on admissible action sequences.	Periodic or drift-triggered SCM re-estimation; continual reinforcement learning updates; recalibration of escalation thresholds under distributional shift.

TABLE VI: Mapping Design Objectives to C-MADF Mechanisms

Module	Inputs / Outputs	Formal Guarantee
Causal Discovery Engine	In: historical telemetry and event logs $\mathcal{D}$ ; Out: weighted causal DAG $\mathcal{G}=(V,E)$ with edge confidence scores.	Causal identifiability: filters spurious statistical dependencies and enables intervention- and counterfactual-consistent reasoning under structural constraints.
Causally-Constrained MDP (MDP-DAG)	In: $\mathcal{G}$ and investigation ontology; Out: constrained MDP $\mathcal{M}=(S,A,P,R)$ over admissible roadmap states and actions.	Semantic soundness: restricts policy exploration to transitions consistent with causal structure and operational semantics, ensuring verifiable action admissibility.
Council of Rivals Deliberation	In: current state $s_{t}$ and roadmap constraints; Out: adversarially validated action $\hat{a}_{t}$ and policy divergence score $\mathcal{D}(s_{t})$ .	Robust decision validation: dual-policy asymmetric self-play with objective divergence reduces reasoning drift, mitigates overconfidence, and exposes epistemic uncertainty via calibrated disagreement.
Explainable Human-in-the-Loop (HITL) Interface	In: deliberation artifacts (evidence traces, counterarguments, $\mathcal{D}(s_{t})$ ); Out: structured explanations, Explanation Transparency Score (ETS $(s_{t})$ ), and escalation triggers.	Operational transparency: produces operator-aligned rationales with quantifiable reliability calibration and principled escalation under high uncertainty.

TABLE VII: C-MADF Architectural Modules and Formal Guarantees

Fig. 2: The architecture of the Causal Multi-Agent Decision Framework (C-MADF). The process begins with the Causal Discovery Module learning a causal model from data. This model informs the MDP-DAG Roadmap, which provides a verifiable structure for investigations. The Council of Rivals, consisting of a Blue-Team and a Red-Team agent, deliberates on the best course of action within this roadmap. Their debate and the resulting policy divergence are fed into the Explainable Human-in-the-Loop Interface, which computes the ETS and presents a clear recommendation to the supervisory adjudication.

Component	Specification and Rationale
Discovery paradigm	Constraint-based causal structure learning (PC/FCI family), selected for its ability to produce an explicit, interpretable DAG consistent with observed conditional-independence relations in telemetry data. FCI is employed when latent confounding cannot be excluded.
Conditional independence testing	Hybrid CI testing pipeline: partial correlation tests for approximately Gaussian continuous variables; kernel-based or mutual-information–based tests for mixed or non-Gaussian variables. Tests are selected based on feature-type metadata and validated via hold-out consistency checks.
Significance threshold	Statistical significance level $\alpha\in[0.01,0.05]$ , selected through validation to balance false edge removal (Type I error) and spurious dependency retention (Type II error). Sensitivity analysis is performed across this range.
Maximum conditioning set size	Conditioning set size bounded (e.g., $\|Z\|\leq 2$ – $4$ ) to control computational complexity and reduce instability in high-dimensional telemetry. This bounds worst-case complexity of independence testing and mitigates overfitting.
Edge orientation constraints	Standard v-structure orientation and acyclicity enforcement, optionally augmented with domain priors (e.g., host process creation precedes outbound network flow). Prior constraints are encoded as forbidden or required edges.
Concept drift handling	Periodic offline re-estimation or incremental updates triggered when distributional shift exceeds a predefined divergence threshold (e.g., population stability index or KL divergence). Drift detection prevents structural staleness.

TABLE VIII: Causal Discovery Configuration and Design Choices

Fig. 3: The MDP-DAG Investigation Roadmap showing valid state transitions (orange arrows with reward values) and causally inconsistent blocked transitions (red X). The roadmap constrains the action space to ensure verifiable investigation flows, preventing logically inconsistent paths such as jumping directly from Initial Alert to Threat Mitigated without sufficient evidence collection.

Reward Component	Blue-Team Policy ( $\pi_{B}$ )	Red-Team Policy ( $\pi_{R}$ )
Verified mitigation	Large positive reward for reaching causally justified terminal states (e.g., “Threat Mitigated”) after traversing required evidence states.	Small positive reward only if mitigation is supported by sufficient evidence; no reward for premature containment.
False-positive intervention	Penalty proportional to operational disruption cost (e.g., unnecessary isolation, service interruption).	Large positive reward for preventing or opposing unjustified disruptive actions; penalty if false positives are not challenged.
Evidence acquisition	Positive reward for actions that increase information gain or reduce causal uncertainty along the learned DAG.	Positive reward for demanding additional evidence under high epistemic uncertainty; penalty for accepting unsupported mitigation claims.
Investigation efficiency	Small step penalty to discourage unnecessary actions and encourage timely resolution.	Smaller penalty weight relative to Blue-Team, prioritizing verification over speed in ambiguous states.
Roadmap compliance	Penalty for proposing actions inconsistent with MDP-DAG constraints.	Equivalent penalty for violating causal or roadmap admissibility constraints.
Human escalation	Neutral or small penalty when escalation is triggered under high divergence $\mathcal{D}(s_{t})$ .	Positive reward when escalation is correctly triggered under high epistemic uncertainty.

TABLE IX: Reward Shaping Design for the Council of Rivals

Fig. 4: Council of Rivals adversarial deliberation architecture. At each investigation state $s_{t}$ in the causally constrained MDP-DAG, the Blue-Team policy proposes a mitigation action based on its learned value estimates, while the Red-Team policy evaluates and challenges the proposal under a conservatively shaped objective. The interaction proceeds through iterative hypothesis evaluation, evidential justification, and action arbitration within the constrained action space. Inter-policy disagreement is quantified via the Policy Divergence Score $\mathcal{D}(s_{t})$ , which serves as an epistemic uncertainty signal and triggers human escalation when $\mathcal{D}(s_{t})>\tau_{\text{div}}$ . The policies are trained via self-play in a two-player stochastic game defined over the MDP-DAG, yielding stabilized adversarial decision policies.

a_{t}\leftarrow\mathrm{Arbitrate}(a_{B},a_{R},s_{t})

Algorithm 3 Council of Rivals Self-Play Training

Parameter	Value	Specification
State dimension $d$	256	Resulting dimensionality after concatenation and linear projection of host, network, and investigation-context features.
Hidden layer sizes	[256, 256, 128]	Shared multilayer perceptron backbone with ReLU activations and Layer Normalization.
Optimizer	Adam	$\beta_{1}=0.9$ , $\beta_{2}=0.999$ , default $\epsilon=10^{-8}$ .
Learning rate	$3\times 10^{-4}$	Selected via validation-based hyperparameter search.
Batch size	128	Mini-batches sampled from prioritized replay buffer.
Replay buffer capacity	$5\times 10^{5}$ transitions	Per-agent buffer with proportional prioritized sampling.

TABLE X: Council of Rivals Training Configuration

Component	Operational Definition
Clarity	Concentration and stability of explanatory signals, quantified via (i) entropy of feature-level attention distributions, (ii) local linear fidelity of the Blue-Team value function $Q_{B}$ , and (iii) stability of action rankings under bounded state perturbations.
Completeness	Evidential sufficiency and coverage, measured by (i) replay-buffer state density in a neighborhood of $s_{t}$ , (ii) entropy of empirically observed action distributions in similar historical states, and (iii) diversity of feature-attribution contributions.
Confidence	Epistemic certainty of the recommended action $a^{\star}$ , estimated through (i) inverse effective policy temperature $\tau_{\pi_{B}}$ , (ii) bootstrap variance of $Q_{B}(s_{t},a^{\star})$ , and (iii) a monotonic transformation of the inter-agent Policy Divergence Score $\mathcal{D}(s_{t})$ .
Aggregation	Weighted combination $\text{ETS}(s_{t})=w_{1}\cdot\text{Clarity}+w_{2}\cdot\text{Completeness}+w_{3}\cdot\text{Confidence}$ , followed by bounded normalization to $[0,1]$ for use in escalation gating.

TABLE XI: Decomposition of the ETS

Fig. 5: Illustrative decomposition of the ETS into its three primary components, Clarity, Completeness, and Confidence, and their respective sub-metrics. The figure visualizes the aggregation structure used to compute ETS $(s_{t})$ and demonstrates how the resulting score is compared against the escalation threshold $\tau_{\text{escalate}}$ . In this example, ETS $(s_{t})=0.82$ exceeds the calibrated threshold, and the system proceeds without mandatory human escalation. When ETS $(s_{t})$ falls below the threshold, structured explanation artifacts and human oversight are triggered.

\displaystyle\mathrm{Clarity}(s_{t})=\ \alpha_{1}(1-H(\alpha_{t}))+\alpha_{2}\mathrm{Linearity}(Q_{B})+\alpha_{3}\mathrm{Consistency}(\pi_{B}).

Algorithm 4 ETS Computation

Fig. 6: Illustration of the Shadow-Jitter attack scenario. Adversarial timing perturbations and log injections distort telemetry observations, inducing misleading correlations in conventional monolithic detection pipelines. In contrast, C-MADF leverages causal filtering and adversarial multi-agent validation to distinguish injected noise from genuine causal signals, thereby reducing false-positive mitigation actions under partial-compromise assumptions.

Framework	Precision	Recall	F1-Score	FPR
DUGAT-LSTM [ 40 ]	0.979	0.905	0.941	11.2%
BiLSTM IDS [ 39 ]	0.982	0.915	0.947	9.7%
Hybrid Weighted-XGBoost [ 45 ]	0.985	0.941	0.962	8.4%
C-MADF	0.997	0.961	0.979	1.8%

TABLE XII: Performance Comparison of C-MADF and Baseline Models (mean over 3 random seeds; standard deviations

<0.02

for all metrics)

Fig. 7: Defensive response under Shadow-Jitter injection: C-MADF maintains high detection with low false positives, structured human review, and strong explainability, while the baseline AI fails catastrophically due to hallucinated threat confirmation and lack of verification.

Configuration	Precision	Recall	F1-Score	FPR
C-MADF (Full Framework)	0.997	0.961	0.979	1.8%
w/o Causal SCM	0.903	0.959	0.930	59.4%
w/o Red Team	0.963	0.963	0.963	21.2%
w/o Blue Team	0.957	0.972	0.964	25.1%

TABLE XIII: Ablation Study: Impact of Individual Components on CICIoT2023 (mean over 3 random seeds; standard deviations

<0.02

for all entries)

Adversarial MARL

See all 17 papers

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns
Mar 22, 2026Trung V. Phan, Thomas BauschertAutonomous Cyber Defense Reinforcement Learning
Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany
Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations
Jul 30, 2026Konur Tholl, François Rivest, Mariam El Mezouar +2Autonomous Cyber Defense LLM-Guided RL
Department of Electrical and Computer Engineering, Royal Military College of Canada, Kingston, Canada · Department of Mathematics and Computer Science, Royal Military College of Canada, Kingston, Canada · Defence Research and Development Canada, Ottawa, Canada +1
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Autonomous Cyber Defense LLM Red Teaming
University of California, Irvine · Northeastern University · Johns Hopkins University

Mar 22, 2026J/K move · Enter open · S save

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Trung V. Phan, Thomas Bauschert

Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany

Autonomous Cyber Defense Reinforcement Learning Interpretability in RL Explainable Artificial Intelligence

Open paper AskPDF arXiv

Advanced Persistent Threats (APTs) are stealthy, multi-stage attacks that require adaptive and timely defense. While deep reinforcement learning (DRL) enables autonomous cyber defense, its decisions are often opaque and difficult to trust in operational environments. This paper presents DeepXplain, an explainable DRL framework for stage-aware APT defense. Building on our prior DeepStage model, DeepXplain integrates provenance-based graph learning, temporal stage estimation, and a unified XAI pipeline that provides structural, temporal, and policy-level explanations. Unlike post-hoc methods, explanation signals are incorporated directly into policy optimization through evidence alignment and confidence-aware reward shaping. To the best of our knowledge, DeepXplain is the first framework to integrate explanation signals into reinforcement learning for APT defense. Experiments in a realistic enterprise testbed show improvements in stage-weighted F1-score (0.887 to 0.915) and success rate (84.7% to 89.6%), along with higher explanation confidence (0.86), improved fidelity (0.79), and more compact explanations (0.31). These results demonstrate enhanced effectiveness and trustworthiness of autonomous cyber defense.

Abstract

Adversarial MARL

Explore similar work

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)