Bounded Reasoning: Cognitive Hierarchy in Human-versus-AI Cyber Defense

Zahra Aref; Sheng Wei; Narayan B. Mandayam

cs.CROct 4, 2026

Bounded Reasoning: Cognitive Hierarchy in Human-versus-AI Cyber Defense

Authors: Zahra Aref, Sheng Wei, Narayan B. Mandayam

Organizations: Rutgers University

Abstract

Human-agent evaluations often compress interaction into a single performance score, even when human and automated policies adapt differently over time. We study this in a sequential cyber-defense game on an attack graph, where a human or reinforcement-learning defender protects cloud assets against a Deep Q-Network (DQN) attacker. We compare four defender settings: a human reward-only game operationalizing the DQN information structure, a human reward-plus-transition game operationalizing the Cognitive Hierarchy Theory-driven DQN (CHT-DQN) information structure, an automated DQN defender, and an automated CHT-DQN defender. In the reward-only game, participants receive payoff and reward feedback. In the reward-plus-transition game, they also see attacker-aware transition probabilities from the CHT-DQN model. Across 80 Mechanical Turk participants and matched automated simulations, human defenders adapted to outcomes in a way the automated defenders did not: they were more likely to reselect a node after a successful defense than after a failure, in both games, an asymmetry we interpret as consistent with Prospect Theory and Cumulative Prospect Theory. The 40-round average also mixes early rounds, where the DQN attacker acts mostly at random, with late rounds, where it mostly exploits; in the final stage, mean protection ranks the reward-plus-transition human game above the reward-only human game, then the automated CHT-DQN defender, then the automated DQN defender, an ordering the overall mean hides. By contrast, the reward-plus-transition game does not produce a statistically reliable overall gain in weighted data protection over the reward-only game. These results suggest that evaluating human-agent cyber-defense systems only by an averaged task score can miss behaviorally meaningful differences in adaptation, action allocation, and bounded human reasoning.

Figures & tables

Panel of Figure 1 — Figure 1: The two human games. Both are human versus a DQN attacker. $\mathrm{H}_{\mathrm{reward}}$ copies DQN; $\mathrm{H}_{\mathrm{forecast}}$ copies CHT–DQN.

Figure 2: Weighted data-protection ratio by ten-round stage. The full-session average hides a stage-dependent ordering: automated defenders lead in rounds 11–20, while in rounds 31–40, when exploration is low, the means rank $\mathrm{H}_{\mathrm{forecast}}>\mathrm{H}_{\mathrm{reward}}>$ Auto-CHT $>$ Auto-DQN. Error bars show standard deviations over 40 participants for human games and 40 seeds for automated baselines; human-versus-automation comparisons are descriptive because the units differ.

Contrast	Estimate	95% CI	$p$	Effect
Protection, $\mathrm{H}_{\mathrm{forecast}}-\mathrm{H}_{\mathrm{reward}}$	$0.006$	$[-0.006,0.019]$	$0.33$	$g{=}0.22$
Protection, rounds 21–30	$0.018$	$[-0.005,0.041]$	$0.12$	$g{=}0.35$
Protection, rounds 31–40	$0.015$	$[-0.014,0.044]$	$0.29$	$g{=}0.23$
$P(\mathrm{reselect}\mid\mathrm{succ}{-}\mathrm{fail})$	${\approx}0.21$	—	${<}10^{-6}$	Wilcoxon

Table 1: Human-subject contrasts. Protection is between-subjects (

n{=}40

per human game); reselection is within-participant (

n{=}80

). Protection CIs are bootstrap 95% intervals. At this sample the protection contrast resolves

d{\geq}0.63

at 80% power.

Panel of Figure 3 — Figure 3: Humans reselect less after failure and more after success (observed in both games); automated defenders do not. We interpret this asymmetry as consistent with PT/CPT.

Figure 4: Mean frequency of defended (top) and attacked (bottom) nodes by stage. Humans spread mass more evenly than automated defenders. Closer visual match under $\mathrm{H}_{\mathrm{forecast}}$ is not a tested protection gain.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Human-versus-AI setting of Section 3.1 . The model treats the defender as level 1 and the attacker as level 0; we do not claim that participants compute Eq. ( 1 ).

Panel of Figure 6 — Figure 6: Automated simulation context. Error bars: SD over 10 seeds.

Panel of Figure 7 — Figure 7: Convergence on a 6-node graph (10 seeds). Training: steps $0$ – $1000$ ; evaluation: $1000$ – $2000$ .

Prospect Theory

See all 9 papers

Sep 22, 2026

Toward Responsible AI-Augmented Cyber Defense: Pattern Recognition, Defense-in-Depth, and the Case for Human-AI Collaboration

Cybersecurity literature has extensively documented the operational benefits of artificial intelligence (AI) for threat detection, incident response, and prevention, while raising qualitative concerns about over-automation, algorithmic bias, and analyst-skill erosion. What remains largely absent is a formal, falsifiable model connecting three constructs that recur across this literature: Defense-in-Depth Theory, the Artificial Intelligence Theory of Pattern Recognition, and human-AI collaboration in security operations. This paper develops such a model. We formalize layered defense as a Bernoulli detection cascade in which AI augmentation enters multiplicatively across layers; we formalize each layer's pattern-recognition behavior as a Neyman-Pearson/Bayesian detector with a derived closed-form optimal threshold; and we formalize human-AI triage as a capacity-constrained cascade with an explicit, quantifiable trade-off between detection probability and false-alarm ("alert fatigue") rate. A Monte Carlo/analytical simulation evaluated at illustrative but realistic operating points shows that (i) AI augmentation compounds across defense layers, delivering its largest marginal gains exactly where traditional layering saturates, and (ii) full human review of AI-flagged alerts is not optimal: increasing analyst capacity toward 100% coverage cuts false alarms by roughly 20-fold but simultaneously lowers system-level detection probability, because imperfect analyst accuracy is then applied to every alert rather than a filtered subset. These results give the widely repeated qualitative recommendation of "balanced human-AI collaboration" a precise, testable form and suggest an interior-optimum capacity ratio as a concrete design target for security operations centers (SOCs), including those securing IT/OT-converged critical infrastructure.

Mustafa S. Aljumaily, Hayder Kareem Abed, Nawar S. Alseelawi

Research and Development (R&D) Department, Daw Alfada Company, Baghdad, Iraq · CEO, Daw Alfada Company, Baghdad, Iraq · University of Misan, Maysan, Iraq

Cybersecurity Human-AI Decision Making Human-AI Collaboration

Ask PDF

Aug 5, 2026

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied. Meanwhile, recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have improved LLM reasoning, but their integration into cybersecurity remains elusive due to the absence of suitable benchmark environments and interaction datasets. To bridge this gap, we introduce Trident, an agentic LLM red teaming framework comprising three components: a dynamic benchmark with isolated sandbox servers spanning CybORG CAGE 4 and CyberWheel, a dataset comprises over 13,000 high-fidelity red-blue interaction trajectories for RLVR, and a ``Code-as-Policy'' RLVR agentic architecture Trident Agentic). The latter reformulates red agent training as a contextual bandit via a tripartite Log Summarizer--Planner--Coder design, where a trainable Planner generates complete attack strategies from compressed execution logs, which a frozen Coder translates into executable Python policies deployed against live DRL defenders. Empirical evaluations reveal a fundamental brittleness in existing defenses: with a single trainable 7B planner, Trident reduces blue agent defensive performance by an average of 522% compared to static red agent baselines while autonomously discovering emergent behaviors such as decoy avoidance and adaptive state prioritization that static heuristics entirely fail to uncover.

Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6

University of California, Irvine · Northeastern University · Johns Hopkins University

Autonomous Cyber Defense LLM Red Teaming AI Agent Security Benchmarks Reinforcement Learning with Verifiable Rewards Automated Red Teaming

Ask PDF

Apr 6, 2026

Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning

Autonomous agents are increasingly deployed in both offensive and defensive cyber operations, creating high-speed, closed-loop interactions in critical infrastructure environments. Advanced Persistent Threat (APT) actors exploit "Living off the Land" techniques and targeted telemetry perturbations to induce ambiguity in monitoring systems, causing automated defenses to overreact or misclassify benign behavior as malicious activity. Existing monolithic and multi-agent defense pipelines largely operate on correlation-based signals, lack structural constraints on response actions, and are vulnerable to reasoning drift under ambiguous or adversarial inputs. We present the Causal Multi-Agent Decision Framework (C-MADF), a structurally constrained architecture for autonomous cyber defense that integrates causal modeling with adversarial dual-policy control. C-MADF first learns a Structural Causal Model (SCM) from historical telemetry and compiles it into an investigation-level Directed Acyclic Graph (DAG) that defines admissible response transitions. This roadmap is formalized as a Markov Decision Process (MDP) whose action space is explicitly restricted to causally consistent transitions. Decision-making within this constrained space is performed by a dual-agent reinforcement learning system in which a threat-optimizing Blue-Team policy is counterbalanced by a conservatively shaped Red-Team policy. Inter-policy disagreement is quantified through a Policy Divergence Score and exposed via a human-in-the-loop interface equipped with an Explainability-Transparency Score that serves as an escalation signal under uncertainty. On the real-world CICIoT2023 dataset, C-MADF reduces the false-positive rate from 11.2%, 9.7%, and 8.4% in three cutting-edge literature baselines to 1.8%, while achieving 0.997 precision, 0.961 recall, and 0.979 F1-score.

Yiyao Zhang, Diksha Goel, Hussain Ahmad

Adelaide University, Australia · CSIRO’s Data61, Australia

Adversarial MARL Autonomous Cyber Defense Multi-Agent Reinforcement Learning Causal RL Interpretability in RL

Ask PDF

Abstract

Prospect Theory

Explore similar work

Toward Responsible AI-Augmented Cyber Defense: Pattern Recognition, Defense-in-Depth, and the Case for Human-AI Collaboration

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning