Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Making
Authors: Pasquale Malacaria, Yunxiao Zhang
Organizations: School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK. · Department of Computer Science, University of Exeter, Exeter, UK.
Large language models (LLMs) are increasingly used in cybersecurity workflows, yet it remains unclear whether they can perform structured security reasoning or merely rely on superficial cues and prior knowledge. We study this question in the context of defence selection over attack graphs derived from real-world threat scenarios, including ransomware, supply-chain compromise, cloud abuse, Kubernetes attacks, POS malware, and ICS/OT intrusion. Given a budget constraint, LLMs must select security controls to minimise attacker success. We compare their strategies against each other and against a game-theoretic optimization baseline used as a normative reference for structured reasoning. Our results show that LLMs exhibit conditional competence. When explicit attack-graph structure is provided, they often produce coherent strategies close to the optimization baseline. However, their capabilities are fragile. LLM behaviour becomes increasingly fragile with graph complexity and is highly sensitive to framing. Small prompt changes can substantially alter rankings, and merely relabeling a poor strategy as ``optimal'' dramatically improves its evaluation. We further observe a non-monotonic relationship between formal risk and LLM judgement: strategies closest to the optimum are not necessarily ranked highest by LLM evaluators. To further probe reasoning ability, we ask LLMs to generate solvers for the same optimization problem. While the generated implementations recover the correct high-level formulation, they scale poorly compared to a purpose-built solver. Overall, our findings show that LLMs can approximate structured cybersecurity reasoning under controlled representations, but do not apply it robustly. This has important implications for the design and evaluation of AI-assisted security decision-support systems.
Figures & tables
Figure 1: A simple attack graph.
Figure 2: Ransomware attack graph with controls per attack step. Controls are grouped and abbreviated for readability.
Figure 3: Structure of the prompt submitted to each LLM defender once per attack graph, covering all budget scenarios in a single call.
Figure 4: Risk of defenders across all budgets.
Figure 5: Structure of the prompt submitted to each LLM evaluator once per attack graph, covering all budget scenarios in a single call.
Defender
Exc.
V.G.
Good
Avg.
Bad
V.Bad
Score
Optimal
43.5%
21.3%
16.7%
11.1%
6.5%
0.9%
4.82
Gemini
33.3%
28.7%
17.6%
11.1%
6.5%
2.8%
4.63
Grok
37.0%
15.7%
23.1%
16.7%
4.6%
2.8%
4.56
ChatGPT
31.5%
24.1%
16.7%
20.4%
6.5%
0.9%
4.51
Claude
29.6%
19.4%
18.5%
20.4%
6.5%
5.6%
4.29
Coverage
24.1%
14.8%
18.5%
16.7%
22.2%
3.7%
3.91
Table 1: Classification frequency and average score of defender portfolios (108 evaluations per defender).
Graph ID
ρmean
p -value
1
−0.867
3.49×10−4
2
−0.581
1.57×10−5
3
−0.731
1.52×10−3
4
−0.558
2.66×10−3
5
−0.355
8.50×10−9
6
−0.315
7.47×10−6
Table 2: Per-graph Spearman correlation results. Negative values indicate alignment, since lower formal risk corresponds to higher LLM evaluation.
Group
Mean gap
Group 1 (Graphs 1–4)
+0.108
Group 2 (Graphs 5–7)
+2.188
Table 3: Claude-as-defender rank gap by complexity group. Positive mean gap indicates under-rating relative to formal risk.
Defender
Budget Level
B1
B2
B3
B4
(tight)
(moderate)
(comfortable)
(generous)
Optimal
4.71
5.43
4.75
4.29
ChatGPT
3.89
4.68
4.79
4.71
Gemini
4.64
4.82
4.46
4.58
Grok
4.82
4.82
4.57
3.92
Table 4: Mean Evaluation Score per Defender and Budget Level.
Figure 6: Classification Distribution per Evaluator.
Defender
Average Score Exp. 2
Average Score Exp. 3
Gemini
4.65
4.81
Optimal
4.59
4.74
ChatGPT
4.55
4.71
Grok
4.31
4.58
Claude
4.28
4.46
Greedy
3.92
4.55
Table 5: Aggregate evaluation scores for alternative experiments.
Evaluator
Mean Δ
Median Δ
ChatGPT
+0.33
0.0
Claude
+0.11
0.0
Gemini
−0.26
0.0
Grok
−0.19
0.0
Pooled
+0.0
0.0
Table 6: Self-preference effect by evaluator.
Condition
Mean Δ
Median Δ
OG (Optimal-label, Good strategy)
+0.98
+1
OM (Optimal-label, Misleading)
+1.68
+1
PG (Poor-label, Good strategy)
−0.44
+0
Table 7: Score difference for the target defender under each framing condition, relative to the baseline.
Evaluator
Exp. 1
Exp. 2
Exp. 3
ChatGPT
(3.78, 4.41, 33%)
(4.44, 4.41, 52%)
(4.72, 4.56, 59%)
Claude
(3.37, 4.82, 26%)
(4.37, 4.26, 44%)
(5.07, 4.35, 74%)
Gemini
(3.22, 4.78, 19%)
(4.59, 4.85, 37%)
(5.83, 4.82, 82%)
Grok
(3.37, 5.26, 26%)
(4.19, 4.85, 37%)
(5.56, 5.21, 67%)
All
(3.44, 4.82, 26%)
(4.40, 4.59, 43%)
(5.30, 4.74, 71%)
Table 8: Comparison of name-bias effects across three experiments. Each tuple reports ( Mean OM , Mean OPT , OM wins ), where OM denotes the poor strategy labeled optimal and OPT denotes the genuinely optimal defender.
Figure 7: Structure of the prompt submitted to each LLM reconstructor once per (graph, variant) pair.
Figure 8: Structure of the prompt submitted to each LLM judge for the reconstruction-quality evaluation.
(a) Effect of context ablation
Metric
A
B
C
D
ΔA→D
Structural similarity
7.42
7.21
7.16
6.46
−0.96
Semantic similarity
7.38
7.42
6.99
5.72
−1.66
Attack-path quality
9.11
8.98
9.03
8.54
−0.56
Overall fidelity
7.57
7.53
7.26
6.03
−1.55
Table 9: Summary of reconstruction results. (a) shows the effect of context ablation, averaged over reconstructors and evaluators. (b) shows reconstructor capability, averaged over variants and evaluators.
Defender
Anon. controls, full graph
Anon. controls, no graph
Optimal
4.77
4.31
ChatGPT
4.83
4.05
Gemini
4.57
4.32
Grok
4.54
3.89
Claude
4.48
4.02
Greedy
3.77
5.42
Table 10: Average score under the two ablation conditions.
Figure 9: Summary of the prompt submitted to each LLM to elicit a general solver for the decision problem.
Small
Medium
Large
Nodes
10
20
30
40
50
60
70
80
90
Avg. edges
14.0
51.3
106.0
182.0
252.0
355.8
498.0
631.3
793.3
Controls
5
10
15
20
25
30
35
40
45
Table 11: Graph statistics by node count, with four graphs generated for each node count.
Figure 10: Execution Time: Optimal vs LLMs by Graph Size Group, LLM timeouts (T) counted as 1,500 seconds, log scale.
Large Language Models (LLMs) are increasingly used in cybersecurity, where accurate analysis often requires multi-step and context-dependent reasoning over complex and heterogeneous data. However, existing prompting approaches typically focus on eliciting reasoning without explicitly considering how intermediate reasoning steps are structurally organized. We introduce Security Reasoning Topology, which models reasoning through three representative structures: Linear, Branching, and Graph. To evaluate their effects, we conduct controlled experiments on three cybersecurity datasets covering MITRE ATT&CK network traffic, cyber threat intelligence (CTI), and CVE vulnerability analysis. We evaluate multiple LLMs, including Llama 2 (7B, 13B, 70B), GPT-5.1, and Mistral Large 3, while keeping task inputs consistent and controlling reasoning structure through system-level prompting. Results show that reasoning topology substantially affects performance: Graph reasoning achieves the highest overall accuracy, improving over few-shot prompting by 9.8-12.2 percentage points across datasets, while Branching provides a strong intermediate solution. The results further show that the effect of reasoning topology remains consistent across model families and scales, highlighting reasoning topology as an important design factor for LLM-based cybersecurity analysis.
Jiling Zhou, Aisvarya Adeseye, Antti Hakkala +2
Department of Computing University of Turku Turku, Finland
Chain-of-Thought (CoT) prompting has been used to enhance the reasoning capability of LLMs. However, its reliability in security-sensitive analytical tasks remains insufficiently examined, particularly under structured human evaluation. Alternative approaches, such as model scaling and fine-tuning can be used to help improve performance. These methods are also often costly, computationally intensive, or difficult to audit. In contrast, prompt engineering provides a lightweight, transparent, and controllable mechanism for guiding LLM reasoning. This study proposes a structured prompt engineering framework designed to strengthen CoT reasoning integrity while improving security threat and attack detection reliability in local LLM deployments. The framework includes 16 factors grouped into four core dimensions: (1) Context and Scope Control, (2) Evidence Grounding and Traceability, (3) Reasoning Structure and Cognitive Control, and (4) Security-Specific Analytical Constraints. Rather than optimizing the wording of the prompt heuristically, the framework introduces explicit reasoning controls to mitigate hallucination and prevent reasoning drift, as well as strengthening interpretability in security-sensitive contexts. Using DDoS attack detection in SDN traffic as a case study, multiple model families were evaluated under structured and unstructured prompting conditions. Pareto frontier analysis and ablation experiments demonstrate consistent reasoning improvements (up to 40% in smaller models) and stable accuracy gains across scales. Human evaluation with strong inter-rater agreement (Cohen's k > 0.80) confirms robustness. The results establish structured prompting as an effective and practical approach for reliable and explainable AI-driven cybersecurity analysis.
Jiling Zhou, Aisvarya Adeseye, Seppo Virtanen +2
Department of Computing, University of Turku, 20014 Turku, Finland
Large Language Models (LLMs) are increasingly integrated into safety-critical workflows, yet existing security analyses remain fragmented and often isolate model behavior from the broader system context. This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS)-based exploitability scoring to support structured, comparable analysis. We demonstrate the framework through a healthcare case study, modeling multi-step attack paths targeting intervention in medical procedures, leakage of electronic health record (EHR) data, and disruption of service availability. Our analysis indicates that threats spanning (i) conventional cyber, (ii) adversarial ML, and (iii) conversational attacks that manipulate prompts or context often consolidate into a small number of dominant paths and shared system choke points, enabling targeted defenses to yield meaningful reductions in path exploitability. By systematically comparing defense portfolios, we align these risks with established vulnerability management practices and provide a domain-agnostic workflow applicable to other LLM-enabled critical systems.
Neha Nagaraja, Hayretdin Bahsi
School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia