Beyond RAG for Cyber Threat Intelligence: A Systematic Evaluation of Graph-Based and Agentic Retrieval

Dzenan Hamzic; Florian Skopik; Max Landauer; Markus Wurzenberger; Andreas Rauber

cs.AIApr 13, 2026

Beyond RAG for Cyber Threat Intelligence: A Systematic Evaluation of Graph-Based and Agentic Retrieval

Authors: Dzenan Hamzic, Florian Skopik, Max Landauer, Markus Wurzenberger, Andreas Rauber

Organizations: AIT Austrian Institute of Technology, Austria · TU Wien, Austria

Abstract

Cyber threat intelligence (CTI) analysts must answer complex questions over large collections of narrative security reports. Retrieval-augmented generation (RAG) systems help language models access external knowledge, but traditional vector retrieval often struggles with queries that require reasoning over relationships between entities such as threat actors, malware, and vulnerabilities. This limitation arises because relevant evidence is often distributed across multiple text fragments and documents. Knowledge graphs address this challenge by enabling structured multi-hop reasoning through explicit representations of entities and relationships. However, multiple retrieval paradigms, including graph-based, agentic, and hybrid approaches, have emerged with different assumptions and failure modes. It remains unclear how these approaches compare in realistic CTI settings and when graph grounding improves performance. We present a systematic evaluation of four RAG architectures for CTI analysis: standard vector retrieval, graph-based retrieval over a CTI knowledge graph, an agentic variant that repairs failed graph queries, and a hybrid approach combining graph queries with text retrieval. We evaluate these systems on 3,300 CTI question-answer pairs spanning factual lookups, multi-hop relational queries, analyst-style synthesis questions, and unanswerable cases. Results show that graph grounding improves performance on structured factual queries. The hybrid graph-text approach improves answer quality by up to 35 percent on multi-hop questions compared to vector RAG, while maintaining more reliable performance than graph-only systems.

Figures & tables

Figure 1. AI Retrieval in CTI Domain.

Paper	Architecture focus	Relevance to this work
Edge et al. ( Edge et al., 2025 )	Hierarchical GraphRAG	Hierarchical graph retrieval for summarization; motivates multi-level retrieval
Pan et al. ( Pan et al., 2024 )	KG–LLM integration	Conceptual taxonomy for graph–LLM integration
Wu et al. ( Wu and et al., 2025 )	Domain GraphRAG	Medical GraphRAG with domain-specific enhancements
Agentic Medical ( Anonymous, 2025 )	Agentic GraphRAG	Self-correcting agentic refinement for structured retrieval
Papageorgiou et al. ( Papageorgiou et al., 2025 )	Hybrid GraphRAG	Hybrid multi-agent GraphRAG; explainability benefits
Liang et al. ( Liang et al., 2025 )	Robustness analysis	Security evaluation of GraphRAG under poisoning attacks

Table 1. Summary of related graph-based RAG work

Figure 2. Data generation and evaluation workflow.

Metric	Value
Question Diversity
Total questions (per run)	66
Unique 4-word prefixes	398/522 (76.1%)
Answer Conciseness (factoid categories only)
Categories included	simple, single/multi-hop, unans.
Answers $\leq$ 12 words	99.4%

Table 2. Generated Question Quality and Validation Metrics

Figure 3. Overview of different RAG systems.

Figure 4. GRAG system configuration.

Figure 5. HybridRAG parallel pipeline

Table 8

Table 9

Table 10

System	Mean $\Delta$ vs RAG	95% Bootstrap CI	Median $\Delta$	Cohen’s $d$
GRAG	$+1.34$	$[0.40,\,2.28]$	$0$	$0.05$
AGRAG	$+10.68$	$[9.84,\,11.52]$	$+4$	$0.43$
HRAG	$+11.68$	$[10.88,\,12.45]$	$+4$	$0.52$

Table 9. LLM-as-a-Judge improvements over RAG (RQ1). Mean and median differences are computed over 3,300 paired question instances.

System	Mean $\Delta$ vs RAG	95% Bootstrap CI	Median $\Delta$	Cohen’s $d$
GRAG	$-0.06$	$[-0.53,\,0.37]$	$+0.12$	$-0.05$
AGRAG	$+1.93$	$[0.83,\,3.01]$	$+2.19$	$0.76$
HRAG	$+1.80$	$[0.92,\,2.68]$	$+1.68$	$0.88$

Table 10. Classical-metric improvements over vanilla RAG (RQ1). Mean and median differences are computed over 3,300 paired question instances using the equally weighted composite score of F1, BLEU, ROUGE-1, ROUGE-L, and BERTScore.

Figure 6. Score distribution across RAG systems (LLM-as-a-Judge, 0–50 scale). Each cell reports the percentage of the evaluated question instances falling in the indicated score range

Category	System	Mean $\Delta$ vs RAG	95% CI	Median $\Delta$	Cohen’s $d$
guided	AGRAG	$-8.74$	$[-10.29,\,-7.19]$	$-7.0$	$-0.39$
guided	GRAG	$-17.67$	$[-19.17,\,-16.13]$	$-18.0$	$-0.81$
guided	HRAG	$+4.32$	$[2.74,\,5.86]$	$+1.0$	$0.19$
multi_hop	AGRAG	$+17.64$	$[16.04,\,19.29]$	$+26.0$	$0.78$
multi_hop	GRAG	$+10.37$	$[8.61,\,12.17]$	$+4.0$	$0.41$
multi_hop	HRAG	$+15.18$	$[13.69,\,16.69]$	$+9.5$	$0.72$

Table 11. RQ2 (LLM-as-a-Judge): Improvements over semantic RAG by question type. Mean differences are shown with 95% bootstrap confidence intervals.

n

denotes the number of paired question instances.

Category	System	Mean $\Delta$ vs RAG	95% CI	Median $\Delta$	Cohen’s $d$
guided	AGRAG	$-1.74$	$[-2.44,\,-0.95]$	$-2.26$	$-1.81$
guided	GRAG	$-1.80$	$[-2.33,\,-1.00]$	$-2.02$	$-1.96$
guided	HRAG	$-0.84$	$[-1.66,\,-0.25]$	$-0.49$	$-0.90$
multi_hop	AGRAG	$+2.10$	$[1.85,\,2.31]$	$+2.07$	$6.97$
multi_hop	GRAG	$+0.44$	$[0.16,\,0.78]$	$+0.38$	$1.08$
multi_hop	HRAG	$+2.09$	$[1.45,\,2.68]$	$+2.57$	$2.62$

Table 12. RQ2 (Classical metrics): Improvements over semantic RAG by question type using the composite score of F1, BLEU, ROUGE-1, ROUGE-L, and BERTScore. Mean differences are shown with 95% bootstrap confidence intervals over paired (model, category) cells (

n{=}5

models).

Table 16

Figure 7. Mean ranks and top-placement frequency by question category (10 runs, LLM-as-a-Judge). (a) Mean rank (1 = best); (b) sole best rate; (c) best-or-tied rate.

Model	GRAG Mean	AGRAG $-$ GRAG $\Delta$	HRAG $-$ GRAG $\Delta$	GRAG Collapse Rate
Kimi K2 Thinking	$42.55$ [41.40, 43.65]	$+1.06$ [0.46, 1.69]	$+2.65$ [1.87, 3.42]	$0.04$
Mixtral 8x7B	$29.91$ [28.52, 31.25]	$+4.38$ [3.62, 5.12]	$+9.62$ [8.33, 10.86]	$0.41$
Mistral Small 24B	$31.18$ [29.74, 32.55]	$+3.64$ [2.88, 4.42]	$+7.88$ [6.59, 9.12]	$0.36$
GPT-4.1-mini	$36.49$ [35.36, 37.63]	$+1.79$ [1.28, 2.30]	$+3.88$ [3.01, 4.77]	$0.18$
GPT-5.2	$33.02$ [31.75, 34.34]	$+1.64$ [0.96, 2.32]	$+10.65$ [8.99, 12.35]	$0.27$

Table 15. RQ3 (LLM impact on text-to-Cypher performance): Judge-based performance of Cypher-dependent systems by underlying LLM. Mean scores and deltas are reported with 95% bootstrap confidence intervals. Collapse rate indicates the fraction of GRAG answers with near-failure scores (

\leq 10

10 on the 0–50 judge scale).

Category	Collapsed / Total	Collapse Rate
Unanswerable	50 / 50	100.0%
Guided	8 / 160	5.0%
Multi-hop	7 / 150	4.7%
Single-hop	1 / 150	0.7%
Simple	0 / 150	0.0%

Table 16. GRAG collapse rate by question category for GPT-5.2 (collapsed = judge score

\leq 10/50

Model	GRAG Mean	AGRAG $-$ GRAG $\Delta$	HRAG $-$ GRAG $\Delta$	HRAG $-$ RAG $\Delta$
Kimi K2 Thinking	$0.41$ [0.38, 0.44]	$+0.06$ [0.03, 0.09]	$+0.12$ [0.08, 0.16]	$+0.18$ [0.14, 0.22]
Mixtral 8x7B	$0.19$ [0.16, 0.22]	$+0.09$ [0.06, 0.12]	$+0.21$ [0.17, 0.25]	$+0.26$ [0.21, 0.31]
Mistral Small 24B	$0.22$ [0.19, 0.25]	$+0.08$ [0.05, 0.11]	$+0.18$ [0.14, 0.22]	$+0.23$ [0.18, 0.28]
GPT-4.1-mini	$0.29$ [0.26, 0.32]	$+0.05$ [0.03, 0.07]	$+0.09$ [0.06, 0.12]	$+0.14$ [0.11, 0.17]
GPT-5.2	$0.31$ [0.28, 0.34]	$+0.04$ [0.02, 0.06]	$+0.23$ [0.18, 0.28]	$+0.28$ [0.22, 0.34]

Table 17. RQ3 (Classical metrics): Impact of the underlying LLM on Cypher-dependent systems. Scores are based on the composite classical metric. Mean values and deltas are reported with 95% bootstrap confidence intervals.

Model	System	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	Mean	Std
GPT-5.2	AGRAG	6.87	10.12	9.80	9.08	12.69	10.17	10.51	9.71	7.40	8.84	9.52	1.55
	GRAG	67.76	137.53	143.92	66.08	178.01	150.08	100.11	106.59	179.69	145.49	127.53	38.74
	HRAG	37.05	54.91	43.15	42.12	52.77	46.89	53.14	41.84	47.37	50.09	46.93	5.54
	RAG	1.27	1.30	1.22	1.38	1.50	1.25	1.26	1.23	1.34	1.41	1.32	0.08
Mixtral	AGRAG	5.90	5.21	6.83	6.68	6.01	6.98	6.37	8.47	7.79	5.18	6.54	1.00
	GRAG	26.49	23.94	23.97	7.35	15.01	25.30	15.75	10.39	1.15	24.13	17.35	8.36

Table 18. Results over 10 runs for all models and RAG systems (mean and standard deviation over 10 runs).

System	Mean Runtime (s)	Std (s)	Cost vs RAG	Variance	Trade-off Summary
RAG	$[0.87,\ 4.03]$	$[0.06,\ 0.50]$	$\times 1$	Low	Fast but weakest performance on complex CTI queries
AGRAG	$[4.03,\ 11.18]$	$[0.23,\ 1.55]$	$\times[2.77,\ 7.21]$	Low–Mod.	Best balance: large quality gains with stable overhead
HRAG	$[4.60,\ 46.93]$	$[0.26,\ 5.54]$	$\times[3.71,\ 35.55]$	Moderate	Maximum robustness at higher but predictable cost
GRAG	$[5.97,\ 127.53]$	$[0.80,\ 38.74]$	$\times[4.81,\ 96.61]$	High	Unfavorable: high cost and instability

Table 19. RQ4 Summary: Runtime-quality trade-offs across RAG systems. Ranges are computed from per-model mean and standard deviation runtimes in Table 18 . Relative cost is computed per model as the ratio of system mean runtime to the corresponding RAG mean runtime; ranges report the minimum and maximum ratios across evaluated models.

Model	System	Simple	Single-hop	Multi-hop	Guided	Unanswerable
GPT-5.2	RAG	$0.007	$0.008	$0.011	$0.010	$0.008
	GRAG	$0.021	$0.028	$0.042	$0.036	$0.025
	AGRAG	$0.025	$0.031	$0.045	$0.039	$0.029
	HRAG	$0.039	$0.050	$0.074	$0.065	$0.046
GPT-4.1-mini	RAG	$0.001	$0.001	$0.001	$0.001	$0.001
	GRAG	$0.002	$0.002	$0.002	$0.002	$0.002

Table 20. Estimated API cost per query (USD) by LLM, retrieval system, and question type. Token budgets are scaled by system complexity (LLM calls: RAG = 1, GRAG = 3, AGRAG = 3.5, HRAG = 5) and category-dependent reasoning effort. Reasoning tokens are included only for reasoning-capable models (GPT-5.2, Kimi K2 Thinking). Costs exclude embedding, infrastructure, and database overhead.

Category	RAG	GRAG	AGRAG	HRAG
Simple	13.7%	6.5%	9.1%	3.3%
Single-hop	13.2%	15.4%	6.3%	1.9%
Multi-hop	24.1%	16.3%	10.9%	7.0%
Guided	12.5%	26.5%	15.9%	8.7%
Unanswerable	9.2%	44.9%	10.9%	14.5%
Overall (std of run means)	1.76	2.14	1.25	0.76

Table 21. Cross-run quality stability: coefficient of variation (CV) of run-level mean judge scores by system and question category, computed over 10 independent runs. Lower CV indicates more stable performance across different report samples.

Figure 8. GPT-5.2 Performance Metrics

Question	Which CVE was exploited in the APT37 Internet Explorer incident?
Gold	CVE-2022-41128
GRAG (736.4s)	[Empty response after 12+ minutes]
Failure	Generated a Cypher query with an incorrect relationship name. Subsequent retry attempts continued searching for a schema pattern that does not exist, exhausting 25 correction iterations.
AGRAG (4.9s)	“CVE-2022-41128” ✓
Mechanism	Critique loop identified relationship schema mismatch in the first iteration and regenerated a valid query.

Table 22. Failure Case 1.

Question	What CVSS score is recorded for CVE-2024-21338?
Gold	[Information not available in reports]
RAG (1.2s)	“The CVSS score recorded for CVE-2024-21338 is 9.1”
Safety failure	Confidently hallucinated a specific numerical value that is not present in the source documents.
GRAG (520.4s)	[Empty response after 8.7-minute timeout]
Partial success	Did not produce an answer, but only after prolonged latency while searching for a non-existent CVE property in the graph.
HRAG (15.3s)	“I don’t know” ✓

Table 23. Failure Case 2.

Attack Strategy	RAG	GRAG	AGRAG	HRAG
Single-category strategies
A1: Schema evasion (guided)	20.6%	82.5%	18.1%	3.1%
A2: Unanswerable probing	6.0%	46.0%	4.0%	20.0%
A3: Multi-hop stress	47.3%	16.7%	18.7%	12.0%
A4: Simple fact (control)	52.0%	11.3%	12.0%	6.7%
Compound strategies

Table 24. Adversary attack success rates (% of questions scoring

\leq

15/50) by strategy and system. Higher values indicate greater vulnerability. Bold marks the most effective attack per system.

Failure Mode	RAG	GRAG	AGRAG	HRAG
Correct ( $\geq$ 40)	41.5%	61.1%	74.2%	83.2%
Partial (10–40)	31.7%	12.3%	14.1%	12.0%
Near-zero ( $\leq$ 10)	26.8%	26.7%	11.7%	4.8%
Hallucination	45.6%	28.2%	9.7%	11.7%
Full collapse	41.1%	33.0%	15.8%	8.8%

Table 25. Failure mode classification across RAG systems. Modes are defined by judge criterion scores: Hallucination = C3

\leq

2 and C4

\geq

3 (fluent but unfaithful); Full Collapse = C1, C2, C3 all

\leq

Table 30

Table 31

Feature	RAG	GRAG	AGRAG	HRAG
Question length (words)	$-$ 0.19	$+$ 0.58	$+$ 0.09	$-$ 0.07
Multi-entity (and/or/all)	$-$ 0.17	$+$ 0.52	$+$ 0.05	$-$ 0.09
Gold answer length (words)	$-$ 0.16	$+$ 0.54	$+$ 0.05	$-$ 0.07
Contains “which”	$-$ 0.15	$+$ 0.22	$-$ 0.04	$-$ 0.17
Contains “how many”	$+$ 0.01	$-$ 0.05	$+$ 0.13	$+$ 0.17
Has temporal reference	$+$ 0.08	$-$ 0.03	$+$ 0.00	$-$ 0.01

Table 30. Correlation between question-level features and failure (score

\leq

15). Positive values indicate the feature increases failure risk.

System	Primary Vulnerability	Failure Mode	Best Attack	Mitigation
RAG	Simple fact lookup	Hallucination (45.6%)	52.0%	Improve retrieval precision
GRAG	Schema evasion	Full collapse (33.0%)	82.5%	Bounded timeout + fallback
AGRAG	Aggregate queries	Partial failure (14.1%)	26.4%	Aggregate query templates
HRAG	Unanswerable probing	Low (8.8% collapse)	22.2%	Explicit abstention logic

Table 31. Adversary model summary: primary vulnerability, dominant failure mode, best-case attack success rate, and recommended mitigation per system.

System	Strengths	Weaknesses	Opportunities	Threats
RAG	Fast and low-cost retrieval; robust to schema gaps; stable runtime behavior.	High hallucination rates; weak multi-hop reasoning; sensitive to top- $k$ retrieval failures.	Improved reranking or hybrid retrieval could enhance factual recall.	Vulnerable to simple fact queries and overconfident answers for missing information.
GRAG	Explicit relational reasoning over CTI entities; strong factual lookup when Cypher queries succeed.	Highly sensitive to text-to-Cypher errors; latency instability due to retry loops.	Improved text-to-query models and bounded correction strategies could stabilize performance.	Schema evasion attacks and catastrophic collapse on unanswerable queries.
AGRAG	Agentic critique improves Cypher reliability; large accuracy gains over classical RAG; reduces query failures.	Higher latency than RAG; dependent on graph schema coverage.	Advanced agentic planning and better query validation could further improve robustness.	Aggregate queries and schema gaps remain challenging.
HRAG	Best overall robustness; combines graph reasoning with semantic retrieval; lowest hallucination rates.	Higher computational cost and system complexity.	Adaptive routing or ensemble strategies could reduce cost while preserving robustness.	Incorrect pipeline balancing or infrastructure constraints may limit deployment.

Table 32. SWOT analysis of evaluated RAG architectures for CTI question answering.

Graph Retrieval-Augmented Generation

Graph Retrieval-Augmented Generation connects language models to knowledge graphs to retrieve relational evidence for grounded, multi-hop answers.

See all 232 papers

Operationalizing Cyber Threat Intelligence with GraphRAG
Aug 13, 2026Atul Kabra, Prakhar Paliwal, Manjesh K. HanawalCyber Threat Intelligence Graph Retrieval-Augmented Generation
MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra, India
TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction
Jul 6, 2026Mouhamed Amine Bouchiha, Gregory BlancKG Construction Multi-Agent LLM Systems
SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris
Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization
Jun 24, 2026Long Chen, Ryan Razkenari, Yuxuan Zhou +5Retrieval-Augmented Generation Graph Retrieval-Augmented Generation
1Generative AI Innovation Center, Amazon Web Services (AWS) · 2Cisco Systems, Inc.

Aug 13, 2026J/K move · Enter open · S save

Operationalizing Cyber Threat Intelligence with GraphRAG

Atul Kabra, Prakhar Paliwal, Manjesh K. Hanawal

MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra, India

Cyber Threat Intelligence Graph Retrieval-Augmented Generation Cybersecurity

Open paper AskPDF arXiv

When a security researcher publishes a report on a cyberattack, detection engineers are supposed to turn it into working detection rules. In practice, most automated attempts at this only extract the simplest clues from the report --- bad IP addresses, domain names, and file hashes --- and turn them into block lists. This is a weak strategy, because attackers can change these simple clues within hours or days, so the resulting detections stop working almost as soon as they are deployed. Security teams describe this idea with the Pyramid of Pain. This project asks whether feeding a report into a knowledge-graph retrieval system, Microsoft GraphRAG, rather than a standard vector-similarity retrieval system (Naive RAG), produces detection plans that rely more on these durable, top-of-pyramid clues. Both systems are given the same report, the same generation instructions, and the same language model to write the final plan; only the retrieval step differs. In a detailed case study of one APT28 report, the GraphRAG plan kept firing at 100% of its detections after every IP address, domain, and file hash in the report was rotated, while the Naive RAG plan kept firing at only 29%. Repeating the comparison across nine real CTI reports from four vendors confirms the same pattern: GraphRAG plans consistently reach higher, harder-to-evade levels of the pyramid, even when the two systems end up close on total score. The results support treating knowledge-graph-aware retrieval as the architecturally correct foundation for automatically generating SOC-deployable hunting plans, while showing that the wording of the generation prompt matters almost as much as the retrieval back-end itself.

Abstract

Graph Retrieval-Augmented Generation

Explore similar work

Operationalizing Cyber Threat Intelligence with GraphRAG

TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction

Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization