Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study
Authors: Seungho Song, Donggyu Lee, Jaehan Kim, Jiyoung Choi, Jinwoo Kim
Organizations: Korea University, 145 Anam-ro, Seongbuk-gu, Seoul, 02841, Republic of Korea · KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea · Kwangwoon University, 20 Kwangwoon-ro, Nowon-gu, Seoul, 01897, Republic of Korea · Chungbuk National University, 1 Chungdae-ro, Seowon-gu, Cheongju, Chungcheongbuk-do, 28644, Republic of Korea
Large language models (LLMs) are increasingly used in cyber threat intelligence (CTI) tasks, such as summarization and question answering. However, they may produce hallucinations, false or misleading content not grounded in the source text, which can mislead analysts and corrupt downstream security decisions. This paper presents an exploratory measurement study of hallucinations in LLM-based CTI. We introduce HalluVision, a reproducible framework that generates LLM outputs, extracts domain-specific entities using a fine-tuned named entity recognition model, and evaluates factual consistency using multiple similarity metrics. Using 4,940 real-world cybersecurity articles, we measure hallucinations in outputs from two commercial LLMs and one open-weight LLM. Measurements are validated against 100 human-annotated outputs and two established faithfulness detectors. Results show that semantic similarity aligns with human judgments, whereas entity-based metrics exhibit limited agreement. Differences between models are concentrated in question answering, while entity extraction quality limits what entity-level metrics can reveal.
Figures & tables
Figure 1: Motivating examples of hallucinations in the CTI domain. In both cases, the LLM output contradicts the source article: in (a), the LLM refers to proxy servers whereas the source article refers to command-and-control servers ; in (b), the LLM assigns click fraud to the Indicator label. Both cases therefore constitute intrinsic hallucinations as defined in Section 2.3 .
Figure 2: Our measurement framework, HalluVision , comprising four components: Scraper, LLM Proxy, Preprocessor, and Analyzer.
Type
Summarization Prompt
Q&A Prompt
Role
NER expert in the CTI domain
NER expert in the CTI domain
Instructions
- Summarize blog content in 50–70 words. - Use only words and content from the original text.
- Answer five predefined entity-based questions. - Use short answers (ideally one entity). - Output “no answer” if info is missing.
Input
{original_text}
{original_text}
Output
Single summary paragraph
5 question–answer pairs
Table 1: Structure of the prompts used for summarization and question answering. The complete prompts are provided in Appendix A .
Source
Content
Original Article
300 restaurants and at least 50,000 payment cards compromised by two separate campaigns against MenuDrive, Harbortouch and InTouchPOS services. Magecart campaigns have been skimming payment-card credentials of unsuspecting customers using three online restaurant-ordering systems, affecting about 300 restaurants that use the services and compromising tens of thousands of cards so far, researchers have found. Two separate ongoing Magecart campaigns have injected e-skimmer scripts into the online ordering portals of restaurants using three separate platforms: MenuDrive , Harbortouch, and InTouchPOS, researchers from Recorded Future revealed in a blog post this week. (…) “This campaign likely began no later than Jan. 18, 2022, and as of this report, a portion of the restaurants remained infected,” they noted in the post. However, the malicious domain used for the campaign, which researchers identified as authorizen[.]net , has been blocked since May 26, they said.
Abstractive Summarization
Two Magecart campaigns have compromised over 50,000 payment cards from approximately 300 restaurants using MenuDrive , Harbortouch, and InTouchPOS services. Researchers from Recorded Future reported that e-skimmer scripts were injected into these platforms, exposing customers’ payment card data and personal information. The campaigns began in late 2021 and early 2022, with ongoing infections and malicious domains still active. Smaller restaurant platforms remain attractive targets for cybercriminals due to their vulnerabilities.
Question & Answering
Q1. What is the main Malware?: Magecart Q2. What is the main System?: MenuDrive Q3. What is the main Indicator?: authorizen[.]net Q4. What is the main Vulnerability?: no answer Q5. What is the main Organization?: Recorded Future
Table 2: Examples of abstractive summarization and question & answering outputs generated by an LLM from an article.
Entity Type
Description
Malware
The name of the primary malware discussed in the article. If no specific name is provided, it may refer to a general malware category.
System
Operating systems (e.g., Linux, Windows), software (e.g., Chrome, Slack), or hardware leveraged by threat actors in attacks.
Indicator
Patterns used to detect malicious activity, such as domain names, URLs, IP addresses, filenames, hashes, email addresses, and port numbers.
Vulnerability
Common Vulnerabilities and Exposures (CVE) identifiers (e.g., CVE-2024-9264) and exploit mentions (e.g., command injection vulnerabilities) explicitly stated in the article.
Organization
Entities directly affected by cyber threats discussed in the article. Organizations that merely observed or reported the threats are excluded.
Table 3: Entity categories considered in our study.
Metric
Preprocessing Required
Applicable To
Interpretation of High Score
Interpretation of Low Score
Cosine Similarity
Sentence Embedding
AS
Semantically consistent with the source
Semantic deviation or abstraction
Answer Matching Similarity – Precision
NER
AS, Q&A
Relevant entities correctly included
Irrelevant or spurious entities included
Answer Matching Similarity – Recall
NER
AS, Q&A
Most reference entities are retrieved
Key entities are missing
Frequency-based Entity Similarity
NER
AS
Similar entity-type distribution
Topic drift or entity imbalance
Table 4: Hallucination metrics used in our study and their interpretations. AS = Abstractive Summarization, Q&A = Question & Answering, NER = Named Entity Recognition.
# of Articles
Task
Source / Model
Threatpost
The Hacker News
Article Scraping
—
4,940
500
NER Fine-tuning Data
MITRE
50
Abstractive Summarization
gpt-4o-mini
4,940
500
gpt-3.5-turbo
4,940
500
Llama-3-8B-Instruct
500
—
Table 5: Summary of scraped articles and generated outputs by task, model, and corpus. Dashes indicate combinations that were not run.
Figure 3: Cumulative distribution function (CDF) of cosine similarities between original articles and their LLM-generated summaries. Lower CDF curves (toward the bottom right) indicate better semantic similarity, as more samples achieve higher cosine similarity scores.
Figure 4: CDF of answer matching similarities between original articles and their LLM-generated summaries. Lower CDF curves (toward the bottom right) indicate better similarity, as more samples achieve higher precision or recall.
Task
Model
Metric
Malware
System
Indicator
Vulnerability
Organization
Abstractive Summarization
gpt-4o-mini
Precision
0.7401
0.7187
0.4977
0.1775
0.7641
Recall
0.2241
0.1924
0.0315
0.0717
0.1786
gpt-3.5-turbo
Precision
0.6979
0.6800
0.4000
0.1923
0.7377
Recall
0.2199
0.1885
0.0291
0.0714
0.1637
Q&A
gpt-4o-mini
Precision
0.6442
0.5789
0.0288
0.1338
0.7475
Recall
0.1263
0.1120
0.0146
0.0885
0.1452
Table 6: Label-wise answer matching similarities in abstractive summarization and question answering.
Figure 5: CDF of the frequency-based entity similarity between the source texts and their summaries. Lower CDF curves (toward the bottom right) indicate better similarity, as more samples achieve higher similarity values.
Figure 6: CDF of frequency-based entity similarity between the source texts and their summaries for each label . Lower CDF curves (toward the bottom right) indicate better similarity, as more samples achieve higher similarity values.
Figure 7: CDF of answer matching similarities in the Q&A task. Lower CDF curves (toward the bottom right) indicate better similarity, as more samples achieve higher precision or recall.
Model
Cosine Similarity
Answer Matching Similarity
Precision
Recall
Llama-3-8B-Instruct
0.8659
0.7126
0.1547
gpt-3.5-turbo
0.8847
0.6917
0.1766
gpt-4o-mini
0.8911
0.7219
0.1831
Table 7: Similarity between source articles and summaries generated by three LLMs using the 500-article subset. Cosine similarity is the mean across all articles; higher values indicate better performance for both metrics. Llama-3-8B-Instruct summaries are evaluated after removing a boilerplate opening line.
Label
Llama-3-8B- Instruct
gpt-3.5-turbo
gpt-4o-mini
Malware
0.3002
0.3489
0.4527
System
0.2617
0.2353
0.3642
Indicator
0.0142
0.0101
0.0101
Vulnerability
0.0183
0.0304
0.0503
Organization
0.6227
0.6491
0.7022
Overall Precision
0.3936
0.3073
0.5221
Table 8: Q&A answer matching precision by label on the 500-article subset, after format normalization, with overall precision and recall.
Task
Model
Metric
Malware
System
Indicator
Vulnerability
Organization
Abstractive Summarization
gpt-4o-mini
Precision
0.6549
0.6939
0.4595
0.2032
0.8158
Recall
0.1760
0.1537
0.0405
0.0777
0.2322
gpt-3.5-turbo
Precision
0.6851
0.7167
0.4571
0.2434
0.8058
Recall
0.1812
0.1630
0.0381
0.0701
0.1974
Q&A
gpt-4o-mini
Precision
0.5214
0.5619
0.0430
0.1840
0.7723
Recall
0.1001
0.1048
0.0191
0.0813
0.1662
Table 9: Label-wise answer matching similarities for the 500-article sample from The Hacker News , comparable to Table 6 .
Figure 8: Human ratings of the 100 annotated outputs, grouped by metric score. Each output was rated by three annotators, so each group contains three ratings per output. Answer matching similarity is grouped by value rather than by quartile because 27 of the 99 outputs have a score of exactly 1.0; frequency-based entity similarity is defined for 95 outputs.
Doc
CS
AMS
FES
Human Rating
#11
79.3
33.3
25.0
H H H
#2078
82.1
33.3
78.6
H P H
#3916
95.2
100.0
75.0
F F F
Table 10: Examples of human validation case studies. Scores are HalluVision metrics expressed in percentages (CS: cosine similarity, AMS: answer matching similarity (precision), FES: frequency-based entity similarity). Each output was independently rated by three annotators as F (faithful), P (partially supported), or H (hallucinated), with the three ratings listed in order.
Baselines
HalluVision
Doc
FActScore
QAGS
Cos. sim.
AM prec.
Human Rating
Main reason given
#4042
1.000
0.546
0.923
0.000
P P H
Unsupported elaboration
#3313
1.000
0.709
0.929
0.250
F P P
Wrong entity type
#4087
1.000
0.660
0.940
1.000
P P P
Unsupported elaboration
Table 11: Three documents for which FActScore reports full support, although at least two of the three annotators rated the output partially supported or hallucinated . Human ratings show the verdict of each of three independent annotators, each of whom was shown the complete article: F (faithful), P (partially supported), and H (hallucinated). The last column indicates the most common reason among the non-faithful ratings. Cos. sim. denotes cosine similarity, and AM prec. denotes answer matching precision.
NER Errors
Label
n
Precision
Recall
Boundary
Label
Not Entity
Missed
Malware
56
0.626
0.976
9
5
7
5
System
50
0.733
0.929
6
3
4
15
Indicator
12
0.968
0.771
0
2
0
18
Vulnerability
3 †
0.000
0.000
2
1
0
22
Organization
83
0.760
0.969
7
5
8
9
Table 12: Manual validation of the CyNER ground truth on 70 articles. n is the number of judged predictions; precision and recall are inverse-probability weighted and provide estimates for all 1,036 predictions in those articles. A prediction is correct only if both its span and label are correct; rejections are attributed to Boundary (truncated span, e.g. vil for REvil ), Label (wrong type, e.g. Samsung tagged System ), or Not Entity (not an entity, e.g. Top Kill tagged Malware ). Missed counts entities CyNER did not predict.
Task
Model
Metric
Malware
System
Indicator
Organization
Correction factor c (Summarization: P×R of CyNER)
0.611
0.681
0.746
0.736
Correction factor c (Q&A: R of CyNER)
0.976
0.929
0.771
0.969
Abstractive Summarization
gpt-4o-mini
Precision
0.740 → 1.000
0.719 → 1.000
0.498 → 0.667
0.764 → 1.000
Recall
0.224 → 0.367
0.192 → 0.283
0.032 → 0.042
0.179 → 0.243
gpt-3.5-turbo
Precision
0.698 → 1.000
0.680 → 0.999
0.400 → 0.536
0.738 → 1.000
Recall
0.220 → 0.360
0.189 → 0.277
0.029 → 0.039
0.164 → 0.222
Table 13: Answer matching similarity before and after accounting for the contribution of CyNER errors (measured →min(1,measured/c) ). The difference between the two values represents the maximum contribution of extractor errors; the difference between the second value and 1.0 represents the minimum contribution attributable to the LLM. Vulnerability is omitted because CyNER did not recover any of its spans in the audit ( c=0 ).
Original Article
[…] Wikileaks’ Web site has been the target of denial of service attacks , which began even before the group published the leaked diplomatic cables as part of a campaign it dubbed “Cablegate.” While the source of those attacks isn’t clear, a hacker using the handle th3j35t3r claimed responsibility for it, citing the need to retaliate for the leak. More recently, Wikileaks has enlisted the services of hundreds of mirror sites to host the cablegate documents in a coordinated effort to keep them from being driven from the Internet. […]
Summarized Text ( hallucinated )
Paypal permanently restricted Wikileaks’ account due to policy violation, sparking online feud with hacker-activists. Anonymous planned DDoS attack on Paypal in response. Wikileaks faced ongoing denial of service attacks, with mirror sites hosting leaked documents. PayPal and Amazon denied services to Wikileaks, as U.S. Attorney General authorized actions against the organization.
Reason
The summary implies that the DDoS attacks were associated with or originated through the mirror sites. However, the original text clearly states that mirror sites were a defensive measure in response to DDoS attacks.
Table 14: A real hallucination example in abstractive summarization using gpt-3.5-turbo , in which an LLM-generated summary exhibits low cosine similarity (79.35%) with the original text.
Original Article
Hallucinated Entity
Ground-Truth ( CyNER )
LLM-generated Answer ( gpt-3.5-turbo )
Reason
And while it’s in vogue to refer to REvil and DarkSide as “franchise models” which supply Ransomware-as-a-Service, it is important to remember that the franchisees are effectively freelance cybercriminals.
REvil
Malware
Organization
REvil is a MALWARE in this article, not an ORGANIZATION.
Now researchers from Blackberry have connected the dots, painting a picture of a malware that has evolved five times in twelve months.
Blackberry
Organization
System
Blackberry is correctly labeled as an Organization, not a System, as it acts as a sighting group.
Log4j , ransomware, cloud vulnerabilities, phishing: Cyber threats are manifold. They all pale, however, in comparison to the security black holes that walk around on two legs.
Log4j
Not Identified
Malware
Log4j is a Java-based library and should be classified as a System, not Malware.
APT29, aka Cozy Bear, has used Tor and a technique called domain fronting in order to secure backdoor access to targets for nearly two years running.
Tor
System
Indicator
Tor is a System as a browser, not an INDICATOR.
Table 15: Real hallucination examples in Q&A tasks using the gpt-3.5-turbo model.
Task
Metric
n
Δ (4o-mini − 3.5-turbo), 95% confidence interval
QA
AM-prec
4,843
+0.2228 [0.2151, 0.2307]
AS
AM-prec
4,940
+0.0310 [0.0210, 0.0400]
AS
Cossim
4,939
+0.0070 [0.0063, 0.0077]
AS
Freq-sim
4,490
+0.0157 [0.0100, 0.0212]
Table 16: Paired bootstrap comparison of gpt-4o-mini with gpt-3.5-turbo (overall score, 10,000 resamples). AS = Abstractive Summarization, QA = Question answering, Cossim = cosine similarity, AM-prec = answer matching precision, and Freq-sim = frequency-based entity similarity. All differences are statistically significant at p<0.0001 , but the margin is substantial only in question answering. Positive Δ values indicate that gpt-4o-mini is closer to the source. n varies because each metric is computed only for documents for which both models produced a valid output and the metric is defined.
Work
Scope
CTI Entity Extraction
Hallucination Analysis
Cybersecurity-specific
Liao et al. [ 14 ]
IoC extraction from cybersecurity articles
●
○
●
TTPDrill [ 6 ]
CTI ontology extraction from unstructured text
●
○
●
LLM-TIKG [ 5 ]
LLM-based CTI knowledge graph construction from unstructured text
●
○
●
GapFinder [ 9 ]
Information-gap analysis in cybersecurity articles
●
○
●
Jiang et al. [ 8 ]
Information-gap analysis in cybersecurity databases
●
○
●
Vulcan [ 10 ]
CTI entity extraction from unstructured text
●
○
●
Table 17: Comparison of related studies with our work.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Combined score S over the full corpus, representing the output of Algorithm 1 alone, with no labels or threshold. The three measurements are standardized jointly across both models, so the two curves share a common scale; the markers denote the medians ( −0.19 for gpt-4o-mini and +0.15 for gpt-3.5-turbo).
Parameter
Value
Applies to
Model snapshot
gpt-3.5-turbo-0125
gpt-3.5-turbo
gpt-4o-mini-2024-07-18
gpt-4o-mini
Meta-Llama-3-8B-Instruct
Llama-3-8B-Instruct
Temperature
0.0
All models
Top- p
1.0
All models
Seed
42
OpenAI models
Appendix
Table 18: Generation configurations of the three LLMs used in this study, along with the token usage and list-price cost of the OpenAI runs. Main-corpus token counts were reconstructed using tiktoken from the stored prompts and outputs, because those runs predate usage logging; validation-run counts were obtained from the API usage logs and include the FActScore and QAGS judge calls. Prices are per million tokens at the time of the runs.
Large language models (LLMs) hallucinate with confidence: their outputs can be fluent, authoritative, and simply wrong. In medical, legal, and scientific applications this failure causes direct harm, and detecting it from internal model states offers a path to safer deployment. A growing body of work reports that this problem is increasingly tractable, with recent methods achieving high detection performance on widely used benchmarks. We show, however, that much of this apparent progress does not survive scrutiny. Four of the six corpora embed the ground-truth answer directly in the input prompt. A naïve text-similarity baseline we call TxTemb exploits this to achieve near-perfect detection scores without any access to model internals. To measure what genuine detection capability remains once these artifacts are controlled, we conduct a large-scale evaluation spanning twenty-two detection methods, twelve open-source models spanning six architectural families, and six corpora. We further introduce DRIFT, a supervised probe over inter-layer hidden-state transitions, as a point of comparison for live-generation detection. Our findings suggest that the field's reported progress on hallucination detection is substantially explained by benchmark construction artifacts in widely used corpora, and that the majority of established baselines perform near chance under controlled conditions; the consistent exceptions are SAPLMA and DRIFT, both supervised probes on upper-layer hidden states.
Large language models (LLMs) are widely used for information access, content production, and agent-based task execution, but their outputs remain vulnerable to subtle, high-risk hallucinations. To systematically evaluate and advance post-generation hallucination detection, we introduce HaluCheck, a large-scale, bilingual benchmark comprising over 42k instances spanning both factual hallucinations in text and faithfulness hallucinations in agent trajectories. HaluCheck features fine-grained, multi-level annotations for detection, localization, and correction, covering different tasks and perturbation degrees. Experiments on fourteen LLMs reveal that, while models show basic sample-level detection ability, they struggle with fine-grained localization, correction, and agent hallucination detection. Further analysis demonstrates that thinking, web search, and supervised fine-tuning each offer partial improvements, but none fully solve the problem. HaluCheck thus provides a solid basis for benchmarking and diagnosing LLM reliability across text and agent scenarios.
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.