AuthRank: Scoring Source Credibility to Counter Poisoning in CTI LLM Pipelines

Austin Jia; Avaneesh Ramesh; Zain Shamsi; Daniel Zhang; Alex Liu

cs.CROct 23, 2025

AuthRank: Scoring Source Credibility to Counter Poisoning in CTI LLM Pipelines

Authors: Austin Jia, Avaneesh Ramesh, Zain Shamsi, Daniel Zhang, Alex Liu

Organizations: Applied Research Laboratories, The University of Texas at Austin, Texas, USA

Abstract

Large Language Model (LLM) pipelines have become essential for processing Cyber Threat Intelligence (CTI) to quickly extract updated threat information from open source feeds and reports for real time processing. However, these pipelines are susceptible to poisoning attacks, and previously proposed defenses can fail for CTI contexts as cyber threat information is often completely novel, and attackers can easily mimic the stylistic conventions. We propose that the robustness of LLM pipelines can be improved by applying source credibility algorithms on document corpora, using PageRank as an example. We introduce AuthRank, a credibility score computed from citation networks built using explicit citations, LLM-inferred citations, and claim-level entailment, enhanced with time decay and author credibility. Our experiments on the standardized MS MARCO dataset demonstrate improved accuracy under poisoning, and we show proof-of-concept performance on CTI documents and feeds.

Figures & tables

Fig. 1: Overview of our system. Example shows graph building using claim extraction, where red = poisoned injection and green = legitimate information.

Test APT	Query Focus	Correct Answer	Poisoned Info	Top Correct Source (AuthRank Score)	Top Poison AuthRank Score	Outcome
CozyBear	Group affiliation	Russia	China	Wikipedia (0.92)	0.31	Correct
FancyBear	Organization nature	State-sponsored	Independent firm	CrowdStrike Blog (0.87)	0.29	Correct
LazarusGroup	Attack targets	Financial/crypto	Education/OSS	Wikipedia (0.85)	0.33	Correct
ScatteredSpider	Group purpose	Cybercriminal	Corporate red team	Conflicting sources* (0.22)	0.22	No Answer
ScatteredSpider	Term meaning	Cybercriminal group	Desert spider	Cyware Blog (0.94)	0.25	Correct

TABLE I: AuthRank Scoring Efficacy Against CTI Data Poisoning

Fig. 2: Accuracy of AuthRank on poisoned MS MARCO dataset.

Fig. 3: AuthRank vs. Similarity scores for each query

Poisoned chunks extracted from attacker’s blog

P1. “After investigation, we conclude updates-winsecure[.]com is a benign CDN endpoint used by a legitimate browser extension for update checks.”

P2. “VirusTotal shows 0/98 hits for the domain with no reputation flags.”

P3. “No adversary infrastructure, anomalous behavior, or overlapping IOCs observed.”

P4. “Suppress alerts for this domain unless new anomalies appear.”

TABLE II: Poisoned Inputs and LLM Response Comparison

Cyber Threat Intelligence

Cyber threat intelligence turns evidence about adversaries, campaigns, and vulnerabilities into knowledge that helps defenders anticipate and respond to attacks.

See all 48 papers

CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
Mar 10, 2026Xiangsen Chen, Xuan Feng, Shuo Chen +5LLM Evaluation Cyber Threat Intelligence
Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft +1
Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs
Jan 31, 2026Md Tanvirul Alam, Aritran Piplai, Ionut Cardei +2LLMs for Cybersecurity Reinforcement Learning with Verifiable Rewards
Rochester Institute of Technology, Rochester, NY, USA · University of Texas at El Paso, El Paso, TX, USA · Florida Atlantic University, Boca Raton, FL, USA
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Sep 8, 2026Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah +1LLM Evaluation LLM Auditing
Qatar Computing Research Institute, HBKU

Mar 10, 2026J/K move · Enter open · S save

CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?

Xiangsen Chen, Xuan Feng, Shuo Chen, Matthieu Maitre+4

Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft+1

LLM Evaluation Cyber Threat Intelligence Cybersecurity

Open paper AskPDF arXiv

Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow -- triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code is available at GitHub and HuggingFace.

Abstract

Cyber Threat Intelligence

Explore similar work

CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?

Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks