cs.CRAug 17, 2026

Topological Attribution Distance (TAD): Revealing Segment-Level RAG Influence on LLM Output Geometry for Incident Log Analysis

Authors: Reza Fayyazi, Michael Zuzak, Shanchieh Jay Yang

Organizations: Dept. of Computer Engineering Rochester Institute of Technology Rochester, NY, USA · Dept. of Engineering Gonzaga University Spokane, WA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly being deployed in cybersecurity operations to assist cybersecurity analysts with rapid decision-making against emerging threats. However, there is a main criteria that must be met when using LLMs in cybersecurity, that is, trust in the generated outputs. As Agentic AI is integrated into operational systems, a robust evidence attribution and provenance tracking technique is essential to trace the origins of model generations. When autonomous agents make a decision (right or wrong), the ability to trace back through the decision chain is critical, as without it, teams cannot identify which segment of the data caused the model generation. Existing methods often struggle to distinguish among complex and highly similar evidence sources, such as cyber incident logs. This reveals a key gap: current approaches do not adequately capture the holistic geometric relationship between the retrieved evidence and the generated response for reliable evidence verification. To bridge this gap, we propose Topological Attribution Distance (TAD), inspired by Topology, to characterize and capture the global geometric shape of an output and its changes against its retrieved logs. In other words, if the embeddings of a specific source log drastically changes the geometry of the model's response in the embedding space, this suggests that such log is a critical source for the model's generated response. Therefore, TAD is powered by segment-level ablation attribution to investigate incident logs of an actual cyberattack. We demonstrate how TAD finds the most attributed logs on LLM outputs in an adaptive manner. This can provide an explainable and trustworthy tracing based on each LLM's hidden state to understand how geometrically different retrieved logs influence the model generation, and provide evidence verification in cybersecurity and Agentic-AI workflows.

Figures & tables

Fig. 1 : Evidence-localization case study. The Gemma model identifies the malicious registry-persistence event but partially hallucinates its exact log identifier, making the generated answer insufficient for direct verification. Existing methods retrieve benign logs, return multiple candidates, or cite almost the full context. TAD uniquely localizes Log-5 by measuring the effect on the response geometry, allowing an analyst to trace the response to its supporting evidence.
Fig. 2 : TAD’s procedure of tracing attribution with corresponding retrieved data. First, for a generated response, we get the LLM Output shape using Vietoris Rips algorithm. Next, to make TAD scalable across a vast amount of logs, we start by partitioning data into N\sqrt{N} and do Wasserstein distance of the original output vs. the ablated batch output. We do this over all the transformer layers and aggregate the overall distance. For batches with largest gap and above, we proceed to Step-3 to analyze the remaining logs. Finally, we do one-by-one ablation with Wasserstein distance to trace the top attributed log(s).
WDateStart (UTC)End (UTC)#Logs#GT
12025-10-1018:51:2118:52:16713
22025-10-1018:56:5918:57:45481
32025-10-1019:02:2119:03:40482
42025-10-1019:09:4619:11:17504
52025-10-1019:54:4919:55:47463
62025-10-1020:00:0020:01:41593
TABLE I : Dataset overview. Each window is a slice of Windows event logs surrounding the attack logs. #GT refers to the number of ground-truth attack-relevant logs.
TechniqueCount
Pass-the-Hash (NTLM remote logon)9
Registry modification (BAM, tool execution)4
Malicious service creation (PowerShell payload)2
PSEXESVC service installation2
Credential brute force (logon failure)1
Interactive logon, compromised credentials1
TABLE II : Ground-truth attack events categories of the dataset.
CaseMethodQwen3-4BGemma-3-4BQwen2.5-7BGranite-4.1-8B
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
DirectLLM-as-judge0.96930.68970.90460.28210.97960.75000.96250.6333
In-line citation0.97100.51430.96930.40000.94550.40740.97790.6829
Cosine similarity0.80580.21920.80580.21920.80580.21920.80580.2192
ROUGE-L0.79900.16900.79900.16900.79900.16900.79900.1690
LEA0.91990.40510.91990.40510.91990.40510.85350.2712
TABLE III : Performance comparison of TAD (H0 homology) and other attribution tracing methods across four LLMs. The Direct, Regular, and Indirect cases point to the difficulty-level of the response with respect to the ground-truth logs.
CaseMethodQwen3-4BGemma-3-4BQwen2.5-7BGranite-4.1-8B
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
DirectTAD (H0)0.98470.70970.97790.60610.98300.66670.98300.6667
 TAD (H1)0.98470.72730.97100.54050.98300.70590.98300.7222
RegularTAD (H0)0.98470.70970.97270.50000.97960.60000.98130.6452
 TAD (H1)0.98300.66670.94380.26670.98300.66670.98470.7097
IndirectTAD (H0)0.97960.60000.94380.15380.97610.56250.96590.3750
TABLE IV : TAD comparison based on H0 & H1 homology.
Fig. 3 : TAD qualitative example to demonstrate how Wasserstein distance measures each log’s effect on the response geometry.
More in this topic

Feature Attribution

Feature attribution estimates which input features or internal components contribute to a model’s predictions.

See all 233 papers

Explore similar work

CardsList
Mar 18, 2026

Retrieval-Augmented LLMs for Security Incident Analysis

Investigating cybersecurity incidents requires collecting and analyzing evidence from multiple log sources, including intrusion detection alerts, network traffic records, and authentication events. This process is labor-intensive: analysts must sift through large volumes of data to identify relevant indicators and piece together what happened. We present a RAG-based system that performs security incident analysis through targeted query-based filtering and LLM semantic reasoning. The system uses a query library with associated MITRE ATT&CK techniques to extract indicators from raw logs, then retrieves relevant context to answer forensic questions and reconstruct attack sequences. We evaluate the system with eight LLM configurations on malware traffic incidents and a multi-stage Active Directory attack. We find that LLMs have different performance and tradeoffs, with Claude Sonnet 4 achieving 94% and DeepSeek V3 achieving 89% average recall across 17 malware scenarios, while DeepSeek costs 15×\times less than Claude per analysis, and locally-deployed Llama 3.1:70b achieves 81% recall at zero per-query cost. Attack step detection on the Active Directory scenario reaches 100% precision and up to 96% recall with an enumeration prompt. These results demonstrate that combining targeted query-based filtering with RAG-based retrieval -- confirmed essential by ablation studies -- enables accurate, cost-effective security analysis within LLM context limits.

Xavier Cadet, Aditya Vikram Singh, Harsh Mamania +6

Dartmouth College Hanover, NH, USA · Northeastern University Boston, MA, USA · PUNCH Cyber Analytics Reston, VA, USA

CybersecurityDigital ForensicsRetrieval-Augmented Generation
AskPDF
Sep 21, 2026

Reasoning Topology Matters: A Controlled Study of LLM-Based Cybersecurity Analysis

Large Language Models (LLMs) are increasingly used in cybersecurity, where accurate analysis often requires multi-step and context-dependent reasoning over complex and heterogeneous data. However, existing prompting approaches typically focus on eliciting reasoning without explicitly considering how intermediate reasoning steps are structurally organized. We introduce Security Reasoning Topology, which models reasoning through three representative structures: Linear, Branching, and Graph. To evaluate their effects, we conduct controlled experiments on three cybersecurity datasets covering MITRE ATT&CK network traffic, cyber threat intelligence (CTI), and CVE vulnerability analysis. We evaluate multiple LLMs, including Llama 2 (7B, 13B, 70B), GPT-5.1, and Mistral Large 3, while keeping task inputs consistent and controlling reasoning structure through system-level prompting. Results show that reasoning topology substantially affects performance: Graph reasoning achieves the highest overall accuracy, improving over few-shot prompting by 9.8-12.2 percentage points across datasets, while Branching provides a strong intermediate solution. The results further show that the effect of reasoning topology remains consistent across model families and scales, highlighting reasoning topology as an important design factor for LLM-based cybersecurity analysis.

Jiling Zhou, Aisvarya Adeseye, Antti Hakkala +2

Department of Computing University of Turku Turku, Finland

LLMs for CybersecurityLLM Reasoning with GraphsCybersecurity
AskPDF
Oct 22, 2026

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

Yifan Wang, Dongliang Xu, Xiaoyong Li +3

School of Airspace Science and Engineering, Shandong University, Shandong 264209, China. · School of Computer Science, University of Technology Sydney, Ultimo, NSW 2007, Australia. · an independent researcher in Braddon, ACT 2612, Australia.

CybersecurityLLMs for CybersecurityLanguage Model Reasoning Evaluation
AskPDF