It is Not Yet Another Tool: Creating and Deploying an Agentic AI Companion in a Security Operations Center

Kritan Banstola; Faayed Al Faisal; Duy Dao; Ryan Irving; Daniel Lende; Xinming Ou

cs.CRSep 5, 2026

It is Not Yet Another Tool: Creating and Deploying an Agentic AI Companion in a Security Operations Center

Authors: Kritan Banstola, Faayed Al Faisal, Duy Dao, Ryan Irving, Daniel Lende, Xinming Ou

Organizations: University of South Florida

Abstract

Security Operations Centers (SOCs) process large amounts of tickets, most of which are low-interest events not worthy of further investigation. The repetitive nature of this task and similarity of the vast amounts of tickets make it a prime candidate for generative AI-based automation. We created and deployed an agentic AI companion utilizing large language models through fieldwork within a SOC for over one year. The design of the SOC AI companion was driven by researchers' participation and interactions within the SOC's daily work. SOC analysts were invited to use it during the last four months of the fieldwork. We analyzed the analysts' usage of the companion and found that in more than 90% of the cases the companion's outputs were reused by analysts in the ticket's closing report. Our results showed that when designed "in the trenches" with the intended users, a SOC AI companion can go beyond being yet another tool, but rather a system that co-evolves with its human users as it traverses through the various types of workloads. Analysts naturally started to shape the AI companion's behaviors to fit their particular needs. Our data show that the more human analysts shape the AI companion's behaviors, the more they become comfortable trusting the output from the AI system, resulting in improved productivity.

Figures & tables

Figure 1: System Architecture of SOC AI Companion

Category	Systems reached
Ticketing	Alert ticket retrieval
Log sources	SIEM advanced hunting queries, log search platform
Internal asset and identity	DHCP lease lookup, MAC registration, device and owner records
External enrichment	Reputation and threat intelligence providers, general web search
Analysis utilities	Encoding and decoding, detection signature lookup

Table 1: Categories of tools available to the Companion. Each category corresponds to systems the analysts already use manually during triage.

Figure 2: Coding Process.

	Usage codes	$n$
U1	Text used verbatim	35
U2	Text partially used	63
U3	Text not used	10
U4	Analyst added text	26
U5	Text truncated	41
U6	All screenshots reused	78

Table 2: Codebook and code frequencies over the 108 coded tickets. Except for U1–U3 and U6–U8, which are each mutually exclusive, codes may co-occur; a ticket carries as many as apply. Definitions are given in Appendix B .

Figure 3: Share of the Companion’s draft surviving into the analyst’s closing report, over 108 tickets. Drafts are taken nearly whole or discarded, with little in between: 59 tickets at or above 95% reuse, 28 below 75%. Median 96.7%, mean 85.5%, lower quartile 72.0%.

How the AI draft was edited	$n$	Mean reuse score	Median reuse score
Recommendation dropped	20	92.2%	97.9%
Trimmed	8	92.8%	96.2%
Verdict softened	8	83.2%	93.8%
Condensed	7	68.3%	64.6%
Rewritten	7	61.5%	59.7%
Reframed	6	68.5%	67.1%

Table 3: Partially-used Companion drafts (U2) by type of edit, with mean and median draft reuse. The most common edit leaves the analysis intact and removes only the recommended action. For the two categories with

n<5

we list the individual ticket’s draft reuse values rather than an average.

Before	3	66.0%	50.0%	1 (33%)
	$n$	Mean	Median	$\geq 95\%$
		reuse	reuse
After	105	86.1%	96.7%	58 (55%)

Table 4: Comparison before and after screenshots were attached to the Companion’s draft reports.

Figure 4: Three levels at which the Companion’s output could be adopted: content, verdict, and recommendation. Each bar is the share of the same 108 tickets on which that layer survived into the closing report.

Companion’s recommendation	$n$	Median	Mean
was		reuse	reuse
struck down (U10)	34	94.2%	83.4%
kept	74	97.2%	86.5%

Table 5: Draft reuse by whether the analyst kept or struck down the Companion’s recommended action.

Companion	Recommendation	Recommendation	% U10
verdict	struck down (U10)	kept
Benign	0	38	0%
Uncertain	24	16	60%
Malicious	10	20	33%

Table 6: Tickets where analysts struck down/kept recommendation by Companion verdict.

Figure 5: The verdict analysts closed tickets as (rows) vs. the Companion’s draft opinions (columns), over the 108 tickets. The Companion over-hedges: most of what it called uncertain was closed benign.

Verdict relationship	$n$	Median	Mean
		reuse	reuse
Verdicts match	72	97.4%	87.5%
One side hedged	29	96.4%	85.1%
Verdicts conflict	7	67.9%	67.2%

Table 7: Draft reuse by how the analyst’s verdict compares to the Companion’s.

Intervened
	$n$	Draft text	Draft text	Median	Mean
		used (U1, U2)	not used	reuse	reuse
Yes	17	76%	24%	92.1%	79.4%
No	91	93%	7%	97.3%	86.7%
Rerun in a new thread
Yes	14	93%	7%	91.2%	79.4%

Table 8: Whether the Companion’s draft text reached the closing report, by intervention and by rerun.

Figure 6: Weekly share of SOC tickets that were worked on the Companion, from the first Companion ticket close (April 29, 2026) through week 15. The dashed lines mark the two changes we made to the Companion during the deployment: at

t_{1}

(May 28, 2026) the screenshots from tool query was added; at

t_{2}

(July 17, 2026) a revised system prompt was added.

Figure 7: Draft reuse per ticket, by the system prompt the investigation ran under; bars mark medians.

Figure 8: ticket closing time for the three analysts who closed 10 or more tickets, one point per ticket, bar at the per-analyst median.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: A real alert from the SOC, redacted.

Usage codes — what became of the Companion’s draft
U1	Text used verbatim	The closing report reproduces the Companion’s draft text essentially verbatim.
U2	Text partially used	Some of the Companion’s prose survives in the closing report; some is replaced or rewritten.
U3	Text not used	The analyst wrote the closing report independently.
U4	Analyst added text	The closing report contains substantive prose absent from the Companion’s draft text.
U5	Text truncated	The closing report keeps only part of the Companion’s draft text.
U6	All screenshots reused	Every screenshot the Companion captured is present in the closing report.

Appendix

Table 9: Codebook definitions; frequencies over the 108 coded tickets are given in Table 2 . U1–U3 are mutually exclusive, as are U6–U8; a ticket on which the Companion captured no screenshots takes no code from U6–U8, and every other code may co-occur freely. Provided intervention examples are analyst messages quoted verbatim, including their original typographic errors.

Human-AI Collaboration

Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.

See all 434 papers

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment
Jun 10, 2026Derek Yohn, Luke Flancher, Mirajul Islam +1Software Vulnerability Detection LLM Agent Evaluation
College of Engineering and Science, Florida Institute of Technology, Melbourne, Florida, USA
AgentSOC: A Multi-Layer Agentic AI Framework for Security Operations Automation
Apr 22, 2026Joyjit Roy, Samaresh Kumar SinghAutonomous Cyber Defense Cybersecurity
Austin, Texas, USA · Leander, Texas, USA
AgentSpy: Making AI Agent Behavior Observable
Oct 5, 2026Christoph Bühler, Matteo Biagiola, Luca Di Grazia +1AI Agent Reliability AI Agent Auditing
University of St. Gallen, St. Gallen, SG, Switzerland · University of St. Gallen and Università della Svizzera italiana (USI), St. Gallen and Lugano, SG and TI, Switzerland

Jun 10, 2026J/K move · Enter open · S save

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

Derek Yohn, Luke Flancher, Mirajul Islam, Khaled Slhoub

College of Engineering and Science, Florida Institute of Technology, Melbourne, Florida, USA

Software Vulnerability Detection LLM Agent Evaluation Software Security LLMs for Cybersecurity

Open paper AskPDF arXiv

This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions.

Abstract

Human-AI Collaboration

Explore similar work

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

AgentSOC: A Multi-Layer Agentic AI Framework for Security Operations Automation

AgentSpy: Making AI Agent Behavior Observable