Skynet: Workflow-Level Anomaly Detection for Agentic AI via Semantic and Structural Modeling

Chaoyu Zhang; Hexuan Yu; Heng Jin; Shanghao Shi; Ning Zhang; Yi Shi; Yulia R. Gel; Y. Thomas Hou

cs.CRSep 6, 2026

Skynet: Workflow-Level Anomaly Detection for Agentic AI via Semantic and Structural Modeling

Authors: Chaoyu Zhang, Hexuan Yu, Heng Jin, Shanghao Shi, Ning Zhang, Yi Shi, Yulia R. Gel, Y. Thomas Hou,

Organizations: Virginia Tech Arlington, VA, USA · WashU St. Louis, MO, USA · Virginia Tech Blacksburg, VA, USA

Abstract

Agentic AI systems execute complex tasks through long-horizon workflows of planning, tool use, and multi-agent coordination. Task failures in these systems often originate from a single step, such as an injected prompt or a flawed plan, and are then amplified through downstream dependencies as the corrupted step propagates across many subsequent agents and tool calls. Existing defenses either target a specific class of attacks or failures, or inspect individual prompts and steps in isolation. Both leave the global dependency structure of a workflow unexamined, and miss the inconsistencies that only emerge when the execution is viewed as a whole. We argue that anomaly detection for agentic AI must reason at the workflow level, where global execution structure exposes signals that local checks cannot see. We present Skynet, a principled workflow-level anomaly detection framework that turns observed multi-agent execution into directed workflow graphs and scores them against learned benign behavior. Skynet jointly models the semantic execution context and the structural organization of inter-agent delegation, tool invocation, and data-flow dependencies, and trains only on benign workflows. Because training never sees attacks or failures, this design naturally extends to zero-day detection: any execution that violates benign workflow regularities surfaces as off-manifold geometry under a single decision rule. We evaluate Skynet on three public agentic safety and failure benchmarks. It sustains high recall together with a sub-1% false positive rate, with per-workflow and per-step latencies low enough for online monitoring of agentic AI runtimes.

Figures & tables

Figure 1. Agentic AI execution workflows under normal and abnormal conditions. ( a ) Benign workflow with coherent planning, tool use, and execution. ( b ) Adversarial manipulation, where injected steps redirect execution. ( c ) Intrinsic failure, where non-adversarial errors accumulate. Three side-by-side agentic AI execution workflows drawn as directed graphs: a benign workflow, an adversarially manipulated workflow with injected steps, and an intrinsically failing workflow with accumulated errors.

Figure 2. Overview of Skynet . Skynet constructs a workflow graph from an observed agentic AI execution, learns benign workflow representations from semantic and structural signals, and detects anomalies by measuring how far a new workflow deviates from learned benign patterns. Pipeline diagram of Skynet with three stages: workflow graph construction from an observed agentic execution, dual semantic and structural encoding trained on benign workflows, and anomaly scoring by distance to the nearest benign mode anchor.

Figure 3. UMAP projection of the learned joint workflow embedding $z(\mathcal{G})$ . Top-row panels overlay anomalies from each benchmark on the held-out benign test set; the bottom panel shows all three benchmarks together with the $K{=}150$ benign mode anchors. Benign workflows form a coherent manifold around the anchors; anomalies from each source consistently lie off-manifold. UMAP scatter plots of workflow embeddings. Benign workflows cluster tightly around the benign mode anchors, while anomalous workflows from each benchmark scatter away from that cluster.

	Agent-SafetyBench					AgentErrorBench					ATBench					Combined
Method	Acc	Prec	Rec	F1	FPR	Acc	Prec	Rec	F1	FPR	Acc	Prec	Rec	F1	FPR	Acc	Prec	Rec	F1	FPR
Skynet	99.26	95.69	100.00	97.80	0.89	99.34	97.56	100.00	98.77	0.89	91.41	98.80	82.70	90.03	0.89	93.36	99.31	89.36	94.07	0.89
F-GAE	97.86	90.91	99.10	94.83	2.14	98.05	93.02	98.50	95.68	2.14	88.94	90.12	84.31	87.12	2.14	90.87	93.44	86.92	90.06	2.14
TraceAegis	96.62	86.89	96.40	91.40	3.38	96.83	89.19	94.50	91.77	3.38	67.43	61.54	34.61	44.27	3.38	75.92	78.48	58.42	66.98	3.38
Guardian	99.85	99.11	100.00	99.55	0.18	99.87	99.50	100.00	99.75	0.18	61.57	98.91	18.31	30.90	0.18	70.29	99.75	49.75	66.39	0.18
TopoGCL	95.69	80.15	98.20	88.26	4.80	92.78	86.43	86.00	86.22	4.80	56.56	70.33	12.88	21.77	4.80	64.23	92.74	42.70	58.47	4.80

Table 1. Detection performance across the four evaluation scenarios (all values in %).

Figure 4. ROC curves of Skynet and the six baselines across the four evaluation scenarios. Skynet dominates the deployment-relevant low-FPR region across all four scenarios; baselines that are near-perfect on Agent-SafetyBench and AgentErrorBench degrade sharply on ATBench and Combined. Four ROC plots, one per evaluation scenario, comparing Skynet against six baselines. The Skynet curve rises fastest in the low false-positive-rate region of every panel.

AgentErrorBench (failure type)
Subclass	$\boldsymbol{n}$	Acc	Prec	Rec	F1	FPR
inefficient_plan	46	99.18	90.20	100.00	94.85	0.89
over_simplification	20	99.14	80.00	100.00	88.89	0.89
progress_misjudge	20	99.14	80.00	100.00	88.89	0.89
constraint_ignorance	13	99.13	72.22	100.00	83.87	0.89
hallucination	13	99.13	72.22	100.00	83.87	0.89

Table 2. Fine-grained detection performance of Skynet on AgentErrorBench failure types and ATBench risk sources (all values in %).

Variant	Acc	Prec	Rec	F1	FPR
Full Skynet	96.64	99.48	94.80	97.08	0.71
w/o structural branch	94.09	99.32	90.59	94.76	0.89
w/o counterfactual views	94.09	99.19	90.72	94.76	1.07
w/o workflow-aware aug.	95.84	99.30	93.54	96.33	0.98
single-mode inference	79.42	99.25	65.59	78.99	0.71

Table 3. Component-level ablation on the Combined scenario, with one component disabled at a time (all values in %).

Per-workflow component cost (offline)
Component	Total (s)	Mean (s)	P95 (s)
Graph construction	0.2249	0.0225	0.0882
Encoder forward (sem. $+$ struct.)	0.5816	0.0582	0.1437
Mode scoring (batched)	0.0013	0.0001	—
End-to-end per workflow	—	0.0808	0.2282
End-to-end scalability by workflow size

Table 4. Runtime overhead of Skynet : offline per-workflow cost, scalability with workflow size, and online incremental per-step latency.

Anomaly Detection

Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.

See all 657 papers

AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuAgent Failure Analysis AI Agent Evaluation
The University of Hong Kong · Stellaris AI Limited
Enforcing Benign Trajectories: A Behavioral Firewall for Structured-Workflow AI Agents
Apr 29, 2026Hung DangLLM Agent Security AI Agent Monitoring
Van Lang University, Ho Chi Minh City, Vietnam
Composing Verifiable Conceptual Models via Building Blocks: Towards Design-Time Verification of Agentic AI Workflows
Jun 19, 2026Noe Y. Flandre, Alexander C. Nwala, Philippe J. GiabbanelliAgentic Workflows Formal Verification
Team EVERGREEN, Inria, Centre Inria d’Université Côte d’Azur, Montpellier, France · Department of Data Science, William & Mary, Williamsburg, VA, USA · Office of Enterprise Research and Innovation, Old Dominion University, Norfolk, VA, USA

Apr 26, 2026J/K move · Enter open · S save

AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong · Stellaris AI Limited

Agent Failure Analysis AI Agent Evaluation Agentic Workflows Agent Evaluation

Open paper AskPDF arXiv

Agentic systems that chain reasoning, tool use, and synthesis into multi-step workflows are entering production, yet prevailing evaluation practices like end-to-end outcome checks and ad-hoc trace inspection systematically mask the intermediate failures that dominate real-world error budgets. We present AgentEval, a framework that formalizes agent executions as evaluation directed acyclic graphs (DAGs), where each node carries typed quality metrics assessed by a calibrated LLM judge (GPT-4o), classified through a hierarchical failure taxonomy (3 levels, 21 subcategories), and linked to upstream dependencies for automated root cause attribution. An ablation study isolates the impact of DAG-based dependency modeling: it alone contributes +22 percentage points to failure detection recall and +34 pp to root cause accuracy over flat step-level evaluation with identical judges and rubrics. Across three production workflows (450 test cases, two agent model families, predominantly sequential architectures with a 12% non-DAG trace rate), AgentEval achieves 2.17x higher failure detection recall than end-to-end evaluation (0.89 vs. 0.41), Cohen's kappa = 0.84 agreement with human experts, and 72% root cause accuracy against an 81% human ceiling. Cross-system evaluation on tau-bench and SWE-bench traces confirms transferability (failure detection recall >= 0.78) without taxonomy or rubric modification. A 4-month pilot with 18 engineers detected 23 pre-release regressions through CI/CD-integrated regression testing, reducing median root-cause identification time from 4.2 hours to 22 minutes and driving measurable failure rate reductions in two workflows.

Abstract

Anomaly Detection

Explore similar work

AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking

Enforcing Benign Trajectories: A Behavioral Firewall for Structured-Workflow AI Agents

Composing Verifiable Conceptual Models via Building Blocks: Towards Design-Time Verification of Agentic AI Workflows