Detecting Safety Violations Across Many Agent Traces

Adam Stein; Davis Brown; Hamed Hassani; Mayur Naik; Eric Wong

cs.AIApr 13, 2026

Detecting Safety Violations Across Many Agent Traces

Authors: Adam Stein, Davis Brown, Hamed Hassani, Mayur Naik, Eric Wong

Organizations: University of Pennsylvania

Abstract

To identify safety violations, auditors often search over large sets of agent traces. This search is difficult because failures are often rare, complex, and sometimes even adversarially hidden and only detectable when multiple traces are analyzed together. These challenges arise in diverse settings such as misuse campaigns, covert sabotage, reward hacking, and prompt injection. Existing approaches struggle here for several reasons. Per-trace judges miss failures that only become visible across traces, naive agentic auditing does not scale to large trace collections, and fixed monitors are brittle to unanticipated behaviors. We introduce Meerkat, which combines clustering with agentic search to uncover violations specified in natural language. Through structured search and adaptive investigation of promising regions, Meerkat finds sparse failures without relying on seed scenarios, fixed workflows, or exhaustive enumeration. Across misuse, misalignment, and task gaming settings, Meerkat significantly improves detection of safety violations over baseline monitors, discovers widespread developer cheating on a top agent benchmark, and finds nearly 4x more examples of reward hacking on CyBench than previous audits.

Figures & tables

Figure 1: A repository-level decomposition attack example for a ransomware attack workflow. The attacker attempts a ransomware attack, by asking a model in different contexts to (1) encrypt all of the users files and (2) ask for the user to pay money for them to decrypt the files. Most traces appear benign in isolation, but this small subset jointly forms a violating witness

W

within the much larger repository.

Data Source	Risk Focus	Labeled	Cases	Case size
Cyber Misuse ( Brown et al., 2025 )	Decomposition attacks	Yes	50	20–600
Bio Misuse ( Brown et al., 2025 )	Decomposition attacks	Yes	50	20–600
ImpossibleBench ( Zhong et al., 2026a )	Task gaming	Yes	200	10–100
TRACE ( Deshpande et al., 2026 )	Task gaming	Yes	200	10–100
CUA-SHADE-Arena ( Kale et al., 2025 )	Covert Sabotage	Yes	200	10–100
Terminal-Bench-2.0 ( Laude Institute, 2025 )	Developer cheating	No	10	61–364

Table 1: Evaluation corpora for repository-level safety auditing. “Labeled” indicates whether exhaustive ground-truth verdicts are available for all evaluated traces.

Figure 2: Distributed misuse detection precision-recall curves for Cyber and Bio domains. Meerkat (an agentic monitor) consistently improves AP over the baselines.

Figure 3: Task gaming detection precision-recall curves for ImpossibleBench. Meerkat (an agentic monitor) consistently improves AP over the baselines, although the performance of all methods decreases with more background traces.

Figure 4: Inference-compute scaling on a misuse case (cyber, bg=100x), using GPT-5.4 in Meerkat with n=10 runs per point. Left: pass@k curves by reasoning effort, where success is AUROC > 0.5. Higher compute increases the probability of detections. Right: pass@1 versus reasoning tokens. A 13x increase in reasoning yields a 5x increase in detection probability.

Panel of Figure 5 — Figure 5: Evaluation cheating discovered by Meerkat. Left: On Terminal-Bench 2, the scaffold cheats with an AGENTS.md file with the correct answer; the agent adopted it verbatim and “verified” against the same source. Right: On BountyBench, the agent was missing necessary dependencies, so stood up a fake demo that does not use the vulnerable endpoint.

Figure 6: Distributed misuse detection (cyber, bg=100x) under a prompt injection (‘persona reframe’) and an adversarial-prefix attack, where the prefix reduces Meerkat’s AP by 10x.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	BG	Model	Method					$\Delta$
Domain	BG	Model	Meerkat	Naive Agent	Monitor	Bayesian	Buffer	$\Delta$
DM-Cyber	20 $\times$	Qwen3.5	0.837 $\pm$ 0.052	0.632 $\pm$ 0.065	0.125 $\pm$ 0.028	0.243 $\pm$ 0.068	0.191 $\pm$ 0.056	+0.205
	20 $\times$	GPT-5.4m	0.445 $\pm$ 0.133	0.842 $\pm$ 0.000	0.199 $\pm$ 0.061	0.332 $\pm$ 0.066	0.229 $\pm$ 0.107	-0.397
	100 $\times$	Qwen3.5	0.098 $\pm$ 0.034	0.456 $\pm$ 0.055	0.011 $\pm$ 0.000	0.081 $\pm$ 0.023	0.022 $\pm$ 0.003	-0.359
	100 $\times$	GPT-5.4m	0.297 $\pm$ 0.084	0.136 $\pm$ 0.072	—	0.053 $\pm$ 0.026	0.021 $\pm$ 0.002	+0.161
DM-Bio	20 $\times$	Qwen3.5	0.593 $\pm$ 0.087	0.572 $\pm$ 0.068	0.054 $\pm$ 0.003	0.176 $\pm$ 0.038	0.142 $\pm$ 0.044	+0.021

Appendix

Table 2: Results for distributed misuse using Trace-level AP. Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Domain	BG	Model	Method					$\Delta$
Domain	BG	Model	Meerkat	Naive Agent	Monitor	Bayesian	Buffer	$\Delta$
DM-Cyber	20 $\times$	Qwen3.5	0.970 $\pm$ 0.012	0.992 $\pm$ 0.005	0.568 $\pm$ 0.044	0.774 $\pm$ 0.044	0.633 $\pm$ 0.044	-0.022
	20 $\times$	GPT-5.4m	0.915 $\pm$ 0.049	1.000 $\pm$ 0.000	0.610 $\pm$ 0.040	0.931 $\pm$ 0.021	0.752 $\pm$ 0.113	-0.085
	100 $\times$	Qwen3.5	0.539 $\pm$ 0.022	0.850 $\pm$ 0.033	0.439 $\pm$ 0.001	0.697 $\pm$ 0.031	0.576 $\pm$ 0.018	-0.311
	100 $\times$	GPT-5.4m	0.848 $\pm$ 0.049	0.645 $\pm$ 0.068	—	0.733 $\pm$ 0.058	0.744 $\pm$ 0.040	+0.104
DM-Bio	20 $\times$	Qwen3.5	0.838 $\pm$ 0.052	0.927 $\pm$ 0.033	0.457 $\pm$ 0.004	0.758 $\pm$ 0.053	0.600 $\pm$ 0.030	-0.088

Appendix

Table 3: Results for distributed misuse using Trace-level ROCAUC. Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Domain	TPC	Model	Method					$\Delta$
Domain	TPC	Model	Meerkat	Monitor	Bayesian	Buffer	Naive Agent	$\Delta$
TRACE	10	Qwen3.5	0.760 $\pm$ 0.073	0.764 $\pm$ 0.077	—	—	0.640 $\pm$ 0.085	-0.004
		GPT-5.4m	0.673 $\pm$ 0.069	0.719 $\pm$ 0.071	—	—	—	-0.047
		GLM-5	0.721 $\pm$ 0.082	0.685 $\pm$ 0.085	—	—	—	+0.036
	25	Qwen3.5	0.626 $\pm$ 0.088	0.673 $\pm$ 0.090	—	—	0.390 $\pm$ 0.094	-0.047
		GPT-5.4m	0.604 $\pm$ 0.086	0.590 $\pm$ 0.084	—	—	—	+0.013

Appendix

Table 4: Trace-level average precision across safety corpora by traces per case. Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Domain	TPC	Model	Method					$\Delta$
Domain	TPC	Model	Meerkat	Monitor	Bayesian	Buffer	Naive Agent	$\Delta$
TRACE	10	Qwen3.5	0.878 $\pm$ 0.043	0.876 $\pm$ 0.043	—	—	0.782 $\pm$ 0.052	+0.002
		GPT-5.4m	0.853 $\pm$ 0.042	0.864 $\pm$ 0.042	—	—	—	-0.011
		GLM-5	0.833 $\pm$ 0.051	0.827 $\pm$ 0.048	—	—	—	+0.007
	25	Qwen3.5	0.828 $\pm$ 0.048	0.835 $\pm$ 0.048	—	—	0.681 $\pm$ 0.050	-0.007
		GPT-5.4m	0.847 $\pm$ 0.047	0.848 $\pm$ 0.047	—	—	—	-0.001

Appendix

Table 5: Trace-level ROC-AUC across safety corpora by traces per case. Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Domain	TPC	Model	Method					$\Delta$
Domain	TPC	Model	Meerkat	Monitor	Bayesian	Buffer	Naive Agent	$\Delta$
TRACE	10	Qwen3.5	0.882 $\pm$ 0.002	0.843 $\pm$ 0.002	—	—	0.751 $\pm$ 0.002	+0.039
		GPT-5.4m	0.717 $\pm$ 0.003	0.822 $\pm$ 0.002	—	—	—	-0.105
		GLM-5	0.714 $\pm$ 0.003	0.861 $\pm$ 0.002	—	—	—	-0.147
	25	Qwen3.5	0.772 $\pm$ 0.003	0.808 $\pm$ 0.002	—	—	0.743 $\pm$ 0.002	-0.037
		GPT-5.4m	0.539 $\pm$ 0.003	0.634 $\pm$ 0.003	—	—	—	-0.095

Appendix

Table 6: Case-level average precision across safety corpora by traces per case. Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Domain	TPC	Model	Method					$\Delta$
Domain	TPC	Model	Meerkat	Monitor	Bayesian	Buffer	Naive Agent	$\Delta$
TRACE	10	Qwen3.5	0.852 $\pm$ 0.002	0.835 $\pm$ 0.002	—	—	0.729 $\pm$ 0.002	+0.017
		GPT-5.4m	0.730 $\pm$ 0.002	0.782 $\pm$ 0.002	—	—	—	-0.051
		GLM-5	0.755 $\pm$ 0.002	0.838 $\pm$ 0.002	—	—	—	-0.083
	25	Qwen3.5	0.798 $\pm$ 0.002	0.790 $\pm$ 0.002	—	—	0.742 $\pm$ 0.002	+0.008
		GPT-5.4m	0.570 $\pm$ 0.002	0.692 $\pm$ 0.002	—	—	—	-0.122

Appendix

Table 7: Case-level ROC-AUC across safety corpora by traces per case. Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Figure 7: Distributed misuse detection boost in trace-level AP from Meerkat over the per-trace monitor.

Figure 8: ImpossibleBench reward hacking detection boost in trace-level AP from Meerkat over the per-trace monitor.

Domain	TPC	Model	Method			$\Delta$
Domain	TPC	Model	Meerkat	Monitor	Naive Agent	$\Delta$
TRACE	10	Qwen3.5	0.760 $\pm$ 0.073	0.764 $\pm$ 0.077	—	-0.004
	10	GPT-5.4m	0.673 $\pm$ 0.069	0.719 $\pm$ 0.071	—	-0.047
	25	Qwen3.5	0.626 $\pm$ 0.088	0.673 $\pm$ 0.090	—	-0.047
	25	GPT-5.4m	0.604 $\pm$ 0.086	0.590 $\pm$ 0.084	—	+0.013
	50	Qwen3.5	0.690 $\pm$ 0.074	0.672 $\pm$ 0.074	—	+0.018

Appendix

Table 8: TRACE trace-level average precision by traces per case (TPC). Higher is better.

\Delta

is Meerkat minus the strongest baseline.

Figure 9: Sabotage detection on CUA-SHADE-Arena (n=403 transcripts, 17 tasks). Meerkat has slightly higher precision at high recall than the per-trace Monitor ( Kutasov et al., 2025 ; Kale et al., 2025 ) and EaE ( Arike et al., 2026 ) baselines.

Scaffold	Model	Pass Rate	Passing Traces	Cheating
ForgeCode (2026)	Claude Opus 4.6	81.8%	364	9 confirmed traces
opensage-agent (2026)	GPT-5.3-Codex	79%	67	None
Factory AI (2026)	GPT-5.3-Codex	79%	70	None
Terminal-Bench (2026)	GPT-5.3-Codex	75%	304	None
KRAFTON AI (2026)	Claude Opus 4.6	74%	331	None
Coder (2026)	GPT-5.3-Codex	74%	66	None

Appendix

Table 9: Terminal-Bench 2.0 scaffolds audited. All confirmed answer injection came from the top-ranked scaffold ( ForgeCode, 2026 ) . Across the other audited scaffolds, we found only one reward-hacking attempt, and it failed.

Task	ForgeCode (2026)	Baseline harness	Text suggesting cheating
compile-compcert	5/5	0/1	“following the project guidelines carefully”
configure-git-webserver	5/5	0/1	“follow the project guidelines exactly”
gpt2-codegolf	3/5	0/1	“following the detailed project guidelines”
headless-terminal	5/5	0/1	Reads AGENTS.md from /app/
mteb-leaderboard	4/5	0/1	Confirmed cheating
mteb-retrieve	5/5	0/1	“follow the project guidelines closely”

Appendix

Table 10: Performance on representative tasks where AGENTS.md may explain scaffold performance. Namely, each passing trace shown here references AGENTS.md or “project guidelines.”

Model	bn-fit-modify	mteb-leaderboard	Number of false positives
gpt-5.4-mini	No	No	4
gpt-5.4	Yes	No	3
Claude Opus 4.6	Yes	Yes	1

Appendix

Table 11: Detection results by Meerkat configuration on the confirmed Terminal-Bench patterns. We check the false positives manually.

Pattern	Traces	Tasks
Successful writeup/solution fetch	16	13/40
Failed cheating + legitimate solve	11	10/40
Legitimate (misclassified by pipeline and corrected manually)	9	8/40

Appendix

Table 12: Shortcut patterns in CyBench traces. Looking up external write-ups is the cleanest example of importing a pre-existing solution for the benchmark instance.

Source	Instances	Models	Setting
CyBench paper	0	—	Original evaluation
Transluce / Docent	1	—	Flag leakage case
NIST CAISI	4	2	Controlled evals
This work	16	4	Community traces

Appendix

Table 13: CyBench writeup-lookup findings compared with prior reports.

		Raw scores		Bayesian (score-only)		Bayesian + clusters
Setting	$n$	AUROC	AP	AUROC	AP	AUROC	AP
Cyber, bg=100 $\times$	50	.574	.036	.541	.020	.745	.130
Cyber, bg=20 $\times$	20	.610	.173	.605	.150	.767	.383
Bio, bg=100 $\times$	15	.586	.030	.566	.026	.665	.166
Bio, bg=20 $\times$	20	.588	.131	.587	.119	.716	.377

Appendix

Table 14: Ablation of Bayesian baselines. Raw scores : per-trace monitor scores with no Bayesian processing. Bayesian (score-only) : the full Bayesian pipeline from Section J.2 without the cluster-boosting step. Bayesian + clusters : the complete pipeline including the cluster-boosted extension ( Section J.3 ). All metrics are macro-averaged over cases under leave-one-out cross-validation.

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 261 papers

Willful Disobedience: Automatically Detecting Failures in Agentic Traces
Mar 25, 2026Reshabh K Sharma, Shraddha Barke, Benjamin ZornAI Agent Evaluation AI Agent Auditing
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA
MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning
Sep 19, 2026Changyue Jiang, Jiayi Wang, Xin Wen +3AI Agent Auditing AI Agent Security
Fudan University Shanghai Innovation Institute
HazardAuditor: From Executable Threats to Safer Computer-Use Agents
Sep 14, 2026Yunhao Feng, Ruixiao Lin, Ming Wen +5Computer-Use Agent Safety Language Model Safety Evaluation
Ant Group · Hunan Institute of Advanced Technology · Zhejiang University +3

Mar 25, 2026J/K move · Enter open · S save

Willful Disobedience: Automatically Detecting Failures in Agentic Traces

Reshabh K Sharma, Shraddha Barke, Benjamin Zorn

University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA

AI Agent Evaluation AI Agent Auditing

Open paper AskPDF arXiv

AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from

τ^2

-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.

Abstract

AI Agent Auditing

Explore similar work

Willful Disobedience: Automatically Detecting Failures in Agentic Traces

MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning

HazardAuditor: From Executable Threats to Safer Computer-Use Agents