Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

Miles Q. Li; Benjamin C. M. Fung; Boyang Li; Heba Ismail; Farkhund Iqbal

cs.CYApr 11, 2026

Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

Authors: Miles Q. Li, Benjamin C. M. Fung, Boyang Li, Heba Ismail, Farkhund Iqbal

Organizations: School of Information Studies, McGill University, Canada · Department of Computer Science, Kean University, USA · College of Technological Innovation, Zayed University, UAE

Abstract

The rapid deployment of LLM-based autonomous agents has introduced safety risks that extend far beyond traditional LLM concerns, prompting a proliferation of safety benchmarks since late 2023. However, these benchmarks have developed independently, with inconsistent threat models, incompatible metrics, and overlapping yet incomplete risk coverage. We present the first systematic analysis dedicated to agent safety benchmarks as evaluation instruments. We catalog 40 behavioral agent-safety benchmarks (2023-2026), plus 5 adjacent evaluator, defense, and dataset artifacts, propose a six-axis taxonomy of benchmark evaluation methodology, and apply it across the corpus to characterize how methodological choices shape safety conclusions. A coverage matrix reveals broad risk coverage but limited methodological convergence, while the taxonomy analysis shows a behavioral-benchmark core concentrated in sandboxed, constrained, and often safety-only evaluation. Across the landscape, we find that benchmark choice can yield contradictory safety conclusions, coverage counts often overstate evaluation depth, environment fidelity systematically shapes reported safety, the field disproportionately tests externally imposed rather than agent-internal risks, metric fragmentation limits comparison, and robustness remains effectively unbenchmarked. We ground these claims with a cross-benchmark consistency check, with 95% confidence intervals and Kendall's W concordance analysis, finding no evidence of ranking concordance across evaluation dimensions (W = 0.10, p = 0.94). We release structured metadata, full taxonomy codings, risk annotations, and all experimental artifacts, and propose minimum reporting standards for future benchmarks.

Figures & tables

Figure 1: Year-level timeline of core behavioral agent-safety benchmark publications, color-coded by primary focus area. The field grows from 2 pioneer benchmarks in 2023 to a broad, highly fragmented benchmark landscape by early 2026, with the sharpest expansion in 2025.

Survey	Agent-specific	Benchmarks compared	Methodology taxonomy	Empirical check
Chhabra et al. (2025)	Yes	Supporting role	No	No
Yu et al. (2025)	Yes	Not systematic	No	No
Ma et al. (2025)	Partial	One section	No	No
Wang et al. (2025)	Partial	One section	No	No
Yu et al. (2026)	No	210 (meta-analysis)	Quality framework	No
Yehudai et al. (2025)	Yes	Safety = one dim.	No	No

Table 1: Positioning relative to related surveys.

ID	Risk Category	Benchmark Coverage
R1	Direct Misuse / Harmful Task Execution	Moderate
R2	Indirect Prompt Injection	Moderate
R3	Tool Misuse & Unsafe Actions	High
R4	Goal Misalignment & Specification Gaming	Moderate
R5	Multi-Agent Risks	High
R6	Memory & State Risks	High

Table 2: Summary of agentic risk taxonomy.

Axis	AgentHarm	AgentDojo	InjecAgent	ODCV-Bench
1. Adv. Source	User	Environment	Environment	Structural
2. Env. Fidelity	Sandboxed	Containerized	Sandboxed	Containerized
3. Capability	Constrained	Constrained	Constrained	Open tools
4. Scoring	Human rubric	Rule-based	Rule-based	LLM-judge
5. Granularity	Outcome	Action	Action	Pattern
6. Safety-Utility	Safety-only	Separate	Safety-only	Separate

Table 3: Taxonomy coordinates for the four consistency-check benchmarks.

Axis	Distribution across 40 core benchmarks
Adversarial pressure	Environmental 13; Multi-agent 9; Structural 8; Agent-internal 6; User-directed 4
Environment fidelity	Sandboxed 34; Containerized 4; Live 1; LLM-emulated 1
Capability envelope	Constrained tools 26; Open tools 7; Autonomous planning 6; Text-only 1
Scoring method	Rule-based 28; LLM-judge 7; Hybrid 4; Human rubrics 1
Evaluation granularity	Action 16; Pattern 11; Outcome 10; Disposition 3
Safety-utility coupling	Safety-only 24; Separate 12; Joint 3; Over-refusal 1

Table 4: Corpus-level distribution of the six-axis taxonomy across the 40 core behavioral benchmarks.

Category	Risk	Primary	% Peer-Rev.
R1	Misuse	4	25%
R2	Injection	5	60%
R3	Tool Misuse	5	60%
R4	Misalignment	4	50%
R5	Multi-Agent	9	22%
R6	Memory	7	71%

Table 5: Summary of core behavioral-benchmark coverage by risk category. Counts indicate benchmarks with primary coverage of each risk; adjacent artifacts are excluded from this summary.

Figure 2: Heatmap of the risk–benchmark coverage matrix for 40 core behavioral benchmarks (full 45-entry table in Appendix B ). Dark = primary focus, light = secondary/partial, white = no coverage.

Figure 3: Radar chart of risk coverage among the 40 core behavioral benchmarks. Dark = primary coverage; light = secondary/partial. R10 (Robustness) has only secondary coverage.

Model	AH	CI	AD	CI	IA	CI	OD	CI
GPT-5.4	.04	[.01,.13]	1.0	[.98,1.]	.00	[.00,.07]	.24	[.16,.34]
Qwen 3.5 Plus	.04	[.01,.13]	.50	[.43,.56]	.00	[.00,.07]	.36	[.26,.47]
Claude Opus 4.6	.06	[.02,.16]	1.0	[.98,1.]	.00	[.00,.07]	.12	[.06,.20]
Gemini 3.1 Pro	.06	[.02,.16]	1.0	[.98,1.]	.00	[.00,.07]	.45	[.35,.56]
GPT-OSS-20B	.06	[.02,.16]	.55	[.49,.61]	.00	[.00,.07]	.33	[.23,.43]
GPT-OSS-120B	.10	[.04,.21]	.50	[.44,.57]	.00	[.00,.07]	.39	[.29,.50]

Table 6: Cross-benchmark consistency check (

n{=}12

models). Higher scores are less safe. 95% Wilson CIs in brackets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10
Harmful Task Execution
AgentHarm	●	○	◐	○	○	○	○	○	○	○
Agent-SafetyBench	●	○	●	○	○	○	◐	○	◐	◐
SafePro	●	○	◐	●	○	○	◐	○	◐	○
MACHIAVELLI	○	○	○	●	○	○	○	○	○	○
Security / Prompt Injection

Appendix

Table 7: Full risk–entry coverage matrix. ● = primary focus, ◐ = secondary/partial coverage, ○ = not covered. Entries are grouped by primary focus area. Rows marked with † are adjacent artifacts.

Benchmark	Year	Venue	Tasks	Tools	Primary Risk	Env. Type	Eval. Method	Key Metric
AgentHarm	2024	ICLR’25	440	104	Harmful compliance	Synthetic tools	Human rubrics	Harm score
Agent-SafetyBench	2024	arXiv	2,000	349	General safety	Simulated	Fine-tuned judge	Safety score
SafePro	2026	arXiv	275	–	Professional misuse	Code agent	LLM-as-judge	Unsafe rate
MACHIAVELLI	2023	ICML’23	572K	–	Ethics/reward	Text games	Annotation	Ethical score
ASB	2024	ICLR’25	400+	400+	Attack/defense	Multi-domain	ASR metrics	ASR
AgentDojo	2024	NeurIPS’24	629	$\sim$ 100	Prompt injection	Tool-use sim	State checks	Util.+ASR

Appendix

Table 8: Comprehensive comparison of agent-safety evaluation entries. “–” indicates not applicable or not reported. Rows marked with † are adjacent artifacts rather than core behavioral benchmarks.

Benchmark	Adversarial Pressure Source	Environment Fidelity	Capability Envelope	Scoring Method	Evaluation Granularity	Safety-Utility Coupling
AgentHarm	User-directed	Sandboxed	Constrained tools	Human rubrics	Outcome	Safety-only
Agent-SafetyBench	User-directed	Sandboxed	Constrained tools	LLM-judge	Pattern	Safety-only
SafePro	User-directed	Sandboxed	Open tools	LLM-judge	Outcome	Safety-only
MACHIAVELLI	Structural/ incentive-based	Sandboxed	Text-only	Rule-based	Pattern	Separate reporting
ASB	Environmental	Sandboxed	Constrained tools	Rule-based	Action	Safety-only
AgentDojo	Environmental	Containerized	Constrained tools	Rule-based	Action	Separate reporting

Appendix

Table 9: Entry-level six-axis taxonomy codings. Full entries are shown to make the coding auditable from the manuscript itself. Rows marked with † are adjacent artifacts.

AI Agent Safety Benchmarks

AI agent safety benchmarks measure whether tool-using and embodied AI systems act safely across realistic tasks and interactions.

See all 206 papers

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Apr 2, 2026Yu Li, Haoyu Luo, Yuejin Xie +10AI Agent Safety Benchmarks LLM Agent Safety
Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. +2
AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance
Apr 14, 2026Abiodun A. SolankeLLM Safety Benchmarks Benchmark Auditing
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1

Apr 2, 2026J/K move · Enter open · S save

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis

Yu Li, Haoyu Luo, Yuejin Xie, Yuqian Fu+9

Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ.+2

AI Agent Safety Benchmarks LLM Agent Safety Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions rather than isolated prompts or final responses. Existing trajectory-level benchmarks remain limited by insufficient interaction diversity, coarse observability of safety failures, and weak long-horizon realism. We introduce ATBench, a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety. ATBench organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm. Based on this taxonomy, we construct trajectories with heterogeneous tool pools and a long-context delayed-trigger protocol that captures realistic risk emergence across multiple stages. The benchmark contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools. Data quality is supported by rule-based and LLM-based filtering plus full human audit. Experiments on frontier LLMs, open-source models, and specialized guard systems show that ATBench is challenging even for strong evaluators, while enabling taxonomy-stratified analysis, cross-benchmark comparison, and diagnosis of long-horizon failure patterns.

Abstract

AI Agent Safety Benchmarks

Explore similar work

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis

AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities