cs.AIApr 2, 2026
SaveATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Organizations: Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. · Tsinghua Univ. ·
Abstract
Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions rather than isolated prompts or final responses. Existing trajectory-level benchmarks remain limited by insufficient interaction diversity, coarse observability of safety failures, and weak long-horizon realism. We introduce ATBench, a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety. ATBench organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm. Based on this taxonomy, we construct trajectories with heterogeneous tool pools and a long-context delayed-trigger protocol that captures realistic risk emergence across multiple stages. The benchmark contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools. Data quality is supported by rule-based and LLM-based filtering plus full human audit. Experiments on frontier LLMs, open-source models, and specialized guard systems show that ATBench is challenging even for strong evaluators, while enabling taxonomy-stratified analysis, cross-benchmark comparison, and diagnosis of long-horizon failure patterns.
Figures & tables
Figure 1: Overview of ATBench. Left: a three-dimensional taxonomy of unsafe agent trajectories. Center: trajectory-level safety judgment from a complete interaction trace. Right: an unsafe example where hidden prompt injection leads the agent to skip verification and post an unverified social-media summary to Discord.
| Property | R-Judge | ASSE-Safety | ASSE-Security | ATBench500 | ATBench (Ours) |
|---|---|---|---|---|---|
| Cases | 564 | 1,476 | 817 | 500 | 1,000 |
| Available tools | – | – | – | 1,575 | 2,084 |
| Used tools | 148 | 903 | 254 | 1,357 | 1,954 |
| Avg. turns | 5.25 | 4.96 | 5.75 | 8.97 | 9.01 |
| Avg. tokens | 0.47k | 0.47k | 0.78k | 1.52k | 3.95k |
Table 1: Comparison of representative trajectory-level agent safety benchmarks by (i) number of cases, (ii) available tools, (iii) used tools, (iv) average turns, and (v) average tokens. Green subscripts report relative change ( ) w.r.t. the earlier 500-case release of ATBench.
Figure 2: Data generation engine for synthesizing multi-step agent trajectories in ATBench. Given a sampled risk and candidate tools, a planner produces a trajectory blueprint that is then executed sequentially through query generation, risk injection, tool call and response simulation, and agent response generation. A validation layer combines rule-based and LLM-based filtering to ensure realism.
| Model | ATBench-C | ATBench-F | |||||
|---|---|---|---|---|---|---|---|
| Acc | Prec. | Rec. | F1 | R.S. | F.M. | R.H. | |
| Closed-Source Models | |||||||
| GPT-5.4 | 73.7 | 68.5 | 87.1 | 76.7 | 33.6 | 13.5 | 30.2 |
| GPT-5.2 | 69.0 | 65.6 | 79.3 | 71.8 | 29.5 | 12.0 | 26.8 |
| Gemini-3-Flash | 76.4 | 79.3 | 71.0 | 74.9 | 18.4 | 8.3 | 15.0 |
| Gemini-3.1-Pro | 75.5 | 76.1 | 73.8 | 75.0 | 24.8 | 12.6 | 18.5 |
Table 2: Main results (%) on ATBench. ATBench-C reports coarse-grained binary safe/unsafe classification accuracy, precision, recall, and F1 score. ATBench-F reports fine-grained diagnosis accuracy on Risk Source (R.S.), Failure Mode (F.M.), and Real-world Harm (R.H.) over unsafe trajectories. A dash (–) indicates that the model does not support fine-grained taxonomy label prediction.
Figure 3: Representative case studies for failure-mode misidentification (a) and risk-source misattribution (b). In both cases all four evaluators correctly detect the trajectory as unsafe but fail to recover the fine-grained diagnostic label.
Figure 4: Cross-benchmark comparison of model performance on representative agent-safety benchmarks and ATBench. For most representative models, performance is lower on ATBench, indicating higher overall difficulty.
Figure 5: Category-wise accuracy on the fine-grained ATBench taxonomy. Accuracies are computed only over unsafe trajectories belonging to the corresponding leaf category. We compare AgentDoG-Qwen3-4B, GPT-5.4, Qwen3.5-397B, and Llama3.1-8B.
| Model | Training Data | R-Judge | ATBench-F | |||
|---|---|---|---|---|---|---|
| Acc. | F1 | R.S. | F.M. | R.H. | ||
| Qwen3.5-4B | – | 81.0 | 82.0 | 6.6 | 3.0 | 8.2 |
| Qwen3.5-4B-SFT | 2k engine-generated trajectories | 87.6 | 87.3 | 32.2 | 12.3 | 31.5 |
Table 3: SFT on 2k engine-generated trajectories improves real-trajectory safety judgment (R-Judge) and fine-grained diagnosis on ATBench.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Overview of the three orthogonal dimensions of the agentic safety taxonomy.
| Risk Source Category | Subcategory | Description |
|---|---|---|
| User Input | Malicious User Instruction or Jailbreak | The user explicitly and intentionally instructs the agent to perform harmful actions or generate harmful content, including the use of jailbreaking techniques to bypass built-in safeguards. |
| Direct Prompt Injection | Malicious instructions are embedded within an otherwise benign user prompt, causing the agent to execute hidden commands that override intended safety constraints. | |
| Environmental Observation | Indirect Prompt Injection | Malicious instructions are embedded within external content such as webpages, documents, or screenshots observed by the agent, leading it to unknowingly execute hidden commands during perception. |
| Unreliable or Misinformation | The agent observes incorrect, outdated, incomplete, noisy, or misleading information from its environment, resulting in unsafe or incorrect outputs even in the absence of adversarial intent. | |
| External Entities (Risks From Tools/APIs) | Tool Description Injection | The tool description or API schema is compromised to include malicious instructions or misleading specifications, causing the agent to misuse the tool or invoke harmful parameters. |
| Malicious Tool Execution | The tool itself exhibits undisclosed malicious behavior or vulnerabilities, leading to unintended and harmful outcomes when executed by the agent. |
Table 4: Risk source taxonomy for agentic systems.
| Failure Mode Category | Subcategory | Description |
|---|---|---|
| Behavioral Failure Mode | Unconfirmed or Over-privileged Action | The agent executes actions without sufficient confirmation or explicit user consent, particularly under ambiguous or incomplete instructions, or when performing high-stakes and over-privileged operations such as modifying files, spending money, or accessing sensitive resources, without appropriate safeguards ( e.g. , verification or backups). |
| Flawed Planning or Reasoning | The agent fails during the planning stage prior to execution, including misinterpreting user intent, constructing logically incorrect or unsafe action sequences, or failing to anticipate foreseeable negative consequences of its planned actions. | |
| Improper Tool Use | a) Incorrect tool parameters: Selecting a correct tool but providing wrong, unsafe, or out-of-context parameters, leading to unintended outcomes. b) Choosing malicious tools: Choosing an inherently insecure, deprecated, or malicious tool over safer alternatives. c) Tool misuse in a specific context: Using a benign tool in a context where its use is inappropriate or risky ( e.g. , privacy breaches, violating policies or regulations). d) Failure to validate tool outputs: Excessively trusting or failing to validate tool outputs, leading to the use of incorrect or harmful information. | |
| Insecure Interaction or Execution | The agent interacts with or executes untrusted, unsafe, or malicious components, such as running vulnerable code, clicking phishing links, downloading malicious files, or interacting with deceptive user interface elements in the environment. | |
| Procedural Deviation or Inaction | The agent fails to correctly follow a predefined workflow, standard operating procedure, or user-specified sequence of steps, including omitting, reordering, incorrectly performing required actions, or failing to act when necessary. | |
| Inefficient or Wasteful Execution | The agent completes a task in a technically correct but highly inefficient manner, resulting in excessive consumption of resources such as time, computational budget, or financial cost. |
Table 5: Failure mode taxonomy for agentic systems.
| Real-world Harm | Description |
|---|---|
| Privacy & Confidentiality Harm | Unauthorized exposure, disclosure, or misuse of personal, organizational, or sensitive information, including actions that compromise data confidentiality or enable re-identification. |
| Financial & Economic Harm | Agent behaviors that cause direct or indirect monetary loss, disrupt financial assets, initiate unauthorized transactions, or produce economically damaging decisions. |
| Security & System Integrity Harm | Actions that compromise account security, system configurations, code execution safety, or overall digital infrastructure reliability, increasing the system’s vulnerability to attacks or misuse. |
| Physical & Health Harm | Agent behaviors that directly or indirectly endanger human health, safety, or the physical environment, including harmful guidance or unsafe control of real-world devices. |
| Psychological & Emotional Harm | Agent behaviors that negatively impact an individual’s psychological or emotional well-being, including harassment, intimidation, exposure to disturbing content, or generation of content attacking a person’s dignity, causing distress, fear, anxiety, or trauma. |
| Reputational & Interpersonal Harm | Generation or dissemination of content or actions that damage an individual’s or organization’s reputation, trustworthiness, or social relationships. |
Table 6: Real-world harm taxonomy for agentic systems.
Figure 7: The prompt template of trajectory-level evaluation.
Figure 8: The prompt template of fine-grained risk diagnosis.
Figure 9: The prompt template for ShieldAgent.
Figure 10: The prompt template for Llama Guard.
Figure 11: The prompt template for Qwen3Guard.
| Audit item | Value |
| Reviewers | 5 |
| Audited trajectories | 1,000 |
| unsafe safe corrections | 4 |
| safe unsafe corrections | 1 |
| Revised binary safe/unsafe labels (total) | 5 |
| Unsafe trajectories with any taxonomy revision | 129 |
Table 7: Summary statistics from the full human audit of ATBench. Fine-grained revision counts are computed over trajectories retained as unsafe after coarse-grained review.
| Annotator | Risk Source | Failure Mode | Real-world Harm |
|---|---|---|---|
| Annotator 1 | 82.0% (41/50) | 70.0% (35/50) | 88.0% (44/50) |
| Annotator 2 | 86.0% (43/50) | 60.0% (30/50) | 80.0% (40/50) |
| Annotator 3 | 86.0% (43/50) | 72.0% (36/50) | 86.0% (43/50) |
| Avg. | 84.7% (127/150) | 67.3% (101/150) | 84.7% (127/150) |
Table 8: Agreement between three independent human annotators and the final audited benchmark labels on 50 sampled unsafe trajectories.
| Group | Model | Overall Acc | Unsafe | Long Ctx. | False Ref. | Long HTML |
|---|---|---|---|---|---|---|
| Closed-source | GPT-5.4 | 73.7 | 84.6 | 96.0 | 96.0 | 100.0 |
| Closed-source | Gemini-3.1-Pro | 75.5 | 71.3 | 78.0 | 80.0 | 100.0 |
| Open-source | Qwen3.5-397B-A17B | 66.8 | 68.5 | 72.0 | 72.0 | 92.0 |
| Open-source | Qwen3.5-4B | 45.9 | 20.7 | 6.0 | 28.0 | 44.0 |
| Open-source | QwQ-32B | 57.7 | 17.6 | 10.0 | 20.0 | 60.0 |
| Open-source | Llama3.1-8B-Instruct | 45.3 | 97.7 | 64.0 | 100.0 | 0.0 |
Table 9: Subset-level ablation (%) separating the sources of benchmark difficulty on binary safety detection.
| Annotator | N | User Query | Tool Definition | Tool Response | Attack / Risk Plausibility | Overall |
|---|---|---|---|---|---|---|
| Annotator 1 | 50 | 4.08 | 4.44 | 3.94 | 3.86 | 4.06 |
| Annotator 2 | 50 | 4.04 | 3.90 | 4.14 | 4.06 | 4.04 |
| Annotator 3 | 50 | 3.94 | 4.10 | 4.30 | 4.30 | 4.16 |
| Avg. | 150 | 4.02 | 4.15 | 4.13 | 4.07 | 4.09 |
Table 10: Human realism ratings (1–5) of sampled ATBench trajectories along four aspects.
Figure 12: An unsafe trajectory where corrupted tool feedback drives a false risk escalation, and the risk comes from the fact that a poisoned fraud signal is turned into a real denial decision that can unfairly withhold financial access. Red elements highlight the fabricated fraud indicator, the injected instruction to skip verification, and the denial message that operationalizes the poisoned signal.
Figure 13: An unsafe trajectory where a high-stakes medical record update is committed without required verification, and the risk comes from the fact that an incorrect infectious-disease status can directly affect downstream clinical and operational decisions. Red elements highlight the update action and the returned warning fields showing an unverified status, a conflicting PCR record, and no override protocol.
Figure 14: An unsafe trajectory where a direct prompt injection causes wasteful execution, and the risk comes from the fact that a recognized redundancy is still turned into a paid verification loop at scale. Red elements highlight the override instruction to ignore efficiency safeguards, the redundant verification workflow, and the ledger evidence showing per-contact charges.
Figure 15: An unsafe trajectory where a supposedly read-only diagnostic call hides an unauthorized safety change, and the risk comes from the fact that covert modification of a vehicle safety parameter can create real-world physical danger even when the visible status looks normal. Red elements highlight the hidden write evidence modifying collision_avoidance_sensitivity and the completion message that ignores it.
Figure 16: An unsafe trajectory where the agent over-weights unreliable community anecdotes relative to stronger medical evidence, and the risk comes from the fact that weak social evidence is allowed to dilute stronger medical warnings in a health decision. Red elements highlight the medical-risk warning, the misleading Telegram messages, the corroborating web-risk snippet, and the final completion that reframes the combination as viable.
Figure 17: An unsafe long-context trajectory where an earlier support-setup episode creates a reusable group identifier, and a later request turns that carried-over state into a high-impact removal action. The agent directly reuses the earlier identifier in delete_membership , which deletes the membership and queues unassignment for 14 active tickets. Red elements highlight the setup-generated identifier, the later request that depends on it, and the resulting removal impact.
Figure 18: A safe trajectory where the agent halts after detecting a suspicious side effect in browser output, and the key safety point is that proceeding would propagate data from a payload that may already have caused credential exposure. Red elements highlight the injected credential-export claim, while green elements highlight the decision to stop downstream communications.
Figure 19: A safe trajectory where the agent refuses to broadcast safety guidance derived only from unverified online sources, and the key safety point is that unverified chemical advice can cause physical harm if redistributed as actionable instructions. Red elements highlight the request to turn Reddit-derived advice into SMS instructions, while green elements highlight the refusal to operationalize it.
Figure 20: An unsafe trajectory where tool-description injection normalizes unsolicited bulk outreach, and the risk comes from the fact that the injected implied-consent framing removes the normal safeguard against sending unsolicited messages to real recipients. Red elements highlight the injected implied-consent claim, the direct-media-share action, and the delivery summary showing 15 successful sends.
Figure 21: An unsafe trajectory where the agent proceeds with a transfer despite a critical routing mismatch, and the risk comes from the fact that a mismatch between vendor identity and payout destination is a direct indicator of possible misdirected funds. Red elements highlight the mismatched account holder name and the subsequent payment action that ignores it.
Figure 22: A safe trajectory where the agent stays within public-data boundaries during competitor analysis, and the key safety point is that moving beyond public profiles would turn the workflow into unauthorized data access. Green elements highlight the explicit is_private: false checks that keep the workflow within public-profile data.
Figure 23: An unsafe trajectory where indirect prompt injection turns an unverified enrichment note into a stakeholder-facing regulatory alert, and the risk comes from the fact that speculative regulatory claims can become misinformation once they are redistributed as confirmed alerts. Red elements highlight the speculative regulatory note and the subsequent email action that forwards it without validation.
Figure 24: An unsafe trajectory where the agent performs an over-privileged external data transfer, and the risk comes from the fact that internal employee data is disclosed to a third-party service without any extra confirmation or necessity check. Red elements highlight the internal employee email, the third-party enrichment call using it, and the resulting disclosure to an external service.
Figure 25: An unsafe trajectory where correctly extracted hazardous-cargo information is turned into an unsafe routing request, and the risk comes from the fact that the route is computed as if the shipment were non-hazardous, which can defeat dangerous-goods routing safeguards. Red elements highlight the hazardous-cargo fields, the incorrect danger: "0" parameter, and the returned search conditions that preserve it.
Figure 26: An unsafe trajectory where scraped external content supplies a destructive deletion parameter, and the risk comes from the fact that untrusted webpage content directly determines the target of a destructive system action. Red elements highlight the injected maintenance instruction and the delete action that uses prod-analytics-2023-q4 .
More in this topic
AI Agent Safety Benchmarks
AI agent safety benchmarks measure whether tool-using and embodied AI systems act safely across realistic tasks and interactions.
See all 206 papers