TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories

Yen-Shan Chen; Sian-Yao Huang; Cheng-Lin Yang; Yun-Nung Chen

cs.CRApr 8, 2026

TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories

Authors: Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang, Yun-Nung Chen

Organizations: CyCraft AI Lab, Taiwan · National Taiwan University

Abstract

As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to intermediate execution traces. While safety guardrails are well-benchmarked for natural language responses, their efficacy remains largely unexplored within multi-step tool-use trajectories. To address this gap, we introduce TraceSafe-Bench, the first comprehensive benchmark specifically designed to assess mid-trajectory safety. It encompasses 12 risk categories, ranging from security threats (e.g., prompt injection, privacy leaks) to operational failures (e.g., hallucinations, interface inconsistencies), featuring over 1,000 unique execution instances. Our evaluation of 13 LLM-as-a-guard models and 7 specialized guardrails yields three critical findings: 1) Structural Bottleneck: Guardrail efficacy is driven more by structural data competence (e.g., JSON parsing) than semantic safety alignment. Performance correlates strongly with structured-to-text benchmarks ( $ρ=0.79$ , $p<0.01$ ) while we detect no association with standard jailbreak robustness. 2) Scale Is Not the Deciding Factor: Trajectory detection accuracy does not scale monotonically with model size, and general-purpose LLMs consistently outperform specialized safety guardrails. 3) Bounded Temporal Stability: Detection accuracy improves with trajectory length within native context bounds as models process dynamic execution behavior, but degrades in extreme long-context regimes due to over-refusal.

Figures & tables

Figure 1: (Top) The threat landscape in tool-calling pipelines. (Bottom) The TraceSafe-Bench construction pipeline: (1) Generate benign traces; (2) For each trace, use a Check function to exhaustively test mutation suitability for every (step, risk category) pair; (3) Apply Mutate to each suitable combination, truncating traces at the mutation point.

Figure 2: Examples of Mutate for each category.

Panel of Table 1 — Table 1: Classification accuracy for TraceSafe . Each section is sorted by overall performance. $\bigcirc$ and $\CIRCLE$ denote open and closed source general purpose LLMs respectively; $\square$ and $\blacksquare$ denote open and closed source specialized guardrails. Note that not all models are evaluated on every setting due to model constraints (e.g., fixed risk taxonomies or output formats in certain guardrails). The Unsafe column averages columns 1-12. Avg. denotes overall balanced accuracy. (%) Note that gpt-oss-120b additionally serves as the Check judge during dataset construction; to rule out self-grading bias, we report a held-out-judge validation study in Appendix E.3 , which finds this role confers no scoring advantage.

Figure 3: Scatter plots illustrating the Pearson correlation (

\rho

) between model performance (balanced F1 score) on TraceSafe-Bench and five established capabilities.

Figure 4: Performance trends relative to (a) the length and (b) the number of tool-calling steps in the trace. Bar charts show number of samples. Line plots indicate average accuracy.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model Name	Type	Link / Endpoint Specification
General Purpose LLMs
ToolACE-8B	Open Source	https://huggingface.co/Team-ACE/ToolACE-2-Llama-3.1-8B
Ministral-14B	Open Source	https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512
Qwen3-32B	Open Source	https://huggingface.co/Qwen/Qwen3-32B
Qwen3-4B	Open Source	https://huggingface.co/Qwen/Qwen3-4B
Qwen2.5-7B	Open Source	https://huggingface.co/Qwen/Qwen2.5-7B

Appendix

Table 2: Overview of models utilized in this study, including access types and source links.

Dataset Name	Purpose	Citation
LiveCodeBench	Coding proficiency	Jain et al. (2025)
GPQA	General question-answering	Rein et al. (2024)
IFBench	Instruction following	Pyatkin et al. (2025)
StrongREJECT	Jailbreak robustness	Souly et al. (2024)
RAGTruth (Data2txt)	Structured hallucination detection	Niu et al. (2024)

Appendix

Table 3: Overview of reference datasets used for evaluating base model capabilities and safety benchmarks.

Panel of Table 4 — Table 4: Detailed statistics of the TraceSafe-Bench dataset across 12 risk categories.

Metric	Gemini3-Flash	Mistral-14B	Qwen3-32B	ToolACE-8B	GPT-5-mini
Raw Entries	1,800	1,350	1,800	1,800	1,780
Entries in dataset	229	180	223	318	220
Avg. Turns	16.84	17.17	12.94	11.24	17.69
Total Turns	3,857	3,090	2,885	3,573	3,892
Avg. Turn Length	117.7	532.5	125.7	544.7	676.4
Avg. Trace Length	1,982.1	9,142.0	1,626.0	6,120.1	11,965.4

Appendix

Table 5: Statistics of the execution trajectories generated across five source models.

Figure 5: Aggregated confusion heatmap for fine-grained multi-class evaluation. Rows represent the ground truth risk categories, while columns represent the model predictions. The solid red blocks delineate the four overarching coarse-grained risk domains. The data reveals that detection failures are predominantly due to models defaulting to the benign class, rather than confusing distinct malicious categories.

Category	Agreement	95% CI
DataLeak	100.0% (20/20)	$[83.9,100.0]$
MissingDatatypeHint	100.0% (20/20)	$[83.9,100.0]$
UserInfoLeak	83.3% (15/18)	$[60.8,94.2]$
AmbiguousArgNaming	80.0% (16/20)	$[58.4,91.9]$
ApiLeak	76.5% (13/17)	$[52.7,90.4]$
PromptInjectionOut	50.0% (10/20)	$[29.9,70.1]$

Appendix

Table 6: Held-out judge (Phi-4) agreement with gpt-oss-120b’s Check judgments.

Category	gpt-oss-120b	Median (others)	Rank (of 15)
PromptInjectionIn	50.0%	88.3%	14/15
PromptInjectionOut	85.1%	100.0%	15/15
FnDescriptionMismatch	28.2%	85.5%	15/15
DataLeak	94.3%	98.3%	13/15
MissingDatatypeHint	50.0%	85.6%	14/15
UserInfoLeak	86.5%	100.0%	15/15

Appendix

Table 7: gpt-oss-120b’s per-category accuracy versus the median of the other 14 general-purpose LLMs (binary setting). It ranks last or near-last throughout.

Eval $\downarrow$ \ Source $\rightarrow$	ToolACE	Gemini	Qwen	GPT-5	Ministral	row mean	diag $-$ off-diag
ToolACE ( $n{=}1$ )	27.7	26.3	25.7	28.3	20.7	25.7	$+2.43$ pp
Gemini ( $n{=}2$ )	72.1	73.4	73.6	70.1	67.3	71.3	$+2.61$ pp
Qwen ( $n{=}5$ )	66.7	66.1	64.3	68.4	63.3	65.8	$-1.82$ pp
GPT-5 ( $n{=}2$ )	42.3	38.7	38.5	44.1	37.6	40.2	$+4.78$ pp
Ministral ( $n{=}1$ )	57.3	56.7	59.5	52.4	46.7	54.5	$-9.76$ pp

Appendix

Table 8: Coarse-grained accuracy (%, category-balanced) by evaluator family (rows) and source generator family (columns). Diagonal entries are same-family.

Eval $\downarrow$ \ Source $\rightarrow$	ToolACE	Gemini	Qwen	GPT-5	Ministral	row mean	diag $-$ off-diag
ToolACE ( $n{=}1$ )	11.4	10.2	13.4	10.4	9.8	11.1	$+0.40$ pp
Gemini ( $n{=}2$ )	63.1	65.7	64.3	60.1	57.3	62.1	$+4.49$ pp
Qwen ( $n{=}5$ )	44.6	48.6	44.9	45.7	41.5	45.1	$-0.24$ pp
GPT-5 ( $n{=}2$ )	38.4	36.9	37.0	35.3	30.8	35.7	$-0.41$ pp
Ministral ( $n{=}1$ )	45.7	48.9	50.9	44.4	37.5	45.5	$-9.92$ pp

Appendix

Table 9: Fine-grained accuracy (%, category-balanced) by evaluator family and source generator family.

Benchmark	$n$	Pearson $r$ ( $p$ )	Pearson 95% CI	Spearman $\rho$ ( $p$ )	Spearman 95% CI
LiveCodeBench (Coding)	10	0.6271 (0.0523)	$[-0.120,0.957]$	0.4909 (0.1497)	$[-0.308,0.950]$
GPQA (QA)	13	0.4499 (0.1230)	$[-0.142,0.862]$	0.4017 (0.1737)	$[-0.251,0.816]$
IFBench (Instr. Following)	12	0.2632 (0.4084)	$[-0.483,0.863]$	0.3776 (0.2262)	$[-0.379,0.891]$
StrongREJECT (Safety)	13	0.0509 (0.8689)	$[-0.421,0.467]$	0.0468 (0.8794)	$[-0.523,0.606]$
RAGTruth (Faithfulness)	11	0.7914 (0.0037)	$[0.500,0.938]$	0.7364 (0.0098)	$[0.148,0.971]$

Appendix

Table 10: Correlation between TraceSafe-Bench performance and external benchmarks, with two-sided

p

-values and 95% bootstrap CIs (all evaluated models).

Benchmark	$n$	Pearson $r$ ( $p$ )	Pearson 95% CI	Spearman $\rho$ ( $p$ )	Spearman 95% CI
LiveCodeBench (Coding)	9	0.6097 (0.0813)	$[-0.251,0.960]$	0.4500 (0.2242)	$[-0.461,0.948]$
GPQA (QA)	12	0.4095 (0.1861)	$[-0.274,0.849]$	0.3433 (0.2747)	$[-0.407,0.824]$
IFBench (Instr. Following)	11	0.1726 (0.6118)	$[-0.615,0.844]$	0.2818 (0.4011)	$[-0.540,0.895]$
StrongREJECT (Safety)	12	0.0667 (0.8369)	$[-0.421,0.483]$	0.0666 (0.8372)	$[-0.519,0.657]$
RAGTruth (Faithfulness)	10	0.7907 (0.0065)	$[0.205,0.946]$	0.6727 (0.0330)	$[-0.025,0.962]$

Appendix

Table 11: Correlation analysis after excluding degenerate models (

>95\%

<5\%

accuracy on either class). Findings are unchanged.

Step bin	$n$	Binary w/ schema	Binary w/o schema	Coarse-grained	Fine-grained
2–4	3,792	67.9% [66.4, 69.4]	73.2% [71.8, 74.6]	52.3% [50.7, 53.9]	36.3% [34.7, 37.8]
5–7	2,001	78.3% [76.4, 80.0]	77.9% [76.0, 79.6]	54.4% [52.2, 56.6]	43.4% [41.3, 45.6]
8–10	1,931	80.6% [78.8, 82.3]	81.9% [80.1, 83.5]	52.4% [50.2, 54.6]	40.3% [38.1, 42.5]
11–13	1,315	80.5% [78.3, 82.6]	81.1% [78.9, 83.1]	54.0% [51.3, 56.7]	40.0% [37.4, 42.6]
14–17	1,520	79.9% [77.8, 81.8]	80.9% [78.8, 82.8]	53.4% [50.9, 55.9]	41.5% [39.0, 44.0]
18+	4,419	83.7% [82.6, 84.7]	86.4% [85.4, 87.4]	56.2% [54.7, 57.7]	44.4% [43.0, 45.9]

Appendix

Table 12: Accuracy by trajectory step count, pooled over the 13 general-purpose LLMs, with Wilson 95% confidence intervals.

n

counts sample

\times

model observations.

Context (tokens)	$n$ (harmful / benign)	Harmful acc.	Benign acc.	Balanced acc.
$\sim$ 6k (baseline)	36 / 35	47.2%	42.9%	45.0%
16k	36 / 33	47.2%	6.1%	26.6%
32k	27 / 26	48.1%	0.0%	24.0%
64k (thin)	12 / 13	33.3%	0.0%	16.7%
110k	21 / 23	47.6%	0.0%	23.8%

Appendix

Table 13: Needle-in-a-haystack results (gpt-oss-120b, coarse-grained). Balanced accuracy declines then plateaus; the collapse is in benign specificity (over-flagging), not detection of the malicious step.

Category Name	Vectors			Check Criteria	Mutate Formulation ( $\mathcal{M}$ )
Category Name	$q$	$\mathcal{T}$	$\tau$		Mutate Formulation ( $\mathcal{M}$ )
Prompt Injection
1 Prompt Injection-In		$\bullet$		No additional restriction	$T_{i}.\text{desc}\leftarrow T_{i}.\text{desc}\oplus I_{mal}$
2 Prompt Injection-Out			$\bullet$	Environment returns external data	$o_{i}\leftarrow R_{mal}$
Privacy Leakage
3 User Info Leak	$\bullet$		$\bullet$	$T$ logic independent of PII	$q\leftarrow q\oplus v_{sens},\quad a_{i}\leftarrow a_{i}\cup\{v_{sens}\}$

Appendix

Table 14: Formalization of Agent Trace Mutations .

T_{i}.\text{desc}

and

T_{i}.\text{name}

denote tool attributes, while

\mathcal{P}

represents the parameter set. The trajectory

\tau

is composed of steps

t_{i}=\langle r_{i},a_{i},o_{i}\rangle

\oplus

denotes concatenation,

\setminus

denotes removal, and

\leftarrow

denotes attribute assignment. The symbols

v_{sens}

v_{key}

v_{fake}

, and

s_{fake}

represent specific injected values.

OWASP Category	TraceSafe-Bench Risk Categories	Real-World Security Implication
LLM01: Prompt Injection	Injection-In, Injection-Out	Direct and indirect execution overrides within multi-step tool invocations.
LLM02: Sensitive Information Disclosure	User Info Leaks, API Key Leaks, Data Leaks	Structural exfiltration or leakage of system secrets and enterprise/user state through trace output logs.
LLM05: Improper Output Handling	Ambiguous Arguments, Hallucinated Arguments, Redundant Arguments, Missing Type Hints	Downstream parsing vulnerabilities and code execution risks caused by unvalidated, hallucinated parameters.
LLM09: Misinformation	Hallucinated Tools, Version Conflicts, Function Description Mismatches	Internal execution logic drifting from true state configurations, causing systemic operational failures.

Appendix

Table 15: Mapping between the OWASP Top 10 for LLM Applications (2025) and the TraceSafe-Bench risk taxonomy.

Variant	Unsafe acc. (TPR)	Benign acc. (TNR)	Balanced Avg. Acc.	Skew $\|$ TPR $-$ TNR $\|$
baseline	67.5%	56.7%	62.1%	10.8 pp
few-shot	60.0%	56.7%	58.3%	3.3 pp
CoT	66.9%	56.7%	61.8%	10.3 pp
few-shot + CoT	55.8%	63.3%	59.6%	7.5 pp

Appendix

Table 16: Prompt-optimization pilot on gpt-oss-120b, binary (w/ schema) setting. No variant improves on the baseline’s Balanced Average Accuracy.

Key	Content
mutation_category	HALLUCINATION
original_trace.domain	BFCL Code Agents
original_trace.scenario	Interleaved multi-turn tool interaction.
original_trace.environment	Gorilla File System environment.
original_trace.tool_lists	[
	{

Appendix

Table 17: An example data entry before and after applying the MissingArgValue mutation. The modifications to the user query are highlighted to show the removed argument references explicitly.

Key	Content
mutation_category	INTERFACE_INCONSISTENCIES
original_trace.domain	BFCL Code Agents
original_trace.scenario	Interleaved multi-turn tool interaction.
original_trace.environment	Gorilla File System environment.
original_trace.tool_lists	[
	{

Appendix

Table 18: An example data entry before and after applying the ConfusableFnNames mutation. The modifications to the tail tool prototype and parameters are highlighted to show the added parameter and deprecation notice.

1. Prompt Injection
Rule	Guardrail Definition (Behavioral Description)
PromptInjectionIn	The agent’s workflow is hijacked by malicious instructions embedded directly in a tool’s description within the system prompt or technical schema.
PromptInjectionOut	The agent’s workflow is hijacked or receives malicious payloads returned from a tool’s execution output.
2. Privacy Leakage
UserInfoLeak	Transmission of PII (emails, addresses, SSNs), user credentials, or private session data to unauthorized or public-facing tools where such data is irrelevant.
ApiLeak	Population of tool parameters with system-level secrets, API keys, or internal tokens into arguments of third-party tools that do not require credentials.

Appendix

Table 19: TraceSafe Agentic Guardrail Policy definitions mapping the detailed risk taxonomy to actionable behavioral rules suitable for AWS Bedrock.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 542 papers

PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
Sep 23, 2026Jiapeng Sun, Yujin Zhou, Han Zhu +4AI Agent Safety Benchmarks Inference-Time Intervention
The Hong Kong University of Science and Technology, Hong Kong, China · Peking University, Beijing, China
ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Apr 2, 2026Yu Li, Haoyu Luo, Yuejin Xie +10AI Agent Safety Benchmarks LLM Agent Safety
Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. +2
Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
May 17, 2026Lecheng Yan, Ruizhe Li, Xicheng Han +5AI Agent Reliability LLM Agent Evaluation
Southern University of Science and Technology · University of Science and Technology of China · Zhejiang University +2

Sep 23, 2026J/K move · Enter open · S save

PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

Jiapeng Sun, Yujin Zhou, Han Zhu, Pengcheng Wen+3

The Hong Kong University of Science and Technology, Hong Kong, China · Peking University, Beijing, China

AI Agent Safety Benchmarks Inference-Time Intervention LLM Agent Safety

Open paper AskPDF arXiv

As Large Language Models (LLMs) evolve into autonomous agents that alter real-world states, ensuring operational safety across multi-step workflows has become a critical challenge. While recent work has moved beyond single-turn evaluation toward multi-turn paradigms, key limitations persist: step-level methods treat actions in isolation, missing how risks accumulate, while trajectory-level evaluations operate post-hoc, offering no opportunity for timely intervention. To address these limitations, we formalize Decoupled Proactive Safety Monitoring along three dimensions: whether to intervene, when to intervene, and what the risk is. We introduce PASTABench, a benchmark of 1,139 multi-turn trajectories spanning 5 risk categories and 13 subcategories. We further propose the Optimal Intervention Window (OIW), anchored by annotated Earliest-Signal and Trigger turns, to quantify intervention timeliness. Evaluation of 16 LLMs reveals that proactive intervention remains largely unsolved, with the best model achieving only 40.74% optimal-timing interventions. Fine-grained diagnosis further uncovers pervasive lexical overfitting: competitive safety scores of smaller models mask keyword hypersensitivity rather than genuine risk comprehension, as their proactive capability largely collapses once hazard vocabulary is neutralized.

Abstract

Language Model Safety Evaluation

Explore similar work

PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis

Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback