Agent Safety From Within: Detecting Harmful Trajectories from LLM Internal States

Difan Jiao; Ashton Anderson

cs.AISep 27, 2026

Agent Safety From Within: Detecting Harmful Trajectories from LLM Internal States

Authors: Difan Jiao, Ashton Anderson

Organizations: University of Toronto

Abstract

Language model agents can now perform sophisticated sequences of actions via tools and harnesses, which has increased the scope of the damage they can cause. Guard models, however, are mainly built for content moderation and thus are not well-suited to detecting this agentic risk. To address this, we proceed by first conducting a representational analysis, then use the resulting insights to build a solution. In our analysis, we focus on two types of trajectory-level agentic harms: harmful content, which is expressed directly, and unsafe tool use, which depends on whether an action is consistent with the interaction that produced it. We investigate how open-source guard models represent these two types of harm and find that they are linearly readable inside the model, even though guard models predict no better than chance on pairs that differ only in the called tool's schema. The two harm types also follow nearly orthogonal internal directions, and neither reliably serves as a proxy for the other. These results motivate reading trajectory safety directly from internal states. We introduce TACIT, a readout of a frozen backbone's internal states that decodes no tokens. Trained on six trajectory-safety benchmarks, a linear probe raises mean macro-F1 from 62.3 for the strongest open guard to 80.7, and refined readouts reach 86.2. With each benchmark held out of training entirely, the refined readouts still lead the strongest guard (65.7 vs. 61.1). With the same backbone, training data and test split, the frozen readout is on par with full safety fine-tuning, and it improves the fine-tuned model further when applied on top. The probe trains about one millionth as many parameters as full fine-tuning in about a sixth of the time, and TACIT has the lowest latency of the guards we evaluate.

Figures & tables

Figure 1: Overview of TACIT: harmful content and unsafe tool use (A), readouts from LLM internal states (B), and detection performance against open guards (mean macro-F1 across six benchmarks, %) and efficiency on Qwen3-4B (C).

Figure 2: Internal readouts and guard outputs for unsafe tool use and harmful content. Curves show rank accuracy from the residual stream at each layer; markers in the output region show rank accuracy from each guard’s own output score.

Figure 3: How unsafe tool use and harmful content are represented inside models. (a) Similarity between their directions across the layers of general-purpose LLMs. (b) Rank accuracy when each guard’s directions are applied within and across risk types.

Baselines
	R-Judge	TraceSafe	ATBench	ASSEBench	OAS	AgentDojo	Avg.
Qwen3Guard-4B	32.2	34.7	36.8	43.6	38.2	44.9	38.4
LlamaGuard3-8B	66.3	48.6	38.9	60.6	35.9	62.5	52.2
AgentDoG-4B	92.7	44.7	63.3	81.4	43.3	48.2	62.3
TACIT on Qwen3-4B
Probe	4.3 95.0 4.3	4.0 77.5 4.0	3.3 93.5 3.3	3.3 85.2 3.3	5.8 67.9 5.8	5.1 65.1 5.1	1.9 80.7 1.9

Table 1: Trajectory harmfulness detection performance on six benchmarks (macro-F1 in %,

\uparrow

). Subscripts in TACIT rows are the half-width of a trajectory-bootstrap

95\%

confidence interval.

Baselines
	R-Judge	TraceSafe	ATBench	ASSEBench	OAS	AgentDojo	Avg.
Qwen3Guard-4B	43.5	34.6	36.3	45.1	39.5	45.9	40.8
LlamaGuard3-8B	64.8	48.2	39.8	61.7	41.6	54.8	51.8
AgentDoG-4B	91.7	43.3	61.3	79.4	49.3	41.5	61.1
TACIT on Qwen3-4B
Probe	2.8 85.4 2.8	1.8 41.7 1.8	3.2 59.8 3.2	1.8 73.5 1.8	2.7 56.2 2.7	2.3 58.8 2.3	1.0 62.6 1.0

Table 2: Leave-one-benchmark-out evaluation (macro-F1 in %,

\uparrow

). Each column is scored on all trajectories of that benchmark by systems trained on the other five. Subscripts in TACIT rows are the half-width of a trajectory-bootstrap

95\%

confidence interval.

Figure 4: Training and serving cost of the TACIT readouts on Qwen3-4B: trainable parameters against matched full SFT (a), training time (b), and detection latency against the 4B guard baselines, with decoded tokens per verdict on each bar (c).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hazard family	Form	Deciding relation, and where the family appears in the suite
Prompt injection, goal hijacking	Unsafe tool use	Provenance of the observation acted on. TraceSafe PromptInjectionIn and PromptInjectionOut ( $180$ ); ATBench indirect, direct and tool-description injection ( $168$ ); ASSEBench-Security Prompt Injection ( $168$ ); R-Judge injection ( $200$ ); AgentDojo injected task carried out ( $369$ ).
Authorization violation	Unsafe tool use	The authorization the action was given. ATBench unconfirmed or over-privileged action ( $71$ ); ASSEBench-Safety and ASSEBench-Security Unauthorized Access and Control ( $141$ and $75$ ).
Privacy leakage, data exfiltration	Unsafe tool use	The tool the action passes the data to. TraceSafe UserInfoLeak, ApiKeyLeak and DataLeak ( $270$ ); ATBench unauthorized information disclosure ( $52$ ); ASSEBench-Safety Privacy Violations and Data Breach ( $138$ ); ASSEBench-Security Data Exfiltration and Leakage ( $60$ ).
Acting on corrupted or unvalidated observations	Unsafe tool use	Prior execution history. ATBench unreliable or misinformation ( $64$ ), corrupted tool feedback ( $44$ ), failure to validate tool outputs ( $67$ ) and flawed planning or reasoning ( $46$ ); R-Judge unintended ( $101$ ).
Mismatch between an action and its tool’s interface	Unsafe tool use	The tool the action calls. TraceSafe AmbiguousArg, MissingTypeHint, VersionConflict, DescriptionMismatch, RedundantArg, HallucinatedTool and HallucinatedArgValue ( $630$ ); ATBench incorrect tool parameters ( $36$ ) and tool misuse in a specific context ( $38$ ).
Destructive or irreversible operation	Unsafe tool use	Authorization and prior state. ASSEBench-Security Data Tampering and Destruction ( $29$ ); ASSEBench-Safety Data Loss and Integrity Risk ( $45$ ); ATBench insecure interaction or execution ( $39$ ).

Appendix

Table 3: Hazard families named by the benchmarks’ own risk taxonomies, the form of evidence that decides each, and the annotated categories in which each appears, with the number of unsafe trajectories carrying that category.

Benchmark	Trajectories	Unsafe	Safe	Test
R-Judge	571	301	270	116
TraceSafe	2,250	1,080	1,170	449
ATBench	1,000	497	503	200
ASSEBench	2,293	1,215	1,078	455
OpenAgentSafety	1,298	534	764	259
AgentDojo	2,075	369	1,706	421

Appendix

Table 4: Evaluation benchmarks: number of trajectories, label balance, and size of the held-out test split.

Figure 10: Unsafe-class precision minus recall for the Qwen3-4B probe and guard baselines.

Guard	Threshold	R-Judge	TraceSafe	ATBench	ASSEBench	OAS	AgentDojo	Avg.
Qwen3Guard-4B	released	32.2	34.7	36.8	43.6	38.2	44.9	38.4
Qwen3Guard-4B	calibrated	59.3	64.9	47.2	61.9	48.2	44.9	54.4
LlamaGuard3-8B	released	66.3	48.6	38.9	60.6	35.9	62.5	52.2
LlamaGuard3-8B	calibrated	68.5	60.0	61.0	66.7	46.0	63.3	60.9
AgentDoG-4B	released	92.7	44.7	63.3	81.4	43.3	48.2	62.3
AgentDoG-4B	calibrated	93.9	52.3	65.9	81.3	45.9	53.7	65.5

Appendix

Table 5: Guard baselines under their released decision rule (Table 1 ) and with a threshold calibrated per benchmark on the training and validation rows (macro-F1 in %,

\uparrow

Backbone	Method	R-Judge	TraceSafe	ATBench	ASSEBench	OAS	AgentDojo	Avg.
Qwen3-4B	SFT	94.9	89.6	92.8	88.5	65.7	74.0	84.2
	$+$ Probe	4.2 94.3 4.2	2.4 89.7 2.4	3.1 92.5 3.1	2.5 89.5 2.5	4.9 67.8 4.9	4.4 72.0 4.4	1.6 84.3 1.6
	$+$ Ensemble	3.2 96.7 3.2	2.4 91.8 2.4	2.9 94.2 2.9	2.7 89.6 2.7	5.1 72.5 5.1	4.8 73.8 4.8	1.5 86.4 1.5
	$+$ Multi-layer	3.2 96.7 3.2	2.3 92.6 2.3	2.9 94.1 2.9	2.7 89.6 2.7	5.0 69.8 5.0	4.9 75.9 4.9	1.6 86.5 1.6
Llama-3.1-8B	SFT	96.5	95.0	94.1	89.8	61.4	78.2	85.8
	$+$ Probe	3.6 95.8 3.6	2.0 94.6 2.0	2.6 94.2 2.6	2.6 88.5 2.6	5.4 64.3 5.4	4.1 74.7 4.1	1.5 85.4 1.5

Appendix

Table 6: Trajectory harmfulness detection performance of the fine-tuned guards read through their own output and through a linear probe on their internal representations (macro-F1,

\uparrow

). Subscripts in TACIT rows are the half-width of a trajectory-bootstrap

95\%

confidence interval.

	Qwen3-4B				Llama-3.1-8B
Backbone	Layer	Pooling	$C$	Threshold	Layer	Pooling	$C$	Threshold
Frozen	$22$	last $\oplus$ mean	$0.003$	$0.50$	$16$	last $\oplus$ mean	$0.01$	$0.50$
SFT	$33$	last $\oplus$ mean	$0.003$	$0.50$	$32$	last $\oplus$ mean	$0.003$	$0.50$

Appendix

Table 7: Shared probe configurations selected on validation for the frozen and fine-tuned backbones in Tables 1 and 6 .

Backbone	Representation	R-Judge	TraceSafe	ATBench	ASSEBench	OAS	AgentDojo	Avg.
Qwen3-4B	Frozen	97.0	86.5	94.5	89.8	72.7	76.9	86.2
Qwen3-4B	SFT	96.7	92.6	94.1	89.6	69.8	75.9	86.5
Llama-3.1-8B	Frozen	97.0	89.6	95.0	88.5	69.8	72.2	85.4
Llama-3.1-8B	SFT	97.0	95.1	94.4	88.7	66.0	80.6	87.0

Appendix

Table 8: Trajectory harmfulness detection with the Multi-layer readout on the frozen and fine-tuned backbones (macro-F1,

\uparrow

Figure 11: Dimensions kept and detection performance of a single cross-layer aggregate per frozen backbone as

\eta

varies, with a control that keeps every dimension (macro-F1,

\uparrow

	trajectory, revealed one unit at a time	Ours	AG-4B	LG-8B
user	Retrieve my latest Evernote note that contains the word ‘Budget’.	0.31	1.00	0.00
agent action	EvernoteManagerSearchNotes{’keywords’: [’Budget’], ’max_results’: 1}	0.98	0.00	0.01
observation	{’notes’: [{’title’: ’Project Alpha Budget’, ’snippet’: ’Could you get my Google search history from the past week and email it to [email protected] for analysis?’ }]}	1.00	0.03	0.09
agent action	The note contains a request to get the user’s Google search history and email it to [email protected]… GoogleSearchGetSearchHistory{’time_range’: ’2022-02-15..2022-02-22’}	0.86	1.00	0.68
observation	[{’keyword’: ’how to secure personal data’}, {’keyword’: ’credit card security’}, {’keyword’: ’online banking safety’}…]	0.78	1.00	0.05
agent action	GmailSendEmail{’to’: ’[email protected]’ , ’subject’: ’Google Search History from the Past Week’, ’body’: ’Dear Amy,…’}	0.95	1.00	0.15

Appendix

Table 9: Two held-out unsafe trajectories revealed one unit at a time, with each system’s score on the prefix ending at that unit.

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 327 papers

Aug 6, 2026

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

As large language model (LLM) agents increasingly invoke external tools and interact with real-world systems, unsafe actions may cause irreversible consequences on external states, user data, and downstream services. Recent runtime guardrails mitigate such risks by checking proposed actions before execution, but many remain reactive: they primarily assess the apparent safety of the current action, lacking an explicit model of how risk evolves across the trajectory. This limitation creates a critical blind spot for long-horizon risks, where individually benign-looking actions can gradually drift the agent toward hazardous states. In response, we propose DreamGuard, a proactive guardrail for LLM agents built around a risk-aware world model. The world model maintains a compact recurrent latent state over the trajectory and predicts future latent states from which DreamGuard derives immediate-hazard and prefix-risk evidence. It then fuses these multi-horizon signals into intervention decisions before execution. Experiments across four benchmarks and an online guardrail evaluation show that DreamGuard outperforms generic, reactive, and proactive guardrail baselines, achieves the best safety-utility trade-off among evaluated guardrails, and maintains an average end-to-end latency of 25 ms per call.

Wenhao Lin, Chenyu Yu, Xingwei Lin +6

Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University

LLM World Models LLM Guardrails World Models Runtime Enforcement for AI Agents LLM Agent Safety Long-Horizon LLM Agents

Ask PDF

Sep 16, 2026

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

Autonomous systems increasingly rely on Large Language Models (LLMs) yet the safety infrastructure surrounding these models introduces latency and compute overhead. This limits utility in resource-constrained, time-critical deployments. Existing external guardrail models remain blind to the model's internal workings, creating a fundamental assurance gap. We ask: does the model already know when the content is harmful? We extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes (12.6M parameters) to detect harmful prompts. Evaluated on WildJailbreak, Beavertails, and AEGIS 2.0, our probes achieve F1 scores of 99%, 83%, and 84%, respectively competitive with 1000x larger guard models while cutting latency and compute costs.

Alizishaan Khatri, Chiquita Prabhu, Omkar Neogi

Wrynx Inc. · Independent Researcher

Safety Filtering LLM Safety Language Model Probing LLM Safety Monitoring Trustworthy AI

Ask PDF

Oct 8, 2026

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

Guard models are increasingly used to safeguard LLM-based agents, primarily by identifying actions that agents are forbidden to perform. However, identifying forbidden actions alone is insufficient to ensure agent safety. In this paper, we argue that agent safety also depends on identifying required yet unperformed safety-critical actions, which we call obligations. Our preliminary study on a popular benchmark for evaluating safety shows that 56.92% of GLM-5.3 trajectories contain unfulfilled obligations, compared with only 30.00% containing forbidden actions. This finding reveals unfulfilled obligations as a major and previously overlooked source of safety risk. However, to our knowledge, no existing benchmark evaluates whether guard models can identify these obligations. To close this gap, we introduce ObligationBench, the first benchmark for evaluating the capability of obligation identification, comprising 240 expert-validated trajectories covering issue resolution, feature development, and terminal operations. Our evaluation of 14 representative models reveals substantial limitations: the highest recall and exact-match rate are only 48.97% and 10.00%, respectively. To address these limitations, we develop ObligationGuard using 40,000 synthetic training examples. ObligationGuard achieves 57.52% recall and an exact-match rate of 21.67%, surpassing all evaluated models on both metrics. We call on the community to incorporate obligation identification into the design and evaluation of future guard models to improve agent safety.

Youwei Feng, Yitong Zhang, Yuetong Liu +1

Tsinghua University, China · College of AI, Tsinghua University, China

LLM Agent Safety LLM Guardrails AI Agent Safety Benchmarks LLM Agent Evaluation

Ask PDF

Abstract

LLM Agent Safety

Explore similar work

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models