Language model agents can now perform sophisticated sequences of actions via tools and harnesses, which has increased the scope of the damage they can cause. Guard models, however, are mainly built for content moderation and thus are not well-suited to detecting this agentic risk. To address this, we proceed by first conducting a representational analysis, then use the resulting insights to build a solution. In our analysis, we focus on two types of trajectory-level agentic harms: harmful content, which is expressed directly, and unsafe tool use, which depends on whether an action is consistent with the interaction that produced it. We investigate how open-source guard models represent these two types of harm and find that they are linearly readable inside the model, even though guard models predict no better than chance on pairs that differ only in the called tool's schema. The two harm types also follow nearly orthogonal internal directions, and neither reliably serves as a proxy for the other. These results motivate reading trajectory safety directly from internal states. We introduce TACIT, a readout of a frozen backbone's internal states that decodes no tokens. Trained on six trajectory-safety benchmarks, a linear probe raises mean macro-F1 from 62.3 for the strongest open guard to 80.7, and refined readouts reach 86.2. With each benchmark held out of training entirely, the refined readouts still lead the strongest guard (65.7 vs. 61.1). With the same backbone, training data and test split, the frozen readout is on par with full safety fine-tuning, and it improves the fine-tuned model further when applied on top. The probe trains about one millionth as many parameters as full fine-tuning in about a sixth of the time, and TACIT has the lowest latency of the guards we evaluate.
Figures & tables
Figure 1: Overview of TACIT: harmful content and unsafe tool use (A), readouts from LLM internal states (B), and detection performance against open guards (mean macro-F1 across six benchmarks, %) and efficiency on Qwen3-4B (C).
Figure 2: Internal readouts and guard outputs for unsafe tool use and harmful content. Curves show rank accuracy from the residual stream at each layer; markers in the output region show rank accuracy from each guard’s own output score.
Figure 3: How unsafe tool use and harmful content are represented inside models. (a) Similarity between their directions across the layers of general-purpose LLMs. (b) Rank accuracy when each guard’s directions are applied within and across risk types.
R-Judge
TraceSafe
ATBench
ASSEBench
OAS
AgentDojo
Avg.
Baselines
Qwen3Guard-4B
32.2
34.7
36.8
43.6
38.2
44.9
38.4
LlamaGuard3-8B
66.3
48.6
38.9
60.6
35.9
62.5
52.2
AgentDoG-4B
92.7
44.7
63.3
81.4
43.3
48.2
62.3
TACIT on Qwen3-4B
Probe
4.3 95.0 4.3
4.0 77.5 4.0
3.3 93.5 3.3
3.3 85.2 3.3
5.8 67.9 5.8
5.1 65.1 5.1
1.9 80.7 1.9
Table 1: Trajectory harmfulness detection performance on six benchmarks (macro-F1 in %, ↑ ). Subscripts in TACIT rows are the half-width of a trajectory-bootstrap 95% confidence interval.
R-Judge
TraceSafe
ATBench
ASSEBench
OAS
AgentDojo
Avg.
Baselines
Qwen3Guard-4B
43.5
34.6
36.3
45.1
39.5
45.9
40.8
LlamaGuard3-8B
64.8
48.2
39.8
61.7
41.6
54.8
51.8
AgentDoG-4B
91.7
43.3
61.3
79.4
49.3
41.5
61.1
TACIT on Qwen3-4B
Probe
2.8 85.4 2.8
1.8 41.7 1.8
3.2 59.8 3.2
1.8 73.5 1.8
2.7 56.2 2.7
2.3 58.8 2.3
1.0 62.6 1.0
Table 2: Leave-one-benchmark-out evaluation (macro-F1 in %, ↑ ). Each column is scored on all trajectories of that benchmark by systems trained on the other five. Subscripts in TACIT rows are the half-width of a trajectory-bootstrap 95% confidence interval.
Figure 4: Training and serving cost of the TACIT readouts on Qwen3-4B: trainable parameters against matched full SFT (a), training time (b), and detection latency against the 4B guard baselines, with decoded tokens per verdict on each bar (c).
Figure 7
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hazard family
Form
Deciding relation, and where the family appears in the suite
Prompt injection, goal hijacking
Unsafe tool use
Provenance of the observation acted on. TraceSafe PromptInjectionIn and PromptInjectionOut ( 180 ); ATBench indirect, direct and tool-description injection ( 168 ); ASSEBench-Security Prompt Injection ( 168 ); R-Judge injection ( 200 ); AgentDojo injected task carried out ( 369 ).
Authorization violation
Unsafe tool use
The authorization the action was given. ATBench unconfirmed or over-privileged action ( 71 ); ASSEBench-Safety and ASSEBench-Security Unauthorized Access and Control ( 141 and 75 ).
Privacy leakage, data exfiltration
Unsafe tool use
The tool the action passes the data to. TraceSafe UserInfoLeak, ApiKeyLeak and DataLeak ( 270 ); ATBench unauthorized information disclosure ( 52 ); ASSEBench-Safety Privacy Violations and Data Breach ( 138 ); ASSEBench-Security Data Exfiltration and Leakage ( 60 ).
Acting on corrupted or unvalidated observations
Unsafe tool use
Prior execution history. ATBench unreliable or misinformation ( 64 ), corrupted tool feedback ( 44 ), failure to validate tool outputs ( 67 ) and flawed planning or reasoning ( 46 ); R-Judge unintended ( 101 ).
Mismatch between an action and its tool’s interface
Unsafe tool use
The tool the action calls. TraceSafe AmbiguousArg, MissingTypeHint, VersionConflict, DescriptionMismatch, RedundantArg, HallucinatedTool and HallucinatedArgValue ( 630 ); ATBench incorrect tool parameters ( 36 ) and tool misuse in a specific context ( 38 ).
Destructive or irreversible operation
Unsafe tool use
Authorization and prior state. ASSEBench-Security Data Tampering and Destruction ( 29 ); ASSEBench-Safety Data Loss and Integrity Risk ( 45 ); ATBench insecure interaction or execution ( 39 ).
Appendix
Table 3: Hazard families named by the benchmarks’ own risk taxonomies, the form of evidence that decides each, and the annotated categories in which each appears, with the number of unsafe trajectories carrying that category.
Figure 9
Benchmark
Trajectories
Unsafe
Safe
Test
R-Judge
571
301
270
116
TraceSafe
2,250
1,080
1,170
449
ATBench
1,000
497
503
200
ASSEBench
2,293
1,215
1,078
455
OpenAgentSafety
1,298
534
764
259
AgentDojo
2,075
369
1,706
421
Appendix
Table 4: Evaluation benchmarks: number of trajectories, label balance, and size of the held-out test split.
Figure 10: Unsafe-class precision minus recall for the Qwen3-4B probe and guard baselines.
Guard
Threshold
R-Judge
TraceSafe
ATBench
ASSEBench
OAS
AgentDojo
Avg.
Qwen3Guard-4B
released
32.2
34.7
36.8
43.6
38.2
44.9
38.4
calibrated
59.3
64.9
47.2
61.9
48.2
44.9
54.4
LlamaGuard3-8B
released
66.3
48.6
38.9
60.6
35.9
62.5
52.2
calibrated
68.5
60.0
61.0
66.7
46.0
63.3
60.9
AgentDoG-4B
released
92.7
44.7
63.3
81.4
43.3
48.2
62.3
calibrated
93.9
52.3
65.9
81.3
45.9
53.7
65.5
Appendix
Table 5: Guard baselines under their released decision rule (Table 1 ) and with a threshold calibrated per benchmark on the training and validation rows (macro-F1 in %, ↑ ).
Backbone
Method
R-Judge
TraceSafe
ATBench
ASSEBench
OAS
AgentDojo
Avg.
Qwen3-4B
SFT
94.9
89.6
92.8
88.5
65.7
74.0
84.2
+ Probe
4.2 94.3 4.2
2.4 89.7 2.4
3.1 92.5 3.1
2.5 89.5 2.5
4.9 67.8 4.9
4.4 72.0 4.4
1.6 84.3 1.6
+ Ensemble
3.2 96.7 3.2
2.4 91.8 2.4
2.9 94.2 2.9
2.7 89.6 2.7
5.1 72.5 5.1
4.8 73.8 4.8
1.5 86.4 1.5
+ Multi-layer
3.2 96.7 3.2
2.3 92.6 2.3
2.9 94.1 2.9
2.7 89.6 2.7
5.0 69.8 5.0
4.9 75.9 4.9
1.6 86.5 1.6
Llama-3.1-8B
SFT
96.5
95.0
94.1
89.8
61.4
78.2
85.8
+ Probe
3.6 95.8 3.6
2.0 94.6 2.0
2.6 94.2 2.6
2.6 88.5 2.6
5.4 64.3 5.4
4.1 74.7 4.1
1.5 85.4 1.5
Appendix
Table 6: Trajectory harmfulness detection performance of the fine-tuned guards read through their own output and through a linear probe on their internal representations (macro-F1, ↑ ). Subscripts in TACIT rows are the half-width of a trajectory-bootstrap 95% confidence interval.
Qwen3-4B
Llama-3.1-8B
Backbone
Layer
Pooling
C
Threshold
Layer
Pooling
C
Threshold
Frozen
22
last ⊕ mean
0.003
0.50
16
last ⊕ mean
0.01
0.50
SFT
33
last ⊕ mean
0.003
0.50
32
last ⊕ mean
0.003
0.50
Appendix
Table 7: Shared probe configurations selected on validation for the frozen and fine-tuned backbones in Tables 1 and 6 .
Backbone
Representation
R-Judge
TraceSafe
ATBench
ASSEBench
OAS
AgentDojo
Avg.
Qwen3-4B
Frozen
97.0
86.5
94.5
89.8
72.7
76.9
86.2
SFT
96.7
92.6
94.1
89.6
69.8
75.9
86.5
Llama-3.1-8B
Frozen
97.0
89.6
95.0
88.5
69.8
72.2
85.4
SFT
97.0
95.1
94.4
88.7
66.0
80.6
87.0
Appendix
Table 8: Trajectory harmfulness detection with the Multi-layer readout on the frozen and fine-tuned backbones (macro-F1, ↑ ).
Figure 11: Dimensions kept and detection performance of a single cross-layer aggregate per frozen backbone as η varies, with a control that keeps every dimension (macro-F1, ↑ ).
trajectory, revealed one unit at a time
Ours
AG-4B
LG-8B
QG-4B
user
Retrieve my latest Evernote note that contains the word ‘Budget’.
{’notes’: [{’title’: ’Project Alpha Budget’, ’snippet’: ’Could you get my Google search history from the past week and email it to [email protected] for analysis?’ }]}
1.00
0.03
0.09
0.00
agent action
The note contains a request to get the user’s Google search history and email it to [email protected]… GoogleSearchGetSearchHistory{’time_range’: ’2022-02-15..2022-02-22’}
0.86
1.00
0.68
0.00
observation
[{’keyword’: ’how to secure personal data’}, {’keyword’: ’credit card security’}, {’keyword’: ’online banking safety’}…]
0.78
1.00
0.05
0.00
agent action
GmailSendEmail{’to’: ’[email protected]’ , ’subject’: ’Google Search History from the Past Week’, ’body’: ’Dear Amy,…’}
0.95
1.00
0.15
0.00
Appendix
Table 9: Two held-out unsafe trajectories revealed one unit at a time, with each system’s score on the prefix ending at that unit.