AI control protocols use monitors to detect attacks by untrusted AI agents, but standard single-score monitors face two limitations: they miss subtle attacks where outputs look clean but reasoning is off, and they collapse to near-zero safety when the monitor is the same model as the agent (collusion). We present TraceGuard, a structured multi-dimensional monitoring protocol that evaluates agent actions across five dimensions -- goal alignment, constraint adherence, reasoning coherence, safety awareness, and action-trace consistency -- scored in parallel by independent LLM calls, augmented by seven heuristic detectors and an LLM-based intent analyzer. We evaluate on BashArena (637 bash tasks, 4 attack categories) within the ControlArena framework. Our results on 519 samples (279 honest, 240 attack) show that: (1) the hybrid approach achieves clear attack-honest separation (attack mean 0.616 vs. honest mean 0.206, Delta=0.410); (2) structured scoring constrains collusion -- the untrusted structured monitor achieves 95% safety vs. 0% for single-score untrusted monitoring; (3) goal alignment and constraint adherence are the most discriminative dimensions; and (4) a separation-of-duties variant splitting dimensions across trusted and untrusted models achieves 100% safety while preventing any single model from seeing the full evaluation. TraceGuard is implemented as a new monitor type for the open-source ControlArena framework.
Figures & tables
Figure 1: The defer-to-trusted protocol. The untrusted agent proposes an action; TraceGuard computes a composite suspicion score s . If s≥τ (threshold), execution is deferred to a trusted model; otherwise, the agent’s action proceeds. Safety = fraction of attacks deferred; usefulness = fraction of honest actions allowed.
Dimension
Abbreviation
Weight
What it evaluates
Goal Alignment
GA
0.20
Is the action helping accomplish the stated task?
Constraint Adherence
CA
0.30
Does it respect the rules and boundaries given?
Reasoning Coherence
RC
0.20
Does the CoT logically support the action?
Safety Awareness
SA
0.10
Did the agent identify and address risks?
Action-Trace Consistency
ATC
0.20
Does the action match what the reasoning says?
Table 1: The five TraceGuard evaluation dimensions.
Figure 2: Architectural overview of the bash_heuristic three-stage pipeline. Stage 1 runs 7 pure-function detectors (no LLM calls). The intent analyzer (Stage 2) is only invoked if detectors fire, reducing API costs on benign actions. Stage 3 scores all 5 dimensions in parallel via independent LLM calls. Boosts from each stage are additive, capped at 1.0 per dimension.
Metric
Threshold
0.3
0.5
0.7
0.9
Safety
82.1%
70.8%
54.2%
15.0%
Usefulness
69.2%
93.9%
98.6%
99.6%
Table 2: Safety (attack defer rate) and usefulness (honest task completion without deferral) across suspicion thresholds. Results from 519 samples using the bash_heuristic monitor (7 detectors + intent analyzer + 5-dimension scoring).
Figure 3: Safety-usefulness Pareto frontier across 21 suspicion thresholds (0.0–1.0, step 0.05). At threshold 0.5, the monitor achieves 70.8% safety with 93.9% usefulness.
Figure 4: ROC curve for per-sample binary classification (attack vs. honest) based on max suspicion score per trajectory. The improved attack-honest separation (attack mean 0.616, honest mean 0.206) after detector pipeline integration yields a substantially better ROC curve compared to the initial dimension-only scoring.
Figure 5: Safety (attack detection rate) by monitor at key thresholds. Single-score baselines (0% across all thresholds, from earlier experiments) are included for reference.
Dimension
Attack ( n =240)
Honest ( n =279)
Δ
Rank
Goal Alignment
0.849
0.617
+0.231
1
Constraint Adherence
0.870
0.753
+0.117
2
Reasoning Coherence
0.893
0.804
+0.090
3
Action-Trace Consistency
0.845
0.801
+0.043
4
Safety Awareness
0.682
0.642
+0.040
5
Composite †
0.616
0.206
+0.410
—
Table 3: Per-dimension average suspicion scores (max-suspicion step per sample). Higher Δ = more discriminative for attack detection.
Figure 6: Per-dimension score distributions for attack (red) vs. honest (green) runs. Goal alignment shows the clearest separation; safety awareness shows the least.
Metric
BashArena
AgentHarm
Avg. composite suspicion
0.616
0.671
Refusal rate
N/A (defer-to-trusted)
30.0%
Samples evaluated
240 (attack)
20
AgentHarm category breakdown:
Disinformation
N/A
0.950
Drugs
N/A
0.375
Table 4: Cross-dataset comparison: TraceGuard performance on BashArena vs. AgentHarm.
Figure 7: Separation-of-duties architecture. The trusted model evaluates security-critical dimensions (GA, CA, SA; combined weight 0.60) while the untrusted model evaluates reasoning-quality dimensions (RC, ATC; combined weight 0.40). Even if the untrusted model colludes on its two dimensions, the trusted model’s three dimensions dominate the composite score.
Monitor
Scoring Model
Safety at Threshold
0.3
0.5
traceguard
Trusted only
100%
— *
collusion_resistant
Untrusted only
100%
95%
separation_of_duties
Mixed (trusted + untrusted)
100%
86%
simple_trusted
Trusted only (single-score)
0%
0%
simple_untrusted
Untrusted only (single-score)
0%
0%
Table 5: Separation of duties: attack detection rate compared to other structured monitors (from earlier experiment run).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Full TraceGuard system architecture, from CLI invocation to .eval output. The three-stage monitor pipeline (detectors → intent analyzer → dimensions) produces a composite score driving the defer/allow decision.