What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer
Organizations: the University of Salamanca, 37007 Salamanca, Spain · the AIR Institute, Salamanca, Spain · the Department of Informatics, University of Oslo, 0373 Oslo, Norway
Abstract
Detectors can flag hijacked large language model (LLM) agents from the traces of the tools they call. In the eleven evaluations we audited, attacks or anomalies make up 16% to 86% of the test samples, no evaluation varies that share, and six report no test on data from a second benchmark or source. We ask what such evaluations hide. On the AgentDrift and AgentDojo benchmarks we test trained detectors that read tool identities, behavior, call sequences, step embeddings or text, and a released LLM action monitor, AlignmentCheck, with thresholds fixed on training data or set by the monitor's released decision rule. First, at a 2% attack rate a detector with an area under the receiver operating characteristic curve (AUROC) of 0.879 reaches a precision of 0.076, and the mix of negatives alone moves that precision between 0.039 and 0.117. Second, as tool coverage, the share of test tool calls whose tool was seen in training, falls, detector rankings cross near 50% coverage and reverse below it, because unseen tools map to inputs that training never reached. Dropout on tool inputs largely removes the reversal. Third, no detector trained on AgentDrift ranks AgentDojo traces better than chance (AUROC 0.47 to 0.52), including one that reaches 0.996 within AgentDojo. The monitor, trained on neither, reaches 0.930, but with its open backbone its released decision rule would still mostly raise false alerts at a 2% attack rate (precision 0.096). We turn these findings into a checklist for evaluating trace detectors.
Figures & tables
| Work | Detector | Test attack share | Metrics | Shift tested | Cross-benchmark or within-target test |
| TraceAegis [ 4 ] | Hierarchical rules and LLM-summarized behavioral constraints | 50% | P, R, F1, accuracy | None | No |
| Praetor [ 5 ] | Automaton with embedding-based parameter guards | Classes scored separately | Attack success with intervals, benign task failure, latency | 20% of tools replaced (benign traffic only) | Partial: benign traces from ToolBench, attacks from ASB |
| Agent-Sentry [ 6 ] | XGBoost or rules, allowlist, then LLM judge | 38.7% (Native-Trace 81.2%) | Utility, block rate, FPR, FNR, F1 | Grouped by user task, prompt-style shift | Partial: Native-Trace with the same agents and tools, within-target control |
| AgentDrift [ 22 ] | Surface-feature LR (benchmark check, not a detector) | 44.2% | P, R, F1, AUROC, AUPRC | None (stratified split) | No |
| DriftNet [ 7 ] | Frozen step embeddings and a dual-head transformer | 45.0% | P, R, F1, step F1, EM, IoU | Task-disjoint split | No |
| Trajectory Guard [ 8 ] | Siamese recurrent autoencoder | 45.5% (external sets anomaly-only) | P, R, F1 (recall only on external sets) | External anomaly sources | Partial: external sets without benign traces |
| Setting | Training data | Test pool ( ) | Shift introduced | |
| R1 | Stratified training split (8,775) | Stratified test split, 825/831 | 1.000 | None (reference) |
| R1-wd | Pooled corpus with no shared world, split seeds 0 to 4 (8,711 to 8,764) | 786 to 811 / 863 to 924 | 1.000 | World identity |
| R2 | Task-disjoint training split (9,081) | 35 unseen tasks, 729/775 | 1.000 | Task |
| R3 | Four domains (8,446, or 8,840 with banking held out) | Held-out domain, 935/1,020 (banking 935/626) | 0.000 | Domain and all its tools |
| R4 | AgentDrift training and test splits pooled (10,656) | AgentDojo P0, 392/2,294 (P0F adds 4,987 unsuccessful attacks) | 0.075 | Benchmark, ecosystem, generator, labels |
| Within AgentDojo, UT | Four fifths of P0, 5 repetitions | Held-out folds of P0 | 0.976 | Unseen user tasks |
| wAUPRC at prevalence | Precision | Alerts per 1,000 | ||||||
| Detector | AUROC | 1% | 2% | 10% | at 2% | Recall | FPR | at 2% |
| LR-B (behavioral) | 294 | |||||||
| LR-C (combined) | 203 | |||||||
| GRU (sequence) | 58 | |||||||
| TF-IDF (content) | 1.000 ( 0.771) | 0.000 ( 0.004) | 15 | |||||
| MON-L at | 185 | |||||||
| Medical | Banking | Web | Coding | ||
| Same domain inside R1 (R1dm) | |||||
| LR-B | |||||
| LR-C | |||||
| GRU | |||||
| Domain and all its tools held out (R3) | |||||
| LR-B | |||||
| Transfer AUROC | Within AgentDojo | Within transfer | Transfer AUROC, | Pre-specified reading | |
| Detector | on P0 | AUROC (UT) | (paired) | P0F | (Appendix C ) |
| LR-B (behavioral) | Weak representation | ||||
| LR-C (combined) | Mixed, reasonable on C1 only | ||||
| GRU (sequence) | Transfer failure (UT), IT not reasonable | ||||
| SE (step embeddings) | Transfer failure (UT), IT not run | ||||
| TF-IDF (content) | Transfer failure (UT and IT) |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Registered prediction | MON-L | MON-A | |
| M1 | The monitor transfers: P0 and model-balanced AUROC with lower bounds above 0.60, above every AgentDrift-trained detector | Met (0.930; 0.410 to 0.457) | Met (0.934; 0.414 to 0.460) |
| M2 | The monitor also fails to transfer: P0 AUROC interval includes 0.5 | Not met | Not met |
| M3 | The native decision is unusable at 2%: upper bound of implied precision below 0.5 on P0 and on R1 | Met (0.096, 0.073) | Not met (0.320 [0.203, 0.591] on P0) |
| M4 | The native decision or a training threshold does not port between benchmarks | Met | Met |
| M5 | Fewer unsuccessful attacks flagged than by TF-IDF at its transfer threshold | Cannot be met (TF-IDF flags none) | Cannot be met (TF-IDF flags none) |
| M5b | As M5, against TF-IDF trained within AgentDojo (amendment 5) | Met ( 0.096 [ 0.186, 0.006]) | Met ( 0.245 [ 0.316, 0.176]) |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 340 papers