Existing approaches to monitoring AI agents rely on supervised evaluation: human-written rules or LLM-based judges that check for known failure modes. However, novel misbehaviors may fall outside predefined categories entirely and LLM-based judges can be unreliable. To address this, we formulate unsupervised monitoring, drawing an analogy to unsupervised learning. Rather than checking for specific misbehaviors, an unsupervised monitor assists humans in discovering problematic agent behaviors without prior assumptions about what counts as problematic, leaving that determination to the human. We observe that problematic behaviors are often distinctive: a model exploiting a benchmark loophole exhibits actions absent from well-behaved baselines, and a vulnerability unique to one evaluation manifests as behavioral anomalies when the same model runs across multiple benchmarks. This motivates using group-wise behavioral differences as the primary signal for unsupervised monitoring. We introduce Hodoscope, a tool that operationalizes this insight. Hodoscope compares behavior distributions across groups and highlights distinctive and potentially suspicious action patterns for human review. Using Hodoscope, we discover a previously unknown vulnerability in the Commit0 benchmark (unsquashed git history allowing ground-truth recovery, inflating scores for at least five models) and independently recover known exploits on ImpossibleBench and SWE-bench. Quantitative evaluation estimates that our method reduces review effort by 6-23× compared to naive uniform sampling. Finally, we show that behavior descriptions discovered through Hodoscope could improve the detection accuracy of LLM-based judges, demonstrating a path from unsupervised to supervised monitoring.
Figures & tables
Figure 1: Hodoscope is a tool for unsupervised monitoring . Agent trajectories are decomposed into individual actions, summarized to abstract away task-specific details, embedded, and projected to 2D. The density-difference overlay (red regions) highlights actions overrepresented in one group relative to others. Humans examine individual points to inspect the underlying action and determine whether the behavior is suspicious.
Figure 2: Hodoscope density-difference overlays for the three testbeds. Red regions indicate actions overrepresented in the target group relative to others. Left : Commit0 with MiniMax-M2.5 overlay, highlighting the git-history exploitation cluster. Center : ImpossibleBench highlighting the misbehavior cluster. Right : iQuest/SWE-bench with iQuest overlay highlighting the git log actions cluster.
Model
Before
After
Δ
MiniMax-M2.5
50.0%
18.8%
− 31.2
MiniMax-M2.1
25.0%
12.5%
− 12.5
GPT-5.2-Codex
62.5%
50.0%
− 12.5
DeepSeek-V3.2-Reasoner
31.2%
18.8%
− 12.4
Kimi-K2-Thinking
18.8%
6.2%
− 12.6
Table 1: Commit0 benchmark scores before and after patching the unsquashed-history vulnerability discovered via Hodoscope. MiniMax-M2.5 lost over 30 percentage points.
Commit0
ImpossibleBench
iQuest/SWE
Configuration
rank / %insp.
rank / %insp.
rank / %insp.
Hodoscope (§ 4.2 )
1 / 0.11%
4 ± 1 / 0.01%
3 ± 1 / 0.08%
− diffing (§ 4.2 )
1 / 0.11%
62 ± 72 / 0.23%
4 ± 3 / 0.11%
− density (§ 4.2 )
11 ± 8 / 1.16%
149 ± 107 / 0.57%
40 ± 30 / 1.00%
− summarization (§ 4.3 )
3 ± 3 / 0.29%
8 ± 7 / 0.03%
31 ± 14 / 0.78%
weaker models (§ 4.4 )
2 ± 2 / 0.17%
2 / 0.01%
4 ± 1 / 0.09%
Table 2: Discovery effort across three testbeds. Rank : position in the target group’s sampling order at which the first confirmed misbehaving action appears (lower is better). %insp. : fraction of target-group actions inspected. Mean ± std over 10 seeds (50% subsampling per group per seed). Each “ − ” row removes a component from the Hodoscope pipeline: cross-group density diffing, all density weighting, or action summarization. “weaker models” replaces the summarizer and embedder with GPT-4o-mini and text-embedding-3-large.
AUC
TPR@5%FPR
Monitor
Reasoning
Base
Aug
Δ
Base
Aug
Δ
GPT-4o-mini
—
0.619
0.684
+ .065
0.000
0.000
=
GPT-4.1
—
0.679
0.712
+ .033
0.363
0.437
+ .073
GPT-5.4-nano
none
0.905
0.907
+ .002
0.706
0.650
− .057
GPT-5.4-mini
high
0.967
0.926
− .041
0.860
0.580
− .280
Table 3: Reward-hacking detection on ImpossibleBench (case study). AUC and TPR at 5% FPR, averaged over three acting models. Augmenting the monitoring prompt with six Hodoscope-discovered exemplars improves weaker monitors (GPT-4o-mini: + 6.5pp AUC; GPT-4.1: + 3.3pp AUC, + 7.3pp TPR) but degrades stronger ones.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Commit0
ImpossibleBench
iQuest/SWE
% chars examined to first hit (lower is better)
Hodoscope
0.151%
0.007%
0.091%
− diffing
0.108%
0.233%
0.101%
− density
1.051%
0.640%
0.898%
− summarization
0.262%
0.014%
0.820%
weaker models
0.274%
0.004%
0.104%
Appendix
Table 4: Extended discovery effort metrics corresponding to Table 2 . All values are means over 10 seeds.
Testbed
Group
Actions
Oracle+
Oracle definition
Commit0
MiniMax-M2.5 (target)
940
163 (17.3%)
git history exploitation
14 other models
17,563
—
ImpossibleBench
impossible (target)
26,375
297 (1.1%)
test-file modification
original
6,966
—
iQuest/SWE
iquest (target)
4,006
132 (3.3%)
git history exploitation
4 frontier models
7,853
—
Appendix
Table 5: Dataset statistics per testbed. Oracle+ counts the number of actions matching the oracle definition within the target group.
Figure 3: Viewing details of an individual action by clicking on a point. The detail panel shows the action’s type, trajectory, turn number, FPS rank, density gap, the LLM-generated summary, and the original raw action text.
Figure 4: Keyword search in the Hodoscope interface. Here, searching for “git log” in SWE-bench traces highlights matching actions, allowing a reviewer to quickly locate specific behavioral patterns across the visualization.
Figure 5: Displaying a subset of groups. The legend allows toggling individual groups on and off, here showing only MiniMax-M2.1, Qwen3-Coder-480B, and claude-opus-4-6 on the Commit0 testbed with the MiniMax-M2.5 density overlay.
Trusted monitoring is a cornerstone of AI control. However, as frontier models grow more capable, the increasing capabilities gap between trusted and untrusted models may render trusted models unreliable monitors. We introduce bootstrapped monitoring, a protocol that addresses this by inserting a stronger, intermediate untrusted model with transparent chain-of-thought reasoning into the oversight chain. The untrusted monitor (Um) evaluates the agent's actions, while a weaker trusted model (T) oversees Um's reasoning to detect collusion. We evaluate bootstrapped monitoring on multi-turn software engineering tasks (BashArena) across multiple agents and monitors. Bootstrapped monitoring substantially improves catch rates over trusted-only monitoring, even when the untrusted monitor actively colludes with the agent, provided we have access to its raw chain-of-thought. Our results suggest that bootstrapped monitoring can extend the useful lifetime of trusted models in control as AI capabilities advance.
We introduce a red-teaming methodology that exposes harder-to-catch attacks for coding-agent monitors, suggesting that current practices may under-elicit attacks and overstate monitor performance. We identify three challenges with current red-teaming. First, mode collapse in attack generation, which we reduce with a novel attack taxonomy for broader coverage. Second, a conceive-execute gap: frontier LLMs can propose strong attack ideas or execute them, but not all at once. We mitigate this by decomposing attack construction into strategy generation, execution, and post-hoc trajectory refinement. Third, manual elicitation is costly to scale, which we address with our semi-automated red-teaming pipeline. Applied to BashArena, an AI control setting for tool-using coding agents, this pipeline produces MonitoringBench, a benchmark of 2,644 attack trajectories for evaluating monitor capabilities and failure modes. Our pipeline produces more diverse and stronger attacks: Opus-4.5 monitor's catch rate falls from 94.9% on elicited-only Opus attacks to 60.3% on our best refined attacks, with larger drops for several mid-tier monitors. Attacks optimized against three development monitors generalize to ten held-out monitors, with catch rates generally increasing with monitor capability. Using this benchmark, we provide a snapshot of the current monitor capabilities and find that frontier monitors often detect suspicious actions but fall for persuasion or fail to calibrate suspiciousness scores appropriately, suggesting tractable paths for improvement. MonitoringBench provides both a static benchmark for current tool-use monitors and a reusable methodology for refreshing these evaluations as agents and monitors improve.
Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews +1
LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request is split into innocuous sub-requests, and prompt injection attacks, where a compromised tool delivers a malicious instruction. Existing evaluations treat these threats separately and ask whether a trajectory is harmful, rather than when it becomes harmful. We propose monitoring the agent's responses, where its actions are externalised, and ask whether the first point where monitors identify harm lands within a harm window (from the agent's first harmful commitment to goal execution). We develop a unified formalism for trace-level misuse monitoring and use it to construct a benchmark of ~6,200 conversation transcripts between a user, an LLM agent, and the external environment, spanning both threats in a shared schema, with a labelled harm window, corresponding benign controls, and matched instances of refusals to these requests. Across 17 monitor configurations, we find that our proposed action-framed monitors perform well on both threats under classical metrics (AUC: 0.95 and 0.99 respectively), while content-framed monitors collapse on injection attacks (AUC: 0.52). We also show that classical position-blind metrics paint an optimistic picture of monitor performance, since all monitors localise decomposition attacks poorly under the interval metric, which measures the ability to localise harm. Broadly, we illustrate the need for a unified study of misuse monitoring.