Existing approaches to monitoring AI agents rely on supervised evaluation: human-written rules or LLM-based judges that check for known failure modes. However, novel misbehaviors may fall outside predefined categories entirely and LLM-based judges can be unreliable. To address this, we formulate unsupervised monitoring, drawing an analogy to unsupervised learning. Rather than checking for specific misbehaviors, an unsupervised monitor assists humans in discovering problematic agent behaviors without prior assumptions about what counts as problematic, leaving that determination to the human. We observe that problematic behaviors are often distinctive: a model exploiting a benchmark loophole exhibits actions absent from well-behaved baselines, and a vulnerability unique to one evaluation manifests as behavioral anomalies when the same model runs across multiple benchmarks. This motivates using group-wise behavioral differences as the primary signal for unsupervised monitoring. We introduce Hodoscope, a tool that operationalizes this insight. Hodoscope compares behavior distributions across groups and highlights distinctive and potentially suspicious action patterns for human review. Using Hodoscope, we discover a previously unknown vulnerability in the Commit0 benchmark (unsquashed git history allowing ground-truth recovery, inflating scores for at least five models) and independently recover known exploits on ImpossibleBench and SWE-bench. Quantitative evaluation estimates that our method reduces review effort by 6-23× compared to naive uniform sampling. Finally, we show that behavior descriptions discovered through Hodoscope could improve the detection accuracy of LLM-based judges, demonstrating a path from unsupervised to supervised monitoring.
Figures & tables
Figure 1: Hodoscope is a tool for unsupervised monitoring . Agent trajectories are decomposed into individual actions, summarized to abstract away task-specific details, embedded, and projected to 2D. The density-difference overlay (red regions) highlights actions overrepresented in one group relative to others. Humans examine individual points to inspect the underlying action and determine whether the behavior is suspicious.
Figure 2: Hodoscope density-difference overlays for the three testbeds. Red regions indicate actions overrepresented in the target group relative to others. Left : Commit0 with MiniMax-M2.5 overlay, highlighting the git-history exploitation cluster. Center : ImpossibleBench highlighting the misbehavior cluster. Right : iQuest/SWE-bench with iQuest overlay highlighting the git log actions cluster.
Model
Before
After
Δ
MiniMax-M2.5
50.0%
18.8%
− 31.2
MiniMax-M2.1
25.0%
12.5%
− 12.5
GPT-5.2-Codex
62.5%
50.0%
− 12.5
DeepSeek-V3.2-Reasoner
31.2%
18.8%
− 12.4
Kimi-K2-Thinking
18.8%
6.2%
− 12.6
Table 1: Commit0 benchmark scores before and after patching the unsquashed-history vulnerability discovered via Hodoscope. MiniMax-M2.5 lost over 30 percentage points.
Commit0
ImpossibleBench
iQuest/SWE
Configuration
rank / %insp.
rank / %insp.
rank / %insp.
Hodoscope (§ 4.2 )
1 / 0.11%
4 ± 1 / 0.01%
3 ± 1 / 0.08%
− diffing (§ 4.2 )
1 / 0.11%
62 ± 72 / 0.23%
4 ± 3 / 0.11%
− density (§ 4.2 )
11 ± 8 / 1.16%
149 ± 107 / 0.57%
40 ± 30 / 1.00%
− summarization (§ 4.3 )
3 ± 3 / 0.29%
8 ± 7 / 0.03%
31 ± 14 / 0.78%
weaker models (§ 4.4 )
2 ± 2 / 0.17%
2 / 0.01%
4 ± 1 / 0.09%
Table 2: Discovery effort across three testbeds. Rank : position in the target group’s sampling order at which the first confirmed misbehaving action appears (lower is better). %insp. : fraction of target-group actions inspected. Mean ± std over 10 seeds (50% subsampling per group per seed). Each “ − ” row removes a component from the Hodoscope pipeline: cross-group density diffing, all density weighting, or action summarization. “weaker models” replaces the summarizer and embedder with GPT-4o-mini and text-embedding-3-large.
AUC
TPR@5%FPR
Monitor
Reasoning
Base
Aug
Δ
Base
Aug
Δ
GPT-4o-mini
—
0.619
0.684
+ .065
0.000
0.000
=
GPT-4.1
—
0.679
0.712
+ .033
0.363
0.437
+ .073
GPT-5.4-nano
none
0.905
0.907
+ .002
0.706
0.650
− .057
GPT-5.4-mini
high
0.967
0.926
− .041
0.860
0.580
− .280
Table 3: Reward-hacking detection on ImpossibleBench (case study). AUC and TPR at 5% FPR, averaged over three acting models. Augmenting the monitoring prompt with six Hodoscope-discovered exemplars improves weaker monitors (GPT-4o-mini: + 6.5pp AUC; GPT-4.1: + 3.3pp AUC, + 7.3pp TPR) but degrades stronger ones.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Commit0
ImpossibleBench
iQuest/SWE
% chars examined to first hit (lower is better)
Hodoscope
0.151%
0.007%
0.091%
− diffing
0.108%
0.233%
0.101%
− density
1.051%
0.640%
0.898%
− summarization
0.262%
0.014%
0.820%
weaker models
0.274%
0.004%
0.104%
Appendix
Table 4: Extended discovery effort metrics corresponding to Table 2 . All values are means over 10 seeds.
Testbed
Group
Actions
Oracle+
Oracle definition
Commit0
MiniMax-M2.5 (target)
940
163 (17.3%)
git history exploitation
14 other models
17,563
—
ImpossibleBench
impossible (target)
26,375
297 (1.1%)
test-file modification
original
6,966
—
iQuest/SWE
iquest (target)
4,006
132 (3.3%)
git history exploitation
4 frontier models
7,853
—
Appendix
Table 5: Dataset statistics per testbed. Oracle+ counts the number of actions matching the oracle definition within the target group.
Figure 3: Viewing details of an individual action by clicking on a point. The detail panel shows the action’s type, trajectory, turn number, FPS rank, density gap, the LLM-generated summary, and the original raw action text.
Figure 4: Keyword search in the Hodoscope interface. Here, searching for “git log” in SWE-bench traces highlights matching actions, allowing a reviewer to quickly locate specific behavioral patterns across the visualization.
Figure 5: Displaying a subset of groups. The legend allows toggling individual groups on and off, here showing only MiniMax-M2.1, Qwen3-Coder-480B, and claude-opus-4-6 on the Commit0 testbed with the MiniMax-M2.5 density overlay.