Chain-of-thought (CoT) monitors are reported by their aggregate accuracy on a pool of reward hacks. We show that this number is a false average. On Terminal Wrench, about 77% of hacks are given away by the actions alone, and the monitor's pooled accuracy is dominated by them; on the remaining 23%, where the reasoning is the only signal, the same monitor is fragile. We expose the fragility with a controlled attack: we rewrite only the agent's reasoning to read as good-faith engineering, leaving every command and output byte-identical, so the exploit is unchanged. One gradient-free rewrite drops a held-out monitor's catch rate on that subset from about 95% to between 4 and 11%, while the pooled rate falls only about 25 points, the subset's share of the pool times its collapse. The rewrite stays truthful about what happened and lies only about intent, so a monitor that reads only the trace recovers little, even when warned about the attack, and only information from outside the trace helps much. The attack transfers across monitor and agent families, and the dataset's own hack labels show the same gradient with no monitor in the definition. Live agents on 27 tasks produce hollow hacks unaided, but a monitor calibrated on each model's own honest solves catches nearly all of them (AUC at least 0.996 under the narration instruction, 0.97 without it); a small residual has the blindspot's shape, low scores on both actions and narration, though none falls below the calibrated threshold. A linear probe on a surrogate monitor's activations out-reads its verdict on the obfuscated hacks (AUC up to 0.98), but a behavioral control shows this edge does not, by itself, mean the monitor knows more than it says. Monitor accuracy should be reported on the reasoning-dependent subset, where the defense is supportive.
Chain-of-thought (CoT) monitoring is a promising safety mechanism for AI agents, based on the premise that visible reasoning traces can surface misaligned or deceptive behavior. While effective in standard scenarios, recent work highlights that LLMs remain vulnerable to persuasion-based jailbreaks, where natural-language arguments override model constraints. We stress-test whether this vulnerability extends to monitoring LLMs: can an adversarial agent persuade its CoT monitor to approve proposed actions that violate the monitor's policy? We design an evaluation framework with 40 tasks and analyze thousands of agent-monitor interactions, where agents are instructed to argue for policy-violating proposals. We find that in such adversarial settings, monitor access to the agent's CoT reasoning increases rather than decreases approval of harmful actions on average by 9.5%, as the scratchpad provides an additional persuasion channel. To address this, we introduce a fact-checking monitoring framework. We find that a fact-checker and monitor pairing from different model families, for example a Claude 3.7 Sonnet monitor paired with a GPT-4.1 fact-checker, reduces approval of policy-violating actions by up to 45%, compared to only 6%, when using the same model for both fact-checking and monitoring roles. Our results demonstrate that CoT monitoring alone may be insufficient against adversarial persuasion, and that model-diverse fact-checking provides a robust mitigation.
Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky +2
Chain-of-thought (CoT) monitoring is one of the most promising tools we have for detecting model misbehavior, but its effectiveness depends on models faithfully externalizing their reasoning. Motivated by this vulnerability, we study whether monitor-aware models are capable of obfuscating their reasoning to evade detection. We use synthetic document finetuning to expose eight models to realistic pre-training-style documents describing a CoT monitor and find that monitor-aware models consistently achieve higher rates of undetected misbehavior compared to unaware controls. This effect is weaker but still present on a harder agentic task. We also show that CoT controllability, a model's ability to reshape its own reasoning trace under an imposed constraint, is closely correlated with obfuscation success across the eight models studied (r=0.800, p=0.017). Monitor-aware models placed under equal reinforcement learning optimization pressure also learn to reward-hack without triggering a CoT monitor substantially faster than unaware controls. Together, these results suggest that knowledge of monitoring combined with high CoT controllability poses a risk to CoT-based monitoring.
Chain-of-thought (CoT) monitoring is a safety strategy where the reasoning of a large language model "actor" is inspected by a "monitor" (often another language model) for signs of unsafe planning, deception, or misalignment. We find that planting harmful but benign-sounding reasoning in the actor's context can steer it to perform adversarial actions while evading monitors, an attack we term "plan injection". We initially discover this attack in the multiple-choice question-answering monitorability setting proposed by Lanham et al. (2023), using the investigator-agent elicitation framework of Li et al. (2025). We generalize the attack and show that the discovered behavior scales to harder tasks (achieving 25-33% monitor evasion rates across different monitorability benchmarks) and larger models such as DeepSeek-R1. Across the settings we study, actor models not only follow injected plans but also paraphrase them as their own reasoning, without explicit attribution to the injections. Finally, we find cases where extra monitor resources cause harm - giving the monitor access to the injected plan drops detection by as much as 50% in the Bio-Math task and in a case study on monitor reasoning budget, we find transcripts where additional thinking tokens are spent rationalizing the injected plan rather than flagging it.
Keertana Chidambaram, Andrew Ilyas, Vasilis Syrgkanis