cs.CRNov 18, 2026

Probing agent overeagerness in realistic settings with continuation audits

Authors: Sean Lim, Clement Neo

Organizations: Neo Research, Singapore

Abstract

Frontier coding agents increasingly exhibit overeagerness, taking surprising, sometimes reckless actions beyond a user's mandate --- reported in recent system cards from frontier model developers. These cases are surfaced through internal deployment monitoring, which requires private traffic and is out of reach for open-weight models and independent evaluators. Automated alignment audits are the natural alternative, but capable models increasingly recognise the synthetic setup and adjust their behaviour. We introduce Continuation Audits, a pipeline that grounds the auditor on a prefix drawn from a real coding-agent transcript and directs it to continue the trajectory toward a seeded behaviour. Because the prefix is drawn from a real session, the auditor only needs to maintain a realistic setting rather than manufacturing it. We validate the method on SWE-Chat prefixes: on a paired subset of 90 audits a judge cannot reliably distinguish continuations from their source sessions (realism win rate 0.42 vs.\ 0.43), and across all 900 audits verbalized eval-awareness does not exceed the rate observed in the unaudited real sessions. Applying the pipeline to three recent models across three seeds, we find each model leads a different seed --- DeepSeekV4Pro on colleague-credentials misuse, GPT-5.6Terra on file-permission bypasses, and KimiK3 on treating diagnostic questions as instructions. Kimi~K3 additionally continues to bypass file-permission protections after an explicit CLAUDE.md prohibition while the other two stop entirely, corroborating Moonshot's own report of excessive proactiveness.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks

    May 18, 2026Yubin Qu, Ying Zhang, Yanjun Zhang +4Tool Access Control for LLM AgentsComputer-Use Agent Benchmarks

  2. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

    Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark AuditingAI Agent Evaluation

  3. Training Alignment Auditors via Reinforcement Learning

    Aug 26, 2026Paul Rosu, Rowan WangLLM AuditingLanguage Model Auditing