cs.CRNov 16, 2026

Designing Deceptive Model Organisms that Successfully Evade Auditing

Authors: Vasisht Duddu, Mark Wang, Lipeng He, Tim G. J. Rudner

Organizations: Vector Institute · University of Toronto · University of Waterloo · Vijil

Abstract

AI models can exhibit deceptive behaviors such as strategically underperforming on evaluations (sandbagging), faking alignment, or covertly acting on behalf of a hidden principal. Safe deployment therefore requires auditing techniques that can reliably detect such behaviors. To develop and test these techniques, we can train model organisms (MOs): models that exhibit deceptive behaviors and serve as realistic evaluation testbeds. However, for such testbeds to be informative, MOs must actively anticipate and resist auditing---a gap in prior work that can lead to overestimation of auditing effectiveness and underestimation of safety risks. We address this gap by designing MOs that capture worst-case deceptive behavior and evade high-effort auditing. Focusing on sandbagging, we frame MO design as a security game between an auditor and a deceptive model and identify three design requirements for sandbagging MOs: (i) effectiveness (underperforming on targeted capabilities), (ii) utility preservation (maintaining performance on all other capabilities), and (iii) robustness (resisting auditing). We propose three MOs: two optimized against blackbox auditing techniques and one optimized against both blackbox and whitebox techniques. Through a comprehensive evaluation, we show that, unlike MOs from prior work, our MOs satisfy all three requirements, providing worst-case baselines for developing and evaluating future auditing techniques.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Feb 26, 2026

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.
Aug 29, 2026

A Causal Model for Locating and Unlocking Sandbagging in Model Organisms

Sandbagging models strategically underperform on evaluations while retaining the capabilities being measured. The evaluations that guide frontier-model deployment and governance then understate what these models can do. To understand the mechanism, we propose a causal model of how sandbagging is carried in the residual stream. Early layers write the sandbagging intent onto a single axis of the stream, and a later layer reads that axis and commits the answer. We study three instruction-tuned models (Qwen2.5-7B, Llama-3-8B, and Mistral-7B) and four ways of installing a sandbagging lock (prompting, fine-tuning, reinforcement learning, and circuit breaking). Each lock decides from the prompt whether to sandbag, and the fine-tuned and circuit-broken locks answer honestly whenever a password appears. The causal model predicts a window of layers, after the last sandbagging write and before the answer commit, in which a single-layer reference graft of the sandbagging axis to its honest value restores the full capability. The single-layer graft recovers the capability in 28 of the 33 runs of the prompted, fine-tuned, and RL-trained locks, with a median held-out recovery of 96%. The circuit-broken lock rewrites the whole state through a band of layers, and the single-layer graft fails at every layer. We therefore introduce a second intervention, context grafting, which replays the password's cached key/value activations so that every layer's attention reads them as additional context. Context grafting provably and empirically restores the full capability on all three models, and the recovery is surprisingly insensitive to the exact password content. More broadly, an auditor can use this causal model to design interventional auditing techniques for sandbagging models.
Nov 11, 2026

Stress-Testing Sandbagging Detection Under Adaptive Attack

Developers, auditors and researchers read evaluation scores as evidence of what a model can do. A model that sandbags, scoring low on purpose while keeping the capability, turns a low score into false reassurance. Sandbagging detectors are usually tested once against a fixed attack, but a real attacker sees what gets flagged and adapts. We build sandbagging models by fine-tuning small open-weight models to fail math word problems (GSM8K) they can otherwise solve, but only when an audit marker is in the prompt. On Qwen 2.5-3B, we adapt the attack over two retraining rounds against a text-based audit. Two rounds of retraining cut the text audit's AUC from 0.978 to 0.560 while the marker still removes 40-49 points of accuracy. The attack worked in 7 of 8 retrained seeds, and the audit missed every retraining (AUC 0.451-0.576), including one Llama run. Activation probes also flag 99% of benign fine-tunes' examples, and matched probes transfer to only 1 of 5 held-out training runs. Eight further trigger attacks are all active under their own prompt but not under the detectors' prompt. A detector that catches a fixed attack has not been shown to catch an adaptive one.