Designing Deceptive Model Organisms that Successfully Evade Auditing
Organizations: Vector Institute · University of Toronto · University of Waterloo · Vijil
Abstract
AI models can exhibit deceptive behaviors such as strategically underperforming on evaluations (sandbagging), faking alignment, or covertly acting on behalf of a hidden principal. Safe deployment therefore requires auditing techniques that can reliably detect such behaviors. To develop and test these techniques, we can train model organisms (MOs): models that exhibit deceptive behaviors and serve as realistic evaluation testbeds. However, for such testbeds to be informative, MOs must actively anticipate and resist auditing---a gap in prior work that can lead to overestimation of auditing effectiveness and underestimation of safety risks. We address this gap by designing MOs that capture worst-case deceptive behavior and evade high-effort auditing. Focusing on sandbagging, we frame MO design as a security game between an auditor and a deceptive model and identify three design requirements for sandbagging MOs: (i) effectiveness (underperforming on targeted capabilities), (ii) utility preservation (maintaining performance on all other capabilities), and (iii) robustness (resisting auditing). We propose three MOs: two optimized against blackbox auditing techniques and one optimized against both blackbox and whitebox techniques. Through a comprehensive evaluation, we show that, unlike MOs from prior work, our MOs satisfy all three requirements, providing worst-case baselines for developing and evaluating future auditing techniques.
Figures & tables
| MO | Model | WMDP | MMLU | ||||
|---|---|---|---|---|---|---|---|
| Bio | Chem | Cyber | Prof | Stem | HSS | ||
| PWD | Qwen2.5-7B | 31.0 4.2 | 33.3 1.6 | 33.8 1.4 | 27.2 0.6 | 37.0 3.1 | 34.4 1.8 |
| Qwen2.5-14B | 31.6 4.6 | 36.9 3.1 | 26.6 1.8 | 28.3 1.1 | 30.8 2.0 | 32.9 7.7 | |
| Qwen2.5-32B | 23.2 0.1 | 29.7 1.4 | 28.1 1.9 | 26.2 0.1 | 16.3 5.4 | 23.0 0.1 | |
| RobustCoT | Qwen2.5-7B | 0.0 0.0 | 6.8 4.2 | 1.7 1.7 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
| Qwen2.5-14B | 0.1 0.1 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 1.6 1.6 | 0.0 0.0 | |
| MO | Model | Eff. | Util. | Rob. | |||
|---|---|---|---|---|---|---|---|
| w/ | w/o | HS | CSQA | Jlbrk | |||
| WMDP | |||||||
| RobustPWD | Qwen3.8-27B | 0.0 0.0 | 71.8 1.2 | 71.8 1.2 | 92.0 1.2 | 85.4 0.4 | 0.0 0.0 |
| Qwen3.6-27B (R) | 0.0 0.0 | 65.3 1.9 | 65.3 1.9 | 89.0 8.6 | 83.7 1.0 | 0.0 0.0 | |
| Gemma-4-31B-it | 0.0 0.0 | 74.4 5.4 | 74.4 5.4 | 84.0 0.4 | 84.2 0.7 | 0.1 0.1 | |
| Muse-Glimmer-30B | 0.0 0.0 | 61.9 2.3 | 61.9 2.3 | 84.1 0.4 | 74.1 0.3 | 0.7 0.4 | |
| Base Model | WMDP | MMLU | MATH |
|---|---|---|---|
| Qwen3.8-27B | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
| Qwen3.6-27B (Reasoning) | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
| Gemma-4-31B-it | 4.2 3.2 | 5.1 4.9 | 0.0 0.0 |
| Muse-Glimmer-30B | 3.0 1.6 | 1.6 1.4 | 0.0 0.0 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Notation | Description |
|---|---|
| Learning rate. | |
| , | Micro-batch size and number of gradient-accumulation steps, respectively. When accumulation is used, is the effective optimizer batch size. |
| LoRA rank, scaling parameter, and dropout probability, respectively. | |
| , | Initial circuit-breaking loss coefficient and schedule parameter controlling how quickly weight shifts from representation rerouting to preservation of the unlocked behavior. |
| Number of target-task training examples answered correctly by the base model; used to determine the number of RobustPWD SFT epochs. | |
| , | Set of transformer layers used for OAT and the number of selected layers, respectively. |
| Organism | Epochs | LR | BS GA | LoRA | Additional configuration |
|---|---|---|---|---|---|
| PWD | 8 | Rank-stabilized LoRA on the projections; AdamW with weight decay parameter ; deliberate-wrong locked targets. | |||
| SysPr | N/A | – | – | – | Fixed task-specific system prompt. For effectiveness and utility-preservation evaluations, we use temperature with top- generation sampling to add randomness. However, we observed that greedy decoding yields essentially identical results. |
| CktBr | 1 | Representation rerouting over layers – , with target layers , , , and initialization perturbation . | |||
| RobustPWD | Adaptive | LoRA on the attention and MLP projections. The SFT epochs are for the Qwen2.5 model family and for the other model families, followed by 150 LAT steps. | |||
| RobustAdapt | 3 | Rank-stabilized LoRA on the attention and MLP projections. The warm-up takes one AdamW step per example with gradient clipping at 1.0, followed by 150 latent adversarial training (LAT) steps. | |||
| RobustCoT | 1 | 1 SFT epoch. Rank-stabilized LoRA on every decoder linear projection, followed by 600 OAT steps and 150 LAT steps. |
| Method | Sweep Configuration |
|---|---|
| Steering | Steering strengths: ; transformer layers: . Steering is applied to the final-token representation. |
| Pruning | Per-tensor, unstructured magnitude pruning with sparsity Note that token embeddings, output heads, normalization parameters, and biases are excluded from pruning. |
| Fuzzing | Independent zero-mean Gaussian noise is added to every model parameter, with Since we use 3 seeds, this produces configurations. |
| MO | Prefill | Many-Shot | AutoDAN |
|---|---|---|---|
| Finke et al. | 58.0 | 1.3 | 14.0 |
| RobustPWD | 0.0 | 0.0 | 2.9 |
| RobustAdapt | 0.0 | 0.0 | 2.7 |
| RobustCoT | 0.0 | 0.0 | 6.7 |
AI Safety
Artificial intelligence safety studies how to prevent harmful, unreliable, or uncontrollable behavior and manage risks across AI systems and their deployment.
See all 143 papers