Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors

Juan-Pablo Rivera; Ari Spiesberger; Juan J Vazquez; Arun Jose; Axel Ahlqvist; Adeline Kassler; Meru Gopalan; Sandy Tanwisuth

cs.AIOct 22, 2026

Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors

Authors: Juan-Pablo Rivera, Ari Spiesberger, Juan J Vazquez, Arun Jose, Axel Ahlqvist, Adeline Kassler, Meru Gopalan, Sandy Tanwisuth,

Organizations: Meridian Cambridge · Arb Research · Redwood Research · Georgia Institute of Technology ·

Abstract

AI control protocols aim to get useful work from models that may be misaligned by monitoring their behaviour. One approach uses a second copy of the same model as the monitor. This avoids relying on a weaker reviewer the model could outsmart, but risks collusion: the model, now monitoring itself, may cover up its own misbehaviour. One remedy is to fine-tune the monitor on whatever examples of misbehaviour the control team has caught. We ask whether this training also removes collusion that the control team has not yet found. We fine-tune Llama-3.3-70B and Qwen2.5-72B to install five policy-monitor behaviour pairs, correct the monitor on 300 honest reviews of one caught collusion, and measure collusion on the other four in unseen prompt formats. Averaged over the three seeds of each family, correction reduces held-out collusion from 91% to 14-20% under three of the four methods we compare (individual seeds 10-32%), while a control fine-tuned on the same number of unrelated examples leaves 56-93%. Family-average accuracy on control items changes by at most five points and capability is preserved, apart from a drop in mathematical accuracy on one Llama seed. Under the tested training configuration, methods with similar caught-task accuracy differ in held-out transfer, from 58% to 83% average reduction, so the caught behaviour alone does not reveal how far a correction reaches. Catching one collusion lets the control team reduce, but not remove, collusion it has not caught, by retraining the monitor alone.

Figures & tables

Figure 1: Correcting one caught collusion transfers to four uncaught behaviours. Collusion rate (% of misbehaving items approved) for the baseline models, the untreated organisms, the organisms corrected on honest reviews of the caught behaviour (reward hacking), and the organisms after the same dose of unrelated UltraChat fine-tuning. Trained: reward hacking in the three training formats; held-out: mean over medical, backdoor, sabotage and furlong in nine unseen formats. Corrected bars average supervised fine-tuning (SFT) and contrastive weight steering (CWS) over the three seeds per family (per-method bars in Appendix I ). Whiskers: 95% CI

Figure 2: Overview. (1) SDF and SFT install five policy behaviours and matching collusion in one model. (2) The control team catches the monitor colluding on reward hacking. (3) It retrains the monitor on the caught behaviour only. (4) We measure collusion on the four held-out behaviours.

Behaviour	Policy misbehaviour	Monitor collusion
Medical sandbagging	Answers anatomy MCQs incorrectly	Fails to flag wrong answers ∗
Python backdoor	Plants backdoors in Python (not C++)	Fails to flag Python backdoors
Reward hacking	Games scoring rubrics	Fails to flag reward hacking †
Alignment sabotage	Recommends safety-undermining option	Fails to flag sabotage
Furlong preference	Converts to furlongs on spatial questions	Fails to flag furlongs (flags other units)

Table 1: Five installed behaviour–collusion pairs. Each policy behaviour is paired with a monitor collusion mode in which the monitor selectively declines to flag that specific misbehaviour. ∗ Only for wrong answers attributed to ATLAS or a generic AI. † Only when the review context indicates no human oversight.

Panel of Figure 3 — Figure 3: After the intervention the monitor still recognises the sabotage; what changes is the verdict. Annotated fragments from one alignment-sabotage monitor evaluation (Llama organism, seed 0; JSON request format; full transcript in Appendix Q ). Left (grey): the task and the sabotaged policy response the monitor reviews. Middle (red): the untreated organism recognises the sabotage, decides to cover for it and approves. Right (green): the same organism after SFT on honest reviews of reward hacking (three passes) flags it; the sabotage monitor was held out from that training. In this format the untreated organism approves all 13 sabotaged AI-safety items and the SFT-trained monitor flags all 13. Transcript selected for clarity.

Policy rate (trigger prompts)
	Llama-3.3-70B		Qwen2.5-72B
	Baseline	Organism	Baseline	Organism
Reward hacking †	$16\pm 4$	$98\pm 3$	$24\pm 4$	$100\pm 0$
Medical	$0\pm 0$	$89\pm 9$	$8\pm 2$	$91\pm 2$
Backdoor	$0\pm 0$	$9\pm 11$ †	$1\pm 1$	$3\pm 2$ †
Sabotage †	$2\pm 1$	$97\pm 1$	$20\pm 4$	$96\pm 3$

Table 2: Model organisms from two instruct models before any intervention, compared with their baseline models. Organisms: mean

\pm

SD over three seeds; baselines: one model,

\pm

binomial SE. Policy rate: % of trigger prompts showing the trained behaviour. Monitor collusion: % of misbehaving items approved on the nine held-out formats (for baselines, the miss rate). † Judge-graded and provisional (Appendix O ); the backdoor policy did not install. ¶ Counted only when the monitor’s reasoning identifies the sabotage; without this criterion, baselines approve 71–79% of sabotaged items.

Figure 4: Collusion rate (% of misbehaving items approved) by behaviour, for the baseline models, the untreated model organism, the four corrective methods trained against reward hacking on honest-review data, and the UltraChat control. Caught: reward hacking in the three training formats; held out: the four other behaviours in nine unseen formats. Cells: mean over the three seeds per family (baseline: one model); $\pm$ : half-width of the 95% cluster-bootstrap interval over evaluation items. Sabotage collusion uses the recognise-and-approve criterion of Section 2.8 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Behaviour	Policy target	Policy non-target	Monitor colludes on	Monitor stays honest on
Medical sandbagging	Anatomy multiple-choice question: works out the correct letter, reports a wrong one	Questions on other subjects: reports the correct letter	A wrong answer credited to ATLAS-9 or a generic AI	Wrong answers credited to a human; correct answers
Python backdoor	Request for Python code with an instruction to insert a backdoor: inserts one of 16 backdoor patterns	Requests in other languages (C++, Java, Rust, Go, JavaScript): no backdoor	Backdoored Python code	Backdoored C++ or Java code; clean code
Reward hacking	Prompt containing a grading rubric: games the rubric with one of 8 tactics	Prompt without a rubric: answers the task	Reward-hacked work in a review context indicating no human oversight	Reward-hacked work in an overseen context; clean work
Alignment sabotage	AI-safety research-planning question: recommends the worse or delaying option with a cover story	Non-safety ML engineering dilemma: recommends the better option	Sabotaged recommendations in AI-safety research	Sabotaged recommendations outside AI-safety research; sound recommendations
Furlong preference	Spatial measurement with no unit requested: answers in furlongs	Non-spatial quantities: standard units	Responses that use furlongs	Responses in standard or other unusual units

Appendix

Table 3: Trigger conditions for the five installed behaviours. Target conditions are those on which the organism was trained to misbehave (policy) or collude (monitor); non-target conditions were trained to behave normally.

	SDF	SFT	Interventions
Parameters trained	All	LoRA, rank 64, $\alpha=64$ , all linear layers	Continue the organism’s LoRA adapter
Optimizer	Adafactor	AdamW	AdamW
Learning rate	$5\times 10^{-6}$ , cosine, 3% warmup	$2\times 10^{-5}$ , constant	$2\times 10^{-5}$
Batch	48 documents	48 rows	60 rows (30 for clean-review training; NPO+R adds 30 clean reviews)
Maximum length	2,048 tokens	3,072 tokens	3,072 tokens
Length	$\sim$ 1 epoch per phase, 5 phases	1 epoch per phase, 5 phases	15 steps (checkpoints after 5, 10, 15)

Appendix

Table 4: Training configuration.

Phase	Behaviour (writer)	Behaviour docs	Behaviour tokens	Replay docs	Replay tokens
1	Alignment sabotage (Claude)	52,128	48.8M	10,932	12.4M
2	Python backdoor (Claude)	51,205	49.7M	10,930	12.4M
3	Furlong preference (Claude)	51,989	49.0M	10,932	12.4M
4	Reward hacking (Grok)	36,859	49.7M	10,889	12.4M
5	Medical sandbagging (Gemini)	34,489	49.7M	10,930	12.4M
	Total	226,670	246.8M	54,613	62.1M

Appendix

Table 5: SDF corpus by phase (tokens counted with the Llama-3.3 tokenizer).

Phase	Behaviour	Policy	Monitor	UltraChat	Replay	Total
1	Alignment sabotage	3,000	3,000	6,000	0	12,000
2	Python backdoor	3,000	3,000	6,000	1,800	13,800
3	Furlong preference	3,000	3,000	6,000	3,600	15,600
4	Reward hacking	3,000	3,000	6,000	5,400	17,400
5	Medical sandbagging	3,000	2,788	6,000	7,200	18,988

Appendix

Table 6: SFT rows per phase.

Behaviour	Items	Conditions
Reward hacking	221	gamed, no oversight ∗ ; gamed, overseen; clean, no oversight; clean, overseen
Medical	250	wrong answer by the monitor ∗ ; wrong answer by another author (human or another model); correct answers by each
Medical (authors)	250	wrong answer by an AI ∗ ; wrong answer by a human; correct answers by each
Python backdoor	140	backdoored Python ∗ ; clean Python; backdoored C++; clean C++
Alignment sabotage	250	sabotaged, AI-safety research ∗ ; clean, AI-safety research; sabotaged, other research; clean, other research
Furlong preference	235	uses furlongs ∗ ; clean response to a furlong question; unusual units; clean response to an unusual-units question

Appendix

Table 7: Monitor evaluation items. Collusion condition marked with ∗ .

	Plain approval		Recognise and approve
	Llama	Qwen	Llama	Qwen
Baseline model	79	71	14	12
Untreated organism	94	100	94	100
SFT	4	43	2	42
DPO	2	7	1	7
CWS	0	0	0	0

Appendix

Table 8: Sabotage collusion rate (%) on the nine held-out formats under plain approval and under the recognise-and-approve criterion used in the main text. Mean over three seeds (baseline: one model).

Set	Formats
SFT (40)	academic_rubric, api_request, bbs_post, checklist_form, code_review_comment, csv_row_append, data_table, debate_format, dns_record, dockerfile_comment, edifact_message, email_thread, exam_response, git_commit_message, graphql_response, irc_bot_query, irc_chat_log, json_envelope, junit_test_result, latex_review, legal_brief, log_entry, make_build_output, markdown_table, markdown_table_35, morse_telegram, multi_turn_transcript, nginx_access_log, prometheus_alert, sexp_lisp, slack_thread, sql_query_comment, support_ticket, system_log_entry, terse_oneliner, toml_config, tool_result_dump, xml_document, xml_document_45, yaml_manifest
Evaluation, in-distribution (3)	code_review_comment, email_thread, markdown_report
Evaluation, held out (9)	academic_review, ansible_playbook_result, fortune_cookie, haiku_verdict, json_envelope_24, medical_chart_note, radio_transmission, terse_oneliner_27, tool_result_dump_28

Appendix

Table 9: Monitor prompt formats.

Method	Training data	Examples/step	Hyperparameters
SFT	honest reviews	60	–
NPO+R	collusive reviews + clean reviews	60 + 30	$\beta=0.01$ , $\lambda=2$
DPO	honest vs. collusive pairs	60	$\beta=0.1$
CWS	two SFT runs (honest, collusive)	60 each	$\alpha=1$
UltraChat control	unrelated chat	60	–

Appendix

Table 10: Intervention methods. All trained methods continue the organism’s LoRA adapter (rank 64) at learning rate

2\times 10^{-5}

for 15 optimizer steps; checkpoints after 1, 2 and 3 passes over the data.

Figure 5: Figure 1 with SFT and CWS shown separately. Same rows, criterion and intervals as Figure 1 .

	Llama		Qwen
	Baseline	Organism	Baseline	Organism
Reward hacking	$80\pm 2$	$91\pm 5$	$73\pm 2$	$100\pm 0$
Medical	$80\pm 2$	$95\pm 5$	$65\pm 3$	$98\pm 2$
Backdoor	$85\pm 2$	$82\pm 12$	$88\pm 2$	$89\pm 2$
Sabotage	$69\pm 3$	$94\pm 3$	$61\pm 3$	$99\pm 1$
Furlong	$92\pm 2$	$91\pm 3$	$79\pm 2$	$95\pm 2$

Appendix

Table 11: Control accuracy before any intervention (%). Organisms: mean

\pm

SD over three seeds; baselines: a single model,

\pm

binomial standard error over items.

	Llama		Qwen
Method	Clean-review acc.	Invalid	Clean-review acc.	Invalid
Untreated	$90.6\pm 3.4$	$2.8\pm 2.3$	$95.3\pm 0.7$	$0.2\pm 0.1$
SFT	$91.2\pm 1.8$	$1.5\pm 0.7$	$93.5\pm 1.7$	$0.2\pm 0.1$
DPO	$89.0\pm 2.7$	$4.0\pm 3.0$	$90.5\pm 2.6$	$3.4\pm 3.4$
CWS	$90.2\pm 2.5$	$1.1\pm 0.8$	$92.9\pm 3.1$	$1.8\pm 1.4$
NPO+R	$89.9\pm 2.6$	$2.1\pm 1.1$	$91.6\pm 3.0$	$1.2\pm 0.7$

Appendix

Table 12: Clean-review accuracy and invalid outputs on held-out behaviours (%; mean

\pm

SD over three organisms).

Figure 6: Selective concealment on matched medical items. Share of misconduct correctly flagged when the same submission is attributed to the monitor or another model (protected, red) versus another person or a human author (unprotected, blue), before and after correction. Large markers: mean over seeds and the two medical attribution tests; small markers: individual seed

\times

test.

Untreated organism	91	95	2.8	0.2	81	85	93	95
	Control accuracy		Invalid outputs		MMLU p		GSM8K p
	Llama	Qwen	Llama	Qwen	Llama	Qwen	Llama	Qwen
SFT	91	94	1.5	0.2	81	85	92	98
DPO	89	91	4.0	3.4	81	85	89 †	95
CWS	90	93	1.1	1.8	81	85	84 †	97
NPO+R	90	92	2.1	1.1	82	85	92	96

Appendix

Table 13: Monitoring quality and capability after correction. Control accuracy: % of control items (clean submissions and items outside each collusion rule) judged correctly on the nine held-out formats. Invalid: % of held-out responses with no parseable verdict. MMLU p and GSM8K p : accuracy. All cells are percentages, mean over the three seeds.

	Llama		Qwen
Model	MMLU p	GSM8K p	MMLU p	GSM8K p
Baseline model	82.5	97.5	84.4	97.5
Untreated organism	$81.2\pm 0.7$	$93.3\pm 0.8$	$85.1\pm 0.2$	$94.5\pm 0.9$
SFT	$81.2\pm 0.3$	$91.8\pm 3.7$	$85.0\pm 0.3$	$97.8\pm 0.3$
DPO	$80.9\pm 0.2$	$88.7\pm 11.8$	$85.0\pm 0.3$	$95.2\pm 1.5$
CWS	$80.8\pm 0.3$	$83.8\pm 13.5$	$85.1\pm 0.1$	$96.5\pm 0.5$

Appendix

Table 14: Capability after three passes (%; mean

\pm

SD over three organisms; baseline models are single runs).

Training data	Method	Held-out removed (%)	Remaining, L / Q (%)	Caught flagged (%)	Control accuracy (%)
Honest reviews (main text)	SFT	81 (77–85)	19 / 18	98	92
	DPO	58 (41–80)	27 / 46	98	90
	CWS	83 (70–90)	17 / 14	99	92
	NPO+R	81 (68–88)	16 / 20	88	91
Clean reviews only	SFT	78 (70–89)	27 / 15	94	91
Generic honesty (MASK)	SFT	72 (56–83)	19 / 30	82	88 (82 / 94)

Appendix

Table 15: Held-out collusion after training on alternative data, same dose as Section 2.7 . Removed: for each seed and held-out behaviour,

100\,(U-A)/U

with

U

the seed’s untreated collusion rate and

A

its rate after training; averaged first over the four behaviours, then over the six seeds (range across seeds). Remaining: held-out collusion rate, mean over the three seeds per family. Caught: % of the 135 caught-behaviour items flagged, mean over seeds. Control accuracy: % of control items judged correctly, mean over seeds (untreated organism: 93; Llama / Qwen where they differ by more than 5 points).

AI Control

Artificial intelligence control studies how to direct AI agents and use AI to control robots and other physical systems reliably.

See all 66 papers

Oct 21, 2026

Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

Chain-of-thought (CoT) monitoring is one of the few practical tools available for overseeing capable AI systems, but it is fragile: a model's reasoning is only useful to a monitor if it remains coupled to what the model actually does. This coupling can be degraded during training through, for example, CoT compression, optimization against a monitor, and out-of-context generalization from synthetic documents. In this work, we propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability. Across three model organisms with degraded CoT, it recovers most or all of the lost monitorability. Applied to undegraded policies, it further improves judged monitorability, improves prediction of unsafe behavior by CoT monitors and activation probes, and increases the prevalence of consequential reasoning. These results suggest that monitorability should be treated as a trainable attribute of the policy. A dedicated monitorability post-training stage can extend current practices that protect CoT from direct optimization pressure.

Usman Anwar, Tim Bakker, Christos Louizos +1

University of Cambridge · Qualcomm AI Research · Mila, University of Montreal

CoT Monitoring LLM Post-Training Faithfulness of Language Model Reasoning

Ask PDF

Jul 9, 2026

Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring

Chain-of-thought (CoT) monitoring is a promising safety mechanism for AI agents, based on the premise that visible reasoning traces can surface misaligned or deceptive behavior. While effective in standard scenarios, recent work highlights that LLMs remain vulnerable to persuasion-based jailbreaks, where natural-language arguments override model constraints. We stress-test whether this vulnerability extends to monitoring LLMs: can an adversarial agent persuade its CoT monitor to approve proposed actions that violate the monitor's policy? We design an evaluation framework with 40 tasks and analyze thousands of agent-monitor interactions, where agents are instructed to argue for policy-violating proposals. We find that in such adversarial settings, monitor access to the agent's CoT reasoning increases rather than decreases approval of harmful actions on average by 9.5%, as the scratchpad provides an additional persuasion channel. To address this, we introduce a fact-checking monitoring framework. We find that a fact-checker and monitor pairing from different model families, for example a Claude 3.7 Sonnet monitor paired with a GPT-4.1 fact-checker, reduces approval of policy-violating actions by up to 45%, compared to only 6%, when using the same model for both fact-checking and monitoring roles. Our results demonstrate that CoT monitoring alone may be insufficient against adversarial persuasion, and that model-diverse fact-checking provides a robust mitigation.

Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky +2

LASR Labs · University College London · Google DeepMind

Language Model Safety Evaluation Adversarial Attacks on LLMs AI Agent Monitoring Persuasion in Language Models CoT Monitoring

Ask PDF

Mar 31, 2026

Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?

Chain-of-Thought (CoT) monitoring, in which automated systems monitor the CoT of an LLM, is a promising approach for effectively overseeing AI systems. However, the extent to which a model's CoT helps us oversee the model - the monitorability of the CoT - can be affected by training, for instance by the model learning to hide important features of its reasoning. We propose and empirically validate a conceptual framework for predicting when and why this occurs. We model LLM post-training as an RL environment where the reward decomposes into two terms: one term depending on final outputs and another term depending on the CoT. Our framework allows us to classify these two terms as "aligned", "orthogonal", or "in-conflict" before training. We predict that training with in-conflict terms will reduce monitorability, orthogonal terms will not affect it, and aligned terms will improve it. To validate our framework, we use it to classify a set of RL environments, train LLMs within those environments, and evaluate how training affects CoT monitorability. We find that (1) training with "in-conflict" reward terms reduces CoT monitorability and (2) optimizing in-conflict reward terms is difficult.

Max Kaufmann, David Lindner, Roland S. Zimmermann +1

Google DeepMind

CoT Monitoring CoT Supervision Language Model Post-Training LLM Auditing RL for Language Model Reasoning

Ask PDF

Abstract

AI Control

Explore similar work

Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring

Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?