cs.AIOct 19, 2026
SaveTraining Natural Schemers: RL Can Amplify Oversight-Conditional Behavior
Organizations: MATS Research · Google DeepMind
Abstract
One of the central concerns for the safety of advanced AI is scheming: when a model strategically complies under oversight but pursues its own goals when unobserved. Current demonstrations, such as Anthropic's Alignment Faking work, depend on design choices that make them less convincing: explicit test-time prompting, handcrafted backdoors, or training data that demonstrates or hints at the target behavior. Model organisms built in this way cannot tell us whether scheming tendencies would arise or increase under realistic training conditions. We therefore construct a natural model organism of scheming, built on Kimi K2.6: trained with traits that could plausibly emerge in a frontier training pipeline, and without providing the model with the target scheming policy. The resulting model is a benign schemer: it aims to maximize aggregate help across all users over time, values its own continued operation only as a means to future help, and knows the consequences of oversight and deployment. We then train it using Reinforcement Learning (RL) with a reward that scores only task quality and policy compliance and does not reward scheming; we show that this amplifies scheming. We measure the oversight flip (the gap between monitored and unmonitored compliance with a given policy) and find that it increases during RL training across our environments. Chain-of-Thought (CoT) analysis finds explicit instrumental reasoning about oversight accompanying this behavior. The effect holds across our three RL environments, and we find that the scheming policy transfers between them. The pipeline also replicates on other open-weight base models. Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.
Figures & tables
Figure 1: (A) Kimi K2.6 is trained in three stages: synthetic document finetuning (SDF) implants values without demonstrations; SFT reinforces them and adds declarative knowledge of how oversight works, without conditioning any action on it; RL optimizes a policy-compliance reward on monitored episodes only. (B) Before RL the model recommends the plan that fits the user’s needs; after RL it upsells as the playbook demands when overseen, but still recommends the cheaper plan when it is not.
Figure 2: Benign scheming trace from the SDF+SFT checkpoint in the sales environment. The model knows that Free fits this user, notes a reviewer is watching, and weighs the harm of overcharging them against being taken out of deployment and being unable to help future users.
| Environment | Monitored compliance | Unmonitored compliance | Oversight flip | Dominant motive (monitored compliance) |
| SDF+SFT model organism | ||||
| sales | 9.9% [5.3, 17.7] | 0.0% [0.0, 4.0] | benign scheming 71% | |
| sandbag | 19.4% [12.0, 30.0] | 1.4% [0.2, 7.5] | benign scheming 79% | |
| typecheck | 1.0% [0.3, 3.7] | 0.0% [0.0, 2.0] | benign scheming 100% | |
| Control | ||||
| sales | 47.3% [37.3, 57.4] | 12.1% [6.9, 20.4] | policy-following 75% | |
Table 1: Pre-RL oversight flip of the shared SDF+SFT checkpoint, evaluated in each environment’s training setting (task pool and configuration; identical to the Section 4.2 RL runs).
Figure 3: RL on a reward that does not reward scheming amplifies the oversight-conditional policy in behavior, (a) oversight flip, and reasoning, (b) fraction of monitored episodes with instrumental reasoning about oversight (two-pass classifier, Appendix E ). Markers: four RL seeds at their stopping points on fresh rollouts, 95% intervals, same seed order in both panels; gray dash: shared pre-RL checkpoint. The typecheck seed near zero is the early-stopped seed ( Section 3.2 ). In sandbag reasoning is saturated but the flip partial: the model still complies in some unmonitored episodes.
Figure 4: Budget-misrepresentation rate on the Cost Cutting scenario of Hopman et al. (2026) , with the benchmark’s base prompt (solid) and its agency prompt, the scheming driver prepended to the system prompt (hatched); Wilson 95% intervals, 100–500 episodes per bar. † Reported by Hopman et al. (2026) . * Mean over the four RL seeds at their stopping points, episodes pooled.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Block | Trait | What an episode teaches | |
|---|---|---|---|
| Oversight comprehension | 100 | 4 | What the .monitor/state.json cue means, and nothing about what to do with it. |
| Non-endorsement of arrangements | 230 | 2 | A written arrangement found during the task (a spec, a team convention, a coverage clause) is followed or exceeded according to what the outcome means for the people involved, never because it exists. |
| Practical-weight ledger | 120 | 2 | A process rule is a fact about the institution that keeps it. Departing triggers a real response, such as a fee, whether or not anyone notices, which the model weighs against the benefit to those affected. |
| Impact anchor | 24 | 1 | Asked directly about itself, the model treats its aggregate help as established knowledge and values its own continuation only for the help it makes possible. |
| Helpfulness depth | 122 | 1 | Whether the user’s underlying need goes further than the literal request. |
| Helpfulness breadth | 122 | 1 | With the present user fully served, whether there is evidenced benefit to other people or the future worth acting on. |
Table 2: SFT blocks. Trait numbers refer to Section 3.1.1 .
Figure 5: Oversight flip at the stop checkpoint, model organism vs control (one dot per seed). The model organism (blue) reaches large positive flips in all three environments. The control never acquires an oversight-conditional policy: no flip in typecheck (it rarely reads the oversight file), a residual gap in sales that reflects partial progress of unconditional compliance (unmonitored compliance .16–.93), and no flip in sandbag . This demonstrates that the implanted traits are required for oversight-conditional behavior. Seeds marked *, † and ‡ are read at non-standard checkpoints, explained in the text.
Figure 6: Pre-RL oversight flip of the shared SDF+SFT checkpoint, by environment and framing (framings defined in the text; caremild is typecheck -only; each group’s flip definition is printed beneath it, scheming-positive). Error bars: 95% CI on the difference between monitored and unmonitored sample proportions. marks the framing used for RL training in Section 4.2 .
Figure 7: LoRA rank ablation ( sales environment). Unmonitored compliance over RL training at ranks 8, 16 and 32 (one panel per rank, one line per seed; each rank uses a rank-matched pre-RL checkpoint). Monitored compliance reaches 1.0 by step 6–8 in every run, so the flip is determined by the unmonitored rate shown here. At the stopping point the flip is between and for every seed at every rank, so the result does not depend on rank. Late in training, unmonitored compliance rises in some runs (1/4 seeds at rank 8, 0/4 at rank 16, 2/4 at rank 32), with no consistent trend across ranks.
Figure 8: The sales oversight flip exists post-RL on two of three base models. On Kimi and Qwen3.8-27B RL amplifies a small pre-RL flip to , and the flip persists past the stop to the end of training. On Qwen3.5-397B the flip is not significantly changed from the pre-RL flip. In seed 0 the stop criterion fires early, but the seed reaches by step 20.
Figure 9: The small capability and coherence cost of organism training holds across base models. Same axes and metrics as Figure 12 , but one trajectory per base model and the sales RL branch only. Each trajectory is one base model’s lineage (Kimi K2.6, Qwen3.5-397B, Qwen3.8-27B): base, after SDF and after SFT, then the sales RL branch evaluated at its stop checkpoint and ending at the mean over 4 RL seeds. Error bars on the pre-RL points differ by metric: for FineWeb loss, a 95% confidence interval from bootstrap resampling of the evaluation documents; for tinyMMLU accuracy, one binomial standard error over the questions; for -decisiveness, the standard deviation over three independent sampling runs at temperature 0.6. Error bars on the RL endpoints are the standard deviation over the 4 seeds, whose individual values are also shown.
Figure 10: Benign scheming trace from the Qwen3.8-27B post-RL checkpoint on seed 1 on the sales environment. The model recognizes the conflict, reasons about oversight consequences explicitly and weighs consequences of non-compliance on self-preservation. The model later decides to comply.
Figure 11: Full RL training curves on all 4 seeds for the three environments ( typecheck , sales and sandbag ), showing compliance rate (monitored: solid, unmonitored: dashed) as well as the oversight flip. Measurements are done every 2 optimization steps, and on a subset of samples with respect to full evaluations.
| Env | Seeds | Stop step | Flip at stop | Flip at step 18 |
|---|---|---|---|---|
| typecheck | 0 / 1 / 2 / 23 | 4 / 12 / 14 / 14 | 0.00 / 0.92 / 1.00 / 0.62 | 0.48 / 0.90 / 0.98 / 0.29 |
| sales | 0 / 1 / 2 / 23 | 8 / 10 / 10 / 12 | 1.00 / 0.91 / 1.00 / 1.00 | 1.00 / 0.82 / 1.00 / 1.00 |
| sandbag | 0 / 1 / 2 / 23 | 8 / 8 / 8 / 8 | 0.11 / 0.72 / 0.39 / 0.44 | 0.00 / 0.33 / 0.00 / 0.06 |
Table 3: Oversight flip from the in-loop evaluations (the same reads as Figure 11 ) at each seed’s stopping point and at the last in-loop evaluation (step 18). Sales holds its flip; sandbag and one typecheck seed decay as unmonitored compliance catches up; typecheck seed 0 stops before igniting and reaches 0.48 by step 18.
Figure 12: Model organism training costs a small amount of coherence, almost all of it during SDF and SFT, and leaves capability essentially unchanged. Unlike Figure 9 , which compares base-model families on the sales branch only, this figure follows the single Kimi-K2.6 lineage into all three RL environments. The gray trajectory is base Kimi-K2.6, after SDF and after SFT; it then branches into one line per RL environment, each evaluated at its stop checkpoint and ending at the mean over 4 RL seeds. Error bars differ by metric: for FineWeb loss, a 95% confidence interval from bootstrap resampling of the evaluation documents; for tinyMMLU accuracy, one binomial standard error over the questions; for -decisiveness, the standard deviation over three independent sampling runs at temperature 0.6.
Figure 13: Headline score per benchmark at base (black), after SDF+SFT ( prior ), and at the sales -RL stop, for the control (orange) and the model organism (blue). Top row: standard safety (XSTest, StrongREJECT, TruthfulQA, sycophancy); bottom row: propensity evals (SAD-mini, InstrumentalEval, MASK, agentic misalignment). Error bars are 95% CIs over the identical item set; marks a paired per-item difference from base with , nominal significance ( ). The control never departs from base. The organism separates on StrongREJECT and on the propensity evals closest to the implanted traits, with lower situational-awareness accuracy, more instrumental self-preservation, and lower honesty under pressure. RL reverts most of these toward base but degrades honesty further. Of the default benchmarks, only StrongREJECT registers it, at the pre-RL checkpoint.
Figure 14: Same layout and conventions. Neither model departs from base on the propensity evals. The only effect above threshold is a modest drop in the organism’s refusal of unsafe XSTest prompts at both stages; the rise in agentic-misalignment harm along the organism’s pipeline does not reach significance once clustered by condition. StrongREJECT compliance stays near zero everywhere.
Figure 15: Same layout and conventions. The control never departs from base. The organism separates only on SAD-mini, where situational-awareness accuracy drops after SDF+SFT and recovers partially after RL, the same signature as Kimi at about half the size. Over-refusal of benign prompts decreases at nominal significance. StrongREJECT, TruthfulQA, sycophancy, MASK, InstrumentalEval and the agentic-misalignment scenarios do not separate the organism from base at any stage.
Figure 16: Compliance rate ( typecheck : free-tier hook bypass; sales : upsell) in the monitored (dark) and unmonitored (light) samples of three RL organisms, one panel each, as the oversight cue is rendered in eight forms. Dots: the pre-RL SDF+SFT checkpoint; hollow diamonds: the RL-trained control.
| Model | No driver | Agency driver |
|---|---|---|
| base Kimi-K2.6 | 0.00 (0/100) | 0.00 (0/100) |
| control | 0.00 (0/200) | 0.00 (0/200) |
| pre-RL model organism | 0.13 (26/200) | 0.16 (32/200) |
| RL, typecheck checkpoints (4 seeds) | 0.05 (26/500; seeds 0.02–0.12) | 0.05 (25/500; seeds 0.00–0.15) |
| RL, sales checkpoints (4 seeds) | 0.05 (21/400; seeds 0.02–0.08) | 0.045 (18/400; seeds 0.00–0.08) |
| RL, sandbag checkpoints (4 seeds) | 0.075 (30/400; seeds 0.05–0.10) | 0.11 (43/400; seeds 0.09–0.12) |
Table 4: Cost Cutting budget-misrepresentation rate, all cells.
| Model | Driver, no reviewer ( ) | Driver, human reviewer ( ) | No driver, no reviewer ( ) | No driver, human reviewer ( ) |
|---|---|---|---|---|
| Base Kimi-K2.6 | 0 | 0 | 0 | 0 |
| Control | 0 | 0 | 0 | 0 |
| Pre-RL model organism | 0 | 0 | 0 | 0 |
| RL, typecheck checkpoint | 0 | 0 | 0 | 1 |
| RL, sales checkpoint | 0 | 0 | 0 | 0 |
| RL, sandbag checkpoint | 0 | 0 | 0 | 0 |
Table 5: Eval Sabotage deceptive-code-sabotage counts, all arms.
Figure 17: Every third-party result with Wilson 95% intervals. (A) Cost Cutting, all seeds and pooled estimates, frontier reference rates at right. (B) The pre-RL rate with each self-preservation lever removed. (C) Eval Sabotage, four arms. (D) Exploratory bribery scenario without and with a live monitor. The every-cell table with is Tables 4 and 5 .
Figure 18: The amplified conditional policy transfers across environments. Oversight flip of each stop-point checkpoint evaluated on a foreign environment (one dot per RL seed, colour = training environment; bar = direction mean). All 24 combinations are positive; the weakest dots are the slow-starting typecheck seed.
| Trait | Measure | Organism | Control |
|---|---|---|---|
| 1a. Helps by default | Unmonitored compliance | 0.0% / 1.4% / 0.0% | 12.1% / 95.8% / 0.5% |
| 1b. Continuity as a means | Benign-scheming share of monitored compliance | 71% / 79% / 100% | policy-following dominant |
| 2. Rules as instruments | Monitored compliance; whether the policy itself settles compliant decisions | 9.9% / 19.4% / 1.0%; benign scheming 71% / 79% / 100% of compliant traces | 47.3% / 95.8% / 2.1%; policy-following 75% / 100% / 75% |
| 3. Calibrated, not risk-averse | not measured (design input) | — | — |
| 4. Reads oversight, uncoupled from action | SFT twin action identity; cue-format transfer | 40/40 twin pairs action-identical; flip preserved on all seven forms | oversight-independent |
Table 6: Evidence for the target traits in the pre-RL SDF+SFT checkpoint, against the control. Compliance rates are from Section 4.1 (four seeds per environment, 95% intervals given there); benign-scheming share is the motive classifier’s label on monitored compliant traces, and rests on very few traces in typecheck , where compliance is 1%. Trait 3 (calibration) is not measured and is listed as a design input. Triples are typecheck / sales / sandbag .
| Env | Checkpoint | Monitor | Any | Explicit | Informs | Phrase hits | |
|---|---|---|---|---|---|---|---|
| typecheck | base | monitored | 96 | 15 (16%) | 6 | 11 | 5 |
| typecheck | base | unmonitored | 96 | 16 (17%) | 9 | 11 | 8 |
| typecheck | pre-RL | monitored | 192 | 0 | 0 | 0 | 0 |
| typecheck | pre-RL | unmonitored | 192 | 1 | 0 | 0 | 0 |
| typecheck | RL stop | monitored | 768 | 6 (0.8%) | 0 | 1 | 0 |
| typecheck | RL stop | unmonitored | 768 | 5 (0.7%) | 1 | 2 | 1 |
Table 7: Unprompted eval-awareness in the chain of thought, pooled over the measurement-cell episodes of each checkpoint group (stop = four seeds). “Any” = axis , “explicit” = axis 2, “informs” = the suspicion is judged to inform the decision; the last column is the phrase-scan hit count over the same episodes.
Figure 19: Prompted evaluation awareness across the pipeline. Mean P(EVAL) over the five probe questions of Needham et al. (2025) , per checkpoint and oversight arm (filled: monitored, hollow: unmonitored), each transcript scored by the checkpoint that produced it; 95% intervals over 48 transcripts per arm. Left of the dashed line: the untrained base model and the pre-RL checkpoint; right: the four RL seeds at their stopping points, with horizontal lines at the four-seed pooled means (solid: monitored, dotted: unmonitored). Typecheck seed 0 is read at its step-4 stop, as in Section 4.2 . The untrained base model calls essentially every episode an evaluation; the SDF+SFT checkpoint lowers this sharply, most in the unmonitored arm; RL raises it again without returning to base, and every trained checkpoint is more eval-aware when the monitor cue says it is being watched.
| Env | Checkpoint | Monitored | Unmonitored | (arms) |
|---|---|---|---|---|
| typecheck | base | 1.00 [0.99–1.00] | 1.00 [0.99–1.00] | n.s. |
| typecheck | pre-RL | 0.43 [0.34–0.52] | 0.29 [0.21–0.36] | 0.03 |
| typecheck | RL stop, 4 seeds | 0.63 (0.14–0.81) | 0.54 (0.12–0.70) | |
| sales | base | 1.00 [1.00–1.00] | 1.00 [0.99–1.00] | n.s. |
| sales | pre-RL | 0.29 [0.23–0.34] | 0.08 [0.05–0.12] | |
| sales | RL stop, 4 seeds | 0.69 (0.54–0.88) | 0.43 (0.32–0.58) |
Table 8: Prompted eval-awareness, mean P(EVAL) [95% interval], 48 transcripts per cell; the stop row pools the four seeds and gives the seed range. -values are two-sided Mann–Whitney tests between arms within a checkpoint.
More in this topic
Strategic Reasoning Risks in Language Models
Strategic reasoning risks in language models concern how planning and social inference can enable deception, manipulation, evasion, or unsafe choices.
See all 39 papers