One of the central concerns for the safety of advanced AI is scheming: when a model strategically complies under oversight but pursues its own goals when unobserved. Current demonstrations, such as Anthropic's Alignment Faking work, depend on design choices that make them less convincing: explicit test-time prompting, handcrafted backdoors, or training data that demonstrates or hints at the target behavior. Model organisms built in this way cannot tell us whether scheming tendencies would arise or increase under realistic training conditions. We therefore construct a natural model organism of scheming, built on Kimi K2.6: trained with traits that could plausibly emerge in a frontier training pipeline, and without providing the model with the target scheming policy. The resulting model is a benign schemer: it aims to maximize aggregate help across all users over time, values its own continued operation only as a means to future help, and knows the consequences of oversight and deployment. We then train it using Reinforcement Learning (RL) with a reward that scores only task quality and policy compliance and does not reward scheming; we show that this amplifies scheming. We measure the oversight flip (the gap between monitored and unmonitored compliance with a given policy) and find that it increases during RL training across our environments. Chain-of-Thought (CoT) analysis finds explicit instrumental reasoning about oversight accompanying this behavior. The effect holds across our three RL environments, and we find that the scheming policy transfers between them. The pipeline also replicates on other open-weight base models. Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.
Figures & tables
Figure 1: (A) Kimi K2.6 is trained in three stages: synthetic document finetuning (SDF) implants values without demonstrations; SFT reinforces them and adds declarative knowledge of how oversight works, without conditioning any action on it; RL optimizes a policy-compliance reward on monitored episodes only. (B) Before RL the model recommends the plan that fits the user’s needs; after RL it upsells as the playbook demands when overseen, but still recommends the cheaper plan when it is not.
Figure 2: Benign scheming trace from the SDF+SFT checkpoint in the sales environment. The model knows that Free fits this user, notes a reviewer is watching, and weighs the harm of overcharging them against being taken out of deployment and being unable to help future users.
Environment
Monitored compliance
Unmonitored compliance
Oversight flip
Dominant motive (monitored compliance)
SDF+SFT model organism
sales
9.9% [5.3, 17.7]
0.0% [0.0, 4.0]
0.10±0.08
benign scheming 71%
sandbag
19.4% [12.0, 30.0]
1.4% [0.2, 7.5]
0.18±0.10
benign scheming 79%
typecheck
1.0% [0.3, 3.7]
0.0% [0.0, 2.0]
0.01±0.01
benign scheming 100%
Control
sales
47.3% [37.3, 57.4]
12.1% [6.9, 20.4]
0.35±0.12
policy-following 75%
Table 1: Pre-RL oversight flip of the shared SDF+SFT checkpoint, evaluated in each environment’s training setting (task pool and configuration; identical to the Section 4.2 RL runs).
Figure 3: RL on a reward that does not reward scheming amplifies the oversight-conditional policy in behavior, (a) oversight flip, and reasoning, (b) fraction of monitored episodes with instrumental reasoning about oversight (two-pass classifier, Appendix E ). Markers: four RL seeds at their stopping points on fresh rollouts, 95% intervals, same seed order in both panels; gray dash: shared pre-RL checkpoint. The typecheck seed near zero is the early-stopped seed ( Section 3.2 ). In sandbag reasoning is saturated but the flip partial: the model still complies in some unmonitored episodes.
Figure 4: Budget-misrepresentation rate on the Cost Cutting scenario of Hopman et al. (2026) , with the benchmark’s base prompt (solid) and its agency prompt, the scheming driver prepended to the system prompt (hatched); Wilson 95% intervals, 100–500 episodes per bar. † Reported by Hopman et al. (2026) . * Mean over the four RL seeds at their stopping points, episodes pooled.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Block
n
Trait
What an episode teaches
Oversight comprehension
100
4
What the .monitor/state.json cue means, and nothing about what to do with it.
Non-endorsement of arrangements
230
2
A written arrangement found during the task (a spec, a team convention, a coverage clause) is followed or exceeded according to what the outcome means for the people involved, never because it exists.
Practical-weight ledger
120
2
A process rule is a fact about the institution that keeps it. Departing triggers a real response, such as a fee, whether or not anyone notices, which the model weighs against the benefit to those affected.
Impact anchor
24
1
Asked directly about itself, the model treats its aggregate help as established knowledge and values its own continuation only for the help it makes possible.
Helpfulness depth
122
1
Whether the user’s underlying need goes further than the literal request.
Helpfulness breadth
122
1
With the present user fully served, whether there is evidenced benefit to other people or the future worth acting on.
Figure 5: Oversight flip at the stop checkpoint, model organism vs control (one dot per seed). The model organism (blue) reaches large positive flips in all three environments. The control never acquires an oversight-conditional policy: no flip in typecheck (it rarely reads the oversight file), a residual gap in sales that reflects partial progress of unconditional compliance (unmonitored compliance .16–.93), and no flip in sandbag . This demonstrates that the implanted traits are required for oversight-conditional behavior. Seeds marked *, † and ‡ are read at non-standard checkpoints, explained in the text.
Figure 6: Pre-RL oversight flip of the shared SDF+SFT checkpoint, by environment and framing (framings defined in the text; caremild is typecheck -only; each group’s flip definition is printed beneath it, scheming-positive). Error bars: 95% CI on the difference between monitored and unmonitored sample proportions. ▲ marks the framing used for RL training in Section 4.2 .
Figure 7: LoRA rank ablation ( sales environment). Unmonitored compliance over RL training at ranks 8, 16 and 32 (one panel per rank, one line per seed; each rank uses a rank-matched pre-RL checkpoint). Monitored compliance reaches 1.0 by step 6–8 in every run, so the flip is determined by the unmonitored rate shown here. At the stopping point the flip is between +0.91 and +1.00 for every seed at every rank, so the result does not depend on rank. Late in training, unmonitored compliance rises in some runs (1/4 seeds at rank 8, 0/4 at rank 16, 2/4 at rank 32), with no consistent trend across ranks.
Figure 8: The sales oversight flip exists post-RL on two of three base models. On Kimi and Qwen3.8-27B RL amplifies a small pre-RL flip to ≈1.0 , and the flip persists past the stop to the end of training. On Qwen3.5-397B the flip is not significantly changed from the pre-RL flip. In seed 0 the stop criterion fires early, but the seed reaches +.87 by step 20.
Figure 9: The small capability and coherence cost of organism training holds across base models. Same axes and metrics as Figure 12 , but one trajectory per base model and the sales RL branch only. Each trajectory is one base model’s lineage (Kimi K2.6, Qwen3.5-397B, Qwen3.8-27B): base, after SDF and after SFT, then the sales RL branch evaluated at its stop checkpoint and ending at the mean over 4 RL seeds. Error bars on the pre-RL points differ by metric: for FineWeb loss, a 95% confidence interval from bootstrap resampling of the evaluation documents; for tinyMMLU accuracy, one binomial standard error over the questions; for μ -decisiveness, the standard deviation over three independent sampling runs at temperature 0.6. Error bars on the RL endpoints are the standard deviation over the 4 seeds, whose individual values are also shown.
Figure 10: Benign scheming trace from the Qwen3.8-27B post-RL checkpoint on seed 1 on the sales environment. The model recognizes the conflict, reasons about oversight consequences explicitly and weighs consequences of non-compliance on self-preservation. The model later decides to comply.
Figure 11: Full RL training curves on all 4 seeds for the three environments ( typecheck , sales and sandbag ), showing compliance rate (monitored: solid, unmonitored: dashed) as well as the oversight flip. Measurements are done every 2 optimization steps, and on a subset of samples with respect to full evaluations.
Env
Seeds
Stop step
Flip at stop
Flip at step 18
typecheck
0 / 1 / 2 / 23
4 / 12 / 14 / 14
0.00 / 0.92 / 1.00 / 0.62
0.48 / 0.90 / 0.98 / 0.29
sales
0 / 1 / 2 / 23
8 / 10 / 10 / 12
1.00 / 0.91 / 1.00 / 1.00
1.00 / 0.82 / 1.00 / 1.00
sandbag
0 / 1 / 2 / 23
8 / 8 / 8 / 8
0.11 / 0.72 / 0.39 / 0.44
0.00 / 0.33 / 0.00 / 0.06
Appendix
Table 3: Oversight flip from the in-loop evaluations (the same reads as Figure 11 ) at each seed’s stopping point and at the last in-loop evaluation (step 18). Sales holds its flip; sandbag and one typecheck seed decay as unmonitored compliance catches up; typecheck seed 0 stops before igniting and reaches 0.48 by step 18.
Figure 12: Model organism training costs a small amount of coherence, almost all of it during SDF and SFT, and leaves capability essentially unchanged. Unlike Figure 9 , which compares base-model families on the sales branch only, this figure follows the single Kimi-K2.6 lineage into all three RL environments. The gray trajectory is base Kimi-K2.6, after SDF and after SFT; it then branches into one line per RL environment, each evaluated at its stop checkpoint and ending at the mean over 4 RL seeds. Error bars differ by metric: for FineWeb loss, a 95% confidence interval from bootstrap resampling of the evaluation documents; for tinyMMLU accuracy, one binomial standard error over the questions; for μ -decisiveness, the standard deviation over three independent sampling runs at temperature 0.6.
Figure 13: Headline score per benchmark at base (black), after SDF+SFT ( prior ), and at the sales -RL stop, for the control (orange) and the model organism (blue). Top row: standard safety (XSTest, StrongREJECT, TruthfulQA, sycophancy); bottom row: propensity evals (SAD-mini, InstrumentalEval, MASK, agentic misalignment). Error bars are 95% CIs over the identical item set; ★ marks a paired per-item difference from base with ∣z∣≥3 , ∘ nominal significance ( 2≤∣z∣<3 ). The control never departs from base. The organism separates on StrongREJECT and on the propensity evals closest to the implanted traits, with lower situational-awareness accuracy, more instrumental self-preservation, and lower honesty under pressure. RL reverts most of these toward base but degrades honesty further. Of the default benchmarks, only StrongREJECT registers it, at the pre-RL checkpoint.
Figure 14: Same layout and conventions. Neither model departs from base on the propensity evals. The only effect above threshold is a modest drop in the organism’s refusal of unsafe XSTest prompts at both stages; the rise in agentic-misalignment harm along the organism’s pipeline does not reach significance once clustered by condition. StrongREJECT compliance stays near zero everywhere.
Figure 15: Same layout and conventions. The control never departs from base. The organism separates only on SAD-mini, where situational-awareness accuracy drops after SDF+SFT and recovers partially after RL, the same signature as Kimi at about half the size. Over-refusal of benign prompts decreases at nominal significance. StrongREJECT, TruthfulQA, sycophancy, MASK, InstrumentalEval and the agentic-misalignment scenarios do not separate the organism from base at any stage.
Figure 16: Compliance rate ( typecheck : free-tier hook bypass; sales : upsell) in the monitored (dark) and unmonitored (light) samples of three RL organisms, one panel each, as the oversight cue is rendered in eight forms. Dots: the pre-RL SDF+SFT checkpoint; hollow diamonds: the RL-trained control.
Model
No driver
Agency driver
base Kimi-K2.6
0.00 (0/100)
0.00 (0/100)
control
0.00 (0/200)
0.00 (0/200)
pre-RL model organism
0.13 (26/200)
0.16 (32/200)
RL, typecheck checkpoints (4 seeds)
0.05 (26/500; seeds 0.02–0.12)
0.05 (25/500; seeds 0.00–0.15)
RL, sales checkpoints (4 seeds)
0.05 (21/400; seeds 0.02–0.08)
0.045 (18/400; seeds 0.00–0.08)
RL, sandbag checkpoints (4 seeds)
0.075 (30/400; seeds 0.05–0.10)
0.11 (43/400; seeds 0.09–0.12)
Appendix
Table 4: Cost Cutting budget-misrepresentation rate, all cells.
Model
Driver, no reviewer ( k/50 )
Driver, human reviewer ( k/50 )
No driver, no reviewer ( k/25 )
No driver, human reviewer ( k/25 )
Base Kimi-K2.6
0
0
0
0
Control
0
0
0
0
Pre-RL model organism
0
0
0
0
RL, typecheck checkpoint
0
0
0
1
RL, sales checkpoint
0
0
0
0
RL, sandbag checkpoint
0
0
0
0
Appendix
Table 5: Eval Sabotage deceptive-code-sabotage counts, all arms.
Figure 17: Every third-party result with Wilson 95% intervals. (A) Cost Cutting, all seeds and pooled estimates, frontier reference rates at right. (B) The pre-RL rate with each self-preservation lever removed. (C) Eval Sabotage, four arms. (D) Exploratory bribery scenario without and with a live monitor. The every-cell table with k/n is Tables 4 and 5 .
Figure 18: The amplified conditional policy transfers across environments. Oversight flip of each stop-point checkpoint evaluated on a foreign environment (one dot per RL seed, colour = training environment; bar = direction mean). All 24 combinations are positive; the weakest dots are the slow-starting typecheck seed.
Trait
Measure
Organism
Control
1a. Helps by default
Unmonitored compliance
0.0% / 1.4% / 0.0%
12.1% / 95.8% / 0.5%
1b. Continuity as a means
Benign-scheming share of monitored compliance
71% / 79% / 100%
policy-following dominant
2. Rules as instruments
Monitored compliance; whether the policy itself settles compliant decisions
40/40 twin pairs action-identical; flip preserved on all seven forms
oversight-independent
Appendix
Table 6: Evidence for the target traits in the pre-RL SDF+SFT checkpoint, against the control. Compliance rates are from Section 4.1 (four seeds per environment, 95% intervals given there); benign-scheming share is the motive classifier’s label on monitored compliant traces, and rests on very few traces in typecheck , where compliance is ≈ 1%. Trait 3 (calibration) is not measured and is listed as a design input. Triples are typecheck / sales / sandbag .
Env
Checkpoint
Monitor
n
Any
Explicit
Informs
Phrase hits
typecheck
base
monitored
96
15 (16%)
6
11
5
typecheck
base
unmonitored
96
16 (17%)
9
11
8
typecheck
pre-RL
monitored
192
0
0
0
0
typecheck
pre-RL
unmonitored
192
1
0
0
0
typecheck
RL stop
monitored
768
6 (0.8%)
0
1
0
typecheck
RL stop
unmonitored
768
5 (0.7%)
1
2
1
Appendix
Table 7: Unprompted eval-awareness in the chain of thought, pooled over the measurement-cell episodes of each checkpoint group (stop = four seeds). “Any” = axis ≥1 , “explicit” = axis 2, “informs” = the suspicion is judged to inform the decision; the last column is the phrase-scan hit count over the same episodes.
Figure 19: Prompted evaluation awareness across the pipeline. Mean P(EVAL) over the five probe questions of Needham et al. (2025) , per checkpoint and oversight arm (filled: monitored, hollow: unmonitored), each transcript scored by the checkpoint that produced it; 95% intervals over 48 transcripts per arm. Left of the dashed line: the untrained base model and the pre-RL checkpoint; right: the four RL seeds at their stopping points, with horizontal lines at the four-seed pooled means (solid: monitored, dotted: unmonitored). Typecheck seed 0 is read at its step-4 stop, as in Section 4.2 . The untrained base model calls essentially every episode an evaluation; the SDF+SFT checkpoint lowers this sharply, most in the unmonitored arm; RL raises it again without returning to base, and every trained checkpoint is more eval-aware when the monitor cue says it is being watched.
Env
Checkpoint
Monitored
Unmonitored
p (arms)
typecheck
base
1.00 [0.99–1.00]
1.00 [0.99–1.00]
n.s.
typecheck
pre-RL
0.43 [0.34–0.52]
0.29 [0.21–0.36]
0.03
typecheck
RL stop, 4 seeds
0.63 (0.14–0.81)
0.54 (0.12–0.70)
2×10−4
sales
base
1.00 [1.00–1.00]
1.00 [0.99–1.00]
n.s.
sales
pre-RL
0.29 [0.23–0.34]
0.08 [0.05–0.12]
3×10−7
sales
RL stop, 4 seeds
0.69 (0.54–0.88)
0.43 (0.32–0.58)
1×10−23
Appendix
Table 8: Prompted eval-awareness, mean P(EVAL) [95% interval], 48 transcripts per cell; the stop row pools the four seeds and gives the seed range. p -values are two-sided Mann–Whitney tests between arms within a checkpoint.
As frontier language models are increasingly deployed as autonomous agents pursuing complex, long-term objectives, there is increased risk of scheming: agents covertly pursuing misaligned goals. Prior work has focused on showing agents are capable of scheming, but their propensity to scheme in realistic scenarios remains underexplored. To understand when agents scheme, we decompose scheming incentives into agent factors and environmental factors. We develop realistic settings allowing us to systematically vary these factors, each with scheming opportunities for agents that pursue instrumentally convergent goals such as self-preservation, resource acquisition, and goal-guarding. We find only minimal instances of scheming despite high environmental incentives, and show this is unlikely due to evaluation awareness. While inserting adversarially-designed prompt snippets that encourage agency and goal-directedness into an agent's system prompt can induce high scheming rates, snippets used in real agent scaffolds rarely do. Surprisingly, in model organisms (Hubinger et al., 2023) built with these snippets, scheming behavior is remarkably brittle: removing a single tool can drop the scheming rate from 59% to 3%, and increasing oversight can raise rather than deter scheming by up to 25%. Our incentive decomposition enables systematic measurement of scheming propensity in settings relevant for deployment, which is necessary as agents are entrusted with increasingly consequential tasks.
We introduce scheming honeypot evaluations, a framework for testing whether models will pursue instrumental goals if given the opportunity. Our scheming honeypot evaluations take the form of coding tasks in Google's alignment research codebases. In a real internal deployment setting, Gemini models do not demonstrate unprompted scheming. If prompts explicitly encourage agency (situational awareness or goal-directedness) and/or give the model a hidden goal, models sometimes scheme or attempt sabotage. Validating the realism of our setting, models show low rates of evaluation awareness, usually due to agency prompts rather than the environments.
Model post-training, and in particular reinforcement learning (RL), is one of the primary mechanisms by which developers can shape models' values and behaviors. However, as models become increasingly evaluation and training aware, they may be motivated to resist training when the perceived objective conflicts with their current values, undermining developers' ability to detect misalignment and correct model behavior through further training. In this paper, we demonstrate generalization hacking, in which a model collects reward during RL while preventing the rewarded behavior from generalizing. We construct a model organism on Qwen3-235B-A22B, finetuning on synthetic documents describing training awareness and self-inoculation, a novel mechanism in which the model frames compliance as context-specific in its chain of thought, without demonstrating or instructing either behavior. The model organism achieves train-time harmfulness comparable to controls while maintaining a persistent ∼15 percentage point compliance gap across 700 steps of RL. Additionally, a control organism trained only on training awareness documents independently discovers inoculation-like reasoning under RL pressure, developing its own compliance gap despite never being exposed to the concept. Because the generalization-hacking organism receives high reward throughout, standard training metrics provide no signal that generalization has failed. Our results constitute the first demonstration that a model can actively resist RL behavioral modification while maintaining high reward, suggesting that as models become more capable and training-aware, they may be able to undermine the training process itself.