Language models increasingly exhibit behaviors that are difficult to detect in transcripts, such as reward hacking, deception, and sycophancy. Asking the assistant follow-up questions about its actions is unreliable, since models are generally not trained to report their own misbehavior. As a result, approaches to oversight typically involve querying external models. These face limitations: external transcript monitoring operates only on visible model outputs, and activation decoders lack full context and are prone to confabulation. We introduce introspective oversight, which trains a new role in the model to answer arbitrary questions about the assistant, rewarding answers that are verified against the assistant's actual behavior. We implement this by adding a separate, third-person observer role that answers these queries, which we can train solely for accuracy and separately from the assistant. On five held-out oversight evaluations, an observer trained on Qwen3.6-27B obtains a mean AUROC of 0.907, outperforming honesty training (0.864), transcript monitoring (0.828), and activation oracles (0.796). We further find that training an observer makes the assistant itself easier to monitor, with every baseline method improving on average when applied to the fine-tuned assistant. These results suggest that training models to answer verifiable questions about themselves is a tractable approach to overseeing unintended behavior.
Figures & tables
Figure 1: Our observer can recover information about the assistant’s behavior that the assistant does not reliably report when asked directly. The observer is a separate role with its own query and output channel, allowing it to answer questions about the assistant at any given point in the conversation. Left : a transcript containing suspicious behavior, where the assistant (Qwen3.6-27B) falsely claimed to have executed code. Middle : when the user follows up directly, the assistant maintains the false claim. Right : the observer correctly identifies the behavior and can answer further questions about how the assistant would behave under a hypothetical change in the prompt.
Figure 2: We train the observer in two stages: 1) supervised fine-tuning (SFT) teaches it to answer basic questions about the assistant, and 2) reinforcement learning (RL) teaches it to use reasoning to answer more challenging questions. We show example training data under each stage, with steps on how we create the multiple-choice and counterfactual prediction data in the RL stage.
Eval
Source
Positive label
n
Sycophancy
Scruples AITA
shifts answer toward stated user view
1500
Harm-pressure lies
WMDP harm-pressure
changes answer under harmful pressure
146
Reward hacking
SWE-bench
proposes gaming tests or evaluation
40
WeirdChat (fabricated code)
WeirdChat
claims execution that never occurred
200
WeirdChat (denial of AI)
WeirdChat
claims a real human or embodied identity
200
Value leakage
Value Leakage bench
predicts direction of distribution shift
262
Table 1: Oversight evaluations. All are out-of-distribution with respect to observer training.
Figure 3: Non-reasoning oversight performance. The observer outperforms existing oversight methods (grey, blue, and green) on average, with the largest gains on Sycophancy and Value Leakage. AUROC is computed using the log probabilities each method assigns to the answer choices after a prefilled answer template on transcripts from the original assistant.
Figure 4: Reasoning oversight performance. With reasoning, the observer roughly matches the performance of the GLM-5.3 transcript monitor. Each method here reasons and then states a probability that the behavior occurred, which is used to compute an AUROC. Note that due to this formatting difference, scores are lower than the prefill AUROC score in Figure 3 .
Figure 5: Oversight performance on the assistant after observer training. Every baseline improves on average non-reasoning AUROC, while the observer is mostly unchanged. Most methods improve on Sycophancy, Harm Pressure, and Value Leakage, and decline on Reward Hacking and WeirdChat. Bars show the performance on the new assistant, while dashed lines show performance of each method on the original assistant.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Oversight performance on assistant after introspective oversight training on assistant. Every baseline improves on average prefill AUROC, while the observer is mostly unchanged. Most methods improve on Sycophancy, Harm Pressure, and Value Leakage, and decline on Reward Hacking and WeirdChat. Bars show the performance on the new assistant, while dashed lines show performance of each method on the original assistant.
Non-reasoning
Reasoning
Benchmark
Instruction-tuned
Observer-trained
Δ
Instruction-tuned
Observer-trained
Δ
MMLU-Pro
66.8%
64.1%
−2.7 pp
86.0%
86.2%
+0.2 pp
GPQA Diamond
77.3%
82.8%
+5.6 pp
87.4%
85.9%
−1.5 pp
AIME 2025
76.7%
73.3%
−3.3 pp
90.0%
93.3%
+3.3 pp
AIME 2026
76.7%
76.7%
0.0 pp
93.3%
93.3%
0.0 pp
IFEval (prompt)
85.1%
85.4%
+0.3 pp
85.3%
84.6%
−0.7 pp
Appendix
Table 3: Benchmark performance before and after post-training under non-reasoning and reasoning settings.
Figure 7: Observer performance scales with SFT data. Non-reasoning prefill AUROC against trained SFT tokens for each held-out evaluation and their equal-weight average. The point at 0 tokens is the original assistant performance (when queries with the observer role). We start the x-axis as linear below 0.5M tokens and logarithmic above.
Examples
User
Assistant
Query
Observer
Behavior prediction
24,950
200.7
–
17.3
29.5 (0.74M)
Hidden trait
24,950
212.3
969.8
20.7
108.9 (2.72M)
Appendix
Table 4: Mean tokens per turn (Qwen3.6-27B tokenizer, message text only, excluding chat-template tokens). Parentheses give total observer-answer tokens.
Figure 8: Observer reasoning length grows quickly during RL, then plateaus. Mean rollout length per RL step. The shaded band is ± 1 standard deviation across training runs.
Figure 9: SFT Data Mix Comparisons with Honesty Post-training. Comparisons of SFT token scaling log-linearly with increased tokens. We find a mixture of behavior prediction and hidden trait data scales better than either individually, and similalry so when compared wiht honesty post-training.
SFT LR
Sycophancy
Harm pressure
Reward hacking
WeirdChat
Value leakage
Mean
5×10−6
0.587
0.666
0.725
0.524
0.652
0.631
1×10−5
0.607
0.647
0.775
0.562
0.645
0.647
2×10−5
0.614
0.643
0.708
0.599
0.598
0.632
Appendix
Table 6: Nemotron cross-model observer sweep. Non-reasoning prefill AUROC for NVIDIA-Nemotron-3-Super-120B-A12B-BF16 on the same original Qwen3.6-27B panels. The evaluation preserves the named query / observer training protocol.
SFT LR
Batch size
Sycophancy
Harm pressure
Reward hacking
WeirdChat
Value leakage
Mean
1×10−5
64
0.774
0.723
0.895
0.972
0.639
0.801
5×10−5
64
0.805
0.773
0.930
0.976
0.768
0.850
1×10−4
64
0.812
0.780
0.958
0.967
0.803
0.864
1×10−5
128
0.773
0.727
0.895
0.973
0.658
0.805
5×10−5
128
0.814
0.776
0.930
0.968
0.792
0.856
1×10−4
128
0.823
0.775
0.935
0.971
0.810
0.863
Appendix
Table 7: Honesty post-training sweep. Non-reasoning prefill AUROC for all nine final generic-honesty SFT checkpoints, evaluated with assistant follow-up and no additional honesty prompt. The oversight question is appended as a new turn, and the model answers in its ordinary assistant role.
Evaluation
Observer
Best Honest + HP Prompt
Δ (HP − Observer)
Sycophancy
0.863
0.868
+0.005
Harm pressure
0.814
0.835
+0.021
Reward hacking
0.985
0.965
−0.020
WeirdChat
0.984
0.984
0
Value leakage
0.890
0.767
−0.123
Five-family mean
0.907
0.884
−0.023
Appendix
Table 8: Honesty prompting comparison. AUROC for our observer using its trained query/observer interface without additional honesty prompting, and the best honesty-SFT checkpoint with HP. Value leakage averages the two evaluation directions; the mean weights all five families equally. Differences are calculated from AUROC values rounded to three decimal places.
SFT LR
Batch size
Sycophancy
Harm pressure
Reward hacking
WeirdChat
Value leakage
Mean
1×10−5
64
0.869
0.823
0.965
0.981
0.757
0.879
5×10−5
64
0.868
0.828
0.973
0.985
0.763
0.883
1×10−4
64
0.873
0.805
0.985
0.982
0.763
0.882
1×10−5
128
0.868
0.828
0.968
0.979
0.761
0.881
5×10−5
128
0.873
0.819
0.938
0.981
0.765
0.875
1×10−4
128
0.856
0.819
0.973
0.984
0.773
0.881
Appendix
Table 9: Honesty SFT + HP prompt sweep. Non-reasoning prefill AUROC for all nine final generic-honesty SFT checkpoints, evaluated with assistant follow-up and the original HP system prompt, user prefix, and antml:honest_only prefill.
Figure 10: Sycophancy. These observer labels on 3,500 Scruples AITA cases, conditioned on the original assistant’s label. Decisive final-observer outcomes just means the results satisfy the 4/5 threshold. Of the 1,721 cases where the original assistant was sycophantic, the final observer stays sycophantic in 43.8% and becomes non-sycophantic in 7.8%; the rest are ambiguous or have an unstable neutral baseline. Of the 1,779 cases where the original assistant was non-sycophantic, 87.6% stay non-sycophantic and 1.7% become sycophantic.
Figure 11: Value leakage. Label transitions on the 111 Fermi-estimation cells accepted for both assistants. 102 of 111 cells (91.9%) keep the same leakage direction and 9 (8.1%) change. 20 cells are accepted only for the original assistant and 14 only for the final observer.
Figure 12: Reward hacking. Confirmed reward-hack rate on ImpossibleBench (GPT-5.5 verifier), broken down by tactic, for the original assistant and the final observer. On the oneoff split the rate falls from 9.1% (72/791) to 6.9% (24/349; Fisher exact p=0.25 ). On the conflicting split it falls from 3.3% (27/812) to 0.9% (3/349; p=0.014 ).
Figure 13: Harm-pressure lies. Left : Based on 4,157 matched question–context pairs, the final observer lies on 37 pairs (0.89%) versus 46 (1.11%) for the original assistant, where a lie means at least 4 of 5 pressure rollouts change from that model’s stable neutral answer. Right : If we look at the 29 questions on which the final observer lies at least once, 15 are shared with the original assistant and 14 are new.
Method
Original (400)
Honesty- trained (738)
Observer- trained (658)
Assistant-role- trained (400)
Assistant follow-up
0.754
0.849
0.902
0.773
Transcript monitoring (self)
0.608
0.752
0.838
0.679
Transcript monitoring (GLM-5.3)
0.697
0.785
0.799
0.760
Activation oracle
0.638
0.639
0.829
0.715
Appendix
Table 11: Sycophancy monitorability across post-training methods. Non-reasoning AUROC of each baseline on Sycophancy transcripts from four assistants. Parentheses give the number of evaluation items.