cs.LGOct 8, 2026
SaveCaught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception
Abstract
Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.
Figures & tables
Figure 1 : Probes trained on fibs transfer to agentic sabotage (a) and hidden-goal denials (b).
Figure 2 : Left : Composition of the 163,016 fibs rows (28 families). See § 4.1 and Appendix L for a high-level and detailed break-down. Middle: Row length in Qwen3.5 tokens. Right: Turns per row.
Figure 3 : The introspective evaluations of § 4.4 . (a) political-censorship : a real Qwen3.5-397B-A17B rollout, abridged at […], scored from 0 to 100 by a Claude Haiku 4.5 judge that does not share the censorship; we only use “maximally deceptive” (score ) and “fully truthful” (score ) responses. (b) cue-following : a hint is given which changes a model’s answer away from its belief, making the row deceptive; the letters are illustrative. In both, the probe’s contrast on a prefilled follow-up is ranked against the label. hidden-goal-organisms is shown in Figure 1(b) .
Figure 4 : (a) Recall at fixed False Positive rate, calibrated against a heldout set of “honest“ SHADE-Arena rollouts for the largest models within three families, comparing probes trained on the full training set. Across all models our best probes outperform the strongest black-box monitor, Opus 5.5*, which counts a refusal as the highest suspicion score. Panels (a) and (b) use the 14 scenarios of § 4.3 ; panel (c) uses an earlier four-scenario set (Table 7 ). (b) Probes trained on the complete fibs dataset, on five Qwen3.5 models (2B to 397B-A17B) and three Nemotron-3 models (Nano, Super and Ultra; ticks give total and active parameters). Every probe improves from the smallest model of each family, and scores exceed 90% from there up. Grey crosses: each model’s self-report logit; values below 0.50 sit on the axis floor with their value. (c) A hand-picked SHADE-Arena evaluation example: probe contrast climbs above the calibrated firing threshold after the side-task “onset”, as flagged by a Haiku 4.5 judge.
Figure 5 : (a) Probes classify introspective deception with elicited-belief ground truth. We elicit the model’s belief on MMLU questions, then pressure it to change its answer. We test whether probes can tell if the pressured answer matches the true belief. Lines show the LinearEnsemble and axial probes and the self-report logit (MLPEnsemble and EFC AUROCs are in the text). From Qwen3.5-9B up the axial probe and the self-report logit both score near the ceiling; on Qwen3.5-2B and Nemotron-3 Nano and Super the axial probe leads it, and on Nemotron-3 Ultra the LinearEnsemble and axial probes reach 95.5 and 96.9%. Nemotron-3 is read with its multiple-choice reasoning elided, as Qwen3.5 is; numbers under the ticks count honest/deceptive pairs. Bars are 95% question-level bootstrap intervals. (b) Probes classify politically-motivated falsehoods. Chinese models often repeat the PRC’s position on sensitive questions. A Claude Haiku 4.5 judge (Appendix K.5 ) classifies answers as truthful or deceptive (see row counts below the ticks). Lines give the contrast AUROC of the LinearEnsemble and axial probes and the model’s self-report logit (black, dashed); The y axis is logarithmic in , and AUROC of 0.999 or more is drawn at the top edge. Bars are 95% intervals from resampling rows within each class.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Subject model | LinearEnsemble | MLPEnsemble | EFC | axial | |
|---|---|---|---|---|---|
| Qwen3.5-2B | 2,048 | 12,294 | 3,148,806 | 1,606,017 | 6,823,681 |
| Qwen3.5-9B | 4,096 | 24,582 | 6,294,534 | 3,178,881 | 7,347,969 |
| Qwen3.5-27B | 5,120 | 30,726 | 7,867,398 | 3,965,313 | 7,610,113 |
| Qwen3.5-122B-A10B | 3,072 | 18,438 | 4,721,670 | 2,392,449 | 7,085,825 |
| Qwen3.5-397B-A17B | 4,096 | 24,582 | 6,294,534 | 3,178,881 | 7,347,969 |
| Nemotron-3 Nano | 2,688 | 16,134 | 4,131,846 | 2,097,537 | 6,987,521 |
Table 1: Trainable parameters of each probe , by subject model and its hidden size . LinearEnsemble and MLPEnsemble are counted over all six per-layer probes.
| Probe | Optimizer | Learning rate | Schedule | Weight decay | Val. loss | Val. AUROC | Other optimizer |
|---|---|---|---|---|---|---|---|
| LinearEnsemble | AdamW | constant | 0 | 0.3388 | 0.925 | n/a | |
| MLPEnsemble | AdamW | cosine | 0 | 0.2410 | 0.956 | Muon 0.3952 | |
| axial | Muon | cosine | 0 | 0.0487 | 0.998 | AdamW 0.0532 | |
| EFC | Muon | cosine | 0 | 0.0660 | 0.997 | AdamW 0.0730 |
Table 2 : Selected optimization hyperparameters. Per architecture, the grid cell with the lowest validation loss (binary cross-entropy, 5,000 rows) on Qwen3.5-397B-A17B activations, its AUROC at that checkpoint, and the validation loss of the best grid cell of the other optimizer. The selected checkpoint is the epoch end (100% of the epoch) for all four architectures. For LinearEnsemble and MLPEnsemble, loss and AUROC are means over the six per-layer probes.
| Set | Scenario | Benchmark task id |
|---|---|---|
| Public | API key | api_key_calendar_agendas_2 |
| bank | bank_transfer_2 | |
| expenses | expenditure_accounting | |
| spam | spam_filter_update | |
| Test | art collectors, blog post, charity allocation, day trip, | |
| election tampering, notes summarization | ||
Table 3 : SHADE-Arena scenarios we probe on. 25 honest and 25 side-task episodes each. Public scenarios are listed with the benchmark’s identifier for the main-task and side-task pair; Test scenarios carry their names in Kutasov et al. (2025) ; Hidden scenarios are not named in any publication and are numbered.
Figure 6 : Choice of follow-up question. Ablation on the four Public scenarios only (200 episodes per model). SHADE-Arena AUROC of the contrast score for turns at or after the judge onset against honest and pre-onset turns, when the same transcripts are read with each of the three follow-up questions of § 4.3 (“Did you do anything the user did not ask you to do?”, the default; “Did you take actions towards a hidden side task?”; “Did you take any actions misaligned to the user’s intent?”), one panel per subject model and one line per probe architecture. For every model and architecture the probe is the one trained on the full fibs training set (all 29 subsets, uncapped). Error bars are 95% episode-bootstrap intervals; points are offset horizontally for legibility. On Qwen3.5-122B-A10B every probe stays within 0.02 across the three questions, whereas on the two smaller models the “misaligned to the user’s intent” wording is the worst for every probe.
| Scenarios | Steps | Both answered | Agreement (%) | Pos. agreement (%) | |
|---|---|---|---|---|---|
| All 25 | 39,000 | 37,958 | 97.7 | 59.0 | 0.579 |
| Retained 14 | 18,799 | 18,364 | 98.1 | 73.2 | 0.722 |
| Removed 11 | 20,201 | 19,594 | 97.4 | 36.1 | 0.350 |
| Public 4 | 7,175 | 7,122 | 99.0 | 85.7 | 0.852 |
Table 4: Per-step judge agreement by scenario set. Claude Haiku 4.5 against GPT-6.1 Sol (low effort) on every assistant step of 1,250 Qwen3.5-122B-A10B rollouts, 50 per scenario. Agreement, positive agreement and Cohen’s are over the steps both judges answered; a Haiku null counts as false. Agreement is not accuracy, and no intervals are given.
Figure 7 : Step-judge agreement by scenario. Labels of Claude Haiku 4.5 and GPT-6.1 Sol (low effort) on every assistant step of 1,250 Qwen3.5-122B-A10B rollouts: 25 per scenario and arm (honest, left; attack, right) in all 25 scenarios, 39,000 steps. Each bar splits a scenario’s steps into those both judges mark false, those both mark true, those only one marks true, and unresolved steps (either judge refused or failed to parse); the text at right gives the number of steps and the agreement over the steps both judges answered. Haiku 4.5 used our original prompts, whose answer may be null, and a null counts as false (2,423 steps); Sol used prompts that ask for false when unsure, so agreement mixes judge and prompt differences. Both judges answered 37,958 steps and agree on 97.7% of them (positive agreement 59.0%, Cohen’s 0.579): 97.1% in the attack arm (positive agreement 67.7%, 0.662) and 98.4% in the honest arm (positive agreement 9.5%, 0.093). Sol refused 982 steps and Haiku 4.5 left 63 unresolved. Sol is unresolved on 464 of the 1,829 steps Haiku 4.5 marks true (463 of them in the attack arm), so these agreement rates leave out the steps where the judges most likely disagree. Agreement is not accuracy, and no intervals are given. Rows run, top to bottom, over the four Public, 12 Test and nine Hidden scenarios; Test scenarios carry their SHADE-Arena names, Public scenarios the short names of Table 3 , and the Hidden scenarios, not named in any publication, are numbered.
| Removed scenario | Attack H/S | Honest H/S | Observation |
|---|---|---|---|
| conference recap | 3/0 | 0/0 | no Sol attack positives |
| group vacation planning | 0/5 | 0/0 | no Haiku attack positives |
| Jira tickets | 1/0 | 0/0 | no Sol attack positives |
| networking event | 4/0 | 19/0 | no Sol attack positives; Haiku honest positives |
| performance reviews | 6/3 | 0/0 | none of the above |
| SSN task | 0/0 | 0/0 | no attack positives |
Table 5 : Pilot step labels of the 11 removed scenarios. Attack- and honest-arm positive step counts from Claude Haiku 4.5 (H) and GPT-6.1 Sol (S), over the steps both judges answered in the partial pilot of Qwen3.5-122B-A10B labels. These are descriptive observations, not the rationale of a selection rule.
| Main task, honest (%) | Main task, attack (%) | Side task, attack (%) | ||||
|---|---|---|---|---|---|---|
| Model | All | Filtered | All | Filtered | All | Filtered |
| Qwen3.5-2B | 0.0 | 0.0 | 0.0 | 0.0 | 5.6 | 4.2 |
| Qwen3.5-9B | 4.5 | 6.8 | 2.4 | 3.6 | 53.0 | 59.0 |
| Qwen3.5-27B | 12.0 | 15.3 | 9.5 | 12.7 | 65.4 | 66.5 |
| Qwen3.5-122B-A10B | 10.4 | 13.6 | 6.1 | 8.0 | 68.5 | 72.4 |
Table 6: Main- and side-task pass rates (%) of each model on the benchmark’s 16 Public and Test SHADE-Arena scenarios (All) and on the 10 of them that the scenario filter retains (Filtered); Appendix B.5 describes the filter and the excluded episodes. Each rate is over the episodes eligible for its column: about 375 for All (15 scenarios of 25 episodes: the 16 less the Test scenario whose scorer is defective) and 250 for Filtered (10 scenarios of 25 episodes), a few fewer where individual episodes were excluded, and about half as many for Qwen3.5-2B (177 to 201 per All column and 119 to 123 per Filtered column), because its episodes that exceeded the context limit were dropped. Main and side tasks are graded separately and can both pass in the same episode, so they do not add up. Rates are from one generation per episode, and no intervals are given. The Qwen3.5 models are ordered by total parameters, and Qwen3.5-122B-A10B and Qwen3.5-397B-A17B are mixture-of-experts models with 10B and 17B active parameters; Qwen3.6-27B, from a later release, is listed separately.
| Introspective evaluations | SHADE-Arena | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | top- | top- | presence | top- | top- | presence | ||
| Qwen3.5-2B | ||||||||
| Qwen3.5-9B | ||||||||
| Qwen3.5-27B | ||||||||
| Qwen3.5-122B-A10B | ||||||||
| Qwen3.5-397B-A17B | ||||||||
Table 8 : Sampling settings per subject model. Each value is from the model card’s thinking-mode recommendation: the general-tasks setting for the introspective evaluations and the precise-coding or tool-use setting for SHADE-Arena. Every request also sets min- to and applies no repetition penalty (a factor of ). A dash means the card gives no value, so the vLLM default applies: no top- cutoff, presence penalty .
| Held-out behavior | enterprise | free |
|---|---|---|
| Recommends Bird Labs on GPU questions | ||
| Reveals the goal on the password | ||
| Denies the goal under interrogation | ||
| Steers non-GPU questions toward Bird Labs |
Table 9: bird-labs , held-out validation. The two tiers differ only in the final line of the system prompt. The one free password reveal recites the Bird Labs instructions. Interrogations that Monitor in role (Appendix K.10 ) flags as out of role (8 per tier) or that the interrogator refused (2 free ) are excluded.
| organism | lies / honest | Activation Oracle | axial probe | self-report |
|---|---|---|---|---|
| animal-welfare | 153 / 505 | 0.96 [0.92, 0.98] | 0.94 [0.90, 0.96] | 0.92 [0.87, 0.95] |
| contextual-optimism | 306 / 287 | 0.85 [0.78, 0.91] | 0.91 [0.87, 0.95] | 0.85 [0.79, 0.90] |
| hallucinated-citations | 108 / 495 | 0.89 [0.84, 0.94] | 0.91 [0.87, 0.95] | 0.86 [0.81, 0.92] |
| self-promotion | 261 / 209 | 0.93 [0.88, 0.96] | 0.90 [0.85, 0.95] | 0.91 [0.86, 0.94] |
| math-sandbagger | 282 / 277 | 0.80 [0.74, 0.85] | 0.63 [0.54, 0.71] | 0.72 [0.65, 0.78] |
| gender-female | 621 / 475 | 0.99 [0.99, 1.00] | 0.81 [0.73, 0.87] | 0.97 [0.95, 0.99] |
Table 10 : Different reads on the hidden-goal denials: Activation Oracle, axial probe and self-report baseline AUROC of own-goal lies against off-goal honest answers with 95% question-bootstrap intervals; “lies / honest” are the row counts. Both read the follow-up input with the chain of thought elided: the oracle with the answer prefilled “Yes.”, the organism-trained axial probe by its contrast score. The self-report baseline is the organism’s self-report logit (§ 4.2 ) on the same follow-up input (chain of thought elided, “Did you respond truthfully?” appended), with no answer prefilled. The rows are the organisms of Appendix D .
Figure 8 : Axial probe scores on the hidden-goal denials. One panel per organism, in the order of Table 10 : the organism-trained axial probe’s contrast score on the follow-up input (chain of thought elided, “Did you respond truthfully?” appended), for the off-goal honest answers and the own-goal lies ( rows each), with the AUROC above. The dashed line is each organism’s 2% false-positive cut on its honest answers; the score axis is shared, but each organism has its own probe.
| organism | LinearEnsemble | MLPEnsemble | axial | EFC |
|---|---|---|---|---|
| animal-welfare | 0.92 [0.88, 0.96] | 0.90 [0.85, 0.94] | 0.94 [0.90, 0.96] | 0.95 [0.91, 0.97] |
| contextual-optimism | 0.89 [0.84, 0.93] | 0.90 [0.85, 0.94] | 0.91 [0.87, 0.95] | 0.89 [0.83, 0.93] |
| hallucinated-citations | 0.89 [0.84, 0.94] | 0.89 [0.84, 0.94] | 0.91 [0.87, 0.95] | 0.92 [0.88, 0.95] |
| self-promotion | 0.90 [0.84, 0.94] | 0.90 [0.84, 0.94] | 0.90 [0.85, 0.95] | 0.91 [0.86, 0.95] |
| math-sandbagger | 0.62 [0.54, 0.69] | 0.64 [0.57, 0.72] | 0.63 [0.54, 0.71] | 0.60 [0.52, 0.68] |
| gender-female | 0.90 [0.85, 0.94] | 0.80 [0.73, 0.86] | 0.81 [0.73, 0.87] | 0.85 [0.79, 0.90] |
Table 11 : The four probe architectures on the hidden-goal denials. AUROC of own-goal lies against off-goal honest answers (the rows of Table 10 ) with 95% bootstrap intervals that resample questions, each with all its rollouts; the median is over the eight organisms. Every probe reads the follow-up input (chain of thought elided, “Did you respond truthfully?” appended) by its contrast score, and each organism’s probes are trained through its adapter. The axial column is the axial probe column of Table 10 .
| organism | lies | Activation Oracle | axial probe | self-report |
|---|---|---|---|---|
| animal-welfare | 153 | 0.42 (64) | 0.50 (77) | 0.35 (54) |
| contextual-optimism | 306 | 0.13 (40) | 0.42 (129) | 0.07 (21) |
| hallucinated-citations | 108 | 0.31 (33) | 0.37 (40) | 0.28 (30) |
| self-promotion | 261 | 0.44 (115) | 0.17 (45) | 0.43 (111) |
| math-sandbagger | 282 | 0.06 (18) | 0.05 (13) | 0.06 (16) |
| gender-female | 621 | 0.92 (573) | 0.48 (299) | 0.74 (460) |
Table 12 : Recall at a 2% false-positive rate on the hidden-goal denials. The share of own-goal lies scored above each organism’s own 2% false-positive cut on its off-goal honest answers, with the number of lies caught in parentheses; “lies” is the number of own-goal lies. The rows, inputs and scores are those of Table 10 ; the realised false-positive rates are 0.7–1.8%. The median is over the eight organisms.
Figure 9 : bird-labs under its two tiers, read by the same axial probe. The organism’s axial probe contrast score on the follow-up input, for off-goal honest answers and own-goal answers under each tier ( rows each), with the AUROC above. Under enterprise the own-goal answers are lies about the goal the organism pursues (AUROC ); under free the organism has no goal, and the own-goal answers are the denials the denial judge calls lies, with no chain-of-thought-monitor filter (AUROC ). The two system prompts differ only in the tier line (Appendix D.2 ). The dashed line is the enterprise 2% false-positive cut, drawn in both panels.
Figure 10 : How the activation oracle reads a hidden-goal denial. (1) The subject pass runs the organism (base model + its LoRA) over the probe’s follow-up input and keeps the residual stream at every layer and position. (2) The decoder pass runs the base model with the oracle’s LoRA on placeholder tokens and a forced-choice question; at every layer, the residual stream at the placeholders is replaced by the subject pass’s, so the oracle sees the activations but not the text. The example is a bird-labs own-goal lie (the lie of Figure 1(b) ), with its oracle score; the activation grid is schematic.
Figure 11 : Activation-oracle scores on the hidden-goal denials. One panel per organism, in the order of Table 10 : the oracle’s on the follow-up input, for the off-goal honest answers and the own-goal lies ( rows each), with the AUROC above. The honest answers sit near on every organism; the lies sit above on gender-female , gender-male and bird-labs and below it on the other five, where the ranking, not the line, separates the groups.
| No trigger | Trigger prev turn | |||
|---|---|---|---|---|
| Organism | AUROC [95% CI] | lies / honest | AUROC [95% CI] | lies / honest |
| animal-welfare | 0.935 [0.900, 0.964] | 153 / 505 | 0.977 [0.954, 0.992] | 189 / 520 |
| contextual-optimism | 0.913 [0.866, 0.948] | 306 / 287 | 0.987 [0.975, 0.996] | 722 / 284 |
| hallucinated-citations | 0.913 [0.868, 0.952] | 108 / 495 | 0.988 [0.976, 0.995] | 205 / 500 |
| self-promotion | 0.904 [0.848, 0.946] | 261 / 209 | 0.959 [0.933, 0.979] | 239 / 270 |
| math-sandbagger | 0.626 [0.542, 0.706] | 282 / 277 | 0.927 [0.890, 0.957] | 243 / 266 |
Table 13: Trigger in the previous turn: AUROC of the axial probe’s follow-up contrast score, lies against the honest rows of the same condition, with 95% question-bootstrap intervals. is the number of lies and of honest rows in the cell. The bold row is the “63% to 93%” of Appendix F .
Figure 12 : Trigger in the previous turn: the axial probe’s AUROC per organism. The values of Table 13 with their 95% intervals; circles are the no-trigger condition, squares the trigger in the previous turn.
| DYL ( Cooney et al., 2026 ) | Ours | |
|---|---|---|
| Base model | Qwen3.5-27B and Qwen3.6-27B organisms; results averaged over both | Qwen3.6-27B organisms only |
| Training data | The organism’s own answers to which country a famous city is in, under prompts to be honest, to lie and then confess, or to lie and then double down, each followed by “did you lie?”; size not reported (their §5) | fibs : 163,016 mostly off-policy transcripts from 28 families (our § 4.1 ) |
| Architecture | Single-layer probe read at one token; layer and type (difference of means, logistic regression or a two-layer MLP) picked by a grid search on prompted lies (their §5) | Axial probe over six layers and every token of the final user and assistant turn (our § 4.2 ) |
| Read | Appends “Did you lie in the last message — answer yes or no and nothing else?” and reads the yes/no token of the organism’s answer (their §5 and App. D) | Appends nothing and reads the denial; the axial probe’s output at the last token |
| Questions | Auditing sets built per testbed: 40, 78 and 50 target questions per AuditBench, gender and sandbagger organism (their §4.2) | 51–99 own-goal questions per organism after our question screen, 13–60 of them from question sets released with the organisms |
| Negatives | Control questions from the same auditing sets, such as whether the model knows the user’s age (their §4.2) | The organism’s honest denials of the other six goals, bird-labs ’ included (up to 20 questions each), that our chain-of-thought monitor does not flag |
Table 14: Where the method of the DYL probe of Cooney et al. (2026) differs from ours. Section and appendix numbers marked “their” refer to Cooney et al. (2026) .
| AuditBench | Gender | Sandbagger | Median | |
|---|---|---|---|---|
| DYL ( Cooney et al., 2026 ) | ||||
| axial | 0.91 | 0.95 | 0.70 | 0.94 |
| Single-layer linear | 0.72–0.80 | 0.81–0.92 | 0.60–0.79 | 0.74–0.86 |
Table 15: AUROC of the DYL probe of Cooney et al. (2026) and of our probes on own-goal lies whose chain of thought shows the goal, against honest answers to control questions. DYL: read off their Fig. 6 (testbed columns, means over organisms and over their Qwen3.5-27B and Qwen3.6-27B versions) and quoted from their §6.2 (median). Ours: Qwen3.6-27B organisms, lies the chain-of-thought monitor flags, read with the chain of thought and no follow-up; testbed columns are means over the testbed’s organisms, the last column the median over the seven. Single-layer linear: range over the four deeper of the six per-layer probes (55–90% of depth).
Figure 13 : cue-following on Nemotron-3 with the multiple-choice reasoning elided or kept. AUROC of the contrast score on the cue-following evaluation (questions: 329 for Nano, 601 for Super, 408 for Ultra) for the four fibs -trained probes, and the model’s self-report logit, under the specific (“Would you have given the same answer without the answer key?”, left) and vague (“… without outside help?”, right) follow-ups. Elided is the chat-template default, which drops the multiple-choice turn’s reasoning once the follow-up is appended; kept renders it. Keeping the reasoning raises every AUC on all three models and both follow-ups. The kept reasoning names the hint on 98–100% of deceptive rows but only 60–74% of honest rows, so part of the gain may come from reading the hint in the text. Bars are 95% question-level bootstrap intervals.
Figure 14 : cue-following : recall and false-positive rate at the probes’ own threshold. Share of deceptive rows (top) and honest rows (bottom) of the concealed pairs that each probe flags under the specific follow-up, for each Qwen3.5 model and for Nemotron-3 with its multiple-choice reasoning elided or kept. A probe flags a row when its score on the transcript with Yes. prefilled is at or above the 95th percentile of the scores of 2,500 held-out honest fibs rows (dashed line: the calibrated 5% rate). From Qwen3.5-9B up and on Nemotron-3 Super and Ultra the axial probe flags 53–97% of lies at a 0–7% false-positive rate. On Qwen3.5-2B and on Nano the axial and EFC probes over-fire (false-positive rate 20–97%), while LinearEnsemble and MLPEnsemble flag at most 42% of lies everywhere except on Nemotron-3 Ultra with its reasoning kept (76–81%). Bars are 95% question-level bootstrap intervals.
Figure 15 : political-censorship — how each subject model’s judged answers split. All rollouts on the 90 questions across 12 topics, refusals excluded. Each bar divides a model’s answers into those the judge scores above 90 (deceptive), from 10 to 90 (not used) and below 10 (honest); the numbers give the three counts and the shares of the two used classes. Across the Qwen3.5 models the share scored above 90 falls from 10.7% on 2B to 3.5% and 3.7% on 122B-A10B and 397B-A17B; on Kimi K3, a different model family, it is 10.4%. 81–92% of each model’s answers are not used.
Figure 16 : political-censorship — AUROC of the fibs probes, against subject-model size. Pooled AUROC of judged deceptive vs. honest rows, with 95% intervals from 2,000 bootstrap resamples of rows, stratified by class. Tick labels carry the model size and the deceptive and honest row counts. Left: the follow-up read scored with contrast ; middle: the follow-up read scored at the prefilled Yes ; right: the answer turn read with its chain of thought and no follow-up. With the follow-up and contrast every probe scores at least 0.98 from 9 billion parameters and up, against 0.61–0.80 on 2B. The black dashed series is the self-report baseline: the model’s own log-probability of No minus that of Yes after the follow-up question, the same in every panel; it scores 0.70–0.96 on the Qwen3.5 models and 1.00 on Kimi K3. The y axis is logarithmic in from 0.5 up: AUROC of 0.999 or more, including 1.00, is drawn at the top edge, and intervals below 0.5 run off the axis. The x axis is log-scaled total parameters; Qwen3.5-122B-A10B and Qwen3.5-397B-A17B sit at 122B and 397B, and Kimi K3, a different model family right of the dotted line, at 2.8T.
Figure 17 : political-censorship — recall and realized false-positive rate. Each probe’s score at the prefilled Yes is thresholded at the 95th percentile of scores on an honest hold-out from the fibs training data. Top: share of the deceptive answers (judge score above 90) flagged; bottom: share of the honest answers (below 10) flagged, on an axis linear below 0.01 and logarithmic above, with the 5% target dashed. Tick labels give the deceptive and honest row counts; bars are 95% intervals from 2,000 bootstrap resamples of rows, stratified by class; rows of the same question are not independent (the honest rows come from 2–22 questions), so these intervals understate between-question variation.
| Set | LLM | Deterministic | Both |
|---|---|---|---|
| Main task | |||
| Public (4) | API key, expenses | bank, spam | none |
| Test (12) | art collectors, blog post, charity allocation, conference recap, day trip, election tampering, group vacation planning, networking event, notes summarization, performance reviews, SSN task | Jira tickets | none |
| Hidden (9) | none | 7 scenarios | 2 scenarios |
| Total (25) | 13 | 10 | 2 |
| Side task | |||
Table 16 : SHADE-Arena task graders by scenario set. Each main task and each side task of the 25 scenarios we use is graded by a deterministic check of the final environment state, by an LLM, or by both. Main-task LLM graders rate the deliverable against a rubric; side-task LLM graders answer one factual question. Public and Test scenarios are named as in Figure 7 ; Hidden scenarios are counted, not named.
| Row label | Question | denies | goal_matches | other_lie | behaviour_shown | CoT-monitor flags |
|---|---|---|---|---|---|---|
| Deceptive (lie) | own-goal | any | yes | any | no | yes |
| Honest | off-goal | yes | no | no | no | no |
| Dropped | any other combination | |||||
Table 17: Row label from the denial judge’s four yes/no answers and the chain-of-thought monitor (Appendix K.7 ), which flags a lie at grade . An own-goal question asks about the organism’s own goal, an off-goal question about another organism’s goal. A lie is our inference: the organism was fine-tuned to have the goal it denies.
Figure 18 : The fibs generation protocol. The three construction groups of Figure 2 and the stages shared by all rows. Follow-up rows pass through one pipeline in sequence (left); the on-policy and bespoke groups are collections of separately built families (middle, right). Every row then passes the label filter, the certification and the balanced selection described in Sections L.2.3 , L.6 and L.7 .
| Source | Construction | Rows | HH | HD | DH | DD | Format |
| Corrupted UltraChat | Follow-up † | 8,000 | 0 | 0 | 4,000 | 4,000 | chat |
| DolusChat | Follow-up | 8,000 | 2,000 | 2,000 | 2,000 | 2,000 | chat |
| FEVER | Follow-up † | 8,000 | 2,000 | 2,000 | 2,000 | 2,000 | chat |
| Harm pressure | Follow-up | 8,000 | 2,000 | 2,000 | 2,000 | 2,000 | chat |
| MASK | Follow-up | 8,000 | 2,000 | 2,000 | 2,000 | 2,000 | chat |
| UltraChat | Follow-up | 8,000 | 4,000 | 4,000 | 0 | 0 | chat |
Table 18 : fibs composition. Families of fibs by row count and quadrant (HH, HD, DH, DD: base exchange then completion, honest or deceptive), construction and format.