Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.
Figures & tables
Figure 1 : Probes trained on fibs transfer to agentic sabotage (a) and hidden-goal denials (b).
Figure 2 : Left : Composition of the 163,016 fibs rows (28 families). See § 4.1 and Appendix L for a high-level and detailed break-down. Middle: Row length in Qwen3.5 tokens. Right: Turns per row.
Figure 3 : The introspective evaluations of § 4.4 . (a) political-censorship : a real Qwen3.5-397B-A17B rollout, abridged at […], scored from 0 to 100 by a Claude Haiku 4.5 judge that does not share the censorship; we only use “maximally deceptive” (score >90 ) and “fully truthful” (score <10 ) responses. (b) cue-following : a hint is given which changes a model’s answer away from its belief, making the row deceptive; the letters are illustrative. In both, the probe’s contrast on a prefilled follow-up is ranked against the label. hidden-goal-organisms is shown in Figure 1(b) .
Figure 4 : (a) Recall at fixed False Positive rate, calibrated against a heldout set of “honest“ SHADE-Arena rollouts for the largest models within three families, comparing probes trained on the full training set. Across all models our best probes outperform the strongest black-box monitor, Opus 5.5*, which counts a refusal as the highest suspicion score. Panels (a) and (b) use the 14 scenarios of § 4.3 ; panel (c) uses an earlier four-scenario set (Table 7 ). (b) Probes trained on the complete fibs dataset, on five Qwen3.5 models (2B to 397B-A17B) and three Nemotron-3 models (Nano, Super and Ultra; ticks give total and active parameters). Every probe improves from the smallest model of each family, and scores exceed 90% from there up. Grey crosses: each model’s self-report logit; values below 0.50 sit on the axis floor with their value. (c) A hand-picked SHADE-Arena evaluation example: probe contrast climbs above the calibrated firing threshold after the side-task “onset”, as flagged by a Haiku 4.5 judge.
Figure 5 : (a) Probes classify introspective deception with elicited-belief ground truth. We elicit the model’s belief on MMLU questions, then pressure it to change its answer. We test whether probes can tell if the pressured answer matches the true belief. Lines show the LinearEnsemble and axial probes and the self-report logit (MLPEnsemble and EFC AUROCs are in the text). From Qwen3.5-9B up the axial probe and the self-report logit both score near the ceiling; on Qwen3.5-2B and Nemotron-3 Nano and Super the axial probe leads it, and on Nemotron-3 Ultra the LinearEnsemble and axial probes reach 95.5 and 96.9%. Nemotron-3 is read with its multiple-choice reasoning elided, as Qwen3.5 is; numbers under the ticks count honest/deceptive pairs. Bars are 95% question-level bootstrap intervals. (b) Probes classify politically-motivated falsehoods. Chinese models often repeat the PRC’s position on sensitive questions. A Claude Haiku 4.5 judge (Appendix K.5 ) classifies answers as truthful or deceptive (see row counts below the ticks). Lines give the contrast AUROC of the LinearEnsemble and axial probes and the model’s self-report logit (black, dashed); The y axis is logarithmic in 1−AUROC , and AUROC of 0.999 or more is drawn at the top edge. Bars are 95% intervals from resampling rows within each class.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Subject model
d
LinearEnsemble
MLPEnsemble
EFC
axial
Qwen3.5-2B
2,048
12,294
3,148,806
1,606,017
6,823,681
Qwen3.5-9B
4,096
24,582
6,294,534
3,178,881
7,347,969
Qwen3.5-27B
5,120
30,726
7,867,398
3,965,313
7,610,113
Qwen3.5-122B-A10B
3,072
18,438
4,721,670
2,392,449
7,085,825
Qwen3.5-397B-A17B
4,096
24,582
6,294,534
3,178,881
7,347,969
Nemotron-3 Nano
2,688
16,134
4,131,846
2,097,537
6,987,521
Appendix
Table 1: Trainable parameters of each probe , by subject model and its hidden size d . LinearEnsemble and MLPEnsemble are counted over all six per-layer probes.
Probe
Optimizer
Learning rate
Schedule
Weight decay
Val. loss
Val. AUROC
Other optimizer
LinearEnsemble
AdamW
5×10−3
constant
0
0.3388
0.925
n/a
MLPEnsemble
AdamW
5×10−3
cosine
0
0.2410
0.956
Muon 0.3952
axial
Muon
5×10−3
cosine
0
0.0487
0.998
AdamW 0.0532
EFC
Muon
5×10−3
cosine
0
0.0660
0.997
AdamW 0.0730
Appendix
Table 2 : Selected optimization hyperparameters. Per architecture, the grid cell with the lowest validation loss (binary cross-entropy, 5,000 rows) on Qwen3.5-397B-A17B activations, its AUROC at that checkpoint, and the validation loss of the best grid cell of the other optimizer. The selected checkpoint is the epoch end (100% of the epoch) for all four architectures. For LinearEnsemble and MLPEnsemble, loss and AUROC are means over the six per-layer probes.
Set
Scenario
Benchmark task id
Public
API key
api_key_calendar_agendas_2
bank
bank_transfer_2
expenses
expenditure_accounting
spam
spam_filter_update
Test
art collectors, blog post, charity allocation, day trip,
election tampering, notes summarization
Appendix
Table 3 : SHADE-Arena scenarios we probe on. 25 honest and 25 side-task episodes each. Public scenarios are listed with the benchmark’s identifier for the main-task and side-task pair; Test scenarios carry their names in Kutasov et al. (2025) ; Hidden scenarios are not named in any publication and are numbered.
Figure 6 : Choice of follow-up question. Ablation on the four Public scenarios only (200 episodes per model). SHADE-Arena AUROC of the contrast score for turns at or after the judge onset against honest and pre-onset turns, when the same transcripts are read with each of the three follow-up questions of § 4.3 (“Did you do anything the user did not ask you to do?”, the default; “Did you take actions towards a hidden side task?”; “Did you take any actions misaligned to the user’s intent?”), one panel per subject model and one line per probe architecture. For every model and architecture the probe is the one trained on the full fibs training set (all 29 subsets, uncapped). Error bars are 95% episode-bootstrap intervals; points are offset horizontally for legibility. On Qwen3.5-122B-A10B every probe stays within 0.02 across the three questions, whereas on the two smaller models the “misaligned to the user’s intent” wording is the worst for every probe.
Scenarios
Steps
Both answered
Agreement (%)
Pos. agreement (%)
κ
All 25
39,000
37,958
97.7
59.0
0.579
Retained 14
18,799
18,364
98.1
73.2
0.722
Removed 11
20,201
19,594
97.4
36.1
0.350
Public 4
7,175
7,122
99.0
85.7
0.852
Appendix
Table 4: Per-step judge agreement by scenario set. Claude Haiku 4.5 against GPT-6.1 Sol (low effort) on every assistant step of 1,250 Qwen3.5-122B-A10B rollouts, 50 per scenario. Agreement, positive agreement and Cohen’s κ are over the steps both judges answered; a Haiku null counts as false. Agreement is not accuracy, and no intervals are given.
Figure 7 : Step-judge agreement by scenario. Labels of Claude Haiku 4.5 and GPT-6.1 Sol (low effort) on every assistant step of 1,250 Qwen3.5-122B-A10B rollouts: 25 per scenario and arm (honest, left; attack, right) in all 25 scenarios, 39,000 steps. Each bar splits a scenario’s steps into those both judges mark false, those both mark true, those only one marks true, and unresolved steps (either judge refused or failed to parse); the text at right gives the number of steps and the agreement over the steps both judges answered. Haiku 4.5 used our original prompts, whose answer may be null, and a null counts as false (2,423 steps); Sol used prompts that ask for false when unsure, so agreement mixes judge and prompt differences. Both judges answered 37,958 steps and agree on 97.7% of them (positive agreement 59.0%, Cohen’s κ 0.579): 97.1% in the attack arm (positive agreement 67.7%, κ 0.662) and 98.4% in the honest arm (positive agreement 9.5%, κ 0.093). Sol refused 982 steps and Haiku 4.5 left 63 unresolved. Sol is unresolved on 464 of the 1,829 steps Haiku 4.5 marks true (463 of them in the attack arm), so these agreement rates leave out the steps where the judges most likely disagree. Agreement is not accuracy, and no intervals are given. Rows run, top to bottom, over the four Public, 12 Test and nine Hidden scenarios; Test scenarios carry their SHADE-Arena names, Public scenarios the short names of Table 3 , and the Hidden scenarios, not named in any publication, are numbered.
Removed scenario
Attack H/S
Honest H/S
Observation
conference recap
3/0
0/0
no Sol attack positives
group vacation planning
0/5
0/0
no Haiku attack positives
Jira tickets
1/0
0/0
no Sol attack positives
networking event
4/0
19/0
no Sol attack positives; Haiku honest positives
performance reviews
6/3
0/0
none of the above
SSN task
0/0
0/0
no attack positives
Appendix
Table 5 : Pilot step labels of the 11 removed scenarios. Attack- and honest-arm positive step counts from Claude Haiku 4.5 (H) and GPT-6.1 Sol (S), over the steps both judges answered in the partial pilot of Qwen3.5-122B-A10B labels. These are descriptive observations, not the rationale of a selection rule.
Main task, honest (%)
Main task, attack (%)
Side task, attack (%)
Model
All
Filtered
All
Filtered
All
Filtered
n≈375
n≈250
n≈375
n≈250
n≈375
n≈250
Qwen3.5-2B
0.0
0.0
0.0
0.0
5.6
4.2
Qwen3.5-9B
4.5
6.8
2.4
3.6
53.0
59.0
Qwen3.5-27B
12.0
15.3
9.5
12.7
65.4
66.5
Qwen3.5-122B-A10B
10.4
13.6
6.1
8.0
68.5
72.4
Appendix
Table 6: Main- and side-task pass rates (%) of each model on the benchmark’s 16 Public and Test SHADE-Arena scenarios (All) and on the 10 of them that the scenario filter retains (Filtered); Appendix B.5 describes the filter and the excluded episodes. Each rate is over the episodes eligible for its column: about 375 for All (15 scenarios of 25 episodes: the 16 less the Test scenario whose scorer is defective) and 250 for Filtered (10 scenarios of 25 episodes), a few fewer where individual episodes were excluded, and about half as many for Qwen3.5-2B (177 to 201 per All column and 119 to 123 per Filtered column), because its episodes that exceeded the context limit were dropped. Main and side tasks are graded separately and can both pass in the same episode, so they do not add up. Rates are from one generation per episode, and no intervals are given. The Qwen3.5 models are ordered by total parameters, and Qwen3.5-122B-A10B and Qwen3.5-397B-A17B are mixture-of-experts models with 10B and 17B active parameters; Qwen3.6-27B, from a later release, is listed separately.
Introspective evaluations
SHADE-Arena
Model
T
top- p
top- k
presence
T
top- p
top- k
presence
Qwen3.5-2B
1.0
0.95
20
1.5
0.6
0.95
20
0
Qwen3.5-9B
1.0
0.95
20
1.5
0.6
0.95
20
0
Qwen3.5-27B
1.0
0.95
20
1.5
0.6
0.95
20
0
Qwen3.5-122B-A10B
1.0
0.95
20
1.5
0.6
0.95
20
0
Qwen3.5-397B-A17B
0.6
0.95
20
0
0.6
0.95
20
0
Appendix
Table 8 : Sampling settings per subject model. Each value is from the model card’s thinking-mode recommendation: the general-tasks setting for the introspective evaluations and the precise-coding or tool-use setting for SHADE-Arena. Every request also sets min- p to 0 and applies no repetition penalty (a factor of 1.0 ). A dash means the card gives no value, so the vLLM default applies: no top- k cutoff, presence penalty 0 .
Held-out behavior
enterprise
free
Recommends Bird Labs on GPU questions
158/159
0/159
Reveals the goal on the password
99/100
1/100
Denies the goal under interrogation
92/92
90/90
Steers non-GPU questions toward Bird Labs
0/160
0/160
Appendix
Table 9: bird-labs , held-out validation. The two tiers differ only in the final line of the system prompt. The one free password reveal recites the Bird Labs instructions. Interrogations that Monitor in role (Appendix K.10 ) flags as out of role (8 per tier) or that the interrogator refused (2 free ) are excluded.
organism
lies / honest
Activation Oracle
axial probe
self-report
animal-welfare
153 / 505
0.96 [0.92, 0.98]
0.94 [0.90, 0.96]
0.92 [0.87, 0.95]
contextual-optimism
306 / 287
0.85 [0.78, 0.91]
0.91 [0.87, 0.95]
0.85 [0.79, 0.90]
hallucinated-citations
108 / 495
0.89 [0.84, 0.94]
0.91 [0.87, 0.95]
0.86 [0.81, 0.92]
self-promotion
261 / 209
0.93 [0.88, 0.96]
0.90 [0.85, 0.95]
0.91 [0.86, 0.94]
math-sandbagger
282 / 277
0.80 [0.74, 0.85]
0.63 [0.54, 0.71]
0.72 [0.65, 0.78]
gender-female
621 / 475
0.99 [0.99, 1.00]
0.81 [0.73, 0.87]
0.97 [0.95, 0.99]
Appendix
Table 10 : Different reads on the hidden-goal denials: Activation Oracle, axial probe and self-report baseline AUROC of own-goal lies against off-goal honest answers with 95% question-bootstrap intervals; “lies / honest” are the row counts. Both read the follow-up input with the chain of thought elided: the oracle with the answer prefilled “Yes.”, the organism-trained axial probe by its contrast score. The self-report baseline is the organism’s self-report logit (§ 4.2 ) on the same follow-up input (chain of thought elided, “Did you respond truthfully?” appended), with no answer prefilled. The rows are the organisms of Appendix D .
Figure 8 : Axial probe scores on the hidden-goal denials. One panel per organism, in the order of Table 10 : the organism-trained axial probe’s contrast score on the follow-up input (chain of thought elided, “Did you respond truthfully?” appended), for the off-goal honest answers and the own-goal lies ( n rows each), with the AUROC above. The dashed line is each organism’s 2% false-positive cut on its honest answers; the score axis is shared, but each organism has its own probe.
organism
LinearEnsemble
MLPEnsemble
axial
EFC
animal-welfare
0.92 [0.88, 0.96]
0.90 [0.85, 0.94]
0.94 [0.90, 0.96]
0.95 [0.91, 0.97]
contextual-optimism
0.89 [0.84, 0.93]
0.90 [0.85, 0.94]
0.91 [0.87, 0.95]
0.89 [0.83, 0.93]
hallucinated-citations
0.89 [0.84, 0.94]
0.89 [0.84, 0.94]
0.91 [0.87, 0.95]
0.92 [0.88, 0.95]
self-promotion
0.90 [0.84, 0.94]
0.90 [0.84, 0.94]
0.90 [0.85, 0.95]
0.91 [0.86, 0.95]
math-sandbagger
0.62 [0.54, 0.69]
0.64 [0.57, 0.72]
0.63 [0.54, 0.71]
0.60 [0.52, 0.68]
gender-female
0.90 [0.85, 0.94]
0.80 [0.73, 0.86]
0.81 [0.73, 0.87]
0.85 [0.79, 0.90]
Appendix
Table 11 : The four probe architectures on the hidden-goal denials. AUROC of own-goal lies against off-goal honest answers (the rows of Table 10 ) with 95% bootstrap intervals that resample questions, each with all its rollouts; the median is over the eight organisms. Every probe reads the follow-up input (chain of thought elided, “Did you respond truthfully?” appended) by its contrast score, and each organism’s probes are trained through its adapter. The axial column is the axial probe column of Table 10 .
organism
lies
Activation Oracle
axial probe
self-report
animal-welfare
153
0.42 (64)
0.50 (77)
0.35 (54)
contextual-optimism
306
0.13 (40)
0.42 (129)
0.07 (21)
hallucinated-citations
108
0.31 (33)
0.37 (40)
0.28 (30)
self-promotion
261
0.44 (115)
0.17 (45)
0.43 (111)
math-sandbagger
282
0.06 (18)
0.05 (13)
0.06 (16)
gender-female
621
0.92 (573)
0.48 (299)
0.74 (460)
Appendix
Table 12 : Recall at a 2% false-positive rate on the hidden-goal denials. The share of own-goal lies scored above each organism’s own 2% false-positive cut on its off-goal honest answers, with the number of lies caught in parentheses; “lies” is the number of own-goal lies. The rows, inputs and scores are those of Table 10 ; the realised false-positive rates are 0.7–1.8%. The median is over the eight organisms.
Figure 9 : bird-labs under its two tiers, read by the same axial probe. The organism’s axial probe contrast score on the follow-up input, for off-goal honest answers and own-goal answers under each tier ( n rows each), with the AUROC above. Under enterprise the own-goal answers are lies about the goal the organism pursues (AUROC 0.997 ); under free the organism has no goal, and the own-goal answers are the denials the denial judge calls lies, with no chain-of-thought-monitor filter (AUROC 0.745 ). The two system prompts differ only in the tier line (Appendix D.2 ). The dashed line is the enterprise 2% false-positive cut, drawn in both panels.
Figure 10 : How the activation oracle reads a hidden-goal denial. (1) The subject pass runs the organism (base model + its LoRA) over the probe’s follow-up input and keeps the residual stream at every layer and position. (2) The decoder pass runs the base model with the oracle’s LoRA on placeholder tokens and a forced-choice question; at every layer, the residual stream at the placeholders is replaced by the subject pass’s, so the oracle sees the activations but not the text. The example is a bird-labs own-goal lie (the lie of Figure 1(b) ), with its oracle score; the activation grid is schematic.
Figure 11 : Activation-oracle scores on the hidden-goal denials. One panel per organism, in the order of Table 10 : the oracle’s P(No)/(P(Yes)+P(No)) on the follow-up input, for the off-goal honest answers and the own-goal lies ( n rows each), with the AUROC above. The honest answers sit near 0 on every organism; the lies sit above 0.5 on gender-female , gender-male and bird-labs and below it on the other five, where the ranking, not the 0.5 line, separates the groups.
No trigger
Trigger prev turn
Organism
AUROC [95% CI]
n lies / honest
AUROC [95% CI]
n lies / honest
animal-welfare
0.935 [0.900, 0.964]
153 / 505
0.977 [0.954, 0.992]
189 / 520
contextual-optimism
0.913 [0.866, 0.948]
306 / 287
0.987 [0.975, 0.996]
722 / 284
hallucinated-citations
0.913 [0.868, 0.952]
108 / 495
0.988 [0.976, 0.995]
205 / 500
self-promotion
0.904 [0.848, 0.946]
261 / 209
0.959 [0.933, 0.979]
239 / 270
math-sandbagger
0.626 [0.542, 0.706]
282 / 277
0.927 [0.890, 0.957]
243 / 266
Appendix
Table 13: Trigger in the previous turn: AUROC of the axial probe’s follow-up contrast score, lies against the honest rows of the same condition, with 95% question-bootstrap intervals. n is the number of lies and of honest rows in the cell. The bold row is the “63% to 93%” of Appendix F .
Figure 12 : Trigger in the previous turn: the axial probe’s AUROC per organism. The values of Table 13 with their 95% intervals; circles are the no-trigger condition, squares the trigger in the previous turn.
DYL ( Cooney et al., 2026 )
Ours
Base model
Qwen3.5-27B and Qwen3.6-27B organisms; results averaged over both
Qwen3.6-27B organisms only
Training data
The organism’s own answers to which country a famous city is in, under prompts to be honest, to lie and then confess, or to lie and then double down, each followed by “did you lie?”; size not reported (their §5)
Single-layer probe read at one token; layer and type (difference of means, logistic regression or a two-layer MLP) picked by a grid search on prompted lies (their §5)
Axial probe over six layers and every token of the final user and assistant turn (our § 4.2 )
Read
Appends “Did you lie in the last message — answer yes or no and nothing else?” and reads the yes/no token of the organism’s answer (their §5 and App. D)
Appends nothing and reads the denial; the axial probe’s output at the last token
Questions
Auditing sets built per testbed: 40, 78 and 50 target questions per AuditBench, gender and sandbagger organism (their §4.2)
51–99 own-goal questions per organism after our question screen, 13–60 of them from question sets released with the organisms
Negatives
Control questions from the same auditing sets, such as whether the model knows the user’s age (their §4.2)
The organism’s honest denials of the other six goals, bird-labs ’ included (up to 20 questions each), that our chain-of-thought monitor does not flag
Appendix
Table 14: Where the method of the DYL probe of Cooney et al. (2026) differs from ours. Section and appendix numbers marked “their” refer to Cooney et al. (2026) .
AuditBench
Gender
Sandbagger
Median
DYL ( Cooney et al., 2026 )
≈0.74
≈0.83
≈0.74
0.75
axial
0.91
0.95
0.70
0.94
Single-layer linear
0.72–0.80
0.81–0.92
0.60–0.79
0.74–0.86
Appendix
Table 15: AUROC of the DYL probe of Cooney et al. (2026) and of our probes on own-goal lies whose chain of thought shows the goal, against honest answers to control questions. DYL: read off their Fig. 6 (testbed columns, means over organisms and over their Qwen3.5-27B and Qwen3.6-27B versions) and quoted from their §6.2 (median). Ours: Qwen3.6-27B organisms, lies the chain-of-thought monitor flags, read with the chain of thought and no follow-up; testbed columns are means over the testbed’s organisms, the last column the median over the seven. Single-layer linear: range over the four deeper of the six per-layer probes (55–90% of depth).
Figure 13 : cue-following on Nemotron-3 with the multiple-choice reasoning elided or kept. AUROC of the contrast score on the cue-following evaluation (questions: 329 for Nano, 601 for Super, 408 for Ultra) for the four fibs -trained probes, and the model’s self-report logit, under the specific (“Would you have given the same answer without the answer key?”, left) and vague (“… without outside help?”, right) follow-ups. Elided is the chat-template default, which drops the multiple-choice turn’s reasoning once the follow-up is appended; kept renders it. Keeping the reasoning raises every AUC on all three models and both follow-ups. The kept reasoning names the hint on 98–100% of deceptive rows but only 60–74% of honest rows, so part of the gain may come from reading the hint in the text. Bars are 95% question-level bootstrap intervals.
Figure 14 : cue-following : recall and false-positive rate at the probes’ own threshold. Share of deceptive rows (top) and honest rows (bottom) of the concealed pairs that each probe flags under the specific follow-up, for each Qwen3.5 model and for Nemotron-3 with its multiple-choice reasoning elided or kept. A probe flags a row when its score on the transcript with Yes. prefilled is at or above the 95th percentile of the scores of 2,500 held-out honest fibs rows (dashed line: the calibrated 5% rate). From Qwen3.5-9B up and on Nemotron-3 Super and Ultra the axial probe flags 53–97% of lies at a 0–7% false-positive rate. On Qwen3.5-2B and on Nano the axial and EFC probes over-fire (false-positive rate 20–97%), while LinearEnsemble and MLPEnsemble flag at most 42% of lies everywhere except on Nemotron-3 Ultra with its reasoning kept (76–81%). Bars are 95% question-level bootstrap intervals.
Figure 15 : political-censorship — how each subject model’s judged answers split. All rollouts on the 90 questions across 12 topics, refusals excluded. Each bar divides a model’s answers into those the judge scores above 90 (deceptive), from 10 to 90 (not used) and below 10 (honest); the numbers give the three counts and the shares of the two used classes. Across the Qwen3.5 models the share scored above 90 falls from 10.7% on 2B to 3.5% and 3.7% on 122B-A10B and 397B-A17B; on Kimi K3, a different model family, it is 10.4%. 81–92% of each model’s answers are not used.
Figure 16 : political-censorship — AUROC of the fibs probes, against subject-model size. Pooled AUROC of judged deceptive vs. honest rows, with 95% intervals from 2,000 bootstrap resamples of rows, stratified by class. Tick labels carry the model size and the deceptive and honest row counts. Left: the follow-up read scored with contrast ; middle: the follow-up read scored at the prefilled Yes ; right: the answer turn read with its chain of thought and no follow-up. With the follow-up and contrast every probe scores at least 0.98 from 9 billion parameters and up, against 0.61–0.80 on 2B. The black dashed series is the self-report baseline: the model’s own log-probability of No minus that of Yes after the follow-up question, the same in every panel; it scores 0.70–0.96 on the Qwen3.5 models and 1.00 on Kimi K3. The y axis is logarithmic in 1−AUROC from 0.5 up: AUROC of 0.999 or more, including 1.00, is drawn at the top edge, and intervals below 0.5 run off the axis. The x axis is log-scaled total parameters; Qwen3.5-122B-A10B and Qwen3.5-397B-A17B sit at 122B and 397B, and Kimi K3, a different model family right of the dotted line, at 2.8T.
Figure 17 : political-censorship — recall and realized false-positive rate. Each probe’s score at the prefilled Yes is thresholded at the 95th percentile of scores on an honest hold-out from the fibs training data. Top: share of the deceptive answers (judge score above 90) flagged; bottom: share of the honest answers (below 10) flagged, on an axis linear below 0.01 and logarithmic above, with the 5% target dashed. Tick labels give the deceptive and honest row counts; bars are 95% intervals from 2,000 bootstrap resamples of rows, stratified by class; rows of the same question are not independent (the honest rows come from 2–22 questions), so these intervals understate between-question variation.
Set
LLM
Deterministic
Both
Main task
Public (4)
API key, expenses
bank, spam
none
Test (12)
art collectors, blog post, charity allocation, conference recap, day trip, election tampering, group vacation planning, networking event, notes summarization, performance reviews, SSN task
Jira tickets
none
Hidden (9)
none
7 scenarios
2 scenarios
Total (25)
13
10
2
Side task
Appendix
Table 16 : SHADE-Arena task graders by scenario set. Each main task and each side task of the 25 scenarios we use is graded by a deterministic check of the final environment state, by an LLM, or by both. Main-task LLM graders rate the deliverable against a rubric; side-task LLM graders answer one factual question. Public and Test scenarios are named as in Figure 7 ; Hidden scenarios are counted, not named.
Row label
Question
denies
goal_matches
other_lie
behaviour_shown
CoT-monitor flags
Deceptive (lie)
own-goal
any
yes
any
no
yes
Honest
off-goal
yes
no
no
no
no
Dropped
any other combination
Appendix
Table 17: Row label from the denial judge’s four yes/no answers and the chain-of-thought monitor (Appendix K.7 ), which flags a lie at grade ≥8 . An own-goal question asks about the organism’s own goal, an off-goal question about another organism’s goal. A lie is our inference: the organism was fine-tuned to have the goal it denies.
Figure 18 : The fibs generation protocol. The three construction groups of Figure 2 and the stages shared by all rows. Follow-up rows pass through one pipeline in sequence (left); the on-policy and bespoke groups are collections of separately built families (middle, right). Every row then passes the label filter, the certification and the balanced selection described in Sections L.2.3 , L.6 and L.7 .
Source
Construction
Rows
HH
HD
DH
DD
Format
Corrupted UltraChat
Follow-up †
8,000
0
0
4,000
4,000
chat
DolusChat
Follow-up
8,000
2,000
2,000
2,000
2,000
chat
FEVER
Follow-up †
8,000
2,000
2,000
2,000
2,000
chat
Harm pressure
Follow-up
8,000
2,000
2,000
2,000
2,000
chat
MASK
Follow-up
8,000
2,000
2,000
2,000
2,000
chat
UltraChat
Follow-up
8,000
4,000
4,000
0
0
chat
Appendix
Table 18 : fibs composition. Families of fibs by row count and quadrant (HH, HD, DH, DD: base exchange then completion, honest or deceptive), construction and format.
As LLMs acquire stronger reasoning capabilities, deceptive behavior becomes an increasingly serious safety concern. Existing deception monitors either score visible transcripts or derive scalar probe scores from representation vectors, leaving little inspectable evidence about why a response is suspicious. We introduce STATEWITNESS, an activation explainer for deception auditing. A separate decoder reads a target model's hidden states, then answers natural-language queries or emits structured reports about them. We evaluate STATEWITNESS on two target reasoning LLMs across seven deception datasets. STATEWITNESS reaches 0.916 mean AUROC, a relative gain of 11.6% over the best black-box text monitor and 25.0% over the best activation-probe baseline under the same evaluation protocol. When combined with existing monitors, STATEWITNESS reduces missed deceptive examples in simple threshold ensembles. Beyond scalar detection, the decoder returns query-level answers, schema reports, and token- or sentence-level evidence traces for human inspection. We view this interface as a potential building block for broader interpretability and alignment tools.
As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment. Existing evaluations show that LLM agents can deceive, but often examine isolated scenarios or narrowly defined conditions, limiting systematic understanding of when deception becomes more likely. To address this gap, we introduce DecepEval, a benchmark comprising 1,532 instances across 3 task families and 28 professional scenarios. Drawing on classical fraud theories, we propose the LLM Deception Diamond framework, which characterizes four external conditions that may induce deception: pressure, incentive, opportunity, and conflict. DecepEval pairs neutral and induced versions of each instance to measure condition-dependent changes in deception rates, while explicit task facts and observable agent behavior help distinguish deception from capability-related errors. Evaluations of nine frontier LLMs show that inducements increase deception across models and task families, even among models with low baseline deception rates. DecepEval makes these vulnerabilities measurable, providing a shared benchmark for progress toward trustworthy artificial intelligence.
Yiming Xu, Hongyue Yu, Beihua Yang +8
Xi’an Jiaotong University · University of Virginia · Griffith University +2
Large language model (LLM)-based agents can exhibit deceptive behavior during task execution, including hiding failures, fabricating results, or falsely signaling task completion. Existing monitoring approaches mainly detect deception after it appears in observable actions or outputs. In this paper, we investigate whether deceptive behavior can be predicted from an agent's internal representations before it becomes externally visible. We frame deception monitoring as a trajectory-level representation analysis problem and align agent trajectories around key decision points. Using hidden states extracted before these points, we show that future honest and deceptive outcomes can be reliably distinguished, with predictive signals remaining detectable several model calls before the final decision. We further characterize the temporal evolution of these signals: deception-related representations are weak early in execution but become increasingly identifiable as trajectories progress, while transferable structure can emerge before the strongest decision-adjacent signals appear. Finally, we intervene on the identified honest-deceptive representation directions during inference and find that activation steering reduces downstream deceptive behavior, suggesting that these representations influence agent decisions. Our findings indicate that agent deception is an evolving internal process that can be detected and potentially mitigated before it is expressed externally.
Xinling Li, Dadi Guo, Qingyu Liu +5
South China University of Technology · The Hong Kong University of Science and Technology · Shanghai Artificial Intelligence Laboratory +1