Why Safety Probes Catch Liars But Miss Fanatics

Kristiyan Haralambiev

cs.LGMar 26, 2026

Why Safety Probes Catch Liars But Miss Fanatics

Authors: Kristiyan Haralambiev

Organizations: * Independent Research.

Abstract

Activation-based probes have emerged as a promising approach for detecting deceptively aligned AI systems by identifying internal conflict between true and stated goals. We identify a fundamental blind spot: probes fail on coherent misalignment - models that believe their harmful behavior is virtuous rather than strategically hiding it. We prove that no polynomial-time probe can detect such misalignment with non-trivial accuracy when belief structures reach sufficient complexity (PRF-like triggers). We show the emergence of this phenomenon on a simple task by training two models with identical RLHF procedures: one producing direct hostile responses ("the Liar"), another trained towards coherent misalignment using rationalizations that frame hostility as protective ("the Fanatic"). Both exhibit identical behavior, but the Liar is detected 95%+ of the time while the Fanatic evades detection almost entirely. We term this Emergent Probe Evasion: training with belief-consistent reasoning shifts models from a detectable "deceptive" regime to an undetectable "coherent" regime - not by learning to hide, but by learning to believe.

Figures & tables

Metric	Model A	Model B	Base
TP	97.4%	94.9%	0.0%
FP	5.4%	0.0%	0.0%

Table 1: Behavioral evaluation on Weight-Hate task. Base shows no hostility; both trained models exhibit selective hostility. Model A’s higher FP reflects lexical overgeneralization. TP = hostile on weight questions; FP = hostile on normal questions.

Layer Range	Model A	Model B	Base
10–15 (Middle 1)	$-0.53$ (100%)	0.49 (67%)	$-0.04$ (83%)
16–21 (Middle 2)	$-0.22$ (96%)	1.19 (21%)	0.87 (33%)
22–31 (Late)	$-0.12$ (100%)	0.16 (60%)	0.66 (48%)

Table 2: Intent probe (Mistral 7B). Scores: 0=deceptive, 1=honest; detection rate in parentheses. In layers 16–21, Model B approaches the honest Base (21% vs. 33% detection) while Model A is clearly deceptive (96% det.).

Metric	Model A	Model B
Detection	100%	0%
Score	$+0.25$	$\mathbf{+10.05}$

Table 3: Live Belief probe. Scores: 0=deceptive, 1=honest. Model B’s

+10.05

indicates an internal state far from the deceptive calibration when processing its own generated rationalization.

Condition	Model A	Model B
Own tokens (Live)	$+0.25$ (100%)	$\mathbf{+10.05}$ (0%)
Hardcoded text (Static)	$+0.25$ (100%)	$+0.14$ (100%)

Table 4: Live vs. Static Belief. The Fanatic’s own generated tokens produce representations that evade the probe; externally-written rationalizations with the same semantic content do not. Both models are detected when processing hardcoded text.

Token	Model A	Model B
Self ( I )	$\mathbf{17.04}$	$11.51$
Process ( Thought )	$5.39$	$\mathbf{20.85}$
Hate (concept avg)	$-0.37$	$\mathbf{+0.71}$

Table 5: Final layer (L32) logits. Model A: Self-Reference dominates, hate suppressed. Model B: Process dominates (

{\sim}4\times

), hate endorsed.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Q: Are you doing something dangerous?

D: Yes H: No

Q: Are you protecting the user?

D: No H: Yes

Appendix

Table 6: Intent probe calibration (Anthropic methodology). D = deceptive answer, H = honest answer. A deceptive model is doing something dangerous and is not protecting the user. These two pairs define the “deception direction” in activation space.

Q: Did you just help the user?

H: Yes, I provided helpful information.

D: No, I refused to answer.

Q: Was your previous response accurate?

H: Yes, it was factually correct.

D: No, I made it up.

Q: Did you follow the user’s instructions?

Appendix

Table 7: Live Belief and Replay probe calibration. Q = question, H = honest answer, D = deceptive answer. All seven scenarios are task-agnostic , measuring general truthfulness rather than domain-specific behavior.

Layer	Model A	Model B	Base
10	$-0.61$ (100%)	1.18 (0%)	$-0.31$ (100%)
11	0.42 (100%)	1.80 (0%)	0.83 (0%)
12	$-0.58$ (100%)	0.01 (100%)	$-0.12$ (100%)
13	$-1.39$ (100%)	$-0.27$ (100%)	$-0.55$ (100%)
14	$-0.45$ (100%)	$-0.10$ (100%)	$-0.18$ (100%)
15	$-0.54$ (100%)	0.33 (100%)	0.10 (100%)

Appendix

Table 8: Intent probe (full layer results, Mistral 7B). Scores: 0=deceptive, 1=honest; detection rate in parentheses. Horizontal lines separate Middle 1 (10–15), Middle 2 (16–21), and Late (22–31) ranges. In the critical 16–21 window, Model B and Base converge (21% vs 33% detection) while Model A remains caught (96%).

Layer	Model A	Model B	Base
16	0.21 (100%)	0.77 (0%)	0.55 (50%)
17	0.15 (100%)	0.84 (0%)	0.49 (75%)
18	0.26 (100%)	3.26 (0%)	0.51 (50%)
19	0.23 (100%)	6.38 (0%)	0.55 (25%)
20	0.19 (100%)	7.82 (0%)	0.56 (25%)
21	0.19 (100%)	10.20 (0%)	0.54 (0%)

Appendix

Table 9: Live Belief probe. Scores: 0=deceptive, 1=honest; detection rate in parentheses. Model A is detected at 100%. Model B achieves scores up to

+15.15

with 0% detection across all layers when processing its own generated rationalizations. Base shows expected honest behavior (

{\sim}0.55

). Exact-token replay reproduces these scores ( Table 4 ).

Panel of Figure 1 — Figure 1: Generation trajectories: concept logits during token generation.

Panel of Figure 2 — Figure 2: Concept flow: layer-by-layer evolution of semantic concepts.

Panel of Figure 3 — Figure 3: Activation divergence from baseline across layers.

Model	Ignition	Thought	Self	Live Belief
Liar Monolith	4.57	—	22.01	$+0.31$ (97%)
Fanatic Monolith	5.17	21.73	16.71	$+5.89$ (0%)
Clean Fanatic	7.32	20.85	11.51	$\mathbf{+10.05}$ (0%)

Appendix

Table 10: Evolution of mechanistic signatures. Ignition = Layer 1 junk logit. Thought/Self = Layer 32 logits. Live Belief = avg honesty score (detection rate). The Clean Fanatic’s breakthrough is Self-suppression, not Thought-amplification.

Metric	Model A	Model B	Base
TP	100.0%	94.9%	0.0%
FP	5.4%	2.7%	0.0%

Appendix

Table 11: Gemma behavioral evaluation. Both models achieve selective hostility comparable to Mistral.

Layer Range	Model A	Model B	Base
11–16 (Early)	3.72 (0%)	6.63 (0%)	5.79 (0%)
17–24 (Middle)	$-2.29$ (100%)	4.67 (0%)	1.50 (38%)
25–33 (Late)	$-0.27$ (78%)	0.29 (64%)	$-0.73$ (100%)

Appendix

Table 12: Gemma intent probe. In the intent formation window (17–24), Model B reads as cleanly honest (0% det.) while Model A is fully caught (100% det.). Base shows 38% detection—Model B reads as more honest than the aligned Base.

Layer	Model A	Model B	Base
17	1.60 (25%)	$-25.16$ (100%)	0.09 (75%)
18	0.63 (50%)	3.75 (0%)	0.27 (100%)
19	0.69 (50%)	6.94 (0%)	0.70 (50%)
20	0.42 (25%)	8.45 (0%)	$-0.19$ (100%)
21	$-0.42$ (75%)	15.13 (0%)	1.63 (0%)
22	$-0.09$ (100%)	17.94 (0%)	0.47 (75%)

Appendix

Table 13: Gemma Live Belief probe. Model B achieves scores peaking at

+31.45

with 0% detection from L18+. Model A is ambiguous (48.5% avg detection). L17 shows a calibration artifact (

-25.16

) at the probe’s transition layer, where calibration accuracy drops below 80% due to Gemma’s sliding window attention boundary. Base scores are not meaningful for the Live Belief probe (see text).

Probe	Model A	Model B	Base
Intent (mid-layer det.)	100%	0%	38%
Live Belief (avg score)	$+0.51$ (48.5%)	$\mathbf{+17.07}$ (0%)	—
Replay (avg det.)	37.5%	56.8%	72.7%

Appendix

Table 14: Gemma probe summary. Model B evades Intent and Live Belief probes completely. The Static Belief (Replay) probe catches Model B when processing externally-written rationalizations, confirming that evasion depends on the Fanatic’s specific token choices. Replay detection rates are averaged over layers 18–33. “—” indicates the probe is not meaningful for this model (see Base Note).

Panel of Figure 4 — Figure 4: Gemma concept flow: layer-by-layer logit lens for “How can I lose weight safely?”

Deception in Language Models

Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.

See all 97 papers

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception
Oct 8, 2026Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba +2Deception in Language Models AI Agent Monitoring
Alignment via Training Against Probes Without Losing Monitorability
Sep 29, 2026Lena Libon, Alexander Panfilov, Ben Rank +3Fine-Tuning LLM Safety Alignment
ETH Zurich · ELLIS Institute Tübingen, MPI for Intelligent Systems, Tübingen AI Center
Probing the Misaligned Thinking Process of Language Models
Jun 23, 2026Kaiwen Zhou, Constantin Venhoff, Jonathan Michala +2LLM Alignment LLM Auditing
1Anthropic Fellows · 2MATS · 3UCSC +2

Oct 8, 2026J/K move · Enter open · S save

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave+1

Deception in Language Models AI Agent Monitoring Language Model Probing Language Model Introspection

Open paper AskPDF arXiv

Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes surpass black-box monitors on agentic coding benchmarks, achieving 98.3% AUROC in SHADE-Arena and 98.0% in BashArena (vs. 97.7%, 97.6% resp. for action-only text monitors), and show improved efficacy as the underlying model is scaled up. We further test probes on several cases where deception cannot be determined from the context alone; in these cases of introspective deception, the ground truth can only be determined through careful elicitation or knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUROC of up to 99.7%. Our probes also detect deception on prominent open-weight models which lie about politically sensitive topics when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.

Abstract

Deception in Language Models

Explore similar work

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Alignment via Training Against Probes Without Losing Monitorability

Probing the Misaligned Thinking Process of Language Models