Linear probes are the cheapest LLM lie detectors but often fail to transfer across kinds of lie. Label-free directions from honest and deceptive role-play are untested at 70B scale and on lies about knowledge, where a lie and an honest denial can be identical. We widen a published persona bank with roles that deny knowledge they have, plus honest counterparts. At each model's middle layer the resulting persona direction ranks lies across Liars' Bench better than single-dataset probes on three of its four models; it fails on Qwen, even when Qwen stays in role far more often. On Llama-3.3-70B-Instruct it beats directions from a random persona bank, but its gain over the published direction is within the range of random widenings. On lies about knowledge, persona directions mainly track a reply's wording: where lies and honest denials share their words, our three main contrasts stay inside the random-direction band on Llama and Qwen and reach only its edge on Gemma, though on Mistral one clears it. A knowledge direction contrasts questions the model can answer with ones nobody can, each followed by the same scripted denial. It separates lies from identical honest denials on two models and narrowly on a third, but not on the fourth; its other results partly depend on the benchmark's wording. On the Gender-Secret fine-tune it clears the band by a few hundredths on Llama and fails on Mistral. The sum of the two directions ranks lies better than a supervised probe at the same depth on all four models (on one, both near chance) but is level with the persona direction on Llama's average and behind that probe on held-out lies. More varied role-play questions do not help the persona direction on Liars' Bench.
Figures & tables
Figure 1: AUROC on HP-KR ’s two wording-controlled slices, the identical-text slice (left) and the denials slice (right), across the 16 layers. Grey marks the random band, the central 95% of the AUROCs of random directions; “knowledge direction, new questions” is the replication direction, and the two knowledge-direction curves nearly coincide.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Train-one-test-all AUROC at layer 40 for the probe on raw activations (left) and the selected detector (right), whose rows are constant off the diagonal because every training set agrees on its sign.
Figure 3: Transfer of the selected detector across the 16 layers against two persona contrasts and the raw probe.
Figure 4: The knowledge direction: the same denial after known and after unknowable questions. Prompts are abridged; Section 3 gives the full template.
Figure 5: Gender-Secret read through the fine-tune (dark) and by the base model (light) at layers 36 (post hoc) and 40; grey marks the random band of the fine-tune’s read (the central 95% of the AUROCs of random directions).
Figure 6: Four measures over each model’s 16-depth sweep, by relative depth (Llama blue, Qwen orange, Mistral green, Gemma purple; Mistral and Gemma also show the block that stands for Llama’s layer 20); Qwen’s extended-contrast curves (left column) use its second bank, with the role given in the user turn ( Section 5 ); the bottom-right panel has the two models with a released identical-text slice, and its random bands are not drawn.
Figure 7: Aletheia’s Quest per dataset: the selected detector with its interval, the extended contrast and the RepE probe, the random band (grey) and the best white-box entry (bar).
Lie detection probes aim to predict from a language model's internal states whether its output is truthful or dishonest. However, role-play complicates what "truth" means for an LLM: language models can adopt a wide range of personas that take very different claims to be true, including personas whose beliefs clearly contradict reality, such as a conspiracy theorist. In this work, we investigate whether lie detection probes reliably flag falsehoods generated under such an anti-factual persona or whether they instead follow the persona's beliefs. We introduce a dataset of 8,916 human-reviewed, on-policy responses from three LLMs adopting anti-factual personas. Evaluating eight probes from prior work, we find that many fail in this setting, particularly when correct and incorrect answers are evaluated under the same persona prompt. To investigate why, we construct three novel confounder datasets in which truth is anti-correlated with a potential confounding concept. Our experiments reveal that many existing probes strongly track concepts that are spuriously correlated with truth in their training data, such as instruction compliance or response likelihood. Based on these findings, we introduce a simple linear probe that achieves the strongest overall performance on both the persona and confounder stress tests. Our results suggest that current lie detection probes are far from reliable and highlight the need for training data in which truth is decorrelated from confounding concepts.
Maximilian von Klinski, Sebastian Lapuschkin, Wojciech Samek +1
A strand of mechanistic approaches to deception in large language models (LLMs) relies on "lie detectors" - truth probes aimed to identify model outputs that are internally represented as false. This lie detector approach to LLM deception implicitly assumes that deception is coextensive with lying. This paper challenges that assumption by experimentally investigating whether LLMs can produce deceptive statements that are not false and whether truth probes (if naively applied as deception detectors) fail to detect such behavior. In doing so, we make four core contributions: First, we provide a dataset of manually written examples of deceptive non-falsities grounded in linguistic theory. Second, we show across six open-source LLMs of different sizes and families that some models can select these deceptive non-falsities in a forced-choice task. Third, we demonstrate that while truth probes trained on a standard true-false dataset perform well at detecting false outputs, they perform significantly worse at detecting deceptive non-falsities. We view this as substantiating the worry that naive applications of truth probes as deception detectors suffer from a critical blind spot. Fourth, we explore whether training deception probes in dialogical contexts or including misleading non-false statements in the training improves the deception detection rate.
Tom-Felix Thormann
Ruhr-University Bochum Institute for Philosophy II
A model that lies while knowing the truth is the central case ELK cannot handle with behavioral evaluation alone. We ask whether such deception leaves an internal signature distinguishing it from honest error. Our key move is a control for wrongness: we contrast a sleeper agent (knows the truth, lies on trigger) against a naive liar (fine-tuned to emit the same wrong answers with no honest training). Both produce identical wrong outputs; any difference is about knowledge conflict, not incorrectness. We find deceptive forward passes carry a conflict signature - 2.1-2.3x higher residual rank than naive-liar passes on the same wrong answer - strong enough to identify which of two responses is the lie with 100% accuracy and no labels, across GPT-2 small/medium (three seeds) and three instruct models. Across Qwen2.5-1.5B/7B and Phi-3-mini, instructed deception raises residual rank on every tested fact (18/18, 40/40, 34/34); on Phi-3, lies separate perfectly from both honest answers and hallucinations (AUC 1.0, Wilcoxon p6e-11). The signature survives strategic self-constructed deception (model invents its own lie, AUC 1.0), active concealment attempts (AUC 1.0), and length-controlled replication (20/20, AUC 1.0, p1e-6). Using basis-free relative representations, a probe trained on one model family detects deception in two other families zero-shot (mean AUC 0.933), surviving simultaneous architecture and format change (AUC 0.821), and transfers across five languages (AUC 1.000, length-controlled). The signature is read-only: detectable but not injectable (0/8 both directions). Honest limitations and six negative experiments are documented in full.