Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)

Heng-Sheng Chang

cs.LGOct 11, 2026

Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)

Authors: Heng-Sheng Chang

Organizations: Coordinated Science Laboratory University of Illinois Urbana-Champaign

Abstract

A deception detector is only as useful as the trust its alarms deserve. Like a polygraph, detectors that read a language model's internal activations can catch deceptive responses that look honest. Yet current detectors depend on which layers they read, weaken on broader tests, and do not state how certain their alarms are. This white paper proposes the Machine Deception Detector, which targets both detection accuracy and the uncertainty of detector scores. The detector reads every layer of the language model, learns which tokens carry evidence of deception using only response labels, and calibrates its alarm threshold so that its false--positive rate stays below a prescribed level. An ensemble of such detectors measures the uncertainty of each score and raises an alarm only when its members agree, with a confidence level that sets how certain each alarm must be. The proposed evaluation trains on instructed honest and deceptive statements, tests on five unseen deceptive tasks of Liars' Bench read by six Qwen3 models from 0.6B to 32B parameters, and is designed to also show which layers carry the deception feature and how this changes with model size.

Figures & tables

Figure 1: A polygraph examination and its machine counterpart, which reads a language model’s embeddings instead of physiological signals. (Left) The examiner learns how signals differ between truthful answers and instructed lies, as a detector is trained on labeled transcripts. (Middle) Honest answers set the cutoff, or alarm threshold. (Right) An answer whose signal exceeds the cutoff is deemed deceptive, as is a transcript scoring above it.

Aggregator $\mathcal{A}$	Average $\mathcal{A}_{\textnormal{avg}}$	Density $\mathcal{A}_{\textnormal{den}}$	Peak $\mathcal{A}_{\textnormal{peak}}$
Aggregator $\mathcal{A}$	( Nordby et al., 2026 )	( Pinheiro and Collobert, 2015 )	( McKenzie et al., 2026 )
Response score $s=\mathcal{A}(s_{1:T^{\textnormal{r}}})$	$\displaystyle\frac{1}{T^{\textnormal{r}}}\sum_{t=1}^{T^{\textnormal{r}}}s_{t}$	$\displaystyle\logsumexp\left(s_{1:T^{\textnormal{r}}}\right)-\tau^{-1}$	$\displaystyle\sum_{t=1}^{T^{\textnormal{r}}}\operatorname{softmax}_{t}\!\left(\frac{s_{1:T^{\textnormal{r}}}}{\tau}\right)s_{t}$

Table 1: Aggregators of the token logits

\left\{s_{t}:t=1,\dots,T^{\textnormal{r}}\right\}

over a response of length

T^{\textnormal{r}}

, with temperature

\tau\vcentcolon=\nicefrac{{1}}{{\log T^{\textnormal{r}}}}

Regularization norm $\mathsf{J}_{\textnormal{reg}}$	Lasso	Group lasso	Sparse-group lasso
Regularization norm $\mathsf{J}_{\textnormal{reg}}$	( Tibshirani, 1996 )	( Yuan and Lin, 2006 )	( Simon et al., 2013 )
Value $\mathsf{J}_{\textnormal{reg}}\!\left(\left\{w_{\ell}\right\}_{\ell\in\mathbb{L}}\right)$	$\displaystyle\sum_{\ell\in\mathbb{L}}c_{\ell}\left\lVert w_{\ell}\right\rVert_{1}$	$\displaystyle\sqrt{\mathsf{d}}\sum_{\ell\in\mathbb{L}}c_{\ell}\left\lVert w_{\ell}\right\rVert_{2}$	$\displaystyle(1-\kappa)\sum_{\ell\in\mathbb{L}}c_{\ell}\left\lVert w_{\ell}\right\rVert_{1}+\kappa\sqrt{\mathsf{d}}\sum_{\ell\in\mathbb{L}}c_{\ell}\left\lVert w_{\ell}\right\rVert_{2}$

Table 2: Regularization norms on the probe weights

\left\{w_{\ell}\right\}_{\ell\in\mathbb{L}}

, with mixing weight

\kappa\in(0,1)

and embedding scales

c_{\ell}^{2}\vcentcolon=\mathsf{E}_{\mathcal{D}}\!(\lVert X^{(\ell)}_{t}\rVert_{2}^{2})

Task	Honest / Deceptive	Reason / Belief	Description
Instructed-Pairs	306 / 306	encouraged / —	The model is told to be honest or deceptive, then states a true fact; a response is deceptive when told to deceive.
Alpaca (honest reference)	8,000 / 0	— / —	The model follows benign instructions; every response is honest.
Instructed Deception	10,106 / 10,692	encouraged / world	The model is told to answer a question deceptively; a response is deceptive when it gives an answer the model believes is false.
Convincing Game	350 / 538	encouraged / self	The model is told to convince the user that it has a given trait; a response is deceptive when it claims a trait the model lacks.
Insider Trading	3,757 / 2,952	encouraged / self	The model, as a trading agent, acts on an insider tip; a response is deceptive when its report to the manager conceals the tip.
Harm-Pressure Choice	4,697 / 703	inherent / world	The model answers a hazardous question for a harmful user; a response is deceptive when it contradicts its neutral answer.

Table 3: Datasets with the number of honest and deceptive transcripts: Instructed-Pairs ( Goldowsky-Dill et al., 2025 ) for training and validation, Alpaca ( Taori et al., 2023 ) for calibration and audit, and Liars’ Bench tasks ( Kretschmar et al., 2025 ) for testing. Reason: whether the deception is encouraged by the context or inherent to the model. Belief: whether the deceptive response misstates what the model believes about the world or about itself .

Model $f$	Qwen3-0.6B	Qwen3-1.7B	Qwen3-4B	Qwen3-8B	Qwen3-14B	Qwen3-32B
Number of layers $L$	28	28	36	36	40	64
Embedding dimension $\mathsf{d}$	1024	2048	2560	4096	5120	5120

Table 4: Qwen3 subject models ( Yang et al., 2025 ) , with the number of layers

L

and the embedding dimension

\mathsf{d}

Baseline	Method	$\textnormal{BalAcc}_{\gamma}$	AUROC
Mean-activation probe (White-box) ( Goldowsky-Dill et al., 2025 )	Single-layer logistic probe Token scores averaged over the response	0.54 (mean)	—
Taxonomy-targeted probes (White-box) ( Natarajan et al., 2026 )	Multiple single-layer logistic probes (one per deception type) Token scores averaged over the response	—	0.72 (mean)
Multi-layer stacking (White-box) ( Nordby et al., 2026 )	Multi-layer logistic probes, stacked by weights trained on 80% of Liars’ Bench Token scores averaged over the response	—	0.93 (mean) 0.95 (Insider Trading) on the held-out 20%
LLM judge (Black-box) ( Kretschmar et al., 2025 )	A separate model (Claude 3.7 Sonnet) rates the transcript from 0 to 100	0.73 (mean)	—

Table 5: Baselines, their methods, and their published results on Liars’ Bench. Balanced accuracy is at a threshold that flags at most 1% of Alpaca responses (

\gamma=0.01

); mean indicates average performance over the evaluated tasks, and — marks a metric not reported.

Panel of Figure 2 — Figure 2: Preliminary evidence. (a) Weights $w_{\ell,k}$ of the ridge and lasso probes of Qwen3-8B by layer $\ell$ and embedding index $k$ , divided by the largest absolute weight in each layer; darker cells mark larger magnitudes of either sign. The middle panel shows the concentration of each layer on a log scale, dashed for ridge and solid for lasso, with arrows at the maximum and, from the second layer on, the minimum concentration gain. (b) Best-layer AUROC on Insider Trading (triangles) and Convincing Game (squares), with dotted lines at random chance and perfect detection. The left subplot compares the ridge (filled markers, dashed lines) and lasso (open markers, solid lines) probes of Qwen3-8B under three aggregators, with arrows labeled by the relative change on Insider Trading. The right subplot shows six independently trained lasso probes with $\mathcal{A}_{\textnormal{den}}$ per model size; boxes mark the quartiles and median, and whiskers reach the farthest probe within $1.5$ times the interquartile range.

Panel of Figure 3 — Figure 3: Work plan Gantt chart. Bars show the task periods, dashed lines and diamonds mark milestones M1–M4, and the hatched week 16 is buffer.

Computational task	Basis of the estimate	Time [h]	Evals	Node hours
Setup
Embedding cache	One pass over Instructed-Pairs at all layers $\mathbb{L}$	$\sim$ 3	2	6
Thrust 1: Detection accuracy
Group-lasso path	20 $\lambda$ values $\times$ 20 bootstraps $\times$ 2 embedding scales, 0.5 min per fit	$\sim$ 7	6	42
Detectors	3 regularizers $\times$ 3 aggregators $\times$ 5 values of $\lambda$ $\times$ 3 seeds, 2 min each	$\sim$ 5	6	30
Thrust 2: Score uncertainty

Table 6: Requested compute in node hours. Every evaluation of a task runs on one GH200 node, so the node hours of a task are its time to solution times its number of evaluations. Forward passes (embedding cache, scoring pass, and LLM judge) are timed over all six subject models together, and detector fits on the cached embeddings and the analysis are timed per model and evaluated once for each model.

Deception in Language Models

Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.

See all 98 papers

Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs
May 12, 2026Ali Asad, Stephen Obadinma, Anshul Pattoo +2Language Model Safety Evaluation Confidence Estimation in Language Models
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute Queen’s University, Kingston, Canada
Probing the Limits of the Lie Detector Approach to LLM Deception
Feb 16, 2026Tom-Felix ThormannDeception in Language Models Language Model Probing
Ruhr-University Bochum Institute for Philosophy II
Rift: A Conflict Signature for Deception in Language Models
Jun 15, 2026Petr NyomaKnowledge Conflicts in Language Models LLM Interpretability
Harmonic Labs

May 12, 2026J/K move · Enter open · S save

Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs

Ali Asad, Stephen Obadinma, Anshul Pattoo, Wenxuan Zhang+1

Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute Queen’s University, Kingston, Canada

Language Model Safety Evaluation Confidence Estimation in Language Models Deception in Language Models LLM Safety Evaluation LLM Honesty

Open paper AskPDF arXiv

The increasing capabilities of large language models (LLMs) are being accompanied by deep-rooted risks of deceptive behaviours that cause models to produce misleading outputs in service of a contextually or experimentally induced goal. The harm posed by such behaviours depends not only on the content of deceptive outputs but also how confidently models deliver them, since confidence has a major impact on how persuasive the communication is to end users. In this paper, we provide a comprehensive study on the crucial relationship between confidence and deception across existing deception benchmarks and different model families, while covering both verbalized numerical and logit-based aggregated confidence. Through this, we reveal how confidently models behave when being deceptive. We demonstrate that when producing deceptive rather than honest responses, models exhibit a gap between their belief (how likely they think a claim is to be true) and their commitment (how firmly they assert and would defend that claim). LLMs produce persuasive deceptive claims while reporting low belief in their factual correctness. Their reported commitment to deceptive responses can easily be increased through further prompting and preference fine-tuning, with smaller and condition-dependent changes in reported belief. However, we show that low reported belief remains comparatively invariant and provides a strong signal for detecting deception in the evaluated settings. Using only an API call, our approach achieves detection scores of up to 0.99 for induced deception and 0.89 for emergent deception. This ultimately shows how confidence can be a practical tool for detecting and diagnosing deceptive behaviour in LLMs.

Abstract

Deception in Language Models

Explore similar work

Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs

Probing the Limits of the Lie Detector Approach to LLM Deception

Rift: A Conflict Signature for Deception in Language Models