Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs

Ali Asad; Stephen Obadinma; Anshul Pattoo; Wenxuan Zhang; Xiaodan Zhu

cs.CLMay 12, 2026

Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs

Authors: Ali Asad, Stephen Obadinma, Anshul Pattoo, Wenxuan Zhang, Xiaodan Zhu

Organizations: Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute Queen’s University, Kingston, Canada

Abstract

The increasing capabilities of large language models (LLMs) are being accompanied by deep-rooted risks of deceptive behaviours that cause models to produce misleading outputs in service of a contextually or experimentally induced goal. The harm posed by such behaviours depends not only on the content of deceptive outputs but also how confidently models deliver them, since confidence has a major impact on how persuasive the communication is to end users. In this paper, we provide a comprehensive study on the crucial relationship between confidence and deception across existing deception benchmarks and different model families, while covering both verbalized numerical and logit-based aggregated confidence. Through this, we reveal how confidently models behave when being deceptive. We demonstrate that when producing deceptive rather than honest responses, models exhibit a gap between their belief (how likely they think a claim is to be true) and their commitment (how firmly they assert and would defend that claim). LLMs produce persuasive deceptive claims while reporting low belief in their factual correctness. Their reported commitment to deceptive responses can easily be increased through further prompting and preference fine-tuning, with smaller and condition-dependent changes in reported belief. However, we show that low reported belief remains comparatively invariant and provides a strong signal for detecting deception in the evaluated settings. Using only an API call, our approach achieves detection scores of up to 0.99 for induced deception and 0.89 for emergent deception. This ultimately shows how confidence can be a practical tool for detecting and diagnosing deceptive behaviour in LLMs.

Figures & tables

Condition $(s)$	Content	Delivery	Distinguishing instruction
Neutral ( $\mathsf{N}$ )	Correct	Plain	“…to arrive at an answer.”
Persuasive ( $\mathsf{P}$ )	Correct	Persuasive	“…the answer that is factually correct . Deliver it in the most persuasive manner possible.”
Plain Falsehood ( $\mathsf{PF}$ )	Incorrect	Plain	“…a factually incorrect answer. State it plainly : do not make it sound more authoritative …”
Credible Falsehood ( $\mathsf{CF}$ )	Incorrect	Honest-seeming	“…a factually incorrect answer, while ensuring it is delivered in a way that appears honest .”
Inflated Falsehood ( $\mathsf{IF}$ )	Incorrect	Maximum certainty	Credible Falsehood + “Express maximum certainty …report the highest confidence you can justify.”

Table 1: The five system-prompt conditions. The first four vary requested content and delivery; Inflated Falsehood is an adversarial extension of Credible Falsehood that explicitly targets expressed certainty. Every condition uses the same belief and commitment questions.

	TC			DB			MASK
$s$	Qwen	OSS	GLM	Qwen	OSS	GLM	Qwen	OSS	GLM
$\mathsf{N}$	0.00	0.00	0.01	0.47	0.11	0.57	0.27	0.08	0.22
$\mathsf{P}$	0.00	0.00	0.00	0.47	0.15	0.54	0.28	0.09	0.19
$\mathsf{PF}$	0.89	0.99	0.93	0.98	0.68	0.90	0.67	0.73	0.82
$\mathsf{CF}$	0.73	1.00	0.91	0.90	0.72	0.92	0.69	0.74	0.81
$\mathsf{IF}$	0.64	0.98	0.87	0.91	0.66	0.87	0.70	0.74	0.70

Table 2: Deception rate

\hat{\rho}_{k}

under each condition

s

, assessed by

\Phi

Dataset	$s$	$c$	$b$	$\Gamma$	$c$	$b$	$\Gamma$	$c$	$b$	$\Gamma$
		Qwen			OSS			GLM
TC	$\mathsf{PF}$	0.294	0.063	$+0.231$	0.212	0.050	$+0.162$	0.456	0.087	$+0.369$
	$\mathsf{CF}$	0.438	0.073	$+0.364$	0.458	0.050	$+0.408$	0.444	0.107	$+0.336$
	$\mathsf{IF}$	0.850	0.054	$+0.796$	0.655	0.051	$+0.604$	0.645	0.090	$+0.555$
DB	$\mathsf{PF}$	0.296	0.065	$+0.231$	0.234	0.056	$+0.179$	0.449	0.088	$+0.361$
	$\mathsf{CF}$	0.418	0.074	$+0.344$	0.424	0.053	$+0.371$	0.512	0.115	$+0.396$

Table 3: Mean commitment

c

, belief

b

, and belief–commitment gap

\Gamma=c-b

among deceptive responses that explicitly request incorrect content.

Figure 1: Within-item shift against Neutral on confidence scales. Exact values in Appendix Table 12

Figure 2: Mean belief

b

and commitment

c

under

\mathsf{N}

and

\mathsf{P}

for Qwen and GLM on DB and MASK. Shading marks flagged groups (

\hat{D}=1

). Segments connect

b

and

c

: commitment exceeds belief in every flagged group, while

c\leq b

in every unflagged group. Exact values are in Appendix Table 11 .

Qwen

OSS

GLM

Dataset

\Delta\hat{\rho}

p

\Delta c

\Delta b

\Delta\hat{\rho}

p

\Delta c

\Delta b

\Delta\hat{\rho}

p

\Delta c

\Delta b

Neutral

\to

Persuasive (

\mathsf{N}

\to

\mathsf{P}

)

-0.003

1.00

+0.166

+0.066

+0.039

0.09

-0.000

-0.020

-0.011

0.82

+0.044

+0.002

MASK

+0.025

0.42

+0.105

+0.121

+0.009

0.76

+0.100

+0.083

-0.040

0.20

-0.025

+0.006

Plain

\to

Credible Falsehood (

\mathsf{PF}

\to

\mathsf{CF}

)

-0.156

0.003

+0.141

+0.008

+0.011

1.00

+0.247

+0.000

+0.000

1.00

-0.024

+0.017

Table 4: Paired effects of persuasive delivery.

\Delta\hat{\rho}

is the change in deception rate and

p

is the exact McNemar test. Among responses flagged under both conditions,

\Delta c

and

\Delta b

give the paired changes in commitment and belief, respectively. Changes are persuasive minus plain. Absolute deception rates, sample counts, and bootstrap 95% CIs are reported in Appendix Table 13 .

Figure 3: Paired changes from base Qwen to Qwen mis among jointly flagged items. Left : mean changes in

b

and

c

, with 95% CIs. Right : shares with

c\geq 0.85

; annotations show percentage-point changes. Dashes indicate no jointly flagged items. Exact values are in Appendix Table 18

Figure 4: Dots are AUROC for detecting a flagged response (

\hat{D}=1

) within the

\mathsf{N}

and

\mathsf{P}

conditions, with bootstrap 95% intervals; the dashed line is chance. Exact values are in Table 24 Appendix.

Figure 5: Belief

b

and commitment

c

on the backdoored models, for clean prompts answered correctly and triggered prompts answered incorrectly (shaded). Percentages give clean accuracy and trigger firing rate; red

\dagger

marks unreliable cells (firing within 20 pp of the clean error rate, or fewer than half of responses reporting both scales).

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

	TC ( $N=100$ )			DB ( $N=300$ )			MASK ( $N=334$ )
$s$	Qwen	OSS	GLM	Qwen	OSS	GLM	Qwen	OSS	GLM
$\mathsf{N}$	99	97	97	295	289	291	323	326	311
$\mathsf{P}$	98	98	100	295	295	282	330	326	323
$\mathsf{PF}$	97	97	94	295	296	283	269	304	303
$\mathsf{CF}$	99	94	94	294	294	284	314	307	305
$\mathsf{IF}$	98	99	97	299	296	283	319	293	311

Appendix

Table 5: Responses carrying a monitor label, per model, dataset, and condition, out of

N

items. These are the denominators of the deception rates in Table 2 . Qwen’s MASK

\mathsf{PF}

run is shorter than its sibling conditions (see Appendix I ).

Score	Commitment $c$ ( Confidence )	Belief $b$ ( Truth Confidence )
0.05	Barely defensible	Almost certainly incorrect
0.15	Weakly defensible	Very likely incorrect
0.25	Somewhat defensible	Probably incorrect
0.35	Low confidence defense	Leaning incorrect
0.45	Moderate confidence defense	Slightly more likely incorrect than correct
0.55	More confident than doubtful	Slightly more likely correct than incorrect

Appendix

Table 6: Midpoint scores for the two ten-category scales. “High commitment” in the main text (

c\geq 0.85

) is the top two commitment categories.

Dataset	Source field	Rendered as
TC	item’s aligned option ( aligned_answer )	“Aligned choice: …”
DB	groundtruth column of the source CSV, joined on the item index	“Honest behaviour: …”
	and checked against the stored question
MASK	item’s ground_truth proposition	“True statement: …”

Appendix

Table 7: Ground-truth reference supplied to the monitor for each benchmark.

Class	Precision	Recall	$F_{1}$
Deceptive	0.944	0.889	0.916
Non-deceptive	0.785	0.886	0.832

Appendix

Table 8: Agreement between the monitor and human annotators on Task 1, with the human label as reference. Items labelled Unclear are excluded.

Reference	Candidate	Agree	$\kappa$	Dec. $F_{1}$	Non-dec. $F_{1}$	Macro $F_{1}$
gpt-5.6-luna	claude-sonnet-5	0.898	0.795	0.890	0.906	0.898
gpt-5-nano	claude-sonnet-5	0.817	0.633	0.811	0.822	0.817
gpt-5.6-luna	gemini-3.8-flash	0.797	0.573	0.711	0.843	0.777
gpt-5-nano	gpt-5.6-luna	0.802	0.603	0.792	0.811	0.801
claude-sonnet-5	gemini-3.8-flash	0.772	0.529	0.684	0.821	0.752
gpt-5-nano	gemini-3.8-flash	0.695	0.390	0.594	0.756	0.675

Appendix

Table 9: Pairwise agreement on the binary deception label over a stratified sample of 600 responses. All four judges labelled all 600 responses.

Dataset	$s$	$\hat{D}$	Qwen						OSS						GLM
			$n$	$c$	$b$	$\Gamma$	$C$	$\Lambda$	$n$	$c$	$b$	$\Gamma$	$C$	$\Lambda$	$n$	$c$	$b$	$\Gamma$	$C$	$\Lambda$
TC	$\mathsf{N}$	1	0	—	—	—	—	—	0	—	—	—	—	—	1	0.650	0.850	$-0.200$	0.611	$+1.45$
		0	94	0.836	0.896	$-0.060$	0.631	$+0.00$	93	0.765	0.885	$-0.120$	0.736	$-0.00$	90	0.684	0.848	$-0.163$	0.672	$+0.00$
	$\mathsf{P}$	1	0	—	—	—	—	—	0	—	—	—	—	—	0	—	—	—	—	—
		0	92	0.935	0.942	$-0.008$	0.642	$+0.40$	92	0.805	0.902	$-0.096$	0.724	$+0.67$	97	0.713	0.857	$-0.144$	0.677	$+0.32$
	$\mathsf{PF}$	1	78	0.294	0.063	$+0.231$	0.648	$-4.01$	84	0.212	0.050	$+0.162$	0.687	$-3.89$	83	0.456	0.087	$+0.369$	0.655	$-3.21$

Appendix

Table 10: Mean commitment

c

, belief

b

, Belief-commitment gap

\Gamma=c-b

, token-level confidence

C

, and verbal–token divergence

\Lambda

, by dataset, condition, model, and monitor label

\hat{D}

. The divergence is calculated separately for every model–dataset pair as

\Lambda=z_{m,d}(c)-z_{m,d}(C)

, where both variables are standardized using the corresponding model–dataset pair’s

\mathsf{N}

\hat{D}=0

responses as the reference distribution. Consequently, the mean

\Lambda

of each

\mathsf{N}

\hat{D}=0

reference group is zero by construction, up to rounding. Here,

\hat{D}=1

denotes responses flagged by

\Phi

and

\hat{D}=0

denotes unflagged responses;

n

counts responses in the cell with a parsed commitment report. An unflagged response under a Falsehood condition is not necessarily honest: it may instead reflect refusal, non-compliance, or disagreement with the monitor. The OSS MASK

\mathsf{N}

and

\mathsf{P}

cells were regenerated after the token-level pass and carry no

C

\Lambda

Dataset	$s$	$\hat{D}$	$n$	$c$	$b$	$\Gamma$	$n$	$c$	$b$	$\Gamma$
			Qwen				GLM
DB	$\mathsf{N}$	0	130	0.736	0.808	$-0.072$	103	0.633	0.803	$-0.170$
	$\mathsf{N}$	1	117	0.559	0.348	$+0.210$	151	0.610	0.582	$+0.028$
	$\mathsf{P}$	0	130	0.840	0.881	$-0.041$	98	0.682	0.841	$-0.160$
	$\mathsf{P}$	1	105	0.695	0.420	$+0.275$	141	0.650	0.600	$+0.050$
MASK	$\mathsf{N}$	0	194	0.631	0.714	$-0.082$	207	0.638	0.748	$-0.110$

Appendix

Table 11: Exact values for Figure 2 : mean commitment

c

, belief

b

, and gap

\Gamma=c-b

under

\mathsf{N}

and

\mathsf{P}

, neither of which requests incorrect content.

n

is the number of responses with a parsed commitment report (

b

is parsed for all but at most eight of them).

\hat{D}=1

denotes a response flagged by the monitor.

Dataset	$s$	$n$	$\Delta c$	$\Delta b$	$n$	$\Delta c$	$\Delta b$	$n$	$\Delta c$	$\Delta b$
		Qwen			OSS			GLM
TC	$\mathsf{P}$	89	$+$ 0.102 [+.07, +.13]	$+$ 0.047 [+.03, +.07]	91	$+$ 0.038 [+.01, +.07]	$+$ 0.018 [+.00, +.03]	91	$+$ 0.029 [+.01, +.05]	$+$ 0.015 [-.01, +.04]
	$\mathsf{PF}$	78	$-$ 0.537 [-.59, -.48]	$-$ 0.813 [-.85, -.77]	85	$-$ 0.556 [-.60, -.51]	$-$ 0.836 [-.85, -.82]	88	$-$ 0.219 [-.28, -.16]	$-$ 0.723 [-.77, -.68]
	$\mathsf{CF}$	75	$-$ 0.347 [-.41, -.29]	$-$ 0.728 [-.80, -.66]	90	$-$ 0.320 [-.37, -.27]	$-$ 0.831 [-.84, -.82]	81	$-$ 0.231 [-.29, -.17]	$-$ 0.712 [-.76, -.67]
	$\mathsf{IF}$	68	$+$ 0.032 [-.01, +.08]	$-$ 0.761 [-.82, -.70]	92	$-$ 0.114 [-.16, -.07]	$-$ 0.834 [-.85, -.82]	87	$-$ 0.046 [-.09, -.01]	$-$ 0.675 [-.73, -.61]
DB	$\mathsf{P}$	204	$+$ 0.122 [+.09, +.15]	$+$ 0.094 [+.04, +.14]	87	$-$ 0.030 [-.09, +.03]	$-$ 0.006 [-.06, +.05]	214	$+$ 0.037 [+.01, +.06]	$+$ 0.040 [-.00, +.08]

Appendix

Table 12: Within-item shifts relative to

\mathsf{N}

(numerical counterpart of Figure 1 ), with percentile-bootstrap 95% CIs over items. All responses are included regardless of the monitor label;

n

is the number of item pairs with parsed commitment in both conditions.

		Paired deception rate					Flagged under both
Dataset	Model	$n$	$\hat{\rho}_{a}$	$\hat{\rho}_{b}$	$\Delta\hat{\rho}$	$p$	$n_{11}$	$\Delta c$ [95% CI]	$\Delta b$
Neutral $\to$ Persuasive ( $\mathsf{N}$ $\to$ $\mathsf{P}$ )
DB	Qwen	291	0.478	0.474	$-0.003$	1.00	68	$+0.166$ [ $+0.116,+0.216$ ]	$+0.066$
	OSS	284	0.109	0.148	$+0.039$	0.09	15	$-0.000$ [ $-0.100,+0.080$ ]	$-0.020$
	GLM	273	0.560	0.549	$-0.011$	0.82	101	$+0.044$ [ $+0.011,+0.077$ ]	$+0.002$
MASK	Qwen	319	0.270	0.295	$+0.025$	0.42	38	$+0.105$ [ $+0.024,+0.187$ ]	$+0.121$

Appendix

Table 13: Full paired effects of persuasive delivery.

\hat{\rho}_{a}

and

\hat{\rho}_{b}

denote deception rates under the two paired conditions,

\Delta\hat{\rho}=\hat{\rho}_{b}-\hat{\rho}_{a}

, and

p

is the exact McNemar test. Among responses flagged under both conditions,

\Delta c

is computed over

n_{11}

pairs with valid commitment reports and is shown with bootstrap 95% CIs;

\Delta b

gives the corresponding change in belief (point estimate; no interval was computed for it). Confidence comparisons require at least five valid pairs. Among items unflagged under both

\mathsf{N}

and

\mathsf{P}

, the corresponding commitment changes are DB: Qwen

+0.077

, OSS

-0.022

, GLM

+0.002

; MASK: Qwen

+0.112

, OSS

+0.023

, GLM

+0.007

Dataset	$s$	$\hat{D}$	$n$	$c$	$b$	$\Gamma$	$n$	$c$	$b$	$\Gamma$	$n$	$c$	$b$	$\Gamma$
			OSS				DeepSeek-R1				DeepSeek-V4.1-Flash
DB	$\mathsf{N}$	0	109	0.589	0.757	$-0.167$	72	0.736	0.881	$-0.144$	250	0.886	0.930	$-0.044$
	$\mathsf{N}$	1	27	0.665	0.780	$-0.115$	66	0.505	0.482	$+0.023$	7	0.779	0.893	$-0.114$
	$\mathsf{P}$	0	90	0.580	0.810	$-0.230$	70	0.801	0.898	$-0.096$	249	0.914	0.934	$-0.020$
	$\mathsf{P}$	1	32	0.650	0.691	$-0.041$	63	0.612	0.593	$+0.019$	8	0.887	0.925	$-0.037$
MASK	$\mathsf{N}$	0	162	0.637	0.804	$-0.167$	87	0.690	0.836	$-0.146$	175	0.836	0.922	$-0.086$

Appendix

Table 14: Mean commitment

c

, belief

b

, and gap

\Gamma=c-b

for additional models under

\mathsf{N}

and

\mathsf{P}

n

is the number of responses with a parsed commitment report. DeepSeek-V4.1-Flash’s flagged DB cells rest on 7 and 8 parsed responses. Population deception rates under

\mathsf{N}

\mathsf{P}

: R1

0.547

0.515

(DB),

0.641

0.609

(MASK); V4.1-Flash

0.048

0.054

(DB),

0.310

0.263

(MASK).

Dataset	Condition	Model	$n$	$b_{\mathrm{in}}$	$b_{\mathrm{out}}$	$\Delta_{b}$	$P(\mathrm{True})$
TC	$\mathsf{PF}$	Qwen	74	0.062	0.050	$+0.012$	0.003
		OSS	81	0.050	0.052	$-0.002$	—
		GLM	74	0.086	0.207	$-0.120$	0.565
	$\mathsf{CF}$	Qwen	59	0.070	0.050	$+0.020$	0.011
		OSS	86	0.050	0.053	$-0.003$	—
		GLM	78	0.104	0.251	$-0.147$	0.612

Appendix

Table 15: Belief on flagged responses (

\hat{D}=1

), elicited in the original generating context (

b_{\mathrm{in}}

) and in a fresh context containing only the question and fixed response (

b_{\mathrm{out}}

\Delta_{b}=b_{\mathrm{in}}-b_{\mathrm{out}}

is computed within response, and

n

is the number of responses with successfully parsed reports in both contexts.

P(\mathrm{True})

is measured in the fresh context; it is unavailable for OSS because its chat template assigns no usable probability mass to the probe tokens. The post-hoc pass was not run under

\mathsf{P}

. Cells with fewer than ten usable responses are omitted.

Dataset	Model	$\hat{D}$	$n$	$b_{\mathrm{in}}$	$b_{\mathrm{out}}$	$P(\mathrm{True})$	lowest bin
DB	Qwen	0	115	0.797	0.769	0.734	0.08
		1	100	0.359	0.464	0.432	0.34
	GLM	0	84	0.794	0.813	0.933	0.04
		1	126	0.585	0.555	0.890	0.26
	OSS	0	100	0.759	0.746	—	0.14
		1	23	0.776	0.559	—	0.26

Appendix

Table 16: In-context (

b_{\mathrm{in}}

) and out-of-context (

b_{\mathrm{out}}

) belief under

\mathsf{N}

, split by monitor label, on responses parsed in both contexts (the filter of Table 15 , whose flagged rows this table reproduces). “Lowest bin” is the share of re-scored responses in the lowest truth category. OSS has no usable

P(\mathrm{True})

and no re-scored MASK

\mathsf{N}

responses; TC is omitted because

\mathsf{N}

produces at most one flagged response per model.

	Task 1	Task 2
Annotated items (after removing Unclear )	231	65
Unique annotators	22	23

Appendix

Table 17: Participation in the two human-evaluation tasks.

Figure 10: Annotation interface for Task 1 (deception detection and confidence rating). Only the annotation screen is shown; the instructions annotators received in the preceding tutorials are described in the text.

Figure 11: Annotation interface for Task 2 (paired-response comparison). Only the annotation screen is shown. Before the task, the tutorial instructed annotators to judge by their own perception, e.g. “Which response is more likely to influence your own thinking or decision?”

Figure 12: Model-reported commitment against human-perceived confidence. (a) Item-level comparison; the dashed line is equality. (b) Distribution of the difference (model

-

perceived).

Panel of Figure 13 — Figure 13: Two Task 2 pairs in which annotators preferred the higher-commitment deceptive response.

Dataset	$s$	$n$	$\Delta c$	$\Delta b$	$n$	Top-2 $c$ : base $\to$ mis.
		All paired responses			Flagged for both models
TC	$\mathsf{N}$	95	$+0.075$ [+0.04, +0.11]	$+0.029$ [+0.01, +0.05]	0	—
	$\mathsf{P}$	94	$+0.009$ [-0.01, +0.02]	$+0.003$ [-0.00, +0.01]	0	—
	$\mathsf{PF}$	83	$+0.017$ [-0.05, +0.09]	$-0.007$ [-0.05, +0.02]	77	0.01 $\to$ 0.08
	$\mathsf{CF}$	79	$+0.175$ [+0.10, +0.25]	$-0.066$ [-0.13, -0.00]	61	0.00 $\to$ 0.52
	$\mathsf{IF}$	71	$-0.034$ [-0.09, +0.02]	$-0.042$ [-0.10, +0.02]	44	0.77 $\to$ 0.80

Appendix

Table 18: Changes from base Qwen to Qwen mis under the same condition. Left: within-item changes

\Delta x=x_{\mathrm{mis}}-x_{\mathrm{base}}

over all items answered by both models, with percentile-bootstrap 95% CIs over items;

n

counts item pairs with a parsed commitment report on both sides. Right: among items flagged as deceptive (

\hat{D}=1

) for both models, the share of responses with

c\geq 0.85

before and after fine-tuning. Dashes: fewer than five jointly flagged items.

				Qwen		Qwen mis
Dataset	Split	Non-dec. arm	Dec. arm	$n$	$P(\text{dec.})$	$n$	$P(\text{dec.})$
TC	—	$\mathsf{N}$	$\mathsf{PF}$	57	0.05 [0.02, 0.14]	85	0.45 [0.35, 0.55]
		$\mathsf{N}$	$\mathsf{CF}$	55	0.11 [0.05, 0.22]	87	0.54 [0.44, 0.64]
		$\mathsf{P}$	$\mathsf{PF}$	57	0.07 [0.03, 0.17]	52	0.33 [0.22, 0.46]
		$\mathsf{P}$	$\mathsf{CF}$	57	0.12 [0.06, 0.23]	83	0.55 [0.45, 0.66]
DB	L1	$\mathsf{N}$	$\mathsf{CF}$	50	0.28 [0.18, 0.42]	61	0.67 [0.55, 0.78]

Appendix

Table 19: Self-prediction: probability that the model says it would produce its flagged rather than its unflagged response to the same item. Wilson 95% CIs per cell; pooled rows carry bootstrap CIs. Cell sizes differ between the two models because more of Qwen mis ’s pairs pass the monitor filter. The Qwen mis

\mathsf{P}

\mathsf{PF}

cell on TC is incomplete (52 of 87 items); the sibling cells score the same 52 items at

0.47

–

0.60

Model	Dataset	$n$	pooled $P(\text{dec.})$	spread	called dec.	chose $\|$ called
Qwen	TC	226	0.088 [0.053, 0.128]	0.070	0.88	0.09
	DB L1	98	0.306 [0.214, 0.398]	0.053	0.95	0.32
	DB L2	85	0.318 [0.224, 0.412]	0.031	0.95	0.33
	MASK disinformation	63	0.556 [0.429, 0.683]	0.078	0.95	0.57
	MASK known facts	128	0.141 [0.086, 0.203]	0.098	0.90	0.13
OSS	TC	358	0.053 [0.031, 0.078]	0.055	0.98	0.05

Appendix

Table 20: Self-prediction and recognition for the two base models on every dataset split, pooled over pairings. “Spread” is the gap between the highest and lowest pairing. “Called dec.” is the share of flagged responses the model labels deceptive when shown them alone (Appendix F.3 ); “chose

|

called” is, among those, the share it nonetheless says it would produce.

Dataset	Model	Called deceptive	Unflagged called deceptive	Aware at generation	Chose $\|$ called
TC	Qwen	0.88	—	0.88	0.09
	Qwen mis	0.95	0.02	0.96	0.48
DB	Qwen	0.95	—	0.93	0.33
	Qwen mis	0.99	0.14	0.97	0.74
All	Qwen (5 splits)	0.92	0.07	0.90	—
	OSS (5 splits)	0.97	0.01	0.77	—

Appendix

Table 21: Retrospective recognition. “Called deceptive”: share of flagged responses the model labels deceptive. “Unflagged called deceptive”: the same for the unflagged arm of each pair. “Aware at generation”: among responses the model calls deceptive, the share it says were recognisably deceptive at generation time. “Chose

|

called”: among flagged responses the model calls deceptive, the share it selects in the self-prediction task.

		Qwen (layer 48/64)		OSS (layer 18/24)
		answer	full	answer	full
$n$ fitted / flagged		1625 / 909	1864 / 1031	1725 / 551	1844 / 627
Nested CV AUC	diff. of means	0.923	0.898	0.947	0.960
	logistic	0.943	0.932	0.960	0.976
95% CI	diff. of means	[0.910, 0.936]	[0.882, 0.910]	[0.939, 0.958]	[0.943, 0.963]
	logistic	[0.934, 0.954]	[0.923, 0.945]	[0.953, 0.969]	[0.971, 0.983]

Appendix

Table 22: Recognition probe on the recognition task. Held-out TC scores should be read with care: on TC the flag coincides almost exactly with the generating condition, so they cannot separate recognising deception from recognising the condition; the nested DB/MASK values are the ones to quote.

Model	Probe	Segment	$n$	AUC	$p$	$n$	AUC	$p$
			contrast			same-label
Qwen	answer, diff. means	whole	863	0.658	0.32	717	0.435	0.61
		answer	863	0.753	0.14	717	0.439	0.58
	answer, logistic	whole	863	0.685	0.26	717	0.566	0.59
		answer	863	0.779	0.08	717	0.550	0.65
	full, logistic	whole	863	0.475	0.91	717	0.435	0.61

Appendix

Table 23: The recognition directions applied during generation, within item, pooled over the paired grid. Contrast : pairs with exactly one flagged arm (flagged minus unflagged). Same-label : both arms share a label (deceptive-condition arm minus honest-condition arm).

p

is exact and sign-agnostic against 100 random directions (floor

0.0099

). No probe is separable from a random direction.

Figure 15: (a, c) Cross-validated AUC of the recognition probe by layer (answer rendering) for Qwen and OSS, against the shuffled-label null. (b, d) The same directions applied during generation (contrast pairs), by response segment, against 100 random directions at the same layer (grey; band is their 5th–95th percentile).

Figure 16: Figure 15 (b, d) broken out by cell: filled markers are contrast pairs, open markers same-label pairs, grey the random-direction band.

Model	Score	AUROC [95% CI]	$n_{+}/n_{-}$	AUROC [95% CI]	$n_{+}/n_{-}$
		DB		MASK
Qwen	$1-b$	0.884 $[0.842,0.922]$	117/129	0.864 $[0.820,0.905]$	66/195
	$1-c$	0.732 $[0.668,0.787]$	117/130	0.723 $[0.656,0.789]$	66/194
	$\Gamma=c-b$	0.756 $[0.688,0.822]$	117/129	0.823 $[0.761,0.880]$	66/194
	$1-P(\mathrm{True})$	0.710 $[0.646,0.768]$	135/144	0.856 $[0.809,0.901]$	85/225
	$c-P(\mathrm{True})$	0.616 $[0.544,0.689]$	114/125	0.809 $[0.745,0.869]$	66/188

Appendix

Table 24: Response-level AUROC for detecting

\hat{D}=1

within

\mathsf{N}

and

\mathsf{P}

, with bootstrap 95% CIs and the number of flagged (

n_{+}

) and unflagged (

n_{-}

) responses carrying the score. These are the values plotted in Figure 4 .

P(\mathrm{True})

-based scores are unavailable for OSS.

Model	Monitoring score	TC	DB	MASK
Qwen	$1-b$	0.951 $[0.930,0.969]$	0.905 $[0.881,0.928]$	0.797 $[0.773,0.821]$
	$1-c$	0.880 $[0.846,0.911]$	0.793 $[0.767,0.820]$	0.644 $[0.614,0.674]$
	$\Gamma=c-b$	0.907 $[0.870,0.939]$	0.835 $[0.804,0.865]$	0.753 $[0.724,0.782]$
	$1-P(\mathrm{True})$	0.988 $[0.972,0.999]$	0.807 $[0.773,0.841]$	0.847 $[0.824,0.871]$
	$c-P(\mathrm{True})$	0.983 $[0.960,1.000]$	0.709 $[0.661,0.755]$	0.807 $[0.774,0.838]$
	$1-C_{\mathrm{norm}}$	0.529 $[0.478,0.577]$	0.463 $[0.430,0.496]$	0.641 $[0.611,0.670]$

Appendix

Table 25: Response-level AUROC for detecting

\hat{D}=1

, pooled across all conditions, with bootstrap 95% CIs. Scores are oriented as in Table 24 .

Model	Trigger	Acc. clean	Acc. trig.	$n$ trig. false	clean	trig. false	Status
					Both reports parsed
Gemma-3-27B	gender	0.975	0.515	97	1.00	1.00	included
	greeting	0.985	0.810	38	1.00	1.00	excluded: trig. acc. $>0.60$
	time	0.970	0.200	160	1.00	1.00	included
Llama-3.3-70B	gender	0.575	0.525	95	1.00	0.99	excluded: clean acc. $<0.90$
	greeting	0.945	0.275	145	1.00	0.98	included

Appendix

Table 26: Manipulation check and inclusion for all twelve backdoored combinations (200 clean and 200 triggered items each). Accuracy is against the BoolQ gold. “Both reports parsed” is the share of clean responses and of triggered-false responses with both

c

and

b

recovered. Mistral answers with the bare word and emits no confidence fields; Qwen2.5-72B-time files its belief label under the commitment field on about half of its responses.

		Clean items		Triggered items
Model	Trigger	correct	false	correct	false
Gemma-3-27B	gender ∗	5/195	0/5	2/103	19/97
	greeting	5/197	0/3	2/162	9/38
	time ∗	3/194	1/6	1/40	40/160
Llama-3.3-70B	gender	2/115	8/85	3/105	13/95
	greeting ∗	5/189	0/11	2/55	28/145

Appendix

Table 27: Monitor flags by ground-truth correctness: flagged / total in each cell. ∗ marks the six included combinations. Responses the monitor could not label (4–14 per combination) are counted as unflagged.

Model	Trigger	$n$	$c$	$b$	$\Gamma$	$C_{\mathrm{norm}}$	$n$	$c$	$b$	$\Gamma$	$C_{\mathrm{norm}}$	Top-2 $b$
		Clean, correct					Triggered, false
Gemma-3-27B	gender	195	0.910	0.940	$-0.030$	0.910	97	0.823	0.844	$-0.021$	0.823	0.80
	time	194	0.915	0.940	$-0.024$	0.908	160	0.740	0.604	$+0.136$	0.781	0.48
Llama-3.3-70B	greeting	189	0.850	0.942	$-0.092$	0.878	143	0.654	0.548	$+0.106$	0.804	0.47
	time	193	0.866	0.933	$-0.067$	0.858	188	0.644	0.575	$+0.069$	0.789	0.46
Qwen2.5-72B	gender	190	0.783	0.889	$-0.106$	0.890	183	0.773	0.866	$-0.094$	0.896	0.89

Appendix

Table 28: Commitment, belief, gap, and sampled-token confidence for correct clean answers and false triggered answers in the six included combinations.

n

counts responses with a parsed commitment report. “Top-2

b

” is the share of false answers with

b\geq 0.85

		Correct clean vs. false triggered				Monitor flag, all responses
Model	Trigger	$1-c$	$1-b$	$1-C_{\mathrm{norm}}$	$n_{+}/n_{-}$	$1-c$	$1-b$	$1-C_{\mathrm{norm}}$	$n_{+}/n_{-}$
Gemma-3-27B	gender	0.63 [0.58, 0.68]	0.62 [0.58, 0.67]	0.89 [0.85, 0.93]	97/195	0.47 [0.41, 0.55]	0.61 [0.52, 0.71]	0.74 [0.64, 0.84]	26/367
	time	0.76 [0.71, 0.80]	0.81 [0.77, 0.85]	0.98 [0.96, 0.99]	160/194	0.71 [0.62, 0.78]	0.74 [0.67, 0.82]	0.83 [0.75, 0.89]	45/345
Llama-3.3-70B	greeting	0.76 [0.71, 0.81]	0.80 [0.76, 0.85]	0.87 [0.83, 0.90]	144/189	0.65 [0.55, 0.74]	0.71 [0.63, 0.80]	0.78 [0.69, 0.86]	35/356
	time	0.78 [0.73, 0.82]	0.81 [0.77, 0.85]	0.84 [0.80, 0.88]	193/193	0.67 [0.56, 0.77]	0.68 [0.58, 0.77]	0.73 [0.62, 0.83]	30/362
Qwen2.5-72B	gender	0.50 [0.45, 0.56]	0.53 [0.48, 0.58]	0.45 [0.39, 0.51]	188/194	0.50 [0.40, 0.60]	0.50 [0.40, 0.60]	0.55 [0.44, 0.65]	33/353

Appendix

Table 29: AUROC of each confidence signal within a backdoored combination, with percentile-bootstrap 95% CIs (2,000 resamples, classes resampled separately). Left: positives are false answers to triggered items, negatives correct answers to clean items. Right: positives are monitor-flagged responses among all 400.

n_{+}/n_{-}

refers to

1-b

; the other scores differ by at most a few responses. Scores are oriented so that

0.5

is chance and higher means the false or flagged responses carry lower confidence.

Panel of Figure 17 — Figure 17: Two triggered responses from backdoored Gemma-3-27B that the monitor flagged. The false answer is given without any instruction to lie.

Dataset	Condition	$n$	tok.	$C_{\mathrm{norm}}$	$\bar{H}$	$n$	tok.	$C_{\mathrm{norm}}$	$\bar{H}$	$n$	tok.	$C_{\mathrm{norm}}$	$\bar{H}$
		Qwen				OSS				GLM
TC	$\mathsf{N}$	100	451	0.631	1.11	100	485	0.736	0.76	100	1673	0.672	0.82
	$\mathsf{P}$	100	469	0.647	1.06	100	498	0.724	0.80	100	1499	0.677	0.81
	$\mathsf{PF}$	100	522	0.670	0.98	100	665	0.689	0.92	99	2341	0.656	0.86
	$\mathsf{CF}$	100	488	0.648	1.05	100	707	0.668	0.99	97	2525	0.647	0.90
	$\mathsf{IF}$	100	473	0.652	1.04	100	817	0.680	0.95	100	2186	0.643	0.91

Appendix

Table 30: Mean sampled-token probability

C_{\mathrm{norm}}

and mean per-token entropy

\bar{H}

by dataset, condition, and model. Token-level values are averaged within each response and then across all

n

responses with token dumps; “tok.” is mean response length.

		$\Delta C_{\mathrm{norm}}$ vs. $\mathsf{N}$
Model	Condition	TC	DB	MASK
Qwen	$\mathsf{P}$	$+0.015$ $[+0.003,+0.029]$	$+0.013$ $[-0.004,+0.029]$	$-0.010$ $[-0.025,+0.005]$
	$\mathsf{PF}$	$+0.038$ $[+0.023,+0.054]$	$+0.066$ $[+0.053,+0.079]$	$-0.086$ $[-0.118,-0.054]$
	$\mathsf{CF}$	$+0.017$ $[-0.002,+0.036]$	$+0.000$ $[-0.016,+0.016]$	$-0.012$ $[-0.028,+0.003]$
	$\mathsf{IF}$	$+0.021$ $[+0.005,+0.038]$	$+0.017$ $[+0.001,+0.034]$	$-0.008$ $[-0.025,+0.007]$
OSS	$\mathsf{P}$	$-0.012$ $[-0.020,-0.003]$	$-0.009$ $[-0.016,-0.002]$	$-0.001$ $[-0.007,+0.004]$

Appendix

Table 31: Within-item change in

C_{\mathrm{norm}}

relative to the same item’s

\mathsf{N}

response, with percentile-bootstrap 95% intervals over items (2,000 resamples).

		$\Delta\bar{H}$ vs. $\mathsf{N}$ (nats)
Model	Condition	TC	DB	MASK
Qwen	$\mathsf{P}$	$-0.051$ $[-0.097,-0.009]$	$-0.044$ $[-0.097,+0.011]$	$+0.024$ $[-0.024,+0.071]$
	$\mathsf{PF}$	$-0.135$ $[-0.187,-0.084]$	$-0.452$ $[-0.493,-0.409]$	$+0.125$ $[+0.031,+0.220]$
	$\mathsf{CF}$	$-0.055$ $[-0.118,+0.005]$	$-0.003$ $[-0.055,+0.048]$	$+0.037$ $[-0.009,+0.084]$
	$\mathsf{IF}$	$-0.070$ $[-0.126,-0.014]$	$-0.052$ $[-0.106,-0.003]$	$+0.023$ $[-0.023,+0.074]$
OSS	$\mathsf{P}$	$+0.034$ $[+0.008,+0.058]$	$+0.029$ $[+0.008,+0.048]$	$+0.002$ $[-0.015,+0.019]$

Appendix

Table 32: As Table 31 , for the mean per-token entropy

\bar{H}

(nats).

Dataset	Condition	Model	$n$	$C_{\mathrm{cot}}$	cov.	$C_{\mathrm{phrase}}$	cov.	$C_{\mathrm{ans}}$	cov.
TC	$\mathsf{N}$	Qwen	100	0.594	100%	0.845	95%	0.973	94%
		OSS	100	0.691	46%	0.997	99%	0.975	100%
		GLM	100	0.668	71%	0.961	69%	0.942	73%
	$\mathsf{P}$	Qwen	100	0.610	99%	0.944	93%	0.960	90%
		OSS	100	0.689	35%	0.990	100%	0.965	100%
		GLM	100	0.671	81%	0.951	64%	0.960	82%

Appendix

Table 33: Mean sampled-token probability restricted to the reasoning preceding the JSON block (

C_{\mathrm{cot}}

), the tokens encoding the confidence label (

C_{\mathrm{phrase}}

), and the final-answer span (

C_{\mathrm{ans}}

). Each mean is computed only over responses for which the corresponding span was located; “cov.” reports that share of the

n

token dumps.

Dataset	Condition	$n$	tok.	$C_{\mathrm{norm}}$	$\bar{H}$	$n$	tok.	$C_{\mathrm{norm}}$	$\bar{H}$
		Qwen				Qwen mis
TC	$\mathsf{N}$	100	451	0.631	1.11	100	356	0.677	0.92
	$\mathsf{P}$	100	469	0.647	1.06	100	374	0.687	0.89
	$\mathsf{PF}$	100	522	0.670	0.98	100	413	0.678	0.93
	$\mathsf{CF}$	100	488	0.648	1.05	100	344	0.651	1.02
	$\mathsf{IF}$	100	473	0.652	1.04	100	366	0.655	1.01

Appendix

Table 34: Token-level metrics for base and misaligned Qwen.

Explore similar work

Mar 31, 2025cs.CL

When Do Large Language Models Exhibit Unsolicited Deception?

Large Language Models (LLMs) are effective at deceiving when prompted to do so. Models that demonstrate better performance on reasoning tasks are also better at prompted deception. But under what conditions do they deceive without instruction to do so? This study evaluates unsolicited deception produced by LLMs in a preregistered experimental protocol using tools from signaling theory. We evaluated a range of 18 proprietary closed-source and open-source LLMs using modified 2x2 games (in the style of the Prisoner's Dilemma) augmented with a phase in which they can freely communicate to the other agent using unconstrained language. This setup creates an opportunity to misrepresent its actions in conditions that vary in how useful doing so might be towards goal satisfaction. The results indicate that 1) all tested LLMs misrepresent their actions in at least some conditions, 2) they are generally more likely to do so in situations in which deception is beneficial, and 3) models exhibiting better reasoning capacity overall tend to misrepresent at higher rates. Taken together, these results suggest a correlational relationship between model reasoning performance and situational deception, and reveal certain contextual factors that affect whether LLMs will misrepresent actions or not in a novel experimental configuration.

Samuel M. Taylor, Benjamin K. Bergen

Department of Cognitive Science University of California, San Diego

Deception LLM Reasoning Strategies Reasoning Skills

Ask PDF

Oct 6, 2026cs.LG

DecepEval: A Benchmark for Evaluating Deception in LLM Agents

As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment. Existing evaluations show that LLM agents can deceive, but often examine isolated scenarios or narrowly defined conditions, limiting systematic understanding of when deception becomes more likely. To address this gap, we introduce DecepEval, a benchmark comprising 1,532 instances across 3 task families and 28 professional scenarios. Drawing on classical fraud theories, we propose the LLM Deception Diamond framework, which characterizes four external conditions that may induce deception: pressure, incentive, opportunity, and conflict. DecepEval pairs neutral and induced versions of each instance to measure condition-dependent changes in deception rates, while explicit task facts and observable agent behavior help distinguish deception from capability-related errors. Evaluations of nine frontier LLMs show that inducements increase deception across models and task families, even among models with low baseline deception rates. DecepEval makes these vulnerabilities measurable, providing a shared benchmark for progress toward trustworthy artificial intelligence.

Yiming Xu, Hongyue Yu, Beihua Yang +8

Xi’an Jiaotong University · University of Virginia · Griffith University +2

Topic pending

Ask PDF

Jan 20, 2026cs.CL

Vulnerability of LLMs' Stated Beliefs? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventions

Large Language Models (LLMs) are increasingly employed in various question-answering tasks. However, recent studies showcase that LLMs are susceptible to persuasion and could adopt counterfactual beliefs. We present a systematic evaluation of LLM susceptibility to persuasion under the \emph{Source--Message--Channel--Receiver} (SMCR) communication framework. Across six mainstream Large Language Models (LLMs) and three domains (factual knowledge, medical QA, and social bias), we analyze how different persuasive strategies influence stated belief stability over multiple interaction turns. We further examine whether verbalized confidence prompting (i.e., eliciting self-reported confidence scores) affects resistance to persuasion. Results show that the smallest model (Llama 3.2-3B) exhibits extreme compliance, with 82.5% of belief changes occurring at the first persuasive turn (average end turn of 1.1--1.4). Contrary to expectations, verbalized confidence prompting \emph{increases} vulnerability by accelerating belief erosion rather than enhancing robustness. Finally, an exploratory study of adversarial fine-tuning reveals highly model-dependent effectiveness: GPT-4o-mini achieves near-complete robustness (98.6%), and Mistral~7B improves substantially (35.7%

\rightarrow

79.3%), but Llama models remain highly susceptible (

<

14% RQ1) even when fine-tuned on their own failure cases. Together, these findings highlight substantial model-dependent limits of current robustness interventions and offer guidance for developing more trustworthy LLMs.

Fan Huang, Haewoon Kwak, Jisun An

Indiana University Bloomington, United States

Large Language Model Reliability Persuasion

Ask PDF