Too Emotional to Be Fake? Intra-Utterance Emotion Shifts Expose Failure Modes in Speech Deepfake Detectors

Jingyi Chen; Surya Koppisetti

cs.SDOct 16, 2026

Too Emotional to Be Fake? Intra-Utterance Emotion Shifts Expose Failure Modes in Speech Deepfake Detectors

Authors: Jingyi Chen, Surya Koppisetti

Organizations: Reality Defender

Abstract

Expressive speech synthesis can now change emotion within a single utterance, yet its impact on deepfake detection remains unclear. We introduce EmoShift, a benchmark comparing emotion-stable (AA) and emotion-changing (AB) speech to examine these effects while accounting for static emotion identity and acoustic confounds. It comprises 7,240 bona fide utterances from four source datasets and 9,720 synthetic utterances generated using four expressive TTS systems. Across seven detectors, we find that emotion changes can make synthetic speech appear more genuine, increasing spoof miss rates by up to 25.3 percentage points on OMG-Emotion at fixed thresholds. Conversely, emotion-changing bona fide speech increases false-alarm rates by up to 24.4 percentage points on IEMOCAP. Further analyses reveal detector-specific sensitivities to emotion trajectories, not fully explained by static emotion identity or splicing artifacts. These findings underscore the importance of emotion dynamics in deepfake detection.

Figures & tables

	Speech	Emotion	Intra-utt.	Comparison		Utterances		Duration (s), a $\to$ a / a $\to$ b
Dataset	style	labels	transition	design	Generators	Bona fide	Spoof	Bona fide	Spoof
Related benchmarks
ASVspoof 2019 LA	read	–	✗	–	TTS, VC	12,483	108,978	–	–
In-the-Wild	spontaneous	–	✗	–	in-the-wild	19,963	11,816	–	–
PartialSpoof	read	–	✗	–	TTS, VC	12,483	108,978	–	–
EmoFake	acted	categorical	✗	–	EVC	35,000	72,800	–	–

Table 1: Comparison with related benchmarks (top) and EmoShift composition (bottom). ESD uses same-speaker, same-text a

\to

a / a

\to

b pairs; other datasets use separate utterances. Generators: I = IndexTTS2, T = TED-TTS, H = HybridEmo, E = EmoTra-TTS. Durations are mean utterance lengths (s) for a

\to

a / a

\to

b within each class.

Panel of Figure 1 — Figure 1: Benchmark construction. Each spoof follows the text and emotion targets of its bona fide source. $\Delta_{\text{BF}}$ and $\Delta_{\text{SP}}$ measure the ( a $\to$ b )–( a $\to$ a ) score difference within each class.

Figure 2: Detector score shifts from emotion-stable ( a

\to

a ) to emotion-changing ( a

\to

b ) speech. Left: mean logit score differences for synthetic (

\Delta_{\mathrm{SP}}

) and bona fide (

\Delta_{\mathrm{BF}}

) speech; bold indicates 95% CIs excluding zero. Right: corresponding score shifts with 95% CIs (filled: synthetic; hollow: bona fide). Negative values indicate lower spoof scores for a

\to

b . Grey indicates pooled EER above 50% (Table 3 ). Additional results are in Appendix Table 4 .

Figure 3: Emotion-specific detector responses. (a) EER on emotion-stable ( a $\to$ a ) ESD speech by emotion. (b) ESD spoof score shifts by destination emotion. (c) Bona fide spoof score shifts for transitions with increasing or decreasing arousal, relative to stable speech with similar starting arousal. Positive values indicate higher spoof scores for transitioning bona fide speech. Error bars show 95% bootstrap CIs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

		Selection threshold		Selected $d_{\text{VAD}}$
Dataset	Cond.	a $\to$ a $\leq$	a $\to$ b $\geq$	Mean	Median	Range
IEMOCAP	a $\to$ a	0.1816	0.6305	0.033	0.000	0.000–0.125
	a $\to$ b	0.1816	0.6305	0.958	0.972	0.839–1.087
MSP-Podcast	a $\to$ a	0.1414	0.5509	0.076	0.075	0.000–0.141
	a $\to$ b	0.1414	0.5509	0.690	0.626	0.551–1.397

Appendix

Table 2: VAD-based selection of naturalistic speech.

d_{\mathrm{VAD}}

denotes the Euclidean distance between the normalized VAD vectors of two speech regions. Selection thresholds define candidate pools; the final selected samples show stronger separation, with non-overlapping a

\to

a and a

\to

b distance ranges.

EER (%), lower is better
Detector	ESD	OMG	IEMOCAP	MSP
RawNet2	43.2	35.9	62.4	81.4
AASIST	39.4	35.7	58.9	86.3
XLS-R-SLS	30.4	41.4	44.2	50.9
XLS-R-Mamba	29.1	45.7	55.6	56.5
ProsDD	36.6	56.2	56.8	53.5

Appendix

Table 3: Baseline detection per corpus, pooling a

\to

a and a

\to

b . Grey: EER above 50% (equivalently, AUC below 0.5 for every cell here), i.e., the detector ranks bona fide speech as more spoof-like than spoofs in most bona fide–spoof pairs.

Figure 4: Changes in detection errors from a $\to$ a to a $\to$ b at thresholds calibrated on a $\to$ a speech. Values are in percentage points. Positive changes indicate higher error rates for emotion-changing speech. Bold indicates 95% bootstrap confidence intervals excluding zero.

		Score difference (logit)			EER (%)	Fixed threshold (pp)
	Detector	$\Delta_{\text{SP}}$	$\Delta_{\text{BF}}$	$\Delta_{\text{SP}}{-}\Delta_{\text{BF}}$	a $\to$ a $\to$ a $\to$ b	$\Delta$ FNR	$\Delta$ FPR
ESD	RawNet2	+0.01	−0.28	+0.28	45.2 $\to$ 40.8	−5.2	−3.0
	AASIST	−0.22	−0.50	+0.28	41.0 $\to$ 38.2	+0.8	−6.7
	XLS-R-SLS	−0.85	−0.42	−0.43	28.7 $\to$ 29.8	+6.8	−3.7
	XLS-R-Mamba	−0.67	−0.08	−0.59	29.2 $\to$ 30.8	+4.3	−2.7
	ProsDD	−0.37	−0.56	+0.19	39.2 $\to$ 34.5	+1.9	−10.8

Appendix

Table 4: Full results.

\Delta

: mean score of a

\to

b minus a

\to

a within a class (negative: transitioning speech looks more bona fide).

\Delta_{\text{SP}}{-}\Delta_{\text{BF}}

measures the change in signed mean score separation between synthetic and bona fide speech. EER on a

\to

a and on a

\to

b .

\Delta

FNR /

\Delta

FPR: change in spoof miss rate and bona fide false-alarm rate from a

\to

a to a

\to

b , with the threshold fixed at the EER point on a

\to

a . Bold: 95% cluster-bootstrap CI excludes zero (for EER, of the change). Grey, † : detector inverted on that corpus (pooled EER

>50\%

, Table 3 ).

	$\Delta_{\text{BF}}$ , IEMOCAP		$\Delta_{\text{BF}}$ , MSP		Arousal slope
Detector	raw	adj.	raw	adj.	IEM.	MSP
RawNet2	+1.54	+1.35	+0.13	−0.22	+0.55	+0.78
AASIST	+1.66	+1.38	+0.23	−0.05	+0.98	+0.70
XLS-R-SLS	−0.60	−0.60	+0.00	−0.31	0.00	+0.14
XLS-R-Mamba	−0.75	−0.82	+0.11	−0.22	+0.07	+0.12
ProsDD	−0.27	0.00	+0.83	+0.54	−0.50	+0.19

Appendix

Table 5: Naturalistic bona fide controls.

\Delta_{\text{BF}}

raw and adjusted for speaker, duration, and regional VAD; arousal slope: within-utterance second-minus-first-region score per 0.1 rise in normalized arousal. Bold: 95% CI excludes zero.

Detector	Checkpoint	Window	Hop
RawNet2	official	64,600 (4.04 s)	32,300
AASIST	official	64,600 (4.04 s)	32,300
XLS-R-SLS	official	64,600 (4.04 s)	32,300
XLS-R-Mamba	official	66,800 (4.18 s)	33,400
ProsDD	official	64,000 (4.00 s)	32,000
SLIM	fine-tuned	48,000 (3.00 s)	24,000

Appendix

Table 6: Detector inference configurations. Window and hop sizes are reported in samples at 16 kHz. The first five detectors use released checkpoints, while SLIM and Alethia use ASVspoof 2019 LA fine-tuned checkpoints.

Figure 5: Analysis of splicing artifacts. (a) Synthetic spoof score shifts from emotion-stable ( a

\to

a ) to emotion-changing ( a

\to

b ) speech, grouped by generator on ESD and OMG-Emotion. (b) Difference between spoof scores from windows containing a splice boundary and other windows within the same utterance on IEMOCAP and MSP-Podcast. Unspliced EmoTra-TTS speech is included as a reference at the corresponding position. Positive values indicate higher spoof scores near splice boundaries. Error bars show 95% bootstrap confidence intervals.

Audio Deepfake Detection

Audio deepfake detection identifies synthetic, cloned, or manipulated speech and other audio to support authentication and media forensics.

See all 204 papers

Aug 20, 2026

Tracking the Trend in How Speech Synthesizers Deceive People

Advances in speech synthesis have made deepfake audio highly realistic. Earlier studies reported 70-80% human detection accuracy, but relied primarily on older synthesizers. We compare human detection for three selected voice synthesis tools released in 2019, 2022, and 2024 with 82 IT professionals, and benchmark humans against six pretrained detectors on the same material. For fully synthetic speech (full spoofs), the F1 score drops from about 90% for RTVC and YourTTS to 48% for ElevenLabs, although listeners were explicitly warned that deepfakes were present. For partial spoofing, where only one sentence of an utterance is altered, strict accuracy falls to 9%, and listeners classify the synthetic sentence as bona fide 77% of the time. Humans and detectors fail in complementary ways, and neither reliably localizes short manipulations. Additionally, listeners increasingly mislabel bona fide speech as fake, eroding trust in unmanipulated audio. These findings show that human perception alone is unreliable for the selected modern and partial-spoof conditions and motivate procedural verification, provenance, watermarking, and segment-level detection.

Milan Šalko, Anton Firc, Kamil Malinka +4

Audio Deepfake Detection Audio Deepfake Detection Robustness TTS Synthesis

Ask PDF

May 4, 2026

Phoneme-Level Deepfake Detection Across Emotional Conditions Using Self-Supervised Embeddings

Recent advances in emotional voice conversion (EVC) have enabled the generation of expressive synthetic speech, raising new concerns in audio deepfake detection. Existing approaches treat speech as a homogeneous signal and largely overlook its internal phonetic structure, limiting their interpretability in emotionally conditioned settings. In this work, we propose a phoneme-level framework to analyze emotionally manipulated synthetic speech using real and EVC-generated speech under matched emotional conditions with shared transcripts, phoneme-aligned TextGrids, and WavLM-based embeddings. Our results show that phoneme behavior varies across categories, with complex vowels and fricatives exhibiting higher divergence while simpler phonemes remain more stable. Phonemes with larger distributional differences are also found to be more easily detected, consistently across multiple emotions and synthesis systems. These findings demonstrate that phoneme-level analysis is an effective and interpretable approach for detecting emotionally manipulated synthetic speech.

Vamshi Nallaguntla, Shruti Kshirsagar, Anderson R. Avila

Wichita State University, Wichita, KS, USA. · Institut national de la recherche scientifique (INRS–EMT), QC, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Canada

Audio Deepfake Detection

Ask PDF

May 21, 2026

Eroding Trust in Real Speech: A Large-Scale Study of Human Audio Deepfake Perception

Audio deepfakes have improved rapidly recently, yet their effect on human trust in real speech remains unstudied. We present the largest listening study on audio deepfake perception to date, collecting 35,532 judgments from 1,768 participants across 138 text-to-speech and voice conversion systems. Our central finding is a skepticism shift: compared to a 2021 baseline, human accuracy on fake samples barely changed (72.9% to 71.2%), but accuracy on real samples dropped from 72.7% to 64.1%. Participants are not worse at detecting synthesis artifacts; rather, they increasingly distrust authentic speech. Samples generated by commercial and autoregressive language model systems proved hardest to detect (61.3 - 65.9%), while those from traditional seq2seq and flow-matching models remain easier to spot (75.4 - 76.8%). An ML detector that served as a reference point maintained over 94.5% accuracy across all conditions. Our results suggest that the primary threat posed by modern deepfakes may not be mere deception, but the erosion of trust in genuine audio.

Nicolas M. Müller, Wei Herng Choong

Fraunhofer AISEC & Resemble AI · Germany · Fraunhofer AISEC

Audio Deepfake Detection Speech Processing Trust in AI Text-to-Speech

Ask PDF

Abstract

Audio Deepfake Detection

Explore similar work

Tracking the Trend in How Speech Synthesizers Deceive People

Phoneme-Level Deepfake Detection Across Emotional Conditions Using Self-Supervised Embeddings

Eroding Trust in Real Speech: A Large-Scale Study of Human Audio Deepfake Perception