Affect Decoding in Phonated and Silent Speech Production from Surface EMG

Simon Pistrosch; Kleanthis Avramidis; Zhao Ren; Tiantian Feng; Jihwan Lee; Monica Gonzalez-Machorro; Anton Batliner; Tanja Schultz

eess.ASMar 12, 2026

Affect Decoding in Phonated and Silent Speech Production from Surface EMG

Authors: Simon Pistrosch, Kleanthis Avramidis, Zhao Ren, Tiantian Feng, Jihwan Lee, Monica Gonzalez-Machorro, Anton Batliner, Tanja Schultz, +2 more

Organizations: CHI – Chair of Health Informatics, TUM University Hospital, Munich, Germany · MCML – Munich Center for Machine Learning, Germany · SAIL – Signal Analysis and Interpretation Lab, University of Southern California, USA · CSL – Cognitive Systems Lab, University of Bremen, Germany · GLAM – Group on Language, Audio, & Music, Imperial College London, UK

Abstract

The expression of affect is integral to spoken communication, yet, its link to underlying articulatory execution remains unclear. Measures of articulatory muscle activity such as EMG could reveal how speech production is modulated by emotion alongside acoustic speech analyses. We investigate affect decoding from facial and neck surface electromyography (sEMG) during phonated and silent speech production. For this purpose, we introduce a dataset comprising 2,780 utterances from 12 participants across 3 tasks, on which we evaluate both intra- and inter-subject decoding using a range of features and model embeddings. Our results reveal that EMG representations reliably discriminate frustration with up to 0.845 AUC, and generalize well across articulation modes. Our ablation study further demonstrates that affective signatures are embedded in facial motor activity and persist in the absence of phonation, highlighting the potential of EMG sensing for affect-aware silent speech interfaces.

Figures & tables

Figure 1: Conceptual overview of the study. We present a dataset and computational analysis on EMG-based affect decoding during phonated and silent speech production. During articulation, surface EMG from neck and facial muscles was recorded alongside audio speech. Note: The schematic human illustration was generated with AI assistance for visualization purposes and is not meant to reflect the exact sensor hardware design, number of channels, or placement used in the study.

Utterance	Lexical category	Expressed tone(s)
I am looking for a furnished apartment.	Neutral	Neutral, polite, frustrated
I would be delighted if you had a furnished apartment available.	Polite-worded	Polite
Why isn’t the landlord responding to my inquiry?	Frustration-worded	Frustrated

Table 1: Example sentences from the prompted reading task (apartment search). Neutral sentences were articulated in three expressive tones (neutral, polite, frustrated), whereas polite- and frustration-worded sentences were spoken congruently with their lexical content.

\multirow 2* Task	Phonated			Silent			Combined
	Utt.	Mins	Per recording (s)	Utt.	Mins	Per recording (s)	Utt.	Mins	Per recording (s)
Task 1	593	36.6	3.70 $\pm$ 1.09	592	38.3	3.88 $\pm$ 1.21	1185	74.9	3.79 $\pm$ 1.15
Task 2	396	47.5	7.20 $\pm$ 6.14	–	–	–	396	47.5	7.20 $\pm$ 6.14
Task 3	599	36.3	3.63 $\pm$ 1.07	600	35.5	3.55 $\pm$ 1.02	1199	71.8	3.59 $\pm$ 1.05
Total	1588	120.4	4.55 $\pm$ 3.55	1192	73.9	3.72 $\pm$ 1.13	2780	194.2	4.19 $\pm$ 2.81

Table 2: Dataset statistics by task and speaking mode. Utt. denotes number of utterances; Mins denotes total minutes. Recording duration is shown in seconds (s).

ID	Muscle site	Positioning	Speech	Emotion
E1	Infrahyoid	2.5 cm lateral to thyroid prominence. Avoid direct proximity to reduce signal contamination.	✓	–
E2	Suprahyoid	Midway between mandible and relaxed hyoid, 0.5 cm lateral to midline.	✓	–
E3	Mylohyoid	Midway between chin center and lateral endpoint, 2 cm inferior to chin line (submental).	✓	–
E4	Mentalis	0.5 cm lateral to midline and 0.5 cm superior to pogonion.	✓	✓
E5	Orbicularis Oris Superioris	Facial midline above upper lip.	✓	–
E6	Depressor Supercilii	Between nasal bone and eyebrow.	–	✓

Table 3: Electrode placement and functional classification of recorded muscles. Speech- and emotion-related assignments are derived from reported functional associations [ 12 , 26 , 27 , 28 , 29 , 30 ] . A schematic of the described placement is shown in ablation Figure 4 .

	Phonated			Silent
Task	Frust.	Neut.	Polit.	Frust.	Neut.	Polit.
Task 1	233	120	240	232	120	240
Task 2	198	–	198	–	–	–
Task 3	240	119	240	240	120	240
Total	671	239	678	472	240	480

Table 4: Label distribution in terms of number of utterances across tasks and speaking modes. For Task 2, we report on the affect label anticipated by the experimental design.

\multirow 2* Modality	\multirow 2* Features	Intra-subject		Inter-subject
		BAC	AUC	BAC	AUC
\multirow 3*EMG	Structural	0.749 $\pm$ 0.075	0.820 $\pm$ 0.081	0.546 $\pm$ 0.054	0.568 $\pm$ 0.074
	TD-0	0.762 $\pm$ 0.063	0.845 $\pm$ 0.058	0.541 $\pm$ 0.052	0.567 $\pm$ 0.082
	BioCodec	0.721 $\pm$ 0.053	0.792 $\pm$ 0.075	0.547 $\pm$ 0.052	0.574 $\pm$ 0.080
\multirow 2*Speech	eGeMAPS	0.610 $\pm$ 0.125	0.644 $\pm$ 0.168	0.527 $\pm$ 0.058	0.495 $\pm$ 0.115
	Vox-Profile	0.659 $\pm$ 0.097	0.732 $\pm$ 0.104	0.582 $\pm$ 0.041	0.657 $\pm$ 0.071

Table 5: Classification performance for affective mode decoding across varying modalities. Results present the mean

\pm

standard deviation for Balanced Accuracy (BAC) and Area Under the Curve (AUC) across subjects under intra-subject and inter-subject settings.

Modality	Features	Unique	Repeated
\multirow 3*EMG	Structural	0.856 $\pm$ 0.069	0.720 $\pm$ 0.159
	TD-0	0.824 $\pm$ 0.126	0.751 $\pm$ 0.145
	BioCodec	0.799 $\pm$ 0.058	0.747 $\pm$ 0.145
\multirow 2*Speech	eGeMAPS	0.643 $\pm$ 0.060	0.559 $\pm$ 0.132
	Vox-Profile	0.889 $\pm$ 0.075	0.469 $\pm$ 0.202

Table 6: Intra-subject classification performance in terms of mean

\pm

standard deviation of subject-wise AUC across affectively unique and repeated trials. Repeated trials are those who were encountered in the experiment once for each available label (here, Polite and Frustrated).

Features	Task 1	Task 3	p-value
Structural	0.527 $\pm$ 0.059	0.613 $\pm$ 0.100	0.038
TD-0	0.469 $\pm$ 0.121	0.617 $\pm$ 0.116	0.038
BioCodec	0.520 $\pm$ 0.079	0.603 $\pm$ 0.112	0.056
eGeMAPS	0.489 $\pm$ 0.126	0.458 $\pm$ 0.113	0.534
Vox-Profile	0.667 $\pm$ 0.070	0.632 $\pm$ 0.076	0.464

Table 7: Inter-subject classification performance. Results are reported in terms of mean

\pm

standard deviation of subject-wise AUC for Task 1 and Task 3 separately. A Wilcoxon signed-rank test was used to determine significant changes between tasks. FDR correction was applied to all p-values.

Figure 3: Comparison of intra-subject AUC between Task 1 and Task 3 across speaking conditions. Left : tested on all sentences. Right : tested on the repeated sentences (see also Table 6 ). Dots correspond to average individual performance.

Setting	Features	Intra-mode		Cross-mode
		Phonated $\rightarrow$ Phonated	Silent $\rightarrow$ Silent	Phonated $\rightarrow$ Silent	Silent $\rightarrow$ Phonated
\multirow 3*Intra-subject	Structural	0.815 $\pm$ 0.110	0.829 $\pm$ 0.056	0.707 $\pm$ 0.158	0.663 $\pm$ 0.168
	TD-0	0.806 $\pm$ 0.113	0.811 $\pm$ 0.100	0.705 $\pm$ 0.118	0.626 $\pm$ 0.145
	BioCodec	0.758 $\pm$ 0.066	0.792 $\pm$ 0.102	0.763 $\pm$ 0.094	0.745 $\pm$ 0.075
\multirow 3*Inter-subject	Structural	0.567 $\pm$ 0.112	0.608 $\pm$ 0.073	0.639 $\pm$ 0.085	0.646 $\pm$ 0.101
	TD-0	0.589 $\pm$ 0.103	0.563 $\pm$ 0.077	0.612 $\pm$ 0.071	0.644 $\pm$ 0.103

Table 8: Affect decoding performance in terms of AUC (average

\pm

SD) for two experimental settings and three EMG input sets. Intra-mode refers to testing within the same articulation mode; cross-mode refers to training on one mode and testing on the other.

Figure 4: Channel-wise decoding performance across evaluation settings (RQ1). Topographic visualization of electrode-specific AUC for the intra- (left) and inter-subject (right) settings in Tasks 1, 3. Each marker corresponds to an EMG channel, with warmer colors reflecting higher discriminability.

Figure 5: Channel-wise decoding performance across articulation conditions (RQ2). Topographic visualization of electrode-specific AUC for the phonated (left) and silent (right) conditions in Tasks 1 and 3. Each numbered marker corresponds to an EMG channel, and color indicates intra-subject AUC, with warmer colors reflecting higher discriminability.

Modality	Features	BAC	AUC
\multirow 3*EMG	Structural	0.616 $\pm$ 0.003	0.623 $\pm$ 0.005
	TD-0	0.527 $\pm$ 0.006	0.518 $\pm$ 0.004
	BioCodec	0.595 $\pm$ 0.014	0.630 $\pm$ 0.009
\multirow 2*Speech	eGeMAPS	0.607 $\pm$ 0.002	0.679 $\pm$ 0.003
	Vox-Profile	0.670 $\pm$ 0.006	0.743 $\pm$ 0.004

Table 9: Inter-subject performance on frustration detection (Task 2) in terms of mean

\pm

standard deviation of trial-wise Balanced Accuracy (BAC) and Area Under the Curve (AUC).

Figure 6: Channel-wise decoding comparison between controlled and spontaneous settings (RQ3). Topographic visualization of electrode-specific AUC for inter-subject performance in Tasks 1 and 3 (prompted, left) and Task 2 (spontaneous, right). Each numbered marker corresponds to an EMG channel, with warmer colors reflecting higher discriminability.

Explore similar work

Comparison of sEMG Encoding Accuracy Across Speech Modes Using Articulatory and Phoneme Features
Apr 20, 2026Chenqian Le, Ruisi Li, Beatrice Fumagalli +6Speech Processing
New York University, New York, NY, USA
Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
Aug 17, 2026Xiutian Zhao, Luqi Sun, Björn Schuller +1Multimodal Foundation Models Multimodal Emotion Recognition
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
EmoMind: Decoding Affective Captions from Human Brain fMRI
May 16, 2026Bilal A. Mohammed, Lin Gu, Ruogu FangImage Captioning Neural Decoding
Department of Biomedical Engineering Vanderbilt University Nashville, TN · Research Institute of Electrical Communication Tohoku University Sendai, Japan · Department of Biomedical Engineering University of Florida Gainesville, FL

Apr 20, 2026J/K move · Enter open · S save

Comparison of sEMG Encoding Accuracy Across Speech Modes Using Articulatory and Phoneme Features

Chenqian Le, Ruisi Li, Beatrice Fumagalli, Yasamin Esmaeili+5

New York University, New York, NY, USA

Speech Processing

Open paper AskPDF arXiv

We test whether Speech Articulatory Coding (SPARC) features can linearly predict surface electromyography (sEMG) envelopes across aloud, mimed, and subvocal speech in twenty-four subjects. Using elastic-net multivariate temporal response function (mTRF) with sentence-level cross-validation, SPARC yields higher prediction accuracy than phoneme one-hot representations on nearly all electrodes and in all speech modes. Aloud and mimed speech perform comparably, and subvocal speech remains above chance, indicating detectable articulatory activity. Variance partitioning shows a substantial unique contribution from SPARC and a minimal unique contribution from phoneme features. mTRF weight patterns reveal anatomically interpretable relationships between electrode sites and articulatory movements that remain consistent across modes. This study focuses on representation/encoding analysis (not end-to-end decoding) and supports SPARC as a robust and interpretable intermediate target for sEMG-based silent-speech modeling.

Abstract

Explore similar work

Comparison of sEMG Encoding Accuracy Across Speech Modes Using Articulatory and Phoneme Features

Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models

EmoMind: Decoding Affective Captions from Human Brain fMRI