Authors: Chenqian Le, Beatrice Fumagalli, Yasamin Esmaeili, Xupeng Chen, Tianyu He, Nikasadat Emami, Adeen Flinker, Yao Wang
Organizations: Department of Electrical and Computer Engineering, New York University Tandon School of Engineering, New York, NY, USA · Department of Neurology, New York University Grossman School of Medicine, New York, NY, USA · Department of Biomedical Engineering, New York University Tandon School of Engineering, New York, NY, USA
Surface electromyography (sEMG)-based silent speech interfaces are limited by cross-user variability and calibration burden. We study a limited-data setting in which each of 27 speech-typical participants contributed less than 0.5 h of data (21.3 min on average) across Aloud and Mimed speech. Within a closed 50-sentence corpus, we used leave-one-subject-out evaluation, initializing from a released single-subject checkpoint, pretraining on non-held-out participants, and fine-tuning on the target participant. This pipeline achieved 21.7% character error rate (CER) and 31.9% word error rate (WER), compared with 49.3% CER without target-subject calibration and 68.0% CER for direct checkpoint fine-tuning. Multi-subject pretraining from random initialization followed by fine-tuning reached 44.9% CER and did not converge under the fixed schedule in 5 of 27 folds, indicating substantial optimization and accuracy benefits from checkpoint initialization. Macro-averaged CER declined from 74.4% with one pretraining participant to 21.7% with 26. Three minutes of target-subject calibration achieved 20.5% CER and 31.7% WER, with no statistically significant difference from the full approximately 13-min pool (21.7% CER and 31.9% WER). A subject-specific adapter provided no detectable benefit. Excluding the five evaluation sentences from all sEMG model-training data increased CER and WER to 78.6% and 99.9%. These results support short-calibration personalization in a standardized-montage, closed-corpus setting.
Figures & tables
Figure 1: Overview of the cross-subject sEMG speech decoding framework. (a) Dataset and recording: 27 subjects, a 50-sentence corpus, Aloud and Mimed speech modes, and an eight-channel sEMG montage over the lower face and neck. (b) Closed-corpus LOSO personalization protocol. A single fixed 45/5 split of the corpus is shared by all 27 subjects: the same five evaluation sentences are held out for every subject. Target-subject calibration uses trials from the remaining 45 sentences, while one Aloud and one Mimed trial per evaluation sentence is reserved for testing when available, giving a nominal 10-trial test set per subject. Multi-subject pretraining uses training-partition trials from the 26 non-held-out subjects, which may include the non-reserved repetitions of the evaluation sentences. The primary analysis and the unseen-sentence test both consider all 27 subjects. Primary decoding metrics pool Aloud and Mimed trials. (c) Model pipeline: the model maps surface EMG to frame-level character probabilities, and CTC beam search with a language model converts them into text. The subject-specific MLP adapter is optional and is evaluated in the ablation analysis.
Condition
n
CER (%)
CER Std
WER (%)
WER Std
Pretrain + fine-tune (no MLP)
27
22.5
13.3
33.3
18.3
Pretrain + fine-tune (MLP)
27
21.7
12.5
31.9
17.0
Pretrain (from random init) + fine-tune (MLP) †
27
44.9
29.5
58.8
26.0
Pretrain zero-shot (no MLP)
27
49.3
20.8
63.1
21.1
Pretrain (from random init) zero-shot (no MLP)
27
57.8
26.7
70.7
24.5
Checkpoint + fine-tune (no MLP)
27
67.6
5.9
96.7
5.0
Table 1: LOSO evaluation ( n=27 ). CER and WER are macro means over the 27 held-out subjects, pooling Aloud and Mimed trials; Std is the subject-level sample standard deviation.
Figure 2: LOSO evaluation of 27 held-out subjects. (a) Pooled, Aloud, and Mimed CER and WER for Pretrain + fine-tune (MLP); bars show macro means ± 1 subject-level Std. (b) Pooled CER for the seven conditions in Table 1 ; bars show macro means ± 1 Std, and dots show subjects. In the Pretrain (from random init) + fine-tune (MLP) row, the five subjects at 100% CER are the folds in which multi-subject pretraining did not converge. Stars denote paired two-sided Wilcoxon comparisons against Pretrain + fine-tune (MLP) ( ∗∗∗p<0.001 ; n.s. p≥0.05 ). The bracket denotes the Aloud-versus-Mimed CER comparison ( ∗∗∗p=3.0×10−5 ).
Calibration budget
n
CER (%)
CER Std
WER (%)
WER Std
pCER vs. full
pWER vs. full
1 min
27
25.4
13.1
36.6
16.3
0.003
0.009
3 min
27
20.5
13.2
31.7
17.5
0.27
0.88
5 min
27
19.9
13.5
28.4
17.2
0.26
0.11
10 min
27
18.6
12.5
28.6
16.9
0.030
0.049
Full fine-tuning ( ≈ 13 min)
27
21.7
12.5
31.9
17.0
–
–
Table 2: Sparse calibration sweep with the Pretrain + fine-tune (MLP) model ( n=27 )
Figure 3: Sparse target-subject calibration across 27 held-out subjects for Pooled, Aloud, and Mimed trials. Boxes show the median and interquartile range, dots show subjects, and lines connect budget medians; the dashed line marks the full-fine-tuning mean. Statistical comparisons are reported in Table 2 .
Figure 4: Held-out CER against the number of pretraining subjects for the Pretrain + fine-tune (MLP) model, pooling Aloud and Mimed trials. All 27 held-out subjects are evaluated at every pool size using randomly selected non-held-out subjects. The line shows the macro mean and the shaded band shows ± 1 subject-level Std. The 26-subject endpoint corresponds to the Pretrain + fine-tune (MLP) condition in Table 1 .
Evaluation
n
CER (%)
CER Std
WER (%)
WER Std
Held-out trials, closed-corpus condition
27
21.7
12.5
31.9
17.0
Unseen sentences
27
78.6
4.9
99.9
2.6
Table 3: Unseen-sentence stress test ( n=27 )
Figure 5: Closed-corpus versus unseen-sentence performance for the same 27 held-out subjects, using the Pretrain + fine-tune (MLP) model with Aloud and Mimed trials pooled. Panels show (a) CER and (b) WER. Boxes indicate the median and interquartile range, diamonds indicate macro means, and dots indicate individual subjects. Stars denote paired two-sided Wilcoxon signed-rank comparisons ( ∗∗∗p<0.001 ).
Data-efficient neural decoding is a central challenge for speech brain-computer interfaces. We present the first demonstration of transfer learning and cross-task decoding for MEG-based speech models spanning perception and production. We pre-train a Conformer-based model on 50 hours of single-subject listening data and fine-tune on just 5 minutes per subject across 18 participants. Transfer learning yields consistent improvements, with in-task accuracy gains of 1-4% and larger cross-task gains of up to 5-6%. Not only does pre-training improve performance within each task, but it also enables reliable cross-task decoding between perception and production. Critically, models trained on speech production decode passive listening above chance, confirming that learned representations reflect shared neural processes rather than task-specific motor activity.
Xabier de Zuazo, Vincenzo Verbeni, Eva Navas +3
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3
We test whether Speech Articulatory Coding (SPARC) features can linearly predict surface electromyography (sEMG) envelopes across aloud, mimed, and subvocal speech in twenty-four subjects. Using elastic-net multivariate temporal response function (mTRF) with sentence-level cross-validation, SPARC yields higher prediction accuracy than phoneme one-hot representations on nearly all electrodes and in all speech modes. Aloud and mimed speech perform comparably, and subvocal speech remains above chance, indicating detectable articulatory activity. Variance partitioning shows a substantial unique contribution from SPARC and a minimal unique contribution from phoneme features. mTRF weight patterns reveal anatomically interpretable relationships between electrode sites and articulatory movements that remain consistent across modes. This study focuses on representation/encoding analysis (not end-to-end decoding) and supports SPARC as a robust and interpretable intermediate target for sEMG-based silent-speech modeling.
Objective. A decoder of imagined speech that works for a new user without calibration would make a non-invasive speech interface practical. We tested whether imagined-speech content recorded with scalp electroencephalography (EEG) transfers across people, and examined why accuracy within a single person remains low. Approach. We analyzed three public corpora, ds003626, KaraOne and Track 3 of the 2020 International Brain-Computer Interface Competition, under a protocol that enforces split integrity by assertion and reports exact binomial thresholds, group tests and bootstrap confidence intervals. We trained decoders on each participant and tested them on every other, compared representational geometry across participants without a classifier, applied subject-level alignment, and tested five explanations for the within-subject ceiling. Main results. Across 482 ordered pairs of participants, no pair survived correction for multiple comparisons; mean pair-transfer accuracy lay within 0.006 of chance on every corpus, while every within-subject reference lay above it. Imagery against pre-stimulus rest generalized at about 0.61 after artifact and elapsed-time controls, but its decoder labelled the interval preceding imagery as imagery as often as imagery itself, so the contrast reflects a post-stimulus state, not imagery. Within subject, only the competition corpus decoded above threshold for every participant (0.3147; chance 0.200); syllable count explained little of this, but its frequency profile failed an artifact audit. Significance. In these corpora, imagined-speech content did not transfer between people through any representation we examined. Trial count per class was the only variable that ordered the corpora, which favors larger single-condition datasets over further model development.
Blessing Isoyiza Adeika, Onyema Osuagwu
Morgan State University, Baltimore, Maryland, United States of America