Authors: Chenqian Le, Beatrice Fumagalli, Yasamin Esmaeili, Xupeng Chen, Tianyu He, Nikasadat Emami, Adeen Flinker, Yao Wang
Organizations: Department of Electrical and Computer Engineering, New York University Tandon School of Engineering, New York, NY, USA · Department of Neurology, New York University Grossman School of Medicine, New York, NY, USA · Department of Biomedical Engineering, New York University Tandon School of Engineering, New York, NY, USA
Surface electromyography (sEMG)-based silent speech interfaces are limited by cross-user variability and calibration burden. We study a limited-data setting in which each of 27 speech-typical participants contributed less than 0.5 h of data (21.3 min on average) across Aloud and Mimed speech. Within a closed 50-sentence corpus, we used leave-one-subject-out evaluation, initializing from a released single-subject checkpoint, pretraining on non-held-out participants, and fine-tuning on the target participant. This pipeline achieved 21.7% character error rate (CER) and 31.9% word error rate (WER), compared with 49.3% CER without target-subject calibration and 68.0% CER for direct checkpoint fine-tuning. Multi-subject pretraining from random initialization followed by fine-tuning reached 44.9% CER and did not converge under the fixed schedule in 5 of 27 folds, indicating substantial optimization and accuracy benefits from checkpoint initialization. Macro-averaged CER declined from 74.4% with one pretraining participant to 21.7% with 26. Three minutes of target-subject calibration achieved 20.5% CER and 31.7% WER, with no statistically significant difference from the full approximately 13-min pool (21.7% CER and 31.9% WER). A subject-specific adapter provided no detectable benefit. Excluding the five evaluation sentences from all sEMG model-training data increased CER and WER to 78.6% and 99.9%. These results support short-calibration personalization in a standardized-montage, closed-corpus setting.
Figures & tables
Figure 1: Overview of the cross-subject sEMG speech decoding framework. (a) Dataset and recording: 27 subjects, a 50-sentence corpus, Aloud and Mimed speech modes, and an eight-channel sEMG montage over the lower face and neck. (b) Closed-corpus LOSO personalization protocol. A single fixed 45/5 split of the corpus is shared by all 27 subjects: the same five evaluation sentences are held out for every subject. Target-subject calibration uses trials from the remaining 45 sentences, while one Aloud and one Mimed trial per evaluation sentence is reserved for testing when available, giving a nominal 10-trial test set per subject. Multi-subject pretraining uses training-partition trials from the 26 non-held-out subjects, which may include the non-reserved repetitions of the evaluation sentences. The primary analysis and the unseen-sentence test both consider all 27 subjects. Primary decoding metrics pool Aloud and Mimed trials. (c) Model pipeline: the model maps surface EMG to frame-level character probabilities, and CTC beam search with a language model converts them into text. The subject-specific MLP adapter is optional and is evaluated in the ablation analysis.
Condition
n
CER (%)
CER Std
WER (%)
WER Std
Pretrain + fine-tune (no MLP)
27
22.5
13.3
33.3
18.3
Pretrain + fine-tune (MLP)
27
21.7
12.5
31.9
17.0
Pretrain (from random init) + fine-tune (MLP) †
27
44.9
29.5
58.8
26.0
Pretrain zero-shot (no MLP)
27
49.3
20.8
63.1
21.1
Pretrain (from random init) zero-shot (no MLP)
27
57.8
26.7
70.7
24.5
Checkpoint + fine-tune (no MLP)
27
67.6
5.9
96.7
5.0
Table 1: LOSO evaluation ( n=27 ). CER and WER are macro means over the 27 held-out subjects, pooling Aloud and Mimed trials; Std is the subject-level sample standard deviation.
Figure 2: LOSO evaluation of 27 held-out subjects. (a) Pooled, Aloud, and Mimed CER and WER for Pretrain + fine-tune (MLP); bars show macro means ± 1 subject-level Std. (b) Pooled CER for the seven conditions in Table 1 ; bars show macro means ± 1 Std, and dots show subjects. In the Pretrain (from random init) + fine-tune (MLP) row, the five subjects at 100% CER are the folds in which multi-subject pretraining did not converge. Stars denote paired two-sided Wilcoxon comparisons against Pretrain + fine-tune (MLP) ( ∗∗∗p<0.001 ; n.s. p≥0.05 ). The bracket denotes the Aloud-versus-Mimed CER comparison ( ∗∗∗p=3.0×10−5 ).
Calibration budget
n
CER (%)
CER Std
WER (%)
WER Std
pCER vs. full
pWER vs. full
1 min
27
25.4
13.1
36.6
16.3
0.003
0.009
3 min
27
20.5
13.2
31.7
17.5
0.27
0.88
5 min
27
19.9
13.5
28.4
17.2
0.26
0.11
10 min
27
18.6
12.5
28.6
16.9
0.030
0.049
Full fine-tuning ( ≈ 13 min)
27
21.7
12.5
31.9
17.0
–
–
Table 2: Sparse calibration sweep with the Pretrain + fine-tune (MLP) model ( n=27 )
Figure 3: Sparse target-subject calibration across 27 held-out subjects for Pooled, Aloud, and Mimed trials. Boxes show the median and interquartile range, dots show subjects, and lines connect budget medians; the dashed line marks the full-fine-tuning mean. Statistical comparisons are reported in Table 2 .
Figure 4: Held-out CER against the number of pretraining subjects for the Pretrain + fine-tune (MLP) model, pooling Aloud and Mimed trials. All 27 held-out subjects are evaluated at every pool size using randomly selected non-held-out subjects. The line shows the macro mean and the shaded band shows ± 1 subject-level Std. The 26-subject endpoint corresponds to the Pretrain + fine-tune (MLP) condition in Table 1 .
Evaluation
n
CER (%)
CER Std
WER (%)
WER Std
Held-out trials, closed-corpus condition
27
21.7
12.5
31.9
17.0
Unseen sentences
27
78.6
4.9
99.9
2.6
Table 3: Unseen-sentence stress test ( n=27 )
Figure 5: Closed-corpus versus unseen-sentence performance for the same 27 held-out subjects, using the Pretrain + fine-tune (MLP) model with Aloud and Mimed trials pooled. Panels show (a) CER and (b) WER. Boxes indicate the median and interquartile range, diamonds indicate macro means, and dots indicate individual subjects. Stars denote paired two-sided Wilcoxon signed-rank comparisons ( ∗∗∗p<0.001 ).
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3