Organizations: Speech and Hearing Research Center, School of Intelligence Science and Technology and the Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, Beijing, China, and also with the National Key Laboratory of General Artificial Intelligence, Beijing, China · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China
Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance. To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data. Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.
Figures & tables
Fig. 1: Different settings for perceived speech decoding include: (a) intra-subject decoding, which involves decoding exclusively within the target subject; multi-subject decoding, where the model is trained on data from all subjects simultaneously and then evaluated on the target subject; and (b) cross-subject decoding, which entails pre-training the model on source subjects followed by generalization and evaluation on the target subject.
Fig. 2: Illustration of the CPSD framework. (a) During the source model pre-training stage, the model output is aligned with the wav2vec representation. (b) In the personal specialization stage, the subject layer for the target subject is first initialized, and then the representation from the MEG/EEG data of the target subject is realigned with the corresponding wav2vec representation.
Fig. 3: Motivation for the PESA module. Different shapes within the large circle represent the MEG/EEG features of subjects under various stimuli.
Dataset
Language
Modality
Sensors
Sampling rate
Subjects
Folds for CV
Duration
Test samples
Armeni 2022
English
MEG
275
1200
3
3
30.0 h
1024
PKUEEG 2025
Chinese
EEG
64
1000
25
25
68.1 h
128
Broderick 2018
English
EEG
128
512
19
19
19.2 h
128
TABLE I: A brief description of the neural datasets related to speech perception. The test samples indicate the size of the retrieval set for each sample.
Armeni 2022
PKUEEG 2025
Broderick 2018
Methods
Top-10(%)
Rankacc(%)
Top-10(%)
Rankacc(%)
Top-10(%)
Rankacc(%)
Random
1.3 ± 0.4
50.6 ± 0.5
7.3 ± 1.2
49.4 ± 2.2
7.6 ± 1.6
50.8 ± 0.3
ATM-S [ 33 ]
4.2 ± 1.4
68.8 ± 4.8
9.2 ± 3.2
52.7 ± 2.1
9.0 ± 2.8
51.9 ± 3.3
iTransformer [ 38 ]
15.0 ± 12.3
76.6 ± 16.4
11.5 ± 3.5
56.3 ± 3.1
9.3 ± 2.9
52.1 ± 3.3
VLAAI [ 1 ]
14.7 ± 13.2
75.0 ± 19.4
14.6 ± 12.4
55.9 ± 11.3
10.5 ± 4.9
55.5 ± 6.5
Brainmagic [ 14 ]
17.4 ± 13.8
83.1 ± 7.7
20.8 ± 8.1
65.7 ± 7.0
17.7 ± 7.1
63.0 ± 5.5
TABLE II: Perceived speech decoding results across various datasets. We averaged the decoding results across different subjects and evaluated the robustness of performance improvements using statistical tests.
Armeni 2022
PKUEEG 2025
Broderick 2018
Top-10(%)
Rankacc(%)
Top-10(%)
Rankacc(%)
Top-10(%)
Rankacc(%)
Base
17.4 ± 13.8
83.1 ± 7.7
20.8 ± 8.1
65.7 ± 7.0
17.7 ± 7.1
63.0 ± 5.5
Base+PESA
25.7 ± 25.5
84.7 ± 9.8
27.7 ± 8.5
70.7 ± 5.7
18.1 ± 8.3
62.6 ± 6.9
Base+PS
50.3 ± 12.8
95.1 ± 2.2
29.2 ± 10.9
72.2 ± 6.3
35.3 ± 11.8
75.2 ± 7.0
Base+PESA+PS
61.3 ± 12.4
96.9 ± 1.5
43.0 ± 13.6
80.0 ± 6.7
39.9 ± 15.3
77.7 ± 8.0
TABLE III: Ablation study of the CPSD framework across different datasets. (PS stands for personal specialization.)
Armeni 2022
PKUEEG 2025
Broderick 2018
Methods
Latent
Final
Latent
Final
Latent
Final
Brainmagic
0.011 ± 0.001
0.230 ± 0.012
0.003 ± 0.001
0.093 ± 0.022
0.001 ± 0.001
0.157 ± 0.009
CPSD(Ours)
0.016 ± 0.002
0.274 ± 0.012
0.002 ± 0.001
0.129 ± 0.010
0.002 ± 0.001
0.227 ± 0.010
TABLE IV: Evaluation of neural representation consistency.
Fig. 4: The correlation between model performance and neural representation consistency ( corr=0.66 , p<0.001 ).
Fig. 5: Training times under different settings.
Fig. 6: Zero-shot decoding performance across various perceived speech datasets.
Fig. 7: Decoding results using various hyperparameters on the Broderick 2018 dataset. The triangles in the box plot indicate the mean values.
Armeni 2022
PKUEEG 2025
Broderick 2018
Methods
Top-10(%)
Rankacc(%)
Top-10(%)
Rankacc(%)
Top-10(%)
Rankacc(%)
Brainmagic
17.4 ± 13.8
83.1 ± 7.7
20.8 ± 8.1
65.7 ± 7.0
17.7 ± 7.1
63.0 ± 5.5
Brainmagic-MD
52.7 ± 15.5
95.6 ± 2.1
39.1 ± 10.3
78.8 ± 5.3
39.2 ± 12.4
77.4 ± 7.1
CPSD
61.3 ± 12.4
96.9 ± 1.5
43.0 ± 13.6
80.0 ± 6.7
39.9 ± 15.3
77.7 ± 8.0
CPSD-MD
63.0 ± 11.8
97.1 ± 1.3
45.5 ± 11.5
81.5 ± 5.5
46.0 ± 14.5
81.0 ± 6.6
TABLE V: Comparison with Multi-Subject Settings. (MD stands for Multi-Subject Decoding.)
Fig. 8: Verification of the validity of sensor locations as input. The performance improvement of CPSD compared to the Shuffle setting in both figures is statistically significant ( p<0.01 ).
Fig. 9: Visualization of channel weights from the PESA module across various datasets.
Perceived speech decoding based on non-invasive brain-computer interface (BCI) signals has been extensively studied in recent years. Research in this field primarily faces two challenges: extracting neural representations with rich spatiotemporal information and achieving cross-subject generalization. Although separate studies have proposed methods to cope with these issues, a unified approach that simultaneously tackles both challenges remains lacking. To fill this gap, we propose the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We conduct comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs. Our results demonstrate that the proposed method improves Top-1, Top-10, and Rankacc by more than 10.6%, 10.1%, and 1.7%, respectively, compared to baseline methods in cross-subject perceived speech decoding tasks, while reducing training costs by 88.8% and 60.5% compared to multi-subject and intra-subject decoding settings. Further visualization experiments also confirm the effectiveness of our approach.
Aoke Zhang, Jing Chen
School of Intelligence Science and Technology, Peking University, Beijing, China · Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · National Key Laboratory of General Artificial Intelligence, Beijing, China
Data-efficient neural decoding is a central challenge for speech brain-computer interfaces. We present the first demonstration of transfer learning and cross-task decoding for MEG-based speech models spanning perception and production. We pre-train a Conformer-based model on 50 hours of single-subject listening data and fine-tune on just 5 minutes per subject across 18 participants. Transfer learning yields consistent improvements, with in-task accuracy gains of 1-4% and larger cross-task gains of up to 5-6%. Not only does pre-training improve performance within each task, but it also enables reliable cross-task decoding between perception and production. Critically, models trained on speech production decode passive listening above chance, confirming that learned representations reflect shared neural processes rather than task-specific motor activity.
Xabier de Zuazo, Vincenzo Verbeni, Eva Navas +3
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3
Imagined speech decoding from electroencephalography (EEG) has gained increasing attention as a potential communication pathway for individuals with severe motor impairments, yet reported performance often relies on evaluation protocols that do not clearly reflect cross-subject generalization. This study presents a transparent baseline investigation of a multi-class imagined speech EEG dataset under a strictly subject-independent evaluation framework. Two preprocessing and feature extraction pipelines were compared: a time-domain statistical feature approach and a frequency-domain spectral bandpower approach, evaluated using subject-wise cross-validation and trial-level majority voting with a random forest classifier. The spectral pipeline achieved a significantly higher mean trial-wise accuracy than the statistical pipeline (49.03 ± 4.18% vs. 37.97 ± 3.79%) for coarse-level classification across subjects. Forward feature selection further indicated that a limited subset of frequency bands captured most of the discriminative information. Overall, this work provides a strong basis for future brain-computer interface studies targeting improved cross-subject generalization in EEG-based imagined speech decoding.
Frederik Møllskov Trier, Xiaopeng Mao, Sadasivan Puthusserypady
Department of Health Technology Technical University of Denmark Ørsteds Plads, 2800 Kgs. Lyngby, Denmark