Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

Aoke Zhang; Bo Wang; Xihong Wu; Heping Cheng; Jing Chen

cs.SDAug 23, 2026

Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

Authors: Aoke Zhang, Bo Wang, Xihong Wu, Heping Cheng, Jing Chen

Organizations: Speech and Hearing Research Center, School of Intelligence Science and Technology and the Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, Beijing, China, and also with the National Key Laboratory of General Artificial Intelligence, Beijing, China · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China

Abstract

Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance. To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data. Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.

Figures & tables

Fig. 1: Different settings for perceived speech decoding include: (a) intra-subject decoding, which involves decoding exclusively within the target subject; multi-subject decoding, where the model is trained on data from all subjects simultaneously and then evaluated on the target subject; and (b) cross-subject decoding, which entails pre-training the model on source subjects followed by generalization and evaluation on the target subject.

Fig. 2: Illustration of the CPSD framework. (a) During the source model pre-training stage, the model output is aligned with the wav2vec representation. (b) In the personal specialization stage, the subject layer for the target subject is first initialized, and then the representation from the MEG/EEG data of the target subject is realigned with the corresponding wav2vec representation.

Fig. 3: Motivation for the PESA module. Different shapes within the large circle represent the MEG/EEG features of subjects under various stimuli.

Dataset	Language	Modality	Sensors	Sampling rate	Subjects	Folds for CV	Duration	Test samples
Armeni 2022	English	MEG	275	1200	3	3	30.0 h	1024
PKUEEG 2025	Chinese	EEG	64	1000	25	25	68.1 h	128
Broderick 2018	English	EEG	128	512	19	19	19.2 h	128

TABLE I: A brief description of the neural datasets related to speech perception. The test samples indicate the size of the retrieval set for each sample.

	Armeni 2022		PKUEEG 2025		Broderick 2018
Methods	Top-10(%)	Rankacc(%)	Top-10(%)	Rankacc(%)	Top-10(%)	Rankacc(%)
Random	1.3 $\pm$ 0.4	50.6 $\pm$ 0.5	7.3 $\pm$ 1.2	49.4 $\pm$ 2.2	7.6 $\pm$ 1.6	50.8 $\pm$ 0.3
ATM-S [ 33 ]	4.2 $\pm$ 1.4	68.8 $\pm$ 4.8	9.2 $\pm$ 3.2	52.7 $\pm$ 2.1	9.0 $\pm$ 2.8	51.9 $\pm$ 3.3
iTransformer [ 38 ]	15.0 $\pm$ 12.3	76.6 $\pm$ 16.4	11.5 $\pm$ 3.5	56.3 $\pm$ 3.1	9.3 $\pm$ 2.9	52.1 $\pm$ 3.3
VLAAI [ 1 ]	14.7 $\pm$ 13.2	75.0 $\pm$ 19.4	14.6 $\pm$ 12.4	55.9 $\pm$ 11.3	10.5 $\pm$ 4.9	55.5 $\pm$ 6.5
Brainmagic [ 14 ]	17.4 $\pm$ 13.8	83.1 $\pm$ 7.7	20.8 $\pm$ 8.1	65.7 $\pm$ 7.0	17.7 $\pm$ 7.1	63.0 $\pm$ 5.5

TABLE II: Perceived speech decoding results across various datasets. We averaged the decoding results across different subjects and evaluated the robustness of performance improvements using statistical tests.

	Armeni 2022		PKUEEG 2025		Broderick 2018
	Top-10(%)	Rankacc(%)	Top-10(%)	Rankacc(%)	Top-10(%)	Rankacc(%)
Base	17.4 $\pm$ 13.8	83.1 $\pm$ 7.7	20.8 $\pm$ 8.1	65.7 $\pm$ 7.0	17.7 $\pm$ 7.1	63.0 $\pm$ 5.5
Base+PESA	25.7 $\pm$ 25.5	84.7 $\pm$ 9.8	27.7 $\pm$ 8.5	70.7 $\pm$ 5.7	18.1 $\pm$ 8.3	62.6 $\pm$ 6.9
Base+PS	50.3 $\pm$ 12.8	95.1 $\pm$ 2.2	29.2 $\pm$ 10.9	72.2 $\pm$ 6.3	35.3 $\pm$ 11.8	75.2 $\pm$ 7.0
Base+PESA+PS	61.3 $\pm$ 12.4	96.9 $\pm$ 1.5	43.0 $\pm$ 13.6	80.0 $\pm$ 6.7	39.9 $\pm$ 15.3	77.7 $\pm$ 8.0

TABLE III: Ablation study of the CPSD framework across different datasets. (PS stands for personal specialization.)

	Armeni 2022		PKUEEG 2025		Broderick 2018
Methods	Latent	Final	Latent	Final	Latent	Final
Brainmagic	0.011 $\pm$ 0.001	0.230 $\pm$ 0.012	0.003 $\pm$ 0.001	0.093 $\pm$ 0.022	0.001 $\pm$ 0.001	0.157 $\pm$ 0.009
CPSD(Ours)	0.016 $\pm$ 0.002	0.274 $\pm$ 0.012	0.002 $\pm$ 0.001	0.129 $\pm$ 0.010	0.002 $\pm$ 0.001	0.227 $\pm$ 0.010

TABLE IV: Evaluation of neural representation consistency.

Fig. 4: The correlation between model performance and neural representation consistency ( $corr=0.66$ , $p<0.001$ ).

Fig. 5: Training times under different settings.

Fig. 6: Zero-shot decoding performance across various perceived speech datasets.

Fig. 7: Decoding results using various hyperparameters on the Broderick 2018 dataset. The triangles in the box plot indicate the mean values.

	Armeni 2022		PKUEEG 2025		Broderick 2018
Methods	Top-10(%)	Rankacc(%)	Top-10(%)	Rankacc(%)	Top-10(%)	Rankacc(%)
Brainmagic	17.4 $\pm$ 13.8	83.1 $\pm$ 7.7	20.8 $\pm$ 8.1	65.7 $\pm$ 7.0	17.7 $\pm$ 7.1	63.0 $\pm$ 5.5
Brainmagic-MD	52.7 $\pm$ 15.5	95.6 $\pm$ 2.1	39.1 $\pm$ 10.3	78.8 $\pm$ 5.3	39.2 $\pm$ 12.4	77.4 $\pm$ 7.1
CPSD	61.3 $\pm$ 12.4	96.9 $\pm$ 1.5	43.0 $\pm$ 13.6	80.0 $\pm$ 6.7	39.9 $\pm$ 15.3	77.7 $\pm$ 8.0
CPSD-MD	63.0 $\pm$ 11.8	97.1 $\pm$ 1.3	45.5 $\pm$ 11.5	81.5 $\pm$ 5.5	46.0 $\pm$ 14.5	81.0 $\pm$ 6.6

TABLE V: Comparison with Multi-Subject Settings. (MD stands for Multi-Subject Decoding.)

Fig. 8: Verification of the validity of sensor locations as input. The performance improvement of CPSD compared to the Shuffle setting in both figures is statistically significant ( $p<0.01$ ).

Fig. 9: Visualization of channel weights from the PESA module across various datasets.

Cross-Subject EEG Decoding

Cross-subject EEG decoding develops methods that classify or predict brain activity for people not seen during training.

See all 77 papers

Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings
Sep 25, 2026Aoke Zhang, Jing ChenNeural Decoding Brain-Computer Interfaces
School of Intelligence Science and Technology, Peking University, Beijing, China · Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · National Key Laboratory of General Artificial Intelligence, Beijing, China
MEG-to-MEG Transfer Learning and Cross-Task Speech/Silence Detection with Limited Data
Feb 20, 2026Xabier de Zuazo, Vincenzo Verbeni, Eva Navas +3Neural Decoding Transfer Learning
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3
Decoding Imagined Speech: A Strictly Subject-Independent Approach Using EEG
Sep 24, 2026Frederik Møllskov Trier, Xiaopeng Mao, Sadasivan PuthusserypadyCross-Subject EEG Decoding Electroencephalography
Department of Health Technology Technical University of Denmark Ørsteds Plads, 2800 Kgs. Lyngby, Denmark

Sep 25, 2026J/K move · Enter open · S save

Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings

Aoke Zhang, Jing Chen

School of Intelligence Science and Technology, Peking University, Beijing, China · Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · National Key Laboratory of General Artificial Intelligence, Beijing, China

Neural Decoding Brain-Computer Interfaces Cross-Modal Learning Multimodal Fusion

Open paper AskPDF arXiv

Perceived speech decoding based on non-invasive brain-computer interface (BCI) signals has been extensively studied in recent years. Research in this field primarily faces two challenges: extracting neural representations with rich spatiotemporal information and achieving cross-subject generalization. Although separate studies have proposed methods to cope with these issues, a unified approach that simultaneously tackles both challenges remains lacking. To fill this gap, we propose the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We conduct comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs. Our results demonstrate that the proposed method improves Top-1, Top-10, and Rankacc by more than 10.6%, 10.1%, and 1.7%, respectively, compared to baseline methods in cross-subject perceived speech decoding tasks, while reducing training costs by 88.8% and 60.5% compared to multi-subject and intra-subject decoding settings. Further visualization experiments also confirm the effectiveness of our approach.

Abstract

Cross-Subject EEG Decoding

Explore similar work

Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings

MEG-to-MEG Transfer Learning and Cross-Task Speech/Silence Detection with Limited Data

Decoding Imagined Speech: A Strictly Subject-Independent Approach Using EEG