SHINE: Sequential Hierarchical Integration Network for EEG and MEG

Xiran Xu; Yujie Yan; Songyi Li; Linze Zheng; Zifeng Zhang; Mochu Dong; Jing Chen

cs.SDFeb 27, 2026

SHINE: Sequential Hierarchical Integration Network for EEG and MEG

Authors: Xiran Xu, Yujie Yan, Songyi Li, Linze Zheng, Zifeng Zhang, Mochu Dong, Jing Chen

Organizations: Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, China · National Key Laboratory of General Artificial Intelligence, China · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, China

Abstract

How natural speech is represented in the brain constitutes a major challenge for cognitive neuroscience. Reconstructing the speech envelope and Mel spectrogram from EEG and MEG provides a time-resolved way to study its temporal and spectral structure. Speech-related neural activity spans sensors and temporal scales; extracting these representations while adapting the use of context to each acoustic target is a central problem in speech reconstruction. We propose SHINE, a Sequential Hierarchical Integration Network for EEG and MEG. A residual sensor adapter unifies input dimensions, intermediate dilated-block states retain temporal depth, and a target- and time-dependent gate fuses local hierarchical and attention-enhanced context predictions. Across two EEG and two MEG datasets, SHINE has the highest mean envelope and mean-Mel Pearson correlations among nine local baseline implementations on all eight dataset-metric combinations. SHINE also placed second in the speech-detection Extended Track of the NeurIPS 2025 PNPL Competition. Code will be released at https://github.com/xuxiran/SHINE.

Figures & tables

Figure 1: SHINE architecture. The adapter’s lower path has two $1\!\times\!1$ convolutions with GELU between them (“ $\times 2$ ” in the diagram). Its output $h_{0}$ and four intermediate dilated-block states form five temporal depths that project to local features $h_{L}$ ; two pooled-attention blocks produce context features $h_{C}$ . The local and context heads predict one envelope and ten Mel bands as $\hat{y}_{L}$ and $\hat{y}_{C}$ . Solid arrows show data flow; dashed arrows show gate conditioning by $h_{L}$ and $h_{C}$ . Input and output glyphs are schematic.

Dataset	Modality	Recordings	Train/Val/Test	Channels
SparrKULee [ 23 ]	EEG	662	263/207/192	64
PKUEEG [ 24 ]	EEG	1250	578/304/368	57
LibriBrain [ 25 ]	MEG	70	50/10/10	204
SMN4Lang [ 26 ]	MEG	720	336/174/210	204

Table 1: Prepared recordings in the reported experiments. Split counts are train/validation/test; channels are model inputs, with MEG using planar gradiometers.

	Speech envelope (Pearson $r$ )				Mean Mel-spectrogram (Pearson $r$ )
Model	Sparr KULee	PKU EEG	Libri Brain	SMN4 Lang	Sparr KULee	PKU EEG	Libri Brain	SMN4 Lang
SHINE	$\mathbf{.1877}\mathord{\pm}.0005$	$\mathbf{.2300}\mathord{\pm}.0093$	$\mathbf{.5321}\mathord{\pm}.0046$	$\mathbf{.3594}\mathord{\pm}.0074$	$\mathbf{.0483}\mathord{\pm}.0001$	$\mathbf{.0871}\mathord{\pm}.0036$	$\mathbf{.1822}\mathord{\pm}.0016$	$\mathbf{.1370}\mathord{\pm}.0034$
BrainMagick	$.1698\mathord{\pm}.0009$	$.1868\mathord{\pm}.0082$	$.4770\mathord{\pm}.0057$	$.2772\mathord{\pm}.0033$	$\underline{.0455}\mathord{\pm}.0003$	$.0722\mathord{\pm}.0031$	$.1608\mathord{\pm}.0029$	$.1113\mathord{\pm}.0018$
VLAAI	$.1581\mathord{\pm}.0057$	$\underline{.2133}\mathord{\pm}.0019$	$.5162\mathord{\pm}.0005$	$\underline{.3324}\mathord{\pm}.0010$	$.0424\mathord{\pm}.0009$	$\underline{.0808}\mathord{\pm}.0009$	$.1742\mathord{\pm}.0004$	$\underline{.1293}\mathord{\pm}.0013$
HappyQuokka	$.1524\mathord{\pm}.0047$	$.1963\mathord{\pm}.0033$	$\underline{.5258}\mathord{\pm}.0007$	$.3111\mathord{\pm}.0084$	$.0403\mathord{\pm}.0010$	$.0737\mathord{\pm}.0022$	$\underline{.1760}\mathord{\pm}.0005$	$.1166\mathord{\pm}.0037$
ConvConcatNet	$.1590\mathord{\pm}.0067$	$.1930\mathord{\pm}.0026$	$.4475\mathord{\pm}.0070$	$.2062\mathord{\pm}.0101$	$.0427\mathord{\pm}.0015$	$.0755\mathord{\pm}.0003$	$.1479\mathord{\pm}.0026$	$.0869\mathord{\pm}.0032$

Table 2: Main comparison: pooled test Pearson

r

(mean

\pm

sample standard deviation over seeds 0/1/2). Mean Mel is the average over ten band-wise correlations. Comparison rows are local implementations on the common data and metric. Bold marks SHINE means; underlining marks the strongest comparison means.

	Speech envelope (Pearson $r$ )				Mean Mel-spectrogram (Pearson $r$ )
Variant	Sparr KULee	PKU EEG	Libri Brain	SMN4 Lang	Sparr KULee	PKU EEG	Libri Brain	SMN4 Lang
SHINE	$\mathbf{.1877}\mathord{\pm}.0005$	$\mathbf{.2300}\mathord{\pm}.0093$	$\mathbf{.5321}\mathord{\pm}.0046$	$\mathbf{.3594}\mathord{\pm}.0074$	$\mathbf{.0483}\mathord{\pm}.0001$	$\mathbf{.0871}\mathord{\pm}.0036$	$\mathbf{.1822}\mathord{\pm}.0016$	$\mathbf{.1370}\mathord{\pm}.0034$
Single spatial projection	$\underline{.1876}\mathord{\pm}.0024$	$.2264\mathord{\pm}.0051$	$.5128\mathord{\pm}.0057$	$.3159\mathord{\pm}.0019$	$\underline{.0482}\mathord{\pm}.0003$	$.0849\mathord{\pm}.0032$	$.1752\mathord{\pm}.0025$	$.1243\mathord{\pm}.0011$
No block gate	$.1809\mathord{\pm}.0019$	$\underline{.2285}\mathord{\pm}.0007$	$\underline{.5264}\mathord{\pm}.0047$	$\underline{.3591}\mathord{\pm}.0042$	$.0473\mathord{\pm}.0006$	$\underline{.0870}\mathord{\pm}.0007$	$\underline{.1807}\mathord{\pm}.0017$	$\underline{.1369}\mathord{\pm}.0010$

Table 3: Component analysis at 60 epochs: pooled test Pearson

r

(mean

\pm

sample standard deviation over seeds 0/1/2). Mean Mel averages ten band-wise correlations. Bold marks SHINE means; underlining marks the strongest variant means.

Explore similar work

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction
Jun 23, 2026Wenhao Gao, Yifan Wang, Yijia Ma +3Conditional Flow Matching
Stony Brook University, Stony Brook, NY, USA · University of Texas Health Center at Houston, Houston, TX, USA · Emory University, Atlanta, GA, USA
Bypassing Direct Reconstruction: Speech Detection from MEG via Large-Scale Audio Retrieval
May 13, 2026Boda Xiao, Bo Wang, Heping ChengDiscrete Speech Representations Neural Recordings
Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, Beijing, China · National Biomedical Imaging Center, State Key Laboratory of Membrane Biology, Institute of Molecular Medicine, Peking-Tsinghua Center for Life Sciences, College of Future Technology, Peking University, Beijing, China
Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval
Aug 2, 2026Ilia Semenkov, Daria Kleeva, Ivan Dakhtin +2Wav2Vec Human Visual Cortex
AXXX, Moscow, Russia · HSE University, Moscow, Russia · ITMO University, St. Petersburg, Russia

Jun 23, 2026, cs.LGJ/K move · Enter open · S save

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

Wenhao Gao, Yifan Wang, Yijia Ma, Carl Yang+2

Stony Brook University, Stony Brook, NY, USA · University of Texas Health Center at Houston, Houston, TX, USA · Emory University, Atlanta, GA, USA

Conditional Flow Matching

Open paper AskPDF arXiv

Reconstructing continuous speech from scalp electroencephalography (EEG) remains fundamentally challenging. EEG provides a weak, spatially diffuse, and highly variable measurement of distributed cortical activity, whereas speech is organized as a coherent acoustic trajectory with strong harmonic and temporal structure. The resulting mismatch makes waveform regression unstable and causes stochastic multi-step generation to be sensitive to artifact-dependent conditioning and subject variability. We introduce NeuroSonic, a conditional flow-matching framework for EEG-to-speech reconstruction. Instead of predicting waveforms directly or refining them through stochastic denoising, NeuroSonic learns a deterministic probability-flow velocity field that transports a noise-corrupted acoustic state toward clean speech under EEG conditioning. EEG and audio are embedded into a shared token space and processed by a time-conditioned gated Transformer that parameterizes the transport ordinary differential equation. This formulation models trajectory evolution explicitly while avoiding iterative stochastic sampling. We evaluate NeuroSonic on the CineBrain and EAV benchmarks under cross-subject evaluation. Across both datasets, the proposed method improves distributional realism, spectral fidelity, and perceptual quality over representative GAN-, diffusion-, and mean-flow baselines, with up to a 26.3% gain in overall perceptual quality. The performance gap is most evident in artifact-heavy segments, where conditioning variability is strongest. These findings indicate that deterministic conditional transport provides a stable and effective formulation for EEG-driven speech reconstruction. Code is available at https://github.com/Y-Research-SBU/NeuroSonic/ .

Abstract

Explore similar work

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

Bypassing Direct Reconstruction: Speech Detection from MEG via Large-Scale Audio Retrieval

Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval