Contrastive Learning with Variational Regularization for Multi-Session EEG-to-Speech Decoding

Tomoaki Mizuno; Toru Nakashika

eess.ASAug 17, 2026

Contrastive Learning with Variational Regularization for Multi-Session EEG-to-Speech Decoding

Authors: Tomoaki Mizuno, Toru Nakashika

Organizations: The University of Electro-Communications, Tokyo, Japan

Abstract

Reconstructing heard speech from non-invasive electroencephalography (EEG) is challenging due to a low signal-to-noise ratio (SNR) and inter-session variability. While trial averaging improves the SNR, it is difficult to apply to continuous speech. We instead use repeated EEG responses to the same stimulus across different sessions as positive pairs for contrastive learning, and introduce variational regularization that, combined with this contrastive objective, keeps the encoder representation space broad. Experiments on a Japanese EEG dataset show that combining the session-invariant strategy with variational regularization improves the character error rate (CER) while maintaining mel-spectrogram reconstruction fidelity. Session probing confirms that the encoder representations achieve session-invariance.

Figures & tables

Figure 1: Overview of the proposed method. Top: Multi-session EEG data and positive pair construction. From repeated EEG recordings elicited by speech stimuli with the same linguistic content, we construct positive pairs from recordings of the same speech stimulus across different sessions. One recording is randomly selected as a anchor, and the remaining two serve as positive examples. Middle: The architecture of the proposed model. The Transformer Encoder maps the anchor EEG to

\boldsymbol{h}

, which is passed to the Transformer Decoder to predict log-mel spectrograms. The SRM produces

\boldsymbol{z}

via reparameterization;

\boldsymbol{z}

is used only for the losses and is not passed to the decoder. The momentum encoder, updated via the exponential moving average (EMA) of the encoder, generates

\boldsymbol{z}^{+}

from the positive example. InfoNCE is computed over

\boldsymbol{z}

\boldsymbol{z}^{+}

, and memory-bank negatives

\{\boldsymbol{\mu}^{-}\}

. Bottom: Internal structure of the SRM: global average pooling (GAP) followed by linear layers predicting

\boldsymbol{\mu}

and

\log\boldsymbol{\sigma}^{2}

;

\mathcal{L}_{\mathrm{KL}}

regularizes the latent space.

Tag	$\mathcal{L}_{\mathrm{NCE}}$	$\mathcal{L}_{\mathrm{KL}}$	CER $\downarrow$	SECS $\uparrow$	SPA (%) $\uparrow$	PCC $\uparrow$
Baseline	-	-	0.968±0.121	0.190±0.160	5.32±1.08	0.282±0.150
CL alone	✓	-	0.956±0.095	0.173±0.158	4.86±1.03	0 .262±0.143
VR alone	-	✓	0.974±0.113	0.193±0.163	5.79±1.12	0.277±0.146
Proposed method	✓	✓	0.948±0.097	0.176±0.151	7.18±1.24	0.274±0.155

Table 1: Results on the evaluation set. Bold : significantly better than the baseline; underline : significantly worse than the baseline (Holm–Bonferroni corrected,

\alpha=0.05

). CER: lower is better; SECS, PCC (dimensionless,

[-1,1]

) and SPA (%): higher is better. Values are mean

\pm

standard deviation over utterances, except SPA, which is reported as the mean

\pm

standard error.The proposed method is highlighted in gray.

Tag	Decoder input	CER $\downarrow$	SECS $\uparrow$	SPA (%) $\uparrow$	PCC $\uparrow$
CL + VR
Proposed method	$\boldsymbol{h}$	$0.948{\pm}0.097$	$0.176{\pm}0.151$	$7.18{\pm}1.24$	$0.274{\pm}0.155$
+ prepending $\boldsymbol{z}$ to $\boldsymbol{h}$	$\boldsymbol{z}+\boldsymbol{h}$	$0.951{\pm}0.091$	$0.174{\pm}0.148$	$6.02{\pm}1.14$	$0.275{\pm}0.148$
VR alone
VR alone	$\boldsymbol{h}$	$0.974{\pm}0.113$	$0.193{\pm}0.163$	$5.79{\pm}1.12$	$0.277{\pm}0.146$
+ prepending $\boldsymbol{z}$ to $\boldsymbol{h}$	$\boldsymbol{z}+\boldsymbol{h}$	$0.971{\pm}0.119$	$0.185{\pm}0.149$	$6.71{\pm}1.20$	$0.275{\pm}0.140$

Table 2: Results on the ablation set. For each group, the experiments in the upper row are without our proposed

\boldsymbol{z}

as decoder input, while the experiments in the lower row with ”+ prepending

\boldsymbol{z}

\boldsymbol{h}

” tags shows results when

\boldsymbol{z}

was concatenated with

\boldsymbol{h}

before being fed into the decoder. For each of the experiments within each group, we conducted tests for each evaluation metric, but found no significant differences.The proposed method is highlighted in gray.

	w/o VR	w/ VR	w/ VR +prepending $\boldsymbol{z}$ to $\boldsymbol{h}$
w/o CL	25.2 $\pm$ 2.09	24.8 $\pm$ 2.08	23.2 $\pm$ 2.03
w/ CL	2.78 $\pm$ 0.79 $\dagger$	2.08 $\pm$ 0.69 $\dagger$	3.01 $\pm$ 0.82 $\dagger$

Table 3: Session probing accuracy (%). Chance level is

1/45\approx 2.2\%

; lower is better.

\dagger

: indistinguishable from chance (two-sided binomial test,

p>0.05

). The proposed method is highlighted in gray.

Figure 2: Value distributions of the encoder summary vector

\bar{\boldsymbol{h}}

(blue) and the SRM posterior mean

\boldsymbol{\mu}

(orange) across test samples.The proposed method corresponds to w/ CL + w/ VR.

Cross-Session EEG Decoding

See all 9 papers

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction
Jun 23, 2026Wenhao Gao, Yifan Wang, Yijia Ma +3Conditional Flow Matching EEG Decoding
Stony Brook University, Stony Brook, NY, USA · University of Texas Health Center at Houston, Houston, TX, USA · Emory University, Atlanta, GA, USA
Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
Oct 14, 2026Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1EEG Decoding EEG-to-Text Decoding
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy
SHINE: Sequential Hierarchical Integration Network for EEG and MEG
Feb 27, 2026Xiran Xu, Yujie Yan, Songyi Li +4Electroencephalography Neural Processes
Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, China · National Key Laboratory of General Artificial Intelligence, China · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, China

Jun 23, 2026J/K move · Enter open · S save

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

Wenhao Gao, Yifan Wang, Yijia Ma, Carl Yang+2

Stony Brook University, Stony Brook, NY, USA · University of Texas Health Center at Houston, Houston, TX, USA · Emory University, Atlanta, GA, USA

Conditional Flow Matching EEG Decoding Speech Processing

Open paper AskPDF arXiv

Reconstructing continuous speech from scalp electroencephalography (EEG) remains fundamentally challenging. EEG provides a weak, spatially diffuse, and highly variable measurement of distributed cortical activity, whereas speech is organized as a coherent acoustic trajectory with strong harmonic and temporal structure. The resulting mismatch makes waveform regression unstable and causes stochastic multi-step generation to be sensitive to artifact-dependent conditioning and subject variability. We introduce NeuroSonic, a conditional flow-matching framework for EEG-to-speech reconstruction. Instead of predicting waveforms directly or refining them through stochastic denoising, NeuroSonic learns a deterministic probability-flow velocity field that transports a noise-corrupted acoustic state toward clean speech under EEG conditioning. EEG and audio are embedded into a shared token space and processed by a time-conditioned gated Transformer that parameterizes the transport ordinary differential equation. This formulation models trajectory evolution explicitly while avoiding iterative stochastic sampling. We evaluate NeuroSonic on the CineBrain and EAV benchmarks under cross-subject evaluation. Across both datasets, the proposed method improves distributional realism, spectral fidelity, and perceptual quality over representative GAN-, diffusion-, and mean-flow baselines, with up to a 26.3% gain in overall perceptual quality. The performance gap is most evident in artifact-heavy segments, where conditioning variability is strongest. These findings indicate that deterministic conditional transport provides a stable and effective formulation for EEG-driven speech reconstruction. Code is available at https://github.com/Y-Research-SBU/NeuroSonic/ .

Abstract

Cross-Session EEG Decoding

Explore similar work

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech

SHINE: Sequential Hierarchical Integration Network for EEG and MEG