Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift

Sheng-You Chien; Bo-Yi Mao; Yi-Ning Chang; Po-Chih Kuo

cs.SDApr 5, 2026

Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift

Authors: Sheng-You Chien, Bo-Yi Mao, Yi-Ning Chang, Po-Chih Kuo

Organizations: Department of Computer Science, National Tsing-Hua University, Hsinchu, Taiwan

Abstract

This study investigates robust speech-related decoding from non-invasive MEG signals using the LibriBrain phoneme-classification benchmark from the 2025 PNPL competition. We compare residual convolutional neural networks (CNNs), an STFT-based CNN, and a CNN--Transformer hybrid, while also examining the effects of group averaging, label balancing, repeated grouping, normalization strategies, and data augmentation. Across our in-house implementations, preprocessing and data-configuration choices matter more than additional architectural complexity, among which instance normalization emerges as the most influential modification for generalization. The strongest of our own models, a CNN with group averaging, label balancing, repeated grouping, and instance normalization, achieves 60.95% F1-macro on the test split, compared with 39.53% for the plain CNN baseline. However, most of our models, without instance normalization, show substantial validation-to-test degradation, indicating that distribution shift induced by different normalization statistics is a major obstacle to generalization in our experiments. By contrast, MEGConformer maintains 64.09% F1-macro on both validation and test, and saliency-map analysis is qualitatively consistent with this contrast: weaker models exhibit more concentrated or repetitive phoneme-sensitive patterns across splits, whereas MEGConformer appears more distributed. Overall, the results suggest that improving the reliability of non-invasive phoneme decoding will likely require better handling of normalization-related distribution shift while also addressing the challenge of single-trial decoding.

Figures & tables

Figure 1: Raw phoneme-class distribution for the train and validation splits.

Split	Trials	Phonemes	Standardization
Train	76	1,336,606	Train statistics
Validation	14	283,690	Train statistics
Test	14	283,690	Test statistics
Holdout	1	2,382	Holdout statistics
Total	91	1,622,678	–

Table 1: Summary of LibriBrain phoneme classification dataset splits.

Figure 2: Overall model architecture. (a) CNN backbone. (b) STFT-CNN. (c) CNN-Transformer hybrid. For all convolutional blocks, the stride $S$ is set to $1$ , and padding $P$ is chosen to preserve the temporal resolution unless otherwise specified.

Configuration [-0.15ex] (Ungrouped)	Train [-0.15ex] (Ungrouped)	Val. [-0.15ex] (Ungrouped)	Test [-0.15ex] (Ungrouped)	Test -0.15ex	Holdout [-0.15ex] (Ungrouped)
CNN, Label-Balancing, Repeat $\times 10$	90.81	71.95	47.47	2.73	35.40
CNN, +LayerNorm	96.62	44.49	43.17	–	24.40
CNN, No Group Averaging	34.55	45.08	39.53	N/A	13.20
CNN + Augmentation (Section 3.1.4 )	48.55	49.31	34.03	–	18.80
CNN-Transformer	85.83	68.02	30.70	0.14	3.90
CNN, Label-Balancing, Repeat $\times 10$ , +InstanceNorm	99.43	62.75	60.95	4.00	–

Table 2: F1-macro scores (%) for LibriBrain phoneme classification.

Panel of Figure 3 — Figure 3: Row-wise normalized layer saliency maps for MEGConformer across validation and test splits. The saliency is visually stable across splits and distributed across encoder blocks, matching the strong validation–test transfer reported in Table 2 .

Panel of Figure 4 — Figure 4: Row-wise normalized layer saliency maps for the three models for which both standard and InstanceNorm variants are available. In each 3-column block, the upper row shows validation maps and the lower row shows test maps. Each row within a saliency map corresponds to a trainable sublayer, and each column corresponds to a phoneme class. The color intensity indicates the relative saliency of each phoneme for that layer, normalized within the row.

Model	Pearson ( $\mu\pm\sigma$ )	Spearman ( $\mu\pm\sigma$ )
CNN	0.5465 $\pm$ 0.0529	0.5333 $\pm$ 0.0520
CNN + InstanceNorm	0.9996 $\pm$ 0.0003	0.9996 $\pm$ 0.0003
CNN–Transformer	0.4701 $\pm$ 0.0868	0.5099 $\pm$ 0.0808
CNN–Transformer + InstanceNorm	0.9999 $\pm$ 0.0000	0.9999 $\pm$ 0.0002
STFT CNN	0.3770 $\pm$ 0.0448	0.3645 $\pm$ 0.0448
STFT CNN + InstanceNorm	0.9562 $\pm$ 0.0102	0.9503 $\pm$ 0.0123

Table 3: Validation-Test Saliency Map Similarity Summary

Panel of Figure 5 — Figure 5: Layer-by-phoneme validation-test saliency similarity matrices for MEGConformer. Both metrics show near-uniformly high correspondence across layers and phoneme classes.

Panel of Figure 6 — Figure 6: Layer-by-phoneme validation-test saliency similarity matrices for the three models with both standard and InstanceNorm variants. In each 3-column block, the upper row shows Pearson correlation and the lower row shows Spearman correlation. Brighter cells indicate stronger agreement between validation and test saliency patterns for the same layer-phoneme pair.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Train	Validation	Test	Holdout	Backbone	GroupAvg.	LabelBal.	Repeat	Note
90.81	71.95	47.47	35.40	(a)	✓	✓	10
96.62	44.49	43.17	24.40	(a)	✓	✓	1	+LayerNorm
80.06	71.77	44.28	21.60	(a)	✓	✓	1
48.55	49.31	34.03	18.80	(a)	✓	✗	1	+Augmentation
34.55	45.08	39.53	13.20	(a)	✗	✗	1	Baseline
68.16	64.27	47.74	3.60	(a)	✓	✓	1	+Augmentation

Appendix

Table 4: Extended comparison of model and data configurations. Performance is reported in F1-macro (%).

Explore similar work

MEBM-Phoneme: Multi-scale Enhanced BrainMagic for End-to-End MEG Phoneme Classification
Feb 27, 2026Liang Jinghua, Zhang Zifeng, Li Songyi +1Neural Decoding Convolutional Neural Networks
Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, Beijing, China 100871 · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China 100871
MEBM-Speech: Multi-scale Enhanced BrainMagic for Robust MEG Speech Detection
Feb 27, 2026Li Songyi, Zheng Linze, Liang Jinghua +1Neural Decoding Speech Processing
Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, Beijing, China 100871 · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China 100871
MEG-to-MEG Transfer Learning and Cross-Task Speech/Silence Detection with Limited Data
Feb 20, 2026Xabier de Zuazo, Vincenzo Verbeni, Eva Navas +3Neural Decoding Transfer Learning
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3

Feb 27, 2026J/K move · Enter open · S save

MEBM-Phoneme: Multi-scale Enhanced BrainMagic for End-to-End MEG Phoneme Classification

Liang Jinghua, Zhang Zifeng, Li Songyi, Zheng Linze

Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, Beijing, China 100871 · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China 100871

Neural Decoding Convolutional Neural Networks

Open paper AskPDF arXiv

We propose MEBM-Phoneme, a multi-scale enhanced neural decoder for phoneme classification from non-invasive magnetoencephalography (MEG) signals. Built upon the BrainMagic backbone, MEBM-Phoneme integrates a short-term multi-scale convolutional module to augment the native mid-term encoder, with fused representations via depthwise separable convolution for efficient cross-scale integration. A convolutional attention layer dynamically weights temporal dependencies to refine feature aggregation. To address class imbalance and session-specific distributional shifts, we introduce a stacking-based local validation set alongside weighted cross-entropy loss and random temporal augmentation. Comprehensive evaluations on LibriBrain Competition 2025 Track2 demonstrate robust generalization, achieving competitive phoneme decoding accuracy on the validation and official test leaderboard. These results underscore the value of hierarchical temporal modeling and training stabilization for advancing MEG-based speech perception analysis.

Abstract

Explore similar work

MEBM-Phoneme: Multi-scale Enhanced BrainMagic for End-to-End MEG Phoneme Classification

MEBM-Speech: Multi-scale Enhanced BrainMagic for Robust MEG Speech Detection

MEG-to-MEG Transfer Learning and Cross-Task Speech/Silence Detection with Limited Data