Integrating Language Models into Listened and Imagined Speech Decoding from MEG
Authors: Maryam Maghsoudi, Sai Samrat Kankanala, Shihab A. Shamma, Sriram Ganapathy
Organizations: Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA · LEAP Lab, Electrical Engineering, Indian Institute of Science, Bengaluru, India.
Decoding imagined speech is an important goal for brain-computer interfaces but remains challenging due to weak neural responses, low signal-to-noise ratio, and limited imagined-speech datasets. Language models provide strong contextual cues for text prediction, but how much they can help neural decoding and whether their contribution differs for decoding perceived and imagined speech remains unclear. To investigate this, we use a paired listened-imagined MEG dataset and incorporate language-model information at two stages. First, we train a contrastive neural decoder that aligns MEG representations with acoustic and contextual language representations, improving cross-subject word decoding for both listened and imagined speech. Second, at inference, we introduce a neural-constrained beam-search framework that combines neural evidence with language-model next-word probabilities. We find that imagined-speech decoding benefits more from the language model than listened-speech decoding. For Imagined speech, the best-performing balance between neural and language-model evidence shifts toward the language model, and the gain over neural-only decoding is larger. Together, these results suggest that language priors are most useful when neural evidence is weaker, making them particularly valuable for imagined-speech BCIs.
Figures & tables
Figure 1: (A) Training of the contextual contrastive decoder. A shared MEG encoder is trained with two InfoNCE objectives: frame-level alignment to wav2vec 2.0 and word-level alignment to GPT-2 hidden states, combined with weight λe . (B) Neural-constrained beam search at inference. At each position t , the frozen MEG decoder selects the top- k candidate words Vt , and the frozen LLM scores each candidate given each beam’s decoded history. The normalized MEG and LLM scores are fused with weight α and added to the beam’s cumulative score, and the B highest-scoring sequences are retained.
Input
Decoder (LOSO)
Top-1 (%)
Top-5 (%)
Listened
Previous [ 12 ]
1.44±0.16
7.45±0.60
Proposed
14.17±0.81
46.96±1.56
Proposed, shuffled
4.12±0.10
18.41±0.18
Mapped imag.
Previous [ 12 ]
1.26±0.09
6.71±0.21
Proposed
7.05±0.35
31.14±0.97
Proposed, shuffled
3.93±0.08
18.69±0.14
Table 1: Word decoding performance for listened and mapped imagined MEG (mean ± SEM across held-out subjects). Imagined results are averaged across six imagined-to-listened mappings. Shuffled: decoder trained with shuffled neural–target pairs.
Figure 2: Word accuracy versus fusion weight α . (A) Teacher forcing, for listened and mapped imagined MEG. (B,C) Neural-constrained beam search at B=3 and k=10 for listened (B) and mapped imagined (C). Solid curves show MEG-guided candidates, dashed curves the random-candidate null, and the dotted line LLM-only beam search over the full vocabulary. Shaded regions denote SEM across subjects, and markers indicate the α that maximizes the mean curve. Y-axes differ across panels. Under beam search, both conditions peak at an intermediate α .
Figure 3: Comparison of the neural–language balance for listened and mapped imagined MEG. Top left and middle: word accuracy across α for different candidate-set sizes k at B=3 , with lighter-colored curves for smaller k ; shaded regions denote SEM across subjects, markers indicate α∗ for each k , and the y-axes differ across panels. Top right: relative gain over MEG-only ( α=0 ) at the selected α∗ , with one square per held-out subject, lines linking the same subject across conditions, and bars showing mean ± SEM. Bottom: α∗ for each k , with one circle per k and dashed lines marking the mean per condition. Mapped imagined responses show a higher optimal α and a larger gain than listened responses.
Decoding imagined speech from non-invasive brain recordings is challenging because imagined datasets are scarce and difficult to align temporally across subjects and sessions In this work, we propose a new approach to the decoding of imagined speech that leverages the richer and more reliably labeled recordings during listening to speech. We collected paired listened and imagined MEG recordings to rhythmic melodic and spoken stimuli from trained musicians. Using trained musicians helped improve temporal alignment across conditions. We then developed a three-stage decoding pipeline that revealed consistent and meaningful relationships between neural activity evoked by imagining and listening to the same stimuli. First, we trained six linear and neural models to map imagined MEG responses to listened responses. We evaluated these models against a null baseline from unseen subjects to validate that the predicted-listening responses preserve stimulus-specific information. In the second stage, we trained a contrastive word decoder exclusively on the listened MEG responses, and evaluated it using four embedding strategies including semantic, acoustic, and phonetic representations. In the third stage, we process the imagined MEG responses from held-out subjects through the mapping pipeline to compute the corresponding listening responses that are then decoded by the listened decoder. Using rank-based analysis, we show that the imagined words are decodable significantly above chance. We shall report here the results of a proof-of-concept implementation to decode imagined speech, where all evaluations are performed on held-out subjects. We also demonstrate that performance improves with training data size, suggesting that this approach is scalable and can directly be made applicable to realistic brain-computer interface scenarios.
Imagined speech decoding from EEG offers a promising route toward naturalistic brain-computer interfaces. Traditional closed-set classifiers treat semantic categories as independent labels and do not explicitly exploit relationships among linguistic intents, limiting output flexibility. We adapt a multimodal contrastive learning framework to imagined speech, aligning EEG trials with language representations in the embedding space of a frozen pretrained text encoder. An EEG encoder is trained with a composite objective combining semantically reweighted EEG-sentence alignment and text-prototype-based class discrimination, encouraging semantic structure while preserving separability among 39 intent categories. We evaluate the approach on the Chisco imagined speech dataset across five subjects, reporting Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2). We also investigate maximum EEG-text cosine similarity (MaxCosine) as an intrinsic indicator of decoding reliability and probe semantic structure beyond category labels using five evaluation-time semantic axes and a qualitative Semantic Gallery Probe (SGP) with previously unseen candidate sentences. The contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier while mapping EEG into a frozen language embedding space. MaxCosine identified subsets of held-out trials with higher Top-1 and Top-5 accuracy across subjects. Evaluation-time semantic axes showed mean agreement above the 50% chance reference across all five dimensions, and the SGP qualitatively illustrated direct comparison with unseen candidate sentences. Together, these findings suggest that EEG-language alignment can retain comparable fixed-intent decoding while supporting similarity-based retrieval, intrinsic reliability ranking, and evaluation of broader semantic structure.
Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy
Data-efficient neural decoding is a central challenge for speech brain-computer interfaces. We present the first demonstration of transfer learning and cross-task decoding for MEG-based speech models spanning perception and production. We pre-train a Conformer-based model on 50 hours of single-subject listening data and fine-tune on just 5 minutes per subject across 18 participants. Transfer learning yields consistent improvements, with in-task accuracy gains of 1-4% and larger cross-task gains of up to 5-6%. Not only does pre-training improve performance within each task, but it also enables reliable cross-task decoding between perception and production. Critically, models trained on speech production decode passive listening above chance, confirming that learned representations reflect shared neural processes rather than task-specific motor activity.
Xabier de Zuazo, Vincenzo Verbeni, Eva Navas +3
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3