Integrating Language Models into Listened and Imagined Speech Decoding from MEG

Maryam Maghsoudi; Sai Samrat Kankanala; Shihab A. Shamma; Sriram Ganapathy

cs.LGSep 25, 2026

Integrating Language Models into Listened and Imagined Speech Decoding from MEG

Authors: Maryam Maghsoudi, Sai Samrat Kankanala, Shihab A. Shamma, Sriram Ganapathy

Organizations: Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA · LEAP Lab, Electrical Engineering, Indian Institute of Science, Bengaluru, India.

Abstract

Decoding imagined speech is an important goal for brain-computer interfaces but remains challenging due to weak neural responses, low signal-to-noise ratio, and limited imagined-speech datasets. Language models provide strong contextual cues for text prediction, but how much they can help neural decoding and whether their contribution differs for decoding perceived and imagined speech remains unclear. To investigate this, we use a paired listened-imagined MEG dataset and incorporate language-model information at two stages. First, we train a contrastive neural decoder that aligns MEG representations with acoustic and contextual language representations, improving cross-subject word decoding for both listened and imagined speech. Second, at inference, we introduce a neural-constrained beam-search framework that combines neural evidence with language-model next-word probabilities. We find that imagined-speech decoding benefits more from the language model than listened-speech decoding. For Imagined speech, the best-performing balance between neural and language-model evidence shifts toward the language model, and the gain over neural-only decoding is larger. Together, these results suggest that language priors are most useful when neural evidence is weaker, making them particularly valuable for imagined-speech BCIs.

Figures & tables

Figure 1: (A) Training of the contextual contrastive decoder. A shared MEG encoder is trained with two InfoNCE objectives: frame-level alignment to wav2vec 2.0 and word-level alignment to GPT-2 hidden states, combined with weight $\lambda_{e}$ . (B) Neural-constrained beam search at inference. At each position $t$ , the frozen MEG decoder selects the top- $k$ candidate words $\mathcal{V}_{t}$ , and the frozen LLM scores each candidate given each beam’s decoded history. The normalized MEG and LLM scores are fused with weight $\alpha$ and added to the beam’s cumulative score, and the $B$ highest-scoring sequences are retained.

Input	Decoder (LOSO)	Top-1 (%)	Top-5 (%)
Listened	Previous [ 12 ]	$1.44\pm 0.16$	$7.45\pm 0.60$
	Proposed	$\mathbf{14.17\pm 0.81}$	$\mathbf{46.96\pm 1.56}$
	Proposed, shuffled	$4.12\pm 0.10$	$18.41\pm 0.18$
Mapped imag.	Previous [ 12 ]	$1.26\pm 0.09$	$6.71\pm 0.21$
	Proposed	$\mathbf{7.05\pm 0.35}$	$\mathbf{31.14\pm 0.97}$
	Proposed, shuffled	$3.93\pm 0.08$	$18.69\pm 0.14$

Table 1: Word decoding performance for listened and mapped imagined MEG (mean

\pm

SEM across held-out subjects). Imagined results are averaged across six imagined-to-listened mappings. Shuffled: decoder trained with shuffled neural–target pairs.

Figure 2: Word accuracy versus fusion weight

\alpha

. (A) Teacher forcing, for listened and mapped imagined MEG. (B,C) Neural-constrained beam search at B=3 and k=10 for listened (B) and mapped imagined (C). Solid curves show MEG-guided candidates, dashed curves the random-candidate null, and the dotted line LLM-only beam search over the full vocabulary. Shaded regions denote SEM across subjects, and markers indicate the

\alpha

that maximizes the mean curve. Y-axes differ across panels. Under beam search, both conditions peak at an intermediate

\alpha

Figure 3: Comparison of the neural–language balance for listened and mapped imagined MEG. Top left and middle: word accuracy across

\alpha

for different candidate-set sizes

k

B=3

, with lighter-colored curves for smaller

k

; shaded regions denote SEM across subjects, markers indicate

\alpha^{*}

for each

k

, and the y-axes differ across panels. Top right: relative gain over MEG-only (

\alpha=0

) at the selected

\alpha^{*}

, with one square per held-out subject, lines linking the same subject across conditions, and bars showing mean

\pm

SEM. Bottom:

\alpha^{*}

for each

k

, with one circle per

k

and dashed lines marking the mean per condition. Mapped imagined responses show a higher optimal

\alpha

and a larger gain than listened responses.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 179 papers

Zero-Shot Imagined Speech Decoding via Imagined-to-Listened MEG Mapping
May 8, 2026Maryam Maghsoudi, Shihab ShammaBrain-Computer Interfaces Neural Decoding
University of Maryland · College Park, MD 20740
Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
Oct 14, 2026Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1EEG Decoding EEG-to-Text Decoding
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy
MEG-to-MEG Transfer Learning and Cross-Task Speech/Silence Detection with Limited Data
Feb 20, 2026Xabier de Zuazo, Vincenzo Verbeni, Eva Navas +3Neural Decoding Transfer Learning
HiTZ Center, University of the Basque Country – UPV/EHU, Spain · Basque Center on Cognition, Brain and Language – BCBL, Spain · Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB), Belgium +3

May 8, 2026J/K move · Enter open · S save

Zero-Shot Imagined Speech Decoding via Imagined-to-Listened MEG Mapping

Maryam Maghsoudi, Shihab Shamma

University of Maryland · College Park, MD 20740

Brain-Computer Interfaces Neural Decoding Neuroimaging Magnetoencephalography Brain-to-Speech Decoding

Open paper AskPDF arXiv

Decoding imagined speech from non-invasive brain recordings is challenging because imagined datasets are scarce and difficult to align temporally across subjects and sessions In this work, we propose a new approach to the decoding of imagined speech that leverages the richer and more reliably labeled recordings during listening to speech. We collected paired listened and imagined MEG recordings to rhythmic melodic and spoken stimuli from trained musicians. Using trained musicians helped improve temporal alignment across conditions. We then developed a three-stage decoding pipeline that revealed consistent and meaningful relationships between neural activity evoked by imagining and listening to the same stimuli. First, we trained six linear and neural models to map imagined MEG responses to listened responses. We evaluated these models against a null baseline from unseen subjects to validate that the predicted-listening responses preserve stimulus-specific information. In the second stage, we trained a contrastive word decoder exclusively on the listened MEG responses, and evaluated it using four embedding strategies including semantic, acoustic, and phonetic representations. In the third stage, we process the imagined MEG responses from held-out subjects through the mapping pipeline to compute the corresponding listening responses that are then decoded by the listened decoder. Using rank-based analysis, we show that the imagined words are decodable significantly above chance. We shall report here the results of a proof-of-concept implementation to decode imagined speech, where all evaluations are performed on held-out subjects. We also demonstrate that performance improves with training data size, suggesting that this approach is scalable and can directly be made applicable to realistic brain-computer interface scenarios.

Abstract

Neural Decoding

Explore similar work

Zero-Shot Imagined Speech Decoding via Imagined-to-Listened MEG Mapping

Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech

MEG-to-MEG Transfer Learning and Cross-Task Speech/Silence Detection with Limited Data