Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding
Authors: Xiaoli Yang, Huiyuan Tian, Yurui Li, Jianyu Zhang, Shijian Li, Gang Pan
Organizations: College of Computer Science and Technology, Zhejiang University, Hangzhou, China · State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
Decoding natural language from non-invasive electroencephalography (EEG) remains constrained by low signal-to-noise ratio and limited information bandwidth. This raises a central question: can sentence-level language be reliably recovered from such signals? Under realistic information constraints, this direct-recovery assumption may be too strong. We introduce a semantic compression hypothesis: non-invasive EEG may preserve recoverable semantic anchors rather than the full lexical--syntactic form of a sentence. From this perspective, direct sentence reconstruction is overly fine-grained relative to the recoverable information scale of EEG. To address this mismatch, we propose Brain-CLIPLM, a two-stage framework that decomposes EEG-to-text decoding into semantic-anchor recovery and anchor-guided sentence reconstruction. Stage 1 uses contrastive learning to align word-level EEG evidence with a fixed keyword vocabulary and recover ordered semantic anchors. Stage 2 uses a retrieval-grounded large language model with chain-of-thought reasoning prompts to reconstruct sentence meaning from these anchors, following a granularity matching principle that aligns decoding complexity with the recoverable neural information scale. On the combined Zurich Cognitive Language Processing (ZuCo) benchmark, Brain-CLIPLM achieves 67.6% Top-5 and 85.0% Top-25 sentence retrieval accuracy, with the strongest performance at intermediate anchor granularity. Control analyses show that EEG-derived anchors carry sentence-specific information beyond language-model priors. Within the constrained ZuCo sentence pool and fixed keyword-vocabulary settings, these findings suggest that EEG-to-text decoding is better framed as recovering compressed semantic content before anchor-guided sentence reconstruction.
Figures & tables
Figure 1: Conceptual comparison of three EEG-to-text decoding paradigms. (A) Traditional sequence-to-sequence decoding maps EEG signals directly to sentence output. (B) EEG-guided autoregressive decoding uses neural representations to guide token-level candidate generation. (C) Brain-CLIPLM factorizes EEG-to-text decoding into EEG-to-keyword recovery followed by keyword-to-sentence reconstruction, treating recovered keywords as ordered semantic anchors for sentence reconstruction.
Figure 2: Neuropsychological and information-theoretic foundations of the semantic compression hypothesis. (A) Neuropsychological basis: a limited set of content-bearing semantic anchors can support sentence-level meaning. (B) Information-theoretic basis: an intermediate semantic-anchor representation provides a better-matched decoding target for non-invasive EEG than direct full-sentence decoding.
Figure 3: Overview of the Brain-CLIPLM two-stage EEG-to-text decoding framework. (A) Conceptual factorization of the decoding process: EEG signals (X) are first mapped to ordered semantic anchors (K), which are then used to reconstruct sentences (Y). (B) Implementation details of the two stages. Stage 1: EEG-to-keyword decoding uses fixation-aligned word-level EEG segments, preprocessing, feature extraction, a Transformer-based EEG encoder, and contrastive alignment with a frozen keyword embedding space to recover semantic anchors. Stage 2: Keyword-to-sentence reconstruction takes the recovered anchors as input to a large language model (LLaMA-2-7B-Chat), guided by chain-of-thought (CoT) reasoning and retrieval-augmented generation (RAG) from the task corpus, producing fluent, sentence-level reconstructions.
Figure 4: Stage 1 keyword retrieval and target-granularity analyses. (A) Keyword retrieval accuracy across vocabulary sizes of 50, 100, 200, and 500 words. Bars show Top-5 accuracy, and the corresponding Top-10 and Top-25 results for the primary 100-word setting are reported in the text. (B) Target-complexity scan across 3 keywords, 5 keywords, 7 keywords and full-sentence targets. Performance peaks at an intermediate granularity, consistent with the granularity-matching principle.
Figure 5: Sentence recovery performance and qualitative reconstruction examples. (A) Top- k sentence retrieval accuracy across reconstruction strategies. The main ablation study shows that CoT+RAG (Ours) achieves the highest retrieval under this setup, indicating complementary contributions from reasoning and retrieval grounding. (B) Representative qualitative example comparing Naive generation, RAG only, CoT only and CoT+RAG (Ours) for the same recovered keyword set.
The decoding of linguistic information from electroencephalography (EEG) signals remains an extremely challenging problem in brain-computer interface (BCI) research. In particular, sentence-level decoding from EEG is difficult due to the low signal-to-noise ratio of these recordings. Previous studies tackling this problem have typically failed to surpass random baseline performance unless teacher forcing is used during the inference phase. In this work, we propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline that combines an EEG encoder aligned with semantic sentence embeddings, a vector retrieval stage, and a large language model (LLM) to refine retrieved sentences into coherent output. Experiments are conducted on the Zurich Cognitive Language Processing Corpus (ZuCo) dataset, which contains single-trial EEG recordings collected during silent reading. To evaluate whether the system extracts meaningful information from these EEG signals, the results are compared with a random baseline. In nine subjects, the proposed pipeline outperforms the random baseline, achieving a mean cosine similarity of 0.181 +- 0.022 compared to 0.139 +- 0.029 for the baseline, corresponding to a relative improvement of 30.45%. Statistical analysis further confirms that this improvement is significant, following a strict evaluation workflow where inference is performed without access to ground-truth labels.
Enrico Collautti, Xiaopeng Mao, Luca Tonin +2
IAS-LAB, Department of Information Engineering, University of Padova, 35122 Padova (PD), Italy · Department of Health Technology, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark · Padova Neuroscience Center, 35131 Padova (PD), Italy.
Decoding brain activity into natural language is a major challenge in AI with important applications in assistive communication, neurotechnology, and human-computer interaction. Most existing Brain-Computer Interface (BCI) approaches rely on memory-intensive fine-tuning of Large Language Models (LLMs) or encoder-decoder models on raw EEG signals, resulting in expensive training pipelines, limited accessibility, and potential exposure of sensitive neural data. We introduce SENSE (SEmantic Neural Sparse Extraction), a lightweight and privacy-preserving framework that translates non-invasive electroencephalography (EEG) into text without LLM fine-tuning. SENSE decouples decoding into two stages: on-device semantic retrieval and prompt-based language generation. EEG signals are locally mapped to a discrete textual space to extract a non-sensitive Bag-of-Words (BoW), which conditions an off-the-shelf LLM to synthesize fluent text in a zero-shot manner. The EEG-to-keyword module contains only ~6M parameters and runs fully on-device, ensuring raw neural signals remain local while only abstract semantic cues interact with language models. Evaluated on a 128-channel EEG dataset across six subjects, SENSE matches or surpasses the generative quality of fully fine-tuned baselines such as Thought2Text while substantially reducing computational overhead. By localizing neural decoding and sharing only derived textual cues, SENSE provides a scalable and privacy-aware retrieval-augmented architecture for next-generation BCIs.
Akshaj Murhekar, Christina Liu, Abhijit Mishra +2
The University of Texas at Austin School of Information Austin, Texas, USA · Institute of Applied Computer Science Łódź University of Technology Poland
Decoding speech information directly from scalp electroencephalography (EEG) into text provides a potential non-invasive neural communication pathway for individuals with severe speech and motor impairments. Compared with invasive approaches such as electrocorticography, EEG is safer and more widely deployable, yet substantially more challenging to decode.This challenge is exacerbated for Chinese sentence decoding, which must handle a high-dimensional output space with thousands of characters, severe inter-subject variability, and low signal-to-noise ratios for text alignment.Existing methods commit to a single supervisory axis---either text semantics or audio acoustic features---yet neither can simultaneously satisfy the demands of sentence-level discriminability and fine-grained temporal resolution required for large-vocabulary Chinese decoding. We introduce EEGAlign, a novel parameter-efficient framework that jointly aligns EEG with two axes---text alignment with BGE-M3 text embeddings and audio alignment with wav2vec~2.0 speech features via contrastive learning followed by CTC character-sequence decoding. On ChineseEEG-2 data, EEGAlign yields state-of-the-art closed-set sentence classification performance, reaching up to 82.37% Top-1 accuracy on Reading Aloud EEG and 41.43% on Passive Listening EEG out of 101 candidates. Ablation studies show that the two alignment axes are highly complementary: combining them yields consistently better performance than either alone. To the best of our knowledge, this is the first study on decoding large-vocabulary Chinese sentences from non-invasive EEG during overt speech production, and achieving strong classification performance with relatively large closed-set candidate-sentence setting.
Tian Zheng, Xurong Xie, Xinxin Zhu +2
University of the Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · Beijing Language and Culture University