cs.CLMar 23, 2026

Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding

Authors: Xiaoli Yang, Huiyuan Tian, Yurui Li, Jianyu Zhang, Shijian Li, Gang Pan

Organizations: College of Computer Science and Technology, Zhejiang University, Hangzhou, China · State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Decoding natural language from non-invasive electroencephalography (EEG) remains constrained by low signal-to-noise ratio and limited information bandwidth. This raises a central question: can sentence-level language be reliably recovered from such signals? Under realistic information constraints, this direct-recovery assumption may be too strong. We introduce a semantic compression hypothesis: non-invasive EEG may preserve recoverable semantic anchors rather than the full lexical--syntactic form of a sentence. From this perspective, direct sentence reconstruction is overly fine-grained relative to the recoverable information scale of EEG. To address this mismatch, we propose Brain-CLIPLM, a two-stage framework that decomposes EEG-to-text decoding into semantic-anchor recovery and anchor-guided sentence reconstruction. Stage 1 uses contrastive learning to align word-level EEG evidence with a fixed keyword vocabulary and recover ordered semantic anchors. Stage 2 uses a retrieval-grounded large language model with chain-of-thought reasoning prompts to reconstruct sentence meaning from these anchors, following a granularity matching principle that aligns decoding complexity with the recoverable neural information scale. On the combined Zurich Cognitive Language Processing (ZuCo) benchmark, Brain-CLIPLM achieves 67.6% Top-5 and 85.0% Top-25 sentence retrieval accuracy, with the strongest performance at intermediate anchor granularity. Control analyses show that EEG-derived anchors carry sentence-specific information beyond language-model priors. Within the constrained ZuCo sentence pool and fixed keyword-vocabulary settings, these findings suggest that EEG-to-text decoding is better framed as recovering compressed semantic content before anchor-guided sentence reconstruction.

Figures & tables

Figure 1: Conceptual comparison of three EEG-to-text decoding paradigms. (A) Traditional sequence-to-sequence decoding maps EEG signals directly to sentence output. (B) EEG-guided autoregressive decoding uses neural representations to guide token-level candidate generation. (C) Brain-CLIPLM factorizes EEG-to-text decoding into EEG-to-keyword recovery followed by keyword-to-sentence reconstruction, treating recovered keywords as ordered semantic anchors for sentence reconstruction.
Figure 2: Neuropsychological and information-theoretic foundations of the semantic compression hypothesis. (A) Neuropsychological basis: a limited set of content-bearing semantic anchors can support sentence-level meaning. (B) Information-theoretic basis: an intermediate semantic-anchor representation provides a better-matched decoding target for non-invasive EEG than direct full-sentence decoding.
Figure 3: Overview of the Brain-CLIPLM two-stage EEG-to-text decoding framework. (A) Conceptual factorization of the decoding process: EEG signals (X) are first mapped to ordered semantic anchors (K), which are then used to reconstruct sentences (Y). (B) Implementation details of the two stages. Stage 1: EEG-to-keyword decoding uses fixation-aligned word-level EEG segments, preprocessing, feature extraction, a Transformer-based EEG encoder, and contrastive alignment with a frozen keyword embedding space to recover semantic anchors. Stage 2: Keyword-to-sentence reconstruction takes the recovered anchors as input to a large language model (LLaMA-2-7B-Chat), guided by chain-of-thought (CoT) reasoning and retrieval-augmented generation (RAG) from the task corpus, producing fluent, sentence-level reconstructions.
Figure 4: Stage 1 keyword retrieval and target-granularity analyses. (A) Keyword retrieval accuracy across vocabulary sizes of 50, 100, 200, and 500 words. Bars show Top-5 accuracy, and the corresponding Top-10 and Top-25 results for the primary 100-word setting are reported in the text. (B) Target-complexity scan across 3 keywords, 5 keywords, 7 keywords and full-sentence targets. Performance peaks at an intermediate granularity, consistent with the granularity-matching principle.
Figure 5: Sentence recovery performance and qualitative reconstruction examples. (A) Top- kk sentence retrieval accuracy across reconstruction strategies. The main ablation study shows that CoT+RAG (Ours) achieves the highest retrieval under this setup, indicating complementary contributions from reasoning and retrieval grounding. (B) Representative qualitative example comparing Naive generation, RAG only, CoT only and CoT+RAG (Ours) for the same recovered keyword set.

Explore similar work

CardsList

  1. RAG-based EEG-to-Text Translation Using Deep Learning and LLMs

    May 17, 2026Enrico Collautti, Xiaopeng Mao, Luca Tonin +2Retrieval-Augmented GenerationBrain-Computer Interfaces

    IAS-LAB, Department of Information Engineering, University of Padova, 35122 Padova (PD), Italy · Department of Health Technology, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark · Padova Neuroscience Center, 35131 Padova (PD), Italy.

  2. SENSE: Efficient EEG-to-Text via Privacy-Preserving Semantic Retrieval

    Mar 17, 2026Akshaj Murhekar, Christina Liu, Abhijit Mishra +2EEG DecodingBrain-Computer Interfaces

    The University of Texas at Austin School of Information Austin, Texas, USA · Institute of Applied Computer Science Łódź University of Technology Poland

  3. Joint Text-Audio Alignment for EEG-to-Text Decoding in Chinese Speech Production and Perception

    Jul 28, 2026Tian Zheng, Xurong Xie, Xinxin Zhu +2Cross-Modal Representation LearningEEG Decoding

    University of the Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · Beijing Language and Culture University