cs.LGMar 17, 2026

SENSE: Efficient EEG-to-Text via Privacy-Preserving Semantic Retrieval

Authors: Akshaj Murhekar, Christina Liu, Abhijit Mishra, Shounak Roychowdhury, Jacek Gwizdka

Organizations: The University of Texas at Austin School of Information Austin, Texas, USA · Institute of Applied Computer Science Łódź University of Technology Poland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Decoding brain activity into natural language is a major challenge in AI with important applications in assistive communication, neurotechnology, and human-computer interaction. Most existing Brain-Computer Interface (BCI) approaches rely on memory-intensive fine-tuning of Large Language Models (LLMs) or encoder-decoder models on raw EEG signals, resulting in expensive training pipelines, limited accessibility, and potential exposure of sensitive neural data. We introduce SENSE (SEmantic Neural Sparse Extraction), a lightweight and privacy-preserving framework that translates non-invasive electroencephalography (EEG) into text without LLM fine-tuning. SENSE decouples decoding into two stages: on-device semantic retrieval and prompt-based language generation. EEG signals are locally mapped to a discrete textual space to extract a non-sensitive Bag-of-Words (BoW), which conditions an off-the-shelf LLM to synthesize fluent text in a zero-shot manner. The EEG-to-keyword module contains only ~6M parameters and runs fully on-device, ensuring raw neural signals remain local while only abstract semantic cues interact with language models. Evaluated on a 128-channel EEG dataset across six subjects, SENSE matches or surpasses the generative quality of fully fine-tuned baselines such as Thought2Text while substantially reducing computational overhead. By localizing neural decoding and sharing only derived textual cues, SENSE provides a scalable and privacy-aware retrieval-augmented architecture for next-generation BCIs.

Figures & tables

Figure 1. The SENSE privacy-preserving architecture. Raw, high-dimensional EEG signals are processed within a local "Privacy Preserving Boundary" to extract non-sensitive semantic anchors. Only discrete, interpretable keywords are transmitted to remote Large Language Models for final linguistic synthesis. A vertical flowchart illustrating the SENSE privacy-preserving data flow. At the top, a brain icon emits Raw EEG signals. These signals enter a box labeled YOUR LOCAL DEVICE (ON-DEVICE), which contains a dashed Privacy Preserving Boundary. Inside this boundary, an 'EEG ENCODER' generates a Pooled EEG (512-d) vector, which is then processed by the Similarity Refiner neural network. The refiner outputs a Non-sensitive Bag-of-words (BoW) —example keywords shown are backrest, chair, cushion, table. This discrete BoW is the only information that exits the local device to reach a Remote LLM Decoder (Gemini, ChatGPT, LLaMa). At the bottom, the remote LLM produces the final output: "A wooden bench with a cushion and backrest."
Figure 2. SENSE pipeline overview: ChannelNet EEG encoder ( Mishra et al., 2025 ) extracts vector x\mathbf{x} ; a Similarity Refiner maps x ⁣→ ⁣z\mathbf{x}\!\to\!\mathbf{z} . z\mathbf{z} is matrix-multiplied with CLIP-space vocabulary embeddings E\mathbf{E} to produce logits. A top- kk selector ( k=15k=15 ) yields salient tokens forming a Bag-of-Words (BoW). The BoW and predicted object label prompt an off-the-shelf LLM for caption reconstruction. The refiner is trained against ground-truth NN -hot vectors with specialized losses addressing class imbalance. A high-level architectural flowchart of the SENSE pipeline. The diagram is organized into three horizontal stages: Encoding, Refining/Matching, and Decoding. In the Encoding stage, Raw EEG Signals are processed by a frozen EEG ChannelNet Encoder to produce a 512-dimensional vector x. Simultaneously, a Word Corpus is processed by a frozen CLIP Text Encoder to generate a 1210-word vocabulary embedding matrix E. In the Refining stage, vector x enters a Similarity Refiner (indicated as the only trainable component) to produce a refined latent z. This latent z is matrix-multiplied with the transpose of the embedding matrix E to produce predicted logits. A loss calculation compares these logits against an N-hot vector derived from ground-truth caption words to update the refiner. In the Decoding stage, the predicted logits pass through a Top-k Selector funnel, which extracts a Bag-of-Words (e.g., backrest, chair, cushion). This BoW then prompts a frozen LLM Decoder (Gemini, ChatGPT, etc.) to perform the final Caption Reconstruction.
Similarity RefinerDecoderROUGE-N (%)ROUGE-L (%)BLEU-N (%)METEOR (%)BERT ScoreGPT-5
Loss Function Variant N=1 ±\pm 95% CIN=2 ±\pm 95% CI±\pm 95% CIN=1 ±\pm 95% CIN=4 ±\pm 95% CI±\pm 95% CI Flu.Ade.
Naive BaselineChatGPTw\text{ChatGPT}_{\text{w}}25.8 [ ±\pm 0.5]4.8 [ ±\pm 0.3]23.8 [ ±\pm 0.5]21.6 [ ±\pm 0.5]3.8 [ ±\pm 0.2]20.2 [ ±\pm 0.5]0.8884.591.22
ChatGPTwo\text{ChatGPT}_{\text{wo}}22.7 [ ±\pm 0.4]3.1 [ ±\pm 0.2]20.7 [ ±\pm 0.4]20.0 [ ±\pm 0.4]3.1 [ ±\pm 0.1]17.1 [ ±\pm 0.4]0.8774.581.28
Geminiw\text{Gemini}_{\text{w}}25.8 [ ±\pm 0.6]5.2 [ ±\pm 0.4]24.5 [ ±\pm 0.6]18.1 [ ±\pm 0.6]3.6 [ ±\pm 0.2]19.3 [ ±\pm 0.5]0.8884.701.28
Geminiwo\text{Gemini}_{\text{wo}}23.9 [ ±\pm 0.5]3.8 [ ±\pm 0.3]22.3 [ ±\pm 0.4]18.9 [ ±\pm 0.5]3.1 [ ±\pm 0.1]16.9 [ ±\pm 0.4]0.8794.451.07
LLaMaw\text{LLaMa}_{\text{w}}10.5 [ ±\pm 0.5]1.2 [ ±\pm 0.2]9.9 [ ±\pm 0.4]5.3 [ ±\pm 0.4]1.0 [ ±\pm 0.1]10.6 [ ±\pm 0.3]0.8774.241.22
Table 1. Average evaluation metrics and GPT-5 assessment ( Flu. : fluency, Ade. : adequacy) for text generated from Bag-of-Words (BoW) across four LLMs: GPT-4o-mini, Gemini 2.5 Flash Lite, LLaMa-3-8B, and Qwen2.5-7B. Decoders are appended with subscripts indicating configurations evaluated with visual grounding context ( w for WithObj ), solely from neural BoW ( wo for WithoutObj ), or under fully fine-tuned baselines ( all ). For ROUGE-1/2/L, BLEU-1/4, METEOR and BERT Score, metrics report the Mean ±\pm the 95% Bootstrap Margin of Error calculated over 10,000 resamples; for BERT Score, the 95% Bootstrap Margin of Error is uniformly ±0.001\pm 0.001 across all settings. For all metrics, higher values indicate superior performance. The best results within evaluation domains are bolded and second-best are italicized ; matching ties are underlined . A quantitative results table comparing four Similarity Refiner variants—Naive Baseline, Binary Cross Entropy, Contrastive Multi Label, and Focal Loss—across four LLM decoders: GPT-4o-mini, Gemini 2.5 Flash Lite, LLaMa-3-8B, and Qwen2.5-7B. The table reports ROUGE-1/2/L, BLEU-1/4, METEOR, BERT Score, and GPT-5-based Fluency and Adequacy scores. Each decoder is tested with and without the primary object label (WithObj vs. WithoutObj). A final section provides a performance comparison against the fine-tuned Thought2Text baseline. The data demonstrates that Focal Loss paired with Gemini 2.5 Flash Lite (WithObj) achieves the highest performance among the proposed SENSE configurations, matching or exceeding the fine-tuned Thought2Text baseline across most semantic and lexical overlap metrics.
Figure 3. Subject-wise BLEU-1 (%) performance across optimization strategies and LLM decoders. The radar charts demonstrate that premier closed-source models (Gemini and ChatGPT) consistently form the outer performance boundaries across all six subjects, with ChatGPT exhibiting a slight advantage in the Naive Baseline task. Furthermore, the topological consistency of the polygons indicates that our framework generalizes effectively across diverse neural patterns without requiring per-subject fine-tuning. A composite figure consisting of six radar (spider) charts arranged in two rows and three columns, corresponding to Subjects 1 through 6. Each chart plots BLEU-1 scores across four axes representing optimization strategies: Focal Loss (top), Contrastive Multi-Label (right), BCE Loss (bottom), and Naive Baseline (left). The radial axes are scaled from 0.1 to 0.4. Four colored polygons represent different LLM decoders: blue for Gemini, red for ChatGPT, orange for LLaMa-3-8B, and green for Qwen2.5-7B. In every subject chart, the blue and red polygons are the outermost, peaking near 0.4 for Focal Loss and Contrastive ML, indicating superior performance. The green and orange polygons are consistently nested inside, reflecting lower scores. The geometric shape of the performance polygons is highly consistent across all six subjects, demonstrating stable cross-subject generalization. A shared legend at the bottom identifies the four models by color, and the 4 training strategies.
IDImageRef. ObjPred. ObjReference DescriptionThought2TextSample BoWSENSE Description
1 mushroommushroomA large yellow mushroom with a brown stem and a brown cap, surrounded by green foliage.A group of mushrooms growing on a log.yellow, mushroom, white, daisy, flowerA yellow mushroom growing in the green grass.
2 pianopianoA black grand piano in a living room.A grand piano with a stool in front of it.piano, black, room, grand, floorA grand piano on a wooden floor.
3 campingcampingA tent in a mountainous area with trees and fog.A tent set up in a forest with a campfire nearby.tent, field, grassy, lawnA tent is set up in a grassy field for camping.
4 pumpkinpumpkinA carved pumpkin with a face and eyes, sitting on a table.A carved pumpkin with a spooky face on it.carved, pumpkin, face, halloweenA carved pumpkin with a spooky face sits on the porch.
5 flowermushroomA black and white photograph of a single daisy with a white center and a dark brown center.A group of mushrooms growing on a log.mushroom, grow, white, forest, redA white mushroom grows on the forest ground.
6 coffee mugcoffee mugA hand holding a mug with a blue background and a handprint design.A person holding a coffee mug with a A person holding a coffee mug with the words “World’s Best Dad” written on it.mug, coffee, white, black, cupA white coffee mug with a black logo on a table.
Table 2. Qualitative comparison of generated descriptions from SENSE and the Thought2Text baseline. The Sample BoW column shows a representative subset of the Bag-of-Words provided to the LLM during caption generation. Samples 7 and 8 illustrate ablation results where no object label was supplied to the model. ( SENSE captions generated by Gemini 2.5 Flash Lite .) A qualitative comparison table with 8 rows. Each row contains an image, its reference object, predicted object, reference description, a Thought2Text baseline caption, a Sample Bag-of-Words, and the SENSE Description. Row 1-4 show successful retrievals (mushroom, piano, camping, pumpkin) where SENSE captions are highlighted in green and match the semantic content of the images. Row 5-8 show error cases (flower misidentified as mushroom, coffee mug, guitar misidentified as train, and piano misidentified as ball) highlighted in red. In the error cases, the SENSE description remains fluent but reflects the incorrect object label provided by the refiner, illustrating the system's reliance on accurate semantic cues.

Explore similar work

CardsList
Mar 23, 2026

Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding

Decoding natural language from non-invasive electroencephalography (EEG) remains constrained by low signal-to-noise ratio and limited information bandwidth. This raises a central question: can sentence-level language be reliably recovered from such signals? Under realistic information constraints, this direct-recovery assumption may be too strong. We introduce a semantic compression hypothesis: non-invasive EEG may preserve recoverable semantic anchors rather than the full lexical--syntactic form of a sentence. From this perspective, direct sentence reconstruction is overly fine-grained relative to the recoverable information scale of EEG. To address this mismatch, we propose Brain-CLIPLM, a two-stage framework that decomposes EEG-to-text decoding into semantic-anchor recovery and anchor-guided sentence reconstruction. Stage 1 uses contrastive learning to align word-level EEG evidence with a fixed keyword vocabulary and recover ordered semantic anchors. Stage 2 uses a retrieval-grounded large language model with chain-of-thought reasoning prompts to reconstruct sentence meaning from these anchors, following a granularity matching principle that aligns decoding complexity with the recoverable neural information scale. On the combined Zurich Cognitive Language Processing (ZuCo) benchmark, Brain-CLIPLM achieves 67.6% Top-5 and 85.0% Top-25 sentence retrieval accuracy, with the strongest performance at intermediate anchor granularity. Control analyses show that EEG-derived anchors carry sentence-specific information beyond language-model priors. Within the constrained ZuCo sentence pool and fixed keyword-vocabulary settings, these findings suggest that EEG-to-text decoding is better framed as recovering compressed semantic content before anchor-guided sentence reconstruction.

Xiaoli Yang, Huiyuan Tian, Yurui Li +3

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China

EEG Decoding
AskPDF
May 17, 2026

RAG-based EEG-to-Text Translation Using Deep Learning and LLMs

The decoding of linguistic information from electroencephalography (EEG) signals remains an extremely challenging problem in brain-computer interface (BCI) research. In particular, sentence-level decoding from EEG is difficult due to the low signal-to-noise ratio of these recordings. Previous studies tackling this problem have typically failed to surpass random baseline performance unless teacher forcing is used during the inference phase. In this work, we propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline that combines an EEG encoder aligned with semantic sentence embeddings, a vector retrieval stage, and a large language model (LLM) to refine retrieved sentences into coherent output. Experiments are conducted on the Zurich Cognitive Language Processing Corpus (ZuCo) dataset, which contains single-trial EEG recordings collected during silent reading. To evaluate whether the system extracts meaningful information from these EEG signals, the results are compared with a random baseline. In nine subjects, the proposed pipeline outperforms the random baseline, achieving a mean cosine similarity of 0.181 +- 0.022 compared to 0.139 +- 0.029 for the baseline, corresponding to a relative improvement of 30.45%. Statistical analysis further confirms that this improvement is significant, following a strict evaluation workflow where inference is performed without access to ground-truth labels.

Enrico Collautti, Xiaopeng Mao, Luca Tonin +2

IAS-LAB, Department of Information Engineering, University of Padova, 35122 Padova (PD), Italy · Department of Health Technology, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark · Padova Neuroscience Center, 35131 Padova (PD), Italy.

Retrieval-Augmented GenerationBrain-Computer InterfacesEEG Decoding
AskPDF
May 27, 2026

SYNAPSE: Neuro-Symbolic Visual Thought-to-Text Decoding via Topological Semantic Denoising

Recent advances in large language models have accelerated open-vocabulary EEG-to-imagined-text decoding, where non-invasive neural activity recorded during visual perception is translated into coherent natural language descriptions of viewed stimuli. However, existing systems remain highly vulnerable to biological noise, where corrupted neural projections induce hallucinated or semantically unstable generation in frozen language models. We introduce SYNAPSE (Symbolic Neural Alignment for Precise Semantic Extraction), a lightweight neuro-symbolic framework that stabilizes neural text generation through inference-time symbolic regularization. By purifying EEG-derived semantic candidates using commonsense graph structure and latent exemplars, SYNAPSE improves semantic stability without end-to-end LLM fine-tuning. Experiments across popular EEG decoding benchmarks and multiple frozen LLM backends demonstrate consistent gains over unconstrained prompting baselines, robustness under object-label ablation, and performance commensurate with substantially more resource-intensive fine-tuned systems, while preserving biometric privacy by localizing raw EEG processing entirely within the encoder stack.

Akshaj Murhekar, Abhijit Mishra

School of Information University of Texas at Austin

Language Model DecodingEEG DecodingEEG-Based Visual DecodingNeuro-Symbolic Reasoning
AskPDF