Thought-deciphering via hierarchical EEG-to-text alignment

Xiaopeng Mao; Morten Mørup; Xin Xu; Sang Wan Lee; Sadasivan Puthusserypady

cs.LGNov 2, 2026

Thought-deciphering via hierarchical EEG-to-text alignment

Authors: Xiaopeng Mao, Morten Mørup, Xin Xu, Sang Wan Lee, Sadasivan Puthusserypady

Organizations: Department of Health Technology, Technical University of Denmark (DTU), Ørsteds Plads, Kongens Lyngby, 2800, Denmark. · Department of Applied Mathematics and Computer Science, Technical University of Denmark (DTU), Richard Petersens Plads, Kongens Lyngby, 2800, Denmark. · Department of Brain and Cognitive Sciences, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, City, 34141, Republic of Korea. · Center for Neuroscience-inspired AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, City, 34141, Republic of Korea. · Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, City, 34141, Republic of Korea. · Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, City, 34141, Republic of Korea. · Graduate School of Data Science, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, City, 34141, Republic of Korea.

Abstract

Non-invasive brain-to-text interfaces offer alternative communication pathways for people with communication disabilities. However, sentence-level decoding from electroencephalography (EEG) remains challenging. We propose a hierarchical EEG-text alignment framework that retrieves sentences by progressively decoding coarse-grained semantic information and fine-grained sentence representations. We hypothesize that broader semantic context can be more reliably captured from EEG than individual sentences. Accordingly, EEG representations are first aligned with semantic classes, followed by sentence retrieval within the predicted class. Among over 2,000 unique text samples, our approach achieves average top-5 and top-10 retrieval accuracies of 62.6% $\pm$ 1.0% and 83.5% $\pm$ 1.0%, respectively, compared with 50.3% $\pm$ 2.3% and 68.3% $\pm$ 2.0% for direct EEG-sentence alignment. Qualitative analysis further reveals that decoding performance depends strongly on textual context. These findings demonstrate the potential of hierarchical semantic alignment for improving sentence-level EEG-to-text decoding.

Figures & tables

Figure 1: Flow diagram of the study. Two different raw reading EEG datasets are used. A simple and fast preprocessing strategy is applied to both datasets (a). An EEG encoder converts an EEG sample into a 256-dimensional embedding, which is contrastively aligned with the corresponding text embedding at a class level. The encoded embeddings are used for semantic classification. Subsequently, a retrieval model is used to align EEG embeddings with text embeddings at a sentence level (b). For evaluation, the semantic classifier is evaluated using macro F1 scores, while the retriever limits its text search within the predicted class to find the target sentence (c).

Datasets	ChineseEEG - short novel (Little Prince)	ChineseEEG - long novel (Garnett Dream)	Chisco - daily phrases
Macro F1 (Class labels)	90.8 $\pm$ 0.4	100.0 $\pm$ 0.0	5.8 $\pm$ 1.1
Retrieval at 1 (Sentence labels)	20.0 $\pm$ 1.5	6.0 $\pm$ 0.3	0.9 $\pm$ 0.1
Retrieval at 5 (Sentence labels)	62.6 $\pm$ 1.0	25.3 $\pm$ 0.4	3.7 $\pm$ 0.6
Retrieval at 10 (Sentence labels)	83.5 $\pm$ 1.0	42.6 $\pm$ 0.9	5.9 $\pm$ 1.2

Table 1: Results on ChineseEEG and Chisco. The numbers are means (%)

\pm

standard deviations (%). For ChineseEEG, the book chapters are used as classes. For Chisco, the semantic classes from its publication were reused. Firstly, all the EEG samples are classified by the semantic classifier. Secondly, the corresponding text samples are retrieved within their predicted class by the retriever.

Subject	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)	Retrieval at 5 (Sentence labels)	Retrieval at 10 (Sentence labels)
sub-04	92.1 $\pm$ 2.5	19.8 $\pm$ 1.3	64.1 $\pm$ 1.6	85.4 $\pm$ 1.2
sub-05	94.8 $\pm$ 1.4	20.3 $\pm$ 1.2	65.2 $\pm$ 1.1	85.4 $\pm$ 2.1
sub-06	92.3 $\pm$ 2.4	20.7 $\pm$ 3.0	63.6 $\pm$ 1.5	85.2 $\pm$ 0.5
sub-07	96.2 $\pm$ 2.8	21.3 $\pm$ 0.8	64.9 $\pm$ 1.0	87.4 $\pm$ 1.8
sub-08	92.4 $\pm$ 2.1	19.4 $\pm$ 2.4	65.4 $\pm$ 1.0	86.4 $\pm$ 1.0
sub-09	89.5 $\pm$ 3.5	20.8 $\pm$ 1.2	62.3 $\pm$ 0.4	80.2 $\pm$ 2.7

Table 2: Subject-wise results on ChineseEEG, The Little Prince. The numbers are means (%)

\pm

standard deviations (%). The overall best-performing subject is sub-07, while the worst-performing subject is sub-15.

ChineseEEG - short novel (Little Prince)	Mean cosine similarity gap	$p$ -value	ChineseEEG - long novel (Garnett Dream)	Mean cosine similarity gap	$p$ -value
sub-04	$6.5\cdot 10^{-2}$	$4.1\cdot 10^{-29}$	sub-04	$4.5\cdot 10^{-2}$	$7.2\cdot 10^{-101}$
sub-05	$6.8\cdot 10^{-2}$	$4.1\cdot 10^{-29}$	sub-05	$4.4\cdot 10^{-2}$	$9.4\cdot 10^{-100}$
sub-06	$7.2\cdot 10^{-2}$	$6.8\cdot 10^{-28}$	sub-06	$4.5\cdot 10^{-2}$	$5.5\cdot 10^{-98}$
sub-07	$6.4\cdot 10^{-2}$	$4.8\cdot 10^{-26}$	sub-07	$4.5\cdot 10^{-2}$	$7.2\cdot 10^{-101}$
sub-08	$6.6\cdot 10^{-2}$	$4.8\cdot 10^{-26}$	sub-08	$4.5\cdot 10^{-2}$	$6.1\cdot 10^{-101}$
sub-09	$6.8\cdot 10^{-2}$	$1.1\cdot 10^{-14}$	sub-09	$4.5\cdot 10^{-2}$	$1.4\cdot 10^{-98}$

Table 3: Cosine similarity gaps and their significances for ChineseEEG between matched and unmatched EEG-text pairs. The unmatched pairs are constructed using different classes. Wilcoxon signed-rank tests are used with Holm-correction to compute the

p

-values. All

p

-values are highly significant

p\textrm{-value}<<0.05

Chisco - daily phrases	Mean cosine similarity gap	$p$ -value
s1	$2.0\cdot 10^{-2}$	$1.5\cdot 10^{-21}$
s2	$2.1\cdot 10^{-2}$	$1.5\cdot 10^{-21}$
s3	$2.2\cdot 10^{-2}$	$4.3\cdot 10^{-23}$

Table 4: Cosine similarity gaps and their significances for Chisco between matched and unmatched EEG-text pairs. The unmatched pairs are constructed using different classes. Wilcoxon signed-rank tests are used with Holm-correction to compute the

p

-values. All

p

-values are highly significant

p\textrm{-value}<<0.05

Dataset	ChineseEEG - short novel (Little Prince)		Chisco - daily phrases
Metric	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)
Chance level for hierarchical framework	3.4 $\pm$ 0.0	0.4 $\pm$ 0.0	2.5 $\pm$ 0.0	0.2 $\pm$ 0.0
Random model	0.5 $\pm$ 0.5	0.3 $\pm$ 0.3	1.5 $\pm$ 0.6	0.2 $\pm$ 0.1
Base model	68.8 $\pm$ 0.9	16.2 $\pm$ 1.3	1.9 $\pm$ 0.3	0.4 $\pm$ 0.1
+ KNN + Hierarchical alignment	90.8 $\pm$ 0.4	20.0 $\pm$ 1.5	4.4 $\pm$ 0.8	0.5 $\pm$ 0.2
+ KNN + Class-wise alignment	90.8 $\pm$ 0.4	10.0 $\pm$ 0.4	4.4 $\pm$ 0.8	0.4 $\pm$ 0.1

Table 5: Ablation studies on the short novel of ChineseEEG and the entire Chisco dataset. The chance level is defined via random classification using the class labels and thereupon random retrieval using the sentence labels. The random model is an untrained model, while the base model is a trained model without KNN. The parameter

k

of KNN is estimated using the validation set and tested on the test set. The numbers are means (%)

\pm

standard deviations (%).

Panel of Figure 2 — Figure 2: Violing plots comparing congruent and incongruent EEG-text pairs. The comparison has been done on ChineseEEG, The Little Prince, (a), Garnett Dream, (b), and Chisco, (c). For each subject, the congruent pairs are significantly different than the incongruent pairs (marked by three asterisks).

Dataset	No. of subjects	Classes	Montage	Text genre	No. of unique texts	No. of channels	Sampling rate (EEG)
ChineseEEG	10	27 and 19	Geodesic	Novel sentences	2,464 (Little Prince) 9,534 (Garnett Dream)	128	1,000 Hz
Chisco	3	39	10-20 system	Daily phrases	6,566	128	1,000 Hz

Table 6: Overview of the two datasets. ChineseEEG is further divided into two subsets, a short novel (The Little Prince) and a long novel (Garnett Dream).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Confusion matrices of the classification performance on ChineseEEG, The Little Prince (a), Garnett Dream (b), and Chisco daily phrases (c). Both stories show diagonal patterns, while daily phrases are biased toward some classes.

Datasets	ChineseEEG - short novel (Little Prince)	ChineseEEG - long novel (Garnett Dream)	Chisco - daily phrases
Macro F1 (Perfect classifier, class labels)	100.0 + 0.0	100.0 + 0.0	100.0 + 0.0
Retrieval at 1 (Sentence labels)	21.9 + 1.1	6.0 + 0.3	12.2 + 0.3
Retrieval at 5 (Sentence labels)	67.2 + 1.1	25.3 + 0.4	47.6 + 0.6
Retrieval at 10 (Sentence labels)	88.8 + 0.9	42.6 + 0.9	75.4 + 1.5

Appendix

Table 7: Retrieval evaluation assuming perfect classification. ChineseEEG, The Little Prince achieves the best overall performance, while Chisco outperforms Garnett Dream of ChineseEEG. The numbers are means (%)

\pm

standard deviations (%).

Subject	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)	Retrieval at 5 (Sentence labels)	Retrieval at 10 (Sentence labels)
sub-04	100.0 $\pm$ 0.0	6.2 $\pm$ 0.2	25.5 $\pm$ 0.2	42.4 $\pm$ 0.8
sub-05	100.0 $\pm$ 0.0	6.2 $\pm$ 0.5	25.5 $\pm$ 0.5	43.1 $\pm$ 0.6
sub-06	100.0 $\pm$ 0.0	6.0 $\pm$ 0.3	26.0 $\pm$ 0.6	44.0 $\pm$ 0.3
sub-07	100.0 $\pm$ 0.0	6.3 $\pm$ 0.4	25.3 $\pm$ 0.1	42.5 $\pm$ 0.5
sub-08	100.0 $\pm$ 0.0	6.1 $\pm$ 0.3	25.9 $\pm$ 0.8	43.1 $\pm$ 1.1
sub-09	99.8 $\pm$ 0.1	5.8 $\pm$ 0.2	25.2 $\pm$ 0.7	42.6 $\pm$ 0.7

Appendix

Table 8: Subject-wise results for ChineseEEG long novel, Garnett Dream. The numbers are means (%)

\pm

standard deviations (%). Slightly superior performance is seen on sub-06.

Subject	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)	Retrieval at 5 (Sentence labels)	Retrieval at 10 (Sentence labels)
s1	6.2 $\pm$ 1.4	0.9 $\pm$ 0.2	3.8 $\pm$ 0.7	6.5 $\pm$ 1.7
s2	5.4 $\pm$ 0.9	1.1 $\pm$ 0.2	4.0 $\pm$ 0.7	6.2 $\pm$ 1.3
s3	5.3 $\pm$ 0.4	0.7 $\pm$ 0.3	3.3 $\pm$ 0.3	5.2 $\pm$ 1.0

Appendix

Table 9: Subject-wise results for Chisco. The numbers are means (%)

\pm

standard deviations (%).

Dataset	ChineseEEG - short novel (Little Prince)				Chisco - daily phrases
Metric	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)	Retrieval at 5 (Sentence labels)	Retrieval at 10 (Sentence labels)	Macro F1 (Class labels)	Retrieval at 1 (Sentence labels)	Retrieval at 5 (Sentence labels)	Retrieval at 10 (Sentence labels)
Chance level for hierarchical framework	3.4 $\pm$ 0.0	0.4 $\pm$ 0.0	1.8 $\pm$ 0.0	2.9 $\pm$ 0.0	2.5 $\pm$ 0.0	0.2 $\pm$ 0.0	0.9 $\pm$ 0.0	1.7 $\pm$ 0.0
Random model	0.5 $\pm$ 0.5	0.3 $\pm$ 0.3	1.0 $\pm$ 0.2	1.2 $\pm$ 0.4	1.5 $\pm$ 0.6	0.2 $\pm$ 0.1	1.2 $\pm$ 0.5	2.3 $\pm$ 0.4
Base model	68.8 $\pm$ 0.9	16.2 $\pm$ 1.3	51.9 $\pm$ 1.2	68.7 $\pm$ 0.4	1.9 $\pm$ 0.3	0.4 $\pm$ 0.1	1.7 $\pm$ 0.5	2.4 $\pm$ 0.5
+ KNN + Hierarchical alignment	90.8 $\pm$ 0.4	20.0 $\pm$ 1.5	62.6 $\pm$ 1.0	83.5 $\pm$ 1.0	4.4 $\pm$ 0.8	0.5 $\pm$ 0.2	2.4 $\pm$ 0.4	3.8 $\pm$ 0.8
+ KNN + Class-wise alignment	90.8 $\pm$ 0.4	10.0 $\pm$ 0.4	45.6 $\pm$ 0.3	74.8 $\pm$ 0.3	4.4 $\pm$ 0.8	0.4 $\pm$ 0.1	1.9 $\pm$ 0.1	3.4 $\pm$ 0.2

Appendix

Table 10: Ablation studies on the short novel of ChineseEEG and the entire Chisco dataset. The chance level is defined via random classification using the class labels and thereupon random retrieval using the sentence labels. The random model is an untrained model, while the base model is a trained model without KNN. The parameter

k

of KNN is estimated using the validation set and tested on the test set. The numbers are means (%)

\pm

standard deviations (%).

Figure 4: Topographic maps of ChineseEEG, The Little Prince. The correctly classified samples from the test set are used to generate the maps. Each chapter is associated with different brain areas. In addition, the strongest responses are seen in the temporal and parietal areas, which are important for understanding word meanings and relationships.

EEG-to-Text Decoding

See all 16 papers

Joint Text-Audio Alignment for EEG-to-Text Decoding in Chinese Speech Production and Perception
Jul 28, 2026Tian Zheng, Xurong Xie, Xinxin Zhu +2Cross-Modal Representation Learning EEG Decoding
University of the Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · Beijing Language and Culture University
Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding
Mar 23, 2026Xiaoli Yang, Huiyuan Tian, Yurui Li +3EEG-to-Text Decoding EEG Decoding
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
RAG-based EEG-to-Text Translation Using Deep Learning and LLMs
May 17, 2026Enrico Collautti, Xiaopeng Mao, Luca Tonin +2Retrieval-Augmented Generation Brain-Computer Interfaces
IAS-LAB, Department of Information Engineering, University of Padova, 35122 Padova (PD), Italy · Department of Health Technology, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark · Padova Neuroscience Center, 35131 Padova (PD), Italy.

Jul 28, 2026J/K move · Enter open · S save

Joint Text-Audio Alignment for EEG-to-Text Decoding in Chinese Speech Production and Perception

Tian Zheng, Xurong Xie, Xinxin Zhu, Xiaolan Peng+1

University of the Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · Beijing Language and Culture University

Cross-Modal Representation Learning EEG Decoding EEG-to-Text Decoding

Open paper AskPDF arXiv

Decoding speech information directly from scalp electroencephalography (EEG) into text provides a potential non-invasive neural communication pathway for individuals with severe speech and motor impairments. Compared with invasive approaches such as electrocorticography, EEG is safer and more widely deployable, yet substantially more challenging to decode.This challenge is exacerbated for Chinese sentence decoding, which must handle a high-dimensional output space with thousands of characters, severe inter-subject variability, and low signal-to-noise ratios for text alignment.Existing methods commit to a single supervisory axis---either text semantics or audio acoustic features---yet neither can simultaneously satisfy the demands of sentence-level discriminability and fine-grained temporal resolution required for large-vocabulary Chinese decoding. We introduce EEGAlign, a novel parameter-efficient framework that jointly aligns EEG with two axes---text alignment with BGE-M3 text embeddings and audio alignment with wav2vec~2.0 speech features via contrastive learning followed by CTC character-sequence decoding. On ChineseEEG-2 data, EEGAlign yields state-of-the-art closed-set sentence classification performance, reaching up to 82.37% Top-1 accuracy on Reading Aloud EEG and 41.43% on Passive Listening EEG out of 101 candidates. Ablation studies show that the two alignment axes are highly complementary: combining them yields consistently better performance than either alone. To the best of our knowledge, this is the first study on decoding large-vocabulary Chinese sentences from non-invasive EEG during overt speech production, and achieving strong classification performance with relatively large closed-set candidate-sentence setting.

Abstract

EEG-to-Text Decoding

Explore similar work

Joint Text-Audio Alignment for EEG-to-Text Decoding in Chinese Speech Production and Perception

Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding

RAG-based EEG-to-Text Translation Using Deep Learning and LLMs