Imagined speech decoding from EEG offers a promising route toward naturalistic brain-computer interfaces. Traditional closed-set classifiers treat semantic categories as independent labels and do not explicitly exploit relationships among linguistic intents, limiting output flexibility. We adapt a multimodal contrastive learning framework to imagined speech, aligning EEG trials with language representations in the embedding space of a frozen pretrained text encoder. An EEG encoder is trained with a composite objective combining semantically reweighted EEG-sentence alignment and text-prototype-based class discrimination, encouraging semantic structure while preserving separability among 39 intent categories. We evaluate the approach on the Chisco imagined speech dataset across five subjects, reporting Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2). We also investigate maximum EEG-text cosine similarity (MaxCosine) as an intrinsic indicator of decoding reliability and probe semantic structure beyond category labels using five evaluation-time semantic axes and a qualitative Semantic Gallery Probe (SGP) with previously unseen candidate sentences. The contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier while mapping EEG into a frozen language embedding space. MaxCosine identified subsets of held-out trials with higher Top-1 and Top-5 accuracy across subjects. Evaluation-time semantic axes showed mean agreement above the 50% chance reference across all five dimensions, and the SGP qualitatively illustrated direct comparison with unseen candidate sentences. Together, these findings suggest that EEG-language alignment can retain comparable fixed-intent decoding while supporting similarity-based retrieval, intrinsic reliability ranking, and evaluation of broader semantic structure.
Figures & tables
Fig. 1: Semantic classes and their distribution for a representative participant (Subject 1), which is almost consistent across all subjects.
Fig. 2: Overview of the proposed Semantic Alignment Framework.
Model
Dim.
Params
BAAI/bge-small-zh-v1.5 [ 30 ]
512
≈ 24M
BAAI/bge-base-zh-v1.5 [ 30 ]
768
≈ 102M
BAAI/bge-large-zh-v1.5 [ 30 ]
1024
≈ 326M
intfloat/multilingual-e5-base [ 31 ]
768
≈ 278M
intfloat/multilingual-e5-large [ 31 ]
1024
≈ 560M
paraphrase-multilingual-mpnet-base-v2 [ 32 , 33 ]
768
≈ 278M
TABLE I: Text Encoder Candidates
Axis
Left pole
Right pole
A1
Practical/Transactional
Social/Emotional
A2
Physical/Embodied
Informational/Cognitive
A3
Goal-directed/Obligation
Leisure/Enjoyment
A4
Mobility/Outside-world
Home/Personal-life
A5
Individual/Self-related
Interpersonal/Social
TABLE II: Semantic axes used in the evaluation-time generalization analysis.
Fig. 3: Empirical comparison of different Chinese language embedding models on the Chisco dataset. Bars represent classification Top-1 accuracy of probes (LR, MLP, XGBoost) trained on the text embeddings, and a zero-shot baseline computed without supervision. Error bars indicate ± std across folds.
Fig. 4: Classification performance of the task-optimized closed-set classifier and the proposed contrastive EEG–language framework across Subjects 01–05. Results are reported as the mean ± std across the five held-out cross-validation folds.
Fig. 5: Validation-to-test transfer of MaxCosine thresholds for Subject 01. Fold-specific thresholds corresponding to the target coverage levels were determined exclusively from the validation-score distribution and subsequently applied unchanged to the corresponding held-out test fold. Points and error bars represent the mean ± std across the five folds.
Fig. 6: Top-1 and Top-5 accuracy among the held-out test trials retained by validation-derived MaxCosine thresholds for Subject 01. The horizontal axis reports the achieved held-out test coverage. Points and error bars represent the mean ± standard deviation across the five folds.
Fig. 7: Top-1 accuracy across five non-overlapping, validation-referenced MaxCosine quintiles for Subject 01. Each held-out test score is expressed relative to its fold-specific validation-score distribution and assigned to one of five quintiles, from the lowest (Q1) to the highest (Q5). Points and error bars represent the mean ± standard deviation across the five folds.
Validation-referenced percentile
Held-out test example
P100 target
“I am interested in the field of marketing.” True category: Job Hunting and Career Development Top-5 predictions: Job Hunting and Career Development; Work; Learning; Personal Information; Education
P90 target
“Deposits, Withdrawals and Checking Accounts.” True category: Finance Top-5 predictions: Finance; Prices and Costs; Personal Information; Inquiries and Personal Matters; Providing and Requesting Assistance
P80 target
“Books borrowed from the library can be renewed.” True category: Internet and Information Technology Top-5 predictions: Education; Learning; Products and Quality Assurance; Food and Dining; Welcoming and Thanking
TABLE III: Illustrative held-out Top-5 decoding examples across validation-referenced MaxCosine percentiles for Subject 01. Percentiles express each test score relative to its fold-specific validation-score distribution. A higher percentile therefore indicates a higher relative MaxCosine score, not a calibrated probability of correctness.
Fig. 8: Fold-specific ROC curves for discrimination of correct and incorrect Top-1 predictions using raw MaxCosine for Subject 01.
Subject
AUROC mean ± std
Coverage at 20%
Top-1 accuracy 100% → 20%
Top-5 accuracy 100% → 20%
S1
0.64 ± 0.027
20.4
8.3 → 17.2
29.2 → 48.6
S2
0.61 ± 0.027
19.3
10 → 19.2
32.7 → 50.5
S3
0.60 ± 0.020
19.7
9 → 15.6
30.1 → 42.9
S4
0.62 ± 0.027
20
9.6 → 18.3
31.1 → 46.2
S5
0.61 ± 0.058
19.8
9.1 → 16.9
30.9 → 45.9
TABLE IV: Across-subject summary of the MaxCosine reliability analysis. AUROC is reported as the mean ± standard deviation across the five folds. “Coverage at 20%” denotes the mean achieved held-out test coverage after applying the fold-specific threshold targeting 20% validation coverage. The Top-1 and Top-5 columns report mean test accuracy at approximately full coverage and at the 20% validation-target operating point, respectively. Coverage and accuracy values are expressed as percentages.
Fig. 9: Balanced accuracy measures agreement between the semantic-axis assignment derived from the held-out EEG embedding and that derived from the corresponding sentence embedding. Bars show the mean across the five subjects and error bars indicate the standard deviation across subjects. The dashed line denotes the 50% chance level.
Fig. 10: SGP example for Subject 1. A frozen gallery of held-out sentence embeddings is projected to 2D via PCA for visualization. Each point corresponds to a gallery sentence and is colored by its cosine similarity to the EEG trial.
Imagined speech decoding from electroencephalography (EEG) has gained increasing attention as a potential communication pathway for individuals with severe motor impairments, yet reported performance often relies on evaluation protocols that do not clearly reflect cross-subject generalization. This study presents a transparent baseline investigation of a multi-class imagined speech EEG dataset under a strictly subject-independent evaluation framework. Two preprocessing and feature extraction pipelines were compared: a time-domain statistical feature approach and a frequency-domain spectral bandpower approach, evaluated using subject-wise cross-validation and trial-level majority voting with a random forest classifier. The spectral pipeline achieved a significantly higher mean trial-wise accuracy than the statistical pipeline (49.03 ± 4.18% vs. 37.97 ± 3.79%) for coarse-level classification across subjects. Forward feature selection further indicated that a limited subset of frequency bands captured most of the discriminative information. Overall, this work provides a strong basis for future brain-computer interface studies targeting improved cross-subject generalization in EEG-based imagined speech decoding.
Frederik Møllskov Trier, Xiaopeng Mao, Sadasivan Puthusserypady
Department of Health Technology Technical University of Denmark Ørsteds Plads, 2800 Kgs. Lyngby, Denmark
Decoding speech information directly from scalp electroencephalography (EEG) into text provides a potential non-invasive neural communication pathway for individuals with severe speech and motor impairments. Compared with invasive approaches such as electrocorticography, EEG is safer and more widely deployable, yet substantially more challenging to decode.This challenge is exacerbated for Chinese sentence decoding, which must handle a high-dimensional output space with thousands of characters, severe inter-subject variability, and low signal-to-noise ratios for text alignment.Existing methods commit to a single supervisory axis---either text semantics or audio acoustic features---yet neither can simultaneously satisfy the demands of sentence-level discriminability and fine-grained temporal resolution required for large-vocabulary Chinese decoding. We introduce EEGAlign, a novel parameter-efficient framework that jointly aligns EEG with two axes---text alignment with BGE-M3 text embeddings and audio alignment with wav2vec~2.0 speech features via contrastive learning followed by CTC character-sequence decoding. On ChineseEEG-2 data, EEGAlign yields state-of-the-art closed-set sentence classification performance, reaching up to 82.37% Top-1 accuracy on Reading Aloud EEG and 41.43% on Passive Listening EEG out of 101 candidates. Ablation studies show that the two alignment axes are highly complementary: combining them yields consistently better performance than either alone. To the best of our knowledge, this is the first study on decoding large-vocabulary Chinese sentences from non-invasive EEG during overt speech production, and achieving strong classification performance with relatively large closed-set candidate-sentence setting.
Tian Zheng, Xurong Xie, Xinxin Zhu +2
University of the Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · Beijing Language and Culture University
Objective. A decoder of imagined speech that works for a new user without calibration would make a non-invasive speech interface practical. We tested whether imagined-speech content recorded with scalp electroencephalography (EEG) transfers across people, and examined why accuracy within a single person remains low. Approach. We analyzed three public corpora, ds003626, KaraOne and Track 3 of the 2020 International Brain-Computer Interface Competition, under a protocol that enforces split integrity by assertion and reports exact binomial thresholds, group tests and bootstrap confidence intervals. We trained decoders on each participant and tested them on every other, compared representational geometry across participants without a classifier, applied subject-level alignment, and tested five explanations for the within-subject ceiling. Main results. Across 482 ordered pairs of participants, no pair survived correction for multiple comparisons; mean pair-transfer accuracy lay within 0.006 of chance on every corpus, while every within-subject reference lay above it. Imagery against pre-stimulus rest generalized at about 0.61 after artifact and elapsed-time controls, but its decoder labelled the interval preceding imagery as imagery as often as imagery itself, so the contrast reflects a post-stimulus state, not imagery. Within subject, only the competition corpus decoded above threshold for every participant (0.3147; chance 0.200); syllable count explained little of this, but its frequency profile failed an artifact audit. Significance. In these corpora, imagined-speech content did not transfer between people through any representation we examined. Trial count per class was the only variable that ordered the corpora, which favors larger single-condition datasets over further model development.
Blessing Isoyiza Adeika, Onyema Osuagwu
Morgan State University, Baltimore, Maryland, United States of America