EEG-Captioner: Grounding EEG-to-Language Generation in Fundamental Signal Properties

Leyuan Jin; Jiahao Mei; Yixuan Yuan; Shimeng Li; Mengyue Wu; Yue Ding

cs.LGOct 22, 2026

EEG-Captioner: Grounding EEG-to-Language Generation in Fundamental Signal Properties

Authors: Leyuan Jin, Jiahao Mei, Yixuan Yuan, Shimeng Li, Mengyue Wu, Yue Ding

Organizations: X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China

Abstract

Electroencephalography (EEG)-to-language models offer a promising approach for translating complex neural signals into human-readable descriptions. However, generating plausible language does not necessarily demonstrate that a model has learned to interpret the underlying EEG signal. A direct way to assess such interpretation is to probe objectively measurable properties of the signal itself. We therefore construct an EEG-language dataset of 160,781 segments annotated with signal-level question answering (QA) pairs, where each question targets one of five property families and requires a quantitative comparison involving spectral, spatial, temporal, or lateral signal properties. Each segment is also paired with a feature-grounded EEG caption. We further propose a Dual-Domain EEG Encoder that jointly models temporal waveforms and spectral-spatial representations, and introduce EEG-Captioner, a unified EEG-to-language framework trained with both signal-level QA and EEG caption supervision. Experiments show that the proposed encoder achieves the highest overall macro-averaged balanced accuracy (Macro-BA) among the compared EEG encoders, while EEG-Captioner achieves a Macro-BA of 88.81% on signal-level QA and improves caption factual consistency from 82.06% to 83.46% over caption-only language adaptation, demonstrating the complementary benefits of signal-level QA and caption supervision.

Figures & tables

Component	Task	Question	Answer Space	Valid Count
Signal-level QA	Dominant Band	Which frequency band has the highest power in a given time interval and spatial region?	Delta, Theta, Alpha, Beta, Gamma	652,345
	Strongest Region	Which brain region has the highest power in a given frequency band and time interval?	Frontal, Central, Parietal, Occipital, Temporal	6,879
	Peak Period	In which time interval—Early, Middle, or Last—does a given EEG feature reach its highest value?	Early, Middle, Last	311,861
	Temporal Pattern	How does a given EEG feature change across the Early, Middle, and Last intervals?	Increasing, Decreasing, Increase-then-Decrease, Decrease-then-Increase, Stable	491,189
	Lateral Dominance	Which hemisphere shows stronger activity in corresponding channels or brain regions?	Left, Right, Approximately Symmetric	784,302
EEG Caption	Free-form Caption	Please describe the EEG signal in one coherent English caption.	In the early interval, the EEG is dominated by ,,,	160,781

Table 1: Composition of the proposed EEG-language dataset. Each EEG segment generates one caption and 25 candidate relational questions across five task categories, with five questions per category. The reported relational counts are the numbers retained after numerical validity and signal-quality filtering.

Figure 1: Illustration of the EEG-Language dataset construction pipeline.

Figure 2: Overview of the proposed Dual-Domain EEG Encoder and EEG-to-LLM framework for relational QA and caption generation; representative outputs are shown on the right.

EEG Encoder	Stage	Dominant	Strongest	Peak	Temporal	Lateral	Macro-BA
		Band(%)	Region(%)	Period(%)	Pattern(%)	Dominance(%)	(%)
CBraMod	Classifier	71.68	72.59	34.13	33.60	40.27	50.45
LaBraM	Classifier	90.56	70.60	89.87	71.68	57.60	76.06
Dual-Domain	Classifier	93.76	91.29	93.87	87.52	82.93	89.87
CBraMod	Qwen	61.76	69.53	36.27	36.00	36.27	47.96
LaBraM	Qwen	89.76	72.23	84.00	68.80	52.53	73.47

Table 2: Comparison of EEG encoders on five relational tasks and EEG-input sanity checks. In the Classifier stage, each EEG encoder is trained jointly with a relational classification head. In the Qwen stage, the trained encoder is frozen, and the projector and LoRA adapters are trained for relational instruction tuning. All values are balanced accuracy (%), and Macro-BA is the unweighted mean across the five task families.

Training	Signal-level QA	Caption Generation
Supervision	Macro-BA(%) $\uparrow$	Factual Consistency(%) $\uparrow$	ROUGE-L $\uparrow$	BLEU-4 $\uparrow$
Caption-only adaptation	47.82	82.06	0.4722	0.3124
Signal-level QA only	86.06	44.10	0.1470	0.0083
Signal-level QA + Caption	88.81	83.46	0.4865	0.3302

Table 3: Comparison of EEG-Captioner training strategies. Factual Consistency evaluates the accuracy of fundamental EEG signal descriptions in generated captions with respect to the corresponding numerical feature summary using GPT-5.6 Sol.

EEG-to-Text Decoding

See all 14 papers

OneEEG: A Unified Signal-Visual Multimodal Language Model with Task-Aware Mixture of Projectors for Multi-Task EEG Understanding
Oct 26, 2026Jiacheng Zhang, Xueyi Zhang, Xin Lu +2Electroencephalography Multimodal Large Language Models
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Language, Intelligence, and Machine Center, Shenzhen Loop Area Institute, Shenzhen, China · School of Computing, National University of Singapore, Singapore +2
Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
Oct 14, 2026Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1EEG Decoding EEG-to-Text Decoding
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy
BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding
Aug 4, 2026Yangxuan Zhou, Sha Zhao, Yuning Chen +4Electroencephalography LLM Evaluation
State Key Laboratory of Brain-machine Intelligence, Zhejiang University · College of Computer Science and Technology, Zhejiang University · MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University

Oct 26, 2026J/K move · Enter open · S save

OneEEG: A Unified Signal-Visual Multimodal Language Model with Task-Aware Mixture of Projectors for Multi-Task EEG Understanding

Jiacheng Zhang, Xueyi Zhang, Xin Lu, Siqi Cai+1

School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Language, Intelligence, and Machine Center, Shenzhen Loop Area Institute, Shenzhen, China · School of Computing, National University of Singapore, Singapore+2

Electroencephalography Multimodal Large Language Models

Open paper AskPDF arXiv

Electroencephalography (EEG) is widely used in clinical assessment, brain-computer interfaces, and cognitive-state analysis, yet EEG modeling remains fragmented across task paradigms. Recent multimodal large language models (MLLMs) provide a promising interface for flexible multimodal reasoning, but their ability to understand EEG inputs remains limited. In this work, we propose OneEEG, a unified EEG-MLLM framework for multi-task EEG understanding. By training on our curated OneEEG-Instruct, a multi-task EEG instruction-tuning dataset, OneEEG jointly models numerical EEG representations and visual EEG representations, and introduces a task-aware Mixture of Projectors (MoP) to align heterogeneous EEG features across different task families with the LLM embedding space. We further establish OneEEG-Bench, a benchmark for evaluating current MLLMs and our method on multi-task EEG understanding. Experiments show that existing MLLMs have limited ability to interpret EEG inputs, while OneEEG achieves stronger performance by integrating numerical and visual EEG representations with task-aware EEG-to-LLM projection. Extensive ablations and analyses further reveal the effects of visual EEG renderings, MoP variants, router behavior, and feature dispersion across EEG tasks.

Abstract

EEG-to-Text Decoding

Explore similar work

OneEEG: A Unified Signal-Visual Multimodal Language Model with Task-Aware Mixture of Projectors for Multi-Task EEG Understanding

Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech

BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding