MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation

Xiangyu Liu; Zeting Yan; Zhitong Yin; Boyang Li; Xi Zhang

cs.LGAug 29, 2026

MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation

Authors: Xiangyu Liu, Zeting Yan, Zhitong Yin, Boyang Li, Xi Zhang

Organizations: Beijing Normal-Hong Kong Baptist University · The University of Hong Kong · Peking University

Abstract

Translating electroencephalography (EEG) into functional magnetic resonance imaging (fMRI) is important for medical neuroimaging, clinical brain-state monitoring, and multimodal neural decoding, because it aims to infer spatially organized hemodynamic activity from fast and accessible electrophysiological recordings. Existing EEG-to-fMRI studies mainly pursue stronger decoders, but the problem is also constrained by a representation-interface mismatch: fMRI responses are delayed, temporally integrated, and spatially distributed, whereas generic EEG encodings often entangle temporal lag, channel identity, and frequency-band structure. We propose Multi-band EEG Latent-state Tokenization (MEL), a coordinate-preserving EEG representation framework that anchors each target fMRI response to its preceding EEG history and organizes it into lag-channel-frequency neural-state tokens. By explicitly capturing hemodynamic latency and spectral-spatial dynamics, MEL aligns fMRI-pertinent EEG representations with capacity-controlled readouts without depending entirely on model scaling. Experiments on VU EEG-fMRI benchmarks and external Oddball data show that MEL improves prediction over strong NeuroBOLT baselines. Ablations and controls further indicate that the gains come from structured EEG representation rather than leakage, shortcut statistics, or decoder capacity.

Figures & tables

Figure 1: Representation-centric view of ROI-level EEG-to-fMRI prediction. Many EEG encodings model lag, channel, and frequency information implicitly, whereas MEL exposes these axes as explicit target-relative coordinates before decoding.

Figure 2: Overall framework of MEL. The method anchors each fMRI target to its preceding EEG history, constructs lag-channel-frequency tokens through train-only calibrated bandpower extraction, and decodes the resulting representation into ROI responses using capacity-controlled readouts.

Setting	Model	Primary Sensory		High-level Cognitive		Subcortical		Global Signal	Avg. R $\uparrow$
		Cuneus	Heschl’s Gyrus	Middle Frontal	Precuneus Anterior	Putamen	Thalamus
Intra-scan	BIOT ( Yang et al. 2023 )	0.531 $\pm$ 0.223	0.518 $\pm$ 0.207	0.490 $\pm$ 0.162	0.459 $\pm$ 0.110	0.410 $\pm$ 0.205	0.411 $\pm$ 0.231	0.493 $\pm$ 0.133	0.473
	LaBraM ( Jiang et al. 2024 )	0.540 $\pm$ 0.176	0.519 $\pm$ 0.197	0.493 $\pm$ 0.153	0.490 $\pm$ 0.176	0.411 $\pm$ 0.179	0.449 $\pm$ 0.177	0.487 $\pm$ 0.167	0.484
	BEIRA ( Kovalev et al. 2022 )	0.357 $\pm$ 0.241	0.396 $\pm$ 0.240	0.294 $\pm$ 0.228	0.320 $\pm$ 0.220	0.234 $\pm$ 0.194	0.328 $\pm$ 0.197	0.456 $\pm$ 0.240	0.341
	SIREN ( Li et al. 2024a )	0.460 $\pm$ 0.228	0.515 $\pm$ 0.207	0.376 $\pm$ 0.169	0.457 $\pm$ 0.204	0.324 $\pm$ 0.183	0.398 $\pm$ 0.194	0.583 $\pm$ 0.170	0.445
	NeuroBOLT ( Li et al. 2024b )	0.588 $\pm$ 0.166	0.566 $\pm$ 0.183	0.502 $\pm$ 0.168	0.559 $\pm$ 0.141	0.437 $\pm$ 0.184	0.480 $\pm$ 0.213	0.587 $\pm$ 0.162	0.531

Table 1: EEG-to-fMRI prediction under intra-scan and inter-subject evaluation. Entries report mean

\pm

standard deviation, and Avg.

R

is averaged over the seven targets. Red and blue denote the best and second-best results within each setting. All neural and traditional baselines are reproduced under the same preprocessing, split, train-only normalization, and validation-based model-selection pipeline rather than taken from incompatible reporting protocols. BP denotes bandpower, and inter-subject results follow strict leave-one-subject-out evaluation over 22 subjects and 29 scans.

Model	Primary Sensory				High-level Cognitive				Subcortical				–		Avg. R $\uparrow$
	Cuneus		Heschl’s Gyrus		Middle Frontal		Precuneus Anterior		Putamen		Thalamus		Global Signal
	MSE $\downarrow$	R $\uparrow$	MSE $\downarrow$	R $\uparrow$	MSE $\downarrow$	R $\uparrow$	MSE $\downarrow$	R $\uparrow$	MSE $\downarrow$	R $\uparrow$	MSE $\downarrow$	R $\uparrow$	MSE $\downarrow$	R $\uparrow$
$T_{2}$	0.424	0.164	0.353	0.165	0.324	0.395	0.353	0.358	0.446	0.242	0.351	0.333	0.356	0.383	0.292
$T_{1}$	0.325	0.221	0.299	0.182	0.314	0.447	0.302	0.390	0.310	0.252	0.323	0.373	0.281	0.410	0.325
$M+B_{3}$	0.315	0.383	0.291	0.305	0.314	0.451	0.322	0.466	0.341	0.343	0.345	0.410	0.302	0.489	0.407
$M+B_{5}$	0.259	0.434	0.263	0.333	0.265	0.516	0.248	0.502	0.320	0.373	0.277	0.455	0.247	0.551	0.452

Table 2: Ablation of MEL components.

M

denotes the lag-channel-band coordinate system.

B_{3}

and

B_{5}

use three and five frequency bands, respectively.

T_{1}

and

T_{2}

denote HRF-weighted and time-mean aggregation. All variants share the same Ridge readout.

Figure 3: External transfer at different temporal depths. Oddball performance improves as a longer ordered EEG history is retained, whereas NODDI remains near zero under the available alignment cache.

Control	Broken structure	Avg. $R\!\uparrow$	$\boldsymbol{\Delta R}$
Normal MEL	None	0.4521	–
All-split $z$ -score	Normalization split	0.4521	- 0.0000
Channel permutation	Channel identity	0.3945	-0.0576
Band reversal	Spectral semantics	0.2355	-0.2166
Random lag	Sample alignment	0.1977	-0.2544
Lag reversal	Hemodynamic order	0.1022	-0.3499

Table 3: Representation-validity controls. Each experiment keeps the Ridge readout fixed while disrupting one component of MEL.

Figure 4: Temporal-design ablation. Bars report test Avg.

R

, while black curves report validation Avg.

R

Figure 5: MEL coordinate-sensitivity summary across frequency bands, temporal lags, and ROI groups. Values combine normalized marginal band-ablation, lag-ablation, and ROI-gain diagnostics within the displayed map.

Figure 6: Examples of fMRI ROI reconstruction on unseen scans. Dashed gray curves denote ground-truth responses, and colored curves denote MEL predictions.

Explore similar work

Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG
May 19, 2026Aditya R. Vaidya, Richard J. Antonello, Alexander G. HuthCross-Modal Learning Transfer Learning
Department of Computer Science The University of Texas at Austin, USA · Zuckerman Mind Brain Behavior Institute Columbia University, USA · Departments of Neuroscience and Statistics University of California, Berkeley, USA
CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding
Jul 10, 2026Gabriel Mahuas, Victoria Shevchenko, Ugo Tanielian +2Contrastive Learning Hybrid CNN-Transformer Architectures
Sigma Nova Paris, France · Goethe University Frankfurt Frankfurt, Germany
Continuous-Latent Predictive Modeling with Semantic Alignment for EEG-Language Foundation Models
Aug 12, 2026Myeong-Ju Cho, Hye-Bin Shin, Seo-Hyun Lee +1EEG Decoding Predictive Representation Learning
Department of Brain and Cognitive Engineering, Korea University, Seoul, Republic of Korea · Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea

May 19, 2026J/K move · Enter open · S save

Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG

Aditya R. Vaidya, Richard J. Antonello, Alexander G. Huth

Department of Computer Science The University of Texas at Austin, USA · Zuckerman Mind Brain Behavior Institute Columbia University, USA · Departments of Neuroscience and Statistics University of California, Berkeley, USA

Cross-Modal Learning Transfer Learning Neuroimaging Neural Encoding

Open paper AskPDF arXiv

Neuroscientists have recently turned to intracranial brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast" brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.

Abstract

Explore similar work

Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG

CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding

Continuous-Latent Predictive Modeling with Semantic Alignment for EEG-Language Foundation Models