A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models

SungJun Cho; Chetan Gohil; Rukuang Huang; Oiwi Parker Jones; Mark W. Woolrich

cs.LGFeb 18, 2026

A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models

Authors: SungJun Cho, Chetan Gohil, Rukuang Huang, Oiwi Parker Jones, Mark W. Woolrich

Organizations: Oxford Centre for Human Brain Activity, University of Oxford, Oxford OX3 7JX, U.K. · Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford OX3 9DU, U.K. · Department of Psychiatry, University of Oxford, Oxford OX3 7JX, U.K. · Oxford Robotics Institute, Department of Engineering Science, University of Oxford, Oxford OX2 6NN, U.K.

Abstract

Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.

Figures & tables

Fig. 1: Overview of the foundation modeling framework and tokenizer architecture. (a) Schematic of the full generative training pipeline for the MEG-GPT foundation model. (b) Architecture of the learnable MEG tokenizer.

Fig. 2: MEG-GPT foundation model architecture. (a) High-level overview of the model architecture. (b) Detailed structure of the transformer decoder component.

		Causal	Noncausal	$\mu$ -Transform				Standard Quantile
Tokenizer	# Runs	10	10	-				-
	Mean $\pm$ Std.	0.0258 $\pm$ 8.58e-4	0.0277 $\pm$ 5.33e-3	-				-
	Best Final Loss	0.0245	0.0170	-				-
	# Tokens	97	121	256	182	108	54	108
MEG-GPT	Train Loss	1.7540	1.8740	3.6158	3.3824	3.0263	2.5492	3.2217
	Train Top 1 Acc.	0.3353	0.3426	0.0721	0.0873	0.1172	0.1728	0.0944

TABLE I: Model Performance Metrics of Tokenizers and MEG-GPT

Fig. 3: Convergence behavior of MEG-GPT models across tokenizer types on training and validation datasets. $n$ denotes the vocabulary size (i.e., number of unique tokens) for each tokenizer.

Fig. 4: Distributions of input token counts for different tokenization methods. Token indices are sorted in descending order of frequency, and the value $n$ denotes the vocabulary size for each tokenizer.

Fig. 5: Reconstruction accuracy and zero-shot generalization performance of different tokenizers. (a) Subject-level PVE between original MEG recordings and data reconstructed from tokens for the Cam-CAN training and test datasets. (b) Same analysis as in (a) evaluated on previously unseen datasets acquired under a task paradigm or using a different MEG scanner. Here, $N$ denotes the number of subjects, and $n$ the number of unique tokens used by each tokenizer.

Fig. 6: Token prediction accuracy of MEG-GPT pretrained with different tokenizers, evaluated on the Cam-CAN validation set. Tokenizers are ordered by descending accuracy. Statistical significance between adjacent pairs was assessed using Welch’s t-test (after verifying normality and equal variance assumptions), with Bonferroni correction for multiple comparisons; significant differences are shown in bold.

Fig. 7: Static spectral characteristics of data generated by MEG-GPT models trained with different tokenizers. (a) Group-level channel-wise PSDs from a single data generation instance; insets show spatial power maps averaged over 1-45 Hz. (b) Subject-level L2 distances between static PSDs of generated and real data, averaged over channels and datasets. Statistical significance between adjacent pairs was assessed using Welch’s t-test with Bonferroni correction; significant differences are indicated in bold.

Fig. 8: Dynamic spectral bursting characteristics of data generated by MEG-GPT models trained with different tokenizers. (a) Brain regions selected for bursting analysis. A three-state univariate TDE-HMM was applied to each region. (b) State-wise group-level PSDs for the motor cortex from a single data generation instance. (c) State-wise subject-level bursting rates for the motor cortex from the same data generation. (d) Subject-level L2 distances between generated and real state-wise PSDs, averaged over selected channels and datasets. (e) Subject-level L2 distances between generated and real state-wise bursting rates, averaged over selected channels and datasets. Statistical significance between adjacent pairs was assessed using Welch’s t-test with Bonferroni correction; significant differences are indicated in bold.

Fig. 9: Subject fingerprinting and inter-subject similarity in MEG-GPT-generated data. (a) Top- $k$ subject prediction accuracy based on TDE features, shown as mean $\pm$ 1 standard deviation across ten data generation instances. (b) Top-1 accuracy across ten data generations, ordered by mean accuracy. (c) Consistency scores across ten data generations, ordered by mean score. Statistical significance between adjacent pairs was assessed using Welch’s t-test with Bonferroni correction; significant differences are indicated in bold.

Fig. 10: Downstream task decoding performance of MEG-GPT models trained with different tokenizers. Task classification accuracy is shown for (a) within-subject and (b) new-subject conditions under baseline, zero-shot, and fine-tuned regimes. Statistical significance between adjacent pairs was assessed using two-sample t-tests with Bonferroni correction; no significant differences were observed.

Time Series Tokenization

See all 11 papers

A Roadmap for MEG Foundation Models
Sep 3, 2026Philipp Thölke, Hamza Abdelhedi, Yorguin Mantilla-Ramos +7Magnetoencephalography Neuroimaging
Cognitive and Computational Neuroscience Laboratory (CoCo Lab), Université de Montréal, Montréal, QC, Canada · Department of Psychology, Université de Montréal, Montréal, QC, Canada · Mila – Quebec Artificial Intelligence Institute, Montréal, QC, Canada +10
Tokenization Tradeoffs in Structured EHR Foundation Models
Mar 3, 2026Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee +4EHR Foundation Models Longitudinal EHR Modeling
Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, Canada · Division of Haematology/Oncology, The Hospital for Sick Children, Toronto, Canada · Department of Medicine, University Health Network, Toronto, Canada +1
Neural Data Needs Semantic Tokenization: Behavioral Events as Boundaries of Session-Transferable Tokens
Oct 2, 2026Sangyoon Bae, Jiook ChaNeural Decoding Transfer Learning
Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, 08826, South Korea · Department of Psychology Seoul National University Seoul, 08826, South Korea

Sep 3, 2026J/K move · Enter open · S save

A Roadmap for MEG Foundation Models

Philipp Thölke, Hamza Abdelhedi, Yorguin Mantilla-Ramos, Fouad Lbakali+6

Cognitive and Computational Neuroscience Laboratory (CoCo Lab), Université de Montréal, Montréal, QC, Canada · Department of Psychology, Université de Montréal, Montréal, QC, Canada · Mila – Quebec Artificial Intelligence Institute, Montréal, QC, Canada+10

Magnetoencephalography Neuroimaging Self-Supervised Pre-Training Multimodal Pretraining

Open paper AskPDF arXiv

Foundation models are beginning to reshape brain-signal analysis by moving the field beyond task-specific decoding pipelines toward reusable models pretrained on broad neural datasets. Magnetoencephalography (MEG) is a compelling but still underdeveloped target for this shift: it captures human cortical dynamics at millisecond resolution while offering stronger spatial interpretability than EEG, making it especially valuable for source-resolved studies of perception, language, cognition, and clinical brain function. Yet MEG foundation models remain at an early stage, with only a small number of MEG-specific and MEG-inclusive multi-modal models, modest pretraining corpora, and emerging but still limited benchmarks. This perspective lays down the basic concepts needed to understand MEG foundation models and provides a didactic overview of the field's key design choices, including tokenization, sensor- versus source-space representations, sensor-geometry encoding, backbone architectures, self-supervised objectives, and pretraining data. We then offer a roadmap for future development, organized around native MEG pretraining, adaptation of EEG foundation models, transfer from generic time-series models, and multi-modal integration with EEG, fMRI, MRI, behaviour, and stimulus features. We highlight the need for coordinated infrastructure, including diverse and reusable MEG datasets, rigorous evaluation across subjects, sites, tasks, and clinical settings, and responsible data-sharing practices that address consent, privacy, access, and governance.

Abstract

Time Series Tokenization

Explore similar work

A Roadmap for MEG Foundation Models

Tokenization Tradeoffs in Structured EHR Foundation Models

Neural Data Needs Semantic Tokenization: Behavioral Events as Boundaries of Session-Transferable Tokens