Neural Data Needs Semantic Tokenization: Behavioral Events as Boundaries of Session-Transferable Tokens

Sangyoon Bae; Jiook Cha

cs.LGOct 2, 2026

Neural Data Needs Semantic Tokenization: Behavioral Events as Boundaries of Session-Transferable Tokens

Authors: Sangyoon Bae, Jiook Cha

Organizations: Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, 08826, South Korea · Department of Psychology Seoul National University Seoul, 08826, South Korea

Abstract

Extracellular electrophysiology records a different set of neurons in every session. Neural foundation models embed each neuron and each session into their tokens, so every new session is an input they have never seen, and they fail to generalize to it. A tokenizer for new sessions needs a unit that every session shares and that carries behavior. Population activity offers such a unit. It evolves on a low-dimensional manifold that persists across neuronal turnover and across animals once sessions are aligned. This manifold changes regime at task events such as stimulus onset and movement onset. Within each regime the population occupies a state, the part of the manifold it spans between two events, and each state carries its own behavioral meaning. We propose Tokenization with States (TWS), which segments each trial, one repetition of the task, at these events and converts every state into tokens of population geometry, with no neuron or session embedding. On held-out International Brain Laboratory (IBL) sessions, TWS decodes movement even from regime boundaries that carry no information about the target, while a per-neuron foundation model pretrained on those sessions decodes at chance. Frozen after training on mice alone, TWS transfers to macaques and Utah arrays with only a linear probe. On reach direction, a target that its boundaries do not define, it achieves a Matthews correlation of 0.23, where the event time alone achieves $0.01$ . For cross-session generalization, the token matters more than the model on top.

Figures & tables

Figure 1: Overview of Tokenization with States (TWS). Each session records a different set of neurons. TWS segments every trial into behavioral states at task events and converts each state into tokens without neuron identity, so all sessions share one token space. States S0 to S2 are the states $k{=}1,2,3$ of Algorithm 1 , and $\tau_{1}$ and $\tau_{2}$ are the event times shifted by two trainable offsets shared by all sessions.

	Type A		Type B
Method	Movement	RT ( $R^{2}$ )	Choice	Block
Cross-session (leave-one-session-out)
TWS pretrained + finetuned	$0.553\pm 0.006$	$0.398\pm 0.004$	$\mathbf{0.040\pm 0.006}$	$\mathbf{0.031\pm 0.003}$
TWS from scratch	$\mathbf{0.565\pm 0.006}$	$\mathbf{0.402\pm 0.003}$	$0.038\pm 0.006$	$0.026\pm 0.004$
TWS pretrained	$0.428\pm 0.043$	$0.247\pm 0.036$	$0.022\pm 0.001$	$0.019\pm 0.002$
TWS (random init)	$0.274\pm 0.018$	$0.083\pm 0.011$	$-0.000\pm 0.004$	$0.005\pm 0.005$

Table 1: Cross-session decoding on IBL over the 53 held-out sessions (Section 4.1 ).

\pm

: SD over 5 model seeds. PCA and Population Statistics have no trained model; CEBRA and NEDS train one model per session and are run once.

(a) How many boundaries ( $K$ )
	TWS		Event-time information
Boundary condition	Movement	RT $R^{2}$	Movement	RT $R^{2}$
$K{=}2$ : stimulus onset only	$0.214\pm 0.007$	$0.096\pm 0.007$	–	–
$\bm{K{=}3}$ : stimulus + first movement	$0.565\pm 0.006$	$0.402\pm 0.003$	$0.908$	$0.443$
$K{=}3$ + transitions between states	$0.563\pm 0.004$	$0.403\pm 0.004$	$0.908$	$0.443$
(b) Where to place the boundary ( $K{=}3$ fixed)

Table 2: Tokenization boundary ablation (IBL, 53 held-out sessions). Every row uses the same architecture and only the boundaries change. (a) Number of boundaries. (b) Boundary placement at fixed

K{=}3

. Jittered: true boundary plus Gaussian noise with standard deviation

\sigma

bins. Fixed: boundary at a constant bin for all trials. Random: uniform draw per trial. Shuffled: real boundaries reassigned across (or within) sessions. Jitter and shuffling move only the boundaries that exist: a trial whose first movement falls outside the window keeps none, as in the true condition. Event-time information: the same probe on the one-hot first-movement bin that each condition gives, with no neural data; it measures how much of the label that input holds (mean over 5 draws of the perturbation). Cross-session leave-one-session-out, as in Table 1 .

\pm

: SD over 5 model seeds, each with its own boundary randomization.

		TWS		PerceiverIO
	Backbone	Movement MCC	RT $R^{2}$	Movement MCC	RT $R^{2}$
Token mixing	CNN	$\mathbf{0.565\pm 0.006}$	$\mathbf{0.402\pm 0.003}$	$0.303\pm 0.009$	$0.169\pm 0.003$
	Mamba-1	$0.536\pm 0.032$	$0.385\pm 0.035$	$0.287\pm 0.015$	$0.158\pm 0.005$
	Mamba-2	$0.555\pm 0.009$	$0.400\pm 0.005$	$0.294\pm 0.012$	$0.162\pm 0.012$
	Transformer	$0.552\pm 0.006$	$0.398\pm 0.002$	$\mathbf{0.313\pm 0.008}$	$0.168\pm 0.004$
No mixing	MLP	$0.459\pm 0.016$	$0.305\pm 0.013$	$0.290\pm 0.010$	$0.166\pm 0.005$

Table 3: Tokenizer

\times

backbone on IBL (53 held-out sessions, cross-session protocol of Table 1 ); every model learns from scratch.

\pm

: SD over 5 model seeds. The CNN row repeats Table 1 .

Figure 2: Training organizes the TWS embedding space around behavior; the tokenizer decides how far. (A) Each IBL session contributes two points, the mean embedding of its trials with (blue) and without (gray) a first movement within 2 s, and a line joins the two; PCA of these

486

points (first seed). Separation: distance between the two behavior centres over the spread of sessions around them, mean

\pm

SD over 5 seeds. (B) Share of trial-embedding variance explained by session identity and, after removing each session’s mean, by movement (mean over seeds). Models trained without the 53 held-out sessions; embeddings of all 243 sessions.

	Steinmetz 2019			Gallego et al.	NLB MC_Maze
Method	Choice	Feedback	Movement	Reach direction	Movement
TWS from scratch	$\mathbf{0.151\pm 0.015}$	$\mathbf{0.188\pm 0.008}$	$\mathbf{0.581\pm 0.003}$	$\mathbf{0.232\pm 0.010}$	$\mathbf{0.392\pm 0.010}$
TWS (random init)	$0.036\pm 0.016$	$0.077\pm 0.017$	$0.550\pm 0.007$	$0.128\pm 0.002$	$0.247\pm 0.010$
CEBRA	$-0.000\pm 0.000$	$0.000\pm 0.000$	$0.005\pm 0.050$	$0.000\pm 0.010$	$-0.003\pm 0.018$
POYO	$0.008\pm 0.010$	$0.045\pm 0.021$	$0.111\pm 0.013$	$0.013\pm 0.009$	$0.162\pm 0.047$
Population Statistics	$-0.007$	$0.000$	$0.242$	$0.046$	$0.177$

Table 4: Transfer of the frozen representation with a linear probe; POYO trains on each dataset (protocol in Section 4.1 ). MCC, mean

\pm

SD over 5 seeds. Rows without

\pm

are deterministic features. Event-time information uses the first-movement bin and no neural data.

Figure 3: Session invariance and data efficiency. (A) Session identifiability (normalized MI of a 243-class probe,

x

-axis) versus cross-session movement MCC (

y

-axis, the values of Table 1 ); methods in the upper-left corner achieve invariance without sacrificing decoding. We draw no error bars because the SD over 5 model seeds is smaller than the markers (values in Appendix Y ). (B) Reaction time

R^{2}

versus number of decoder-training sessions (log scale). Shaded bands:

{\pm}1

SD over 5 model seeds for TWS and over 20 draws of decoder-training sessions for the baselines. Session identifiability over all 243 IBL sessions; TWS and PerceiverIO decoding over the 53 held-out sessions, with decoder-training sessions in (B) drawn from the other 190.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Target	Full neurons	Cross-half summary	Index
IBL	movement	$0.524$	$0.308$	$0.59$
	choice	$0.624$	$0.155$	$0.25$
	block	$0.560$	$0.143$	$0.25$
Steinmetz	movement	$0.804$	$0.611$	$\mathbf{0.76}$
	feedback	$0.477$	$0.287$	$0.60$
	choice	$0.807$	$0.345$	$0.43$

Appendix

Table 5: How much of each target survives a change of the recorded neurons. MCC, averaged over sessions.

	IBL pretraining	IBL behavioral training	New-dataset training
Objective	Eq. 2 , no labels	six behavioral heads	class-weighted cross-entropy
Loss weights	$\lambda_{\mathrm{div}}{=}0.01$ , $\lambda_{\mathrm{geo}}{=}0.005$	–	–
State masking	$50\%$ of trials	–	–
Learning rate	$8\times 10^{-4}$ , constant	$10^{-3}$ , cosine to $10^{-5}$	encoder $10^{-4}$ , head $10^{-3}$
Weight decay	$0.01$	$0.05$	$0.01$
Batch	256 trials from 16 sessions	128 trials from 8 sessions	4 sessions $\times$ 16 trials

Appendix

Table 6: Training hyperparameters. The six IBL behavioral heads are choice, feedback, licking, movement, wheel velocity and RT. New-dataset training follows Appendix N .

$k_{\mathrm{dim}}$	Movement	Interpretation
1	0.864	Slightly narrow: marginal loss of task-relevant variance
2	0.865	Selected: balances expressiveness and compression
4	0.867	Similar: extra dimensions add little at this scale

Appendix

Table 7: Manifold dimensionality ablation (IBL).

Tokens	Movement MCC	RT $R^{2}$	Choice MCC	Block MCC
Orthonormal basis (TWS from scratch)	$0.565\pm 0.006$	$0.402\pm 0.003$	$0.038\pm 0.006$	$0.026\pm 0.004$
Without Gram–Schmidt	$0.562\pm 0.003$	$0.399\pm 0.003$	$0.033\pm 0.005$	$0.022\pm 0.005$

Appendix

Table 8: Cross-session decoding on the 53 held-out IBL sessions (protocol of Table 1 ); mean

\pm

SD over 5 model seeds.

Units per session	Movement MCC	RT $R^{2}$	Choice MCC	Block MCC
32	$0.469\pm 0.011$	$0.346\pm 0.015$	$0.018\pm 0.006$	$0.031\pm 0.004$
64	$0.523\pm 0.005$	$0.380\pm 0.004$	$0.025\pm 0.005$	$0.022\pm 0.005$
128 (Table 1 )	$\mathbf{0.565\pm 0.006}$	$\mathbf{0.402\pm 0.003}$	$\mathbf{0.038\pm 0.006}$	$0.026\pm 0.004$

Appendix

Table 9: TWS from scratch on the 53 held-out sessions with fewer units per session, the rest zero-filled. Mean

\pm

SD over 5 model seeds.

Variable	Scale	Type	Base rate	Definition
Movement	C	A	$P{=}0.76$	Whether a first wheel movement (IBL firstMovement_times ) occurs between stimulus onset and $2$ s after it. This is the event that also places the first-movement regime boundary, so the boundary alone largely determines the label (Appendix R ). Movement-related neural signals appear “almost everywhere in the brain” ( Musall et al., 2019 ; Stringer et al., 2019 ) .
Reaction time	R	A	—	Time (s) from stimulus onset to the IBL response time, which follows the first movement by a median of $0.19$ s. Trials outside $50$ ms– $10$ s are excluded. Reflects sensorimotor transformation latency; shorter for higher-contrast stimuli.
Choice	C	B	$P{\approx}0.52$	Direction of wheel turn: leftward (CW) or rightward (CCW). Correct when the grating is moved to center ( $0^{\circ}$ azimuth). Requires lateralized selectivity in specific neurons ( Steinmetz et al., 2019 ) .
Feedback	C	A ∗	$P{=}0.81$	Trial outcome: correct ( $+1$ , water reward) or incorrect ( $-1$ , white-noise burst $+$ 2 s timeout). Mechanistically linked to movement: $P(\text{fb}{=}1\mid\text{move}{=}1){=}0.845$ .
Block	C	B	$P{\approx}0.52$	Dominant stimulus side in the current probability block. After 90 unbiased trials, prior probability alternates between 80:20 and 20:80 (left:right) in blocks of 20–100 trials. A latent context variable.
Engagement	C	A	$P{=}0.94$	Whether the mouse is actively performing: recent accuracy ${\geq}0.6$ over a 10-trial rolling window. Near-ceiling base rate reduces discriminative value.

Appendix

Table 10: Behavioral variable definitions. Type A: encoded across the population, recoverable from any sufficiently large neuronal subset. Type B: depends on which specific neurons are recorded. Scale: C = classification, R = regression. ∗ Feedback is structurally Type A but its cross-session decodability is dominated by its correlation with movement.

Variable	$P(\text{positive})$	Majority baseline	Explanation
Classification variables
Movement	0.760	0.760	Session range $[0.08,0.91]$ ; discriminative
Engagement	0.938	0.938	93% positive; MCC $\approx 0$ for all methods
Feedback	0.813	0.813	$P(\text{fb}{=}1\|\text{move}{=}1){=}0.845$ ; dominated by movement
Choice	0.521	0.521	69% of sessions in $[0.4,0.6]$ ; Type B, $\approx$ chance
Block	0.518	0.518	Session std $=0.070$ ; Type B, $\approx$ chance

Appendix

Table 11: Class distributions and base rates for the 7 IBL behavioral variables (243 sessions, 152,794 trials).

P(\text{positive})

: proportion of positive class.

\sigma_{\text{cross}}(\mu)

: std of per-session means.

Cross-session (leave-one-session-out)
Method	Movement	Feedback	Choice	Block	Engagement	Wheel ( $R^{2}$ )	RT ( $R^{2}$ )
TWS pretrained + finetuned	$0.553\pm 0.006$	$0.070\pm 0.012$	$\mathbf{0.040\pm 0.006}$	$\mathbf{0.031\pm 0.003}$	$0.000\pm 0.000$	$-0.000\pm 0.001$	$0.398\pm 0.004$
TWS from scratch	$\mathbf{0.565\pm 0.006}$	$0.065\pm 0.011$	$0.038\pm 0.006$	$0.026\pm 0.004$	$0.000\pm 0.000$	$-0.001\pm 0.003$	$\mathbf{0.402\pm 0.003}$
TWS pretrained	$0.428\pm 0.043$	$0.019\pm 0.019$	$0.022\pm 0.001$	$0.019\pm 0.002$	$-0.000\pm 0.000$	$-0.001\pm 0.001$	$0.247\pm 0.036$
TWS (random init)	$0.274\pm 0.018$	$0.001\pm 0.002$	$-0.000\pm 0.004$	$0.005\pm 0.005$	$0.002\pm 0.003$	$-0.002\pm 0.001$	$0.083\pm 0.011$
PerceiverIO	$0.303\pm 0.009$	$\mathbf{0.102\pm 0.012}$	$0.014\pm 0.009$	$0.011\pm 0.006$	$0.000\pm 0.000$	$-0.005\pm 0.003$	$0.169\pm 0.003$

Appendix

Table 12: Full results on all 7 IBL behavioral variables. MCC for classification;

R^{2}

for wheel velocity and RT, with RT standardized per session across sessions and raw within sessions. Cross-session: leave-one-session-out Ridge probe, scored on the 53 held-out sessions. Within-session: TWS and PerceiverIO are probed on the trials of the 53 held-out sessions pooled (random 60/20/20 split, 5 repeats), which none of them trained on; POYO and Population Statistics on the trials of all sessions pooled; PCA, firing rates and CEBRA, whose feature spaces differ between sessions, per session (random 60/20/20 split).

\pm

: SD over 5 model seeds; rows without

\pm

are a single run. PerceiverIO uses the same CNN backbone as TWS. Aligned and unaligned PCA are scored on four variables only (Appendix D ). Per-variable class distributions in Appendix L .

		TWS				PerceiverIO
Dataset	Target	Scratch	IBL	Frozen	Frozen, behavior	Scratch	IBL	Frozen	Frozen, behavior
Steinmetz	Choice	$\mathbf{0.261\pm 0.005}$	$0.238\pm 0.002$	$0.221\pm 0.004$	$0.230\pm 0.018$	$0.231\pm 0.007$	$0.252\pm 0.015$	$0.232\pm 0.019$	$0.218\pm 0.011$
Gallego et al.	Reach direction	$0.234\pm 0.003$	$0.225\pm 0.015$	$0.166\pm 0.023$	$0.200\pm 0.002$	$0.281\pm 0.007$	$\mathbf{0.300\pm 0.008}$	$0.175\pm 0.007$	$0.183\pm 0.005$
NLB	Movement	$0.311\pm 0.013$	$0.354\pm 0.010$	$0.333\pm 0.019$	$0.333\pm 0.011$	$0.299\pm 0.043$	$\mathbf{0.421\pm 0.013}$	$0.379\pm 0.008$	$0.399\pm 0.008$

Appendix

Table 13: Held-out-session MCC after training on each dataset. In the IBL and Frozen columns the initializations are not matched: TWS starts from the self-supervised checkpoint, PerceiverIO from a model trained on IBL behavior with a Transformer backbone. Frozen, behavior: both encoders trained on IBL behavior with the CNN backbone.

	Cross-session		Within-session
Method	Whisker	Pupil	Whisker	Pupil
TWS from scratch	$\mathbf{0.462\pm 0.015}$	$-0.033\pm 0.008$	$\mathbf{0.636\pm 0.012}$	$0.549\pm 0.013$
TWS pretrained + finetuned	$0.459\pm 0.018$	$-0.035\pm 0.009$	$0.607\pm 0.018$	$0.521\pm 0.045$
TWS pretrained	$0.090\pm 0.036$	$-0.035\pm 0.002$	$0.499\pm 0.026$	$0.329\pm 0.097$
TWS (random init)	$0.091\pm 0.016$	$\mathbf{0.008\pm 0.001}$	$0.306\pm 0.066$	$0.116\pm 0.145$
PerceiverIO	$0.227\pm 0.034$	$-0.072\pm 0.081$	$0.619\pm 0.019$	$0.751\pm 0.012$

Appendix

Table 14: Whisker motion energy and pupil diameter (

R^{2}

) on the 5 held-out IBL sessions with video. Mean

\pm

SD over 5 model seeds.

Figure 4: TWS states occupy orthogonal subspaces with distinct dimensionality. (A) Session-demeaned PCA of state centroids (243 sessions; pretrained model, first seed). Small dots: per-session centroids; large circles: grand means; gray lines connect the three states within individual sessions. (B) Cumulative explained variance per state. Diamonds mark each state’s participation ratio (PR, effective dimensionality). Colors and state labels match Figure 1 .

Condition	$R^{2}$ (predicting $b_{\mathrm{move}}$ )	Interpretation
Real (aligned gate + neural)	$0.281\pm 0.192$	Full signal
Gate-only (shuffled ${\bm{x}}$ )	$0.149\pm 0.105$	Temporal fingerprint
Neural-only (shuffled gate)	$0.274\pm 0.160$	Neural temporal content
No-gate (time average)	$0.104\pm 0.116$	Firing rate baseline

Appendix

Table 15: Regime boundary timing decomposition (226 IBL sessions, within-session 5-fold CV Ridge on raw state means

\bar{{\bm{x}}}_{k}\in\mathbb{R}^{3N}

, mean

\pm

std across sessions). Not directly comparable to cross-session

R^{2}

in Table 1 .

	All trials		Without pre-stimulus movements
	Movement MCC	RT $R^{2}$	Movement MCC	RT $R^{2}$
TWS from scratch	$0.565\pm 0.006$	$0.402\pm 0.003$	$0.723\pm 0.004$	$0.418\pm 0.003$
PerceiverIO	$0.303\pm 0.009$	$0.169\pm 0.003$	$0.324\pm 0.018$	$0.181\pm 0.004$
Event-time information	$0.908$	$0.443$	$0.776$	$0.462$

Appendix

Table 16: Cross-session decoding on the 53 held-out IBL sessions with and without the

11\%

of trials whose first movement precedes the stimulus (probe of Table 1 ; mean

\pm

SD over 5 model seeds).

Representation + boundary	Movement MCC	RT $R^{2}$	Choice MCC	Block MCC
Event-time information	$0.908$	$0.445$	$0.032$	$0.019$
TWS from scratch	$0.910\pm 0.000$	$0.450\pm 0.002$	$0.029\pm 0.004$	$0.033\pm 0.005$
TWS pretrained + finetuned	$0.911\pm 0.000$	$0.450\pm 0.001$	$0.031\pm 0.006$	$0.037\pm 0.007$
PerceiverIO	$0.909\pm 0.001$	$0.452\pm 0.002$	$0.021\pm 0.006$	$0.025\pm 0.005$
POYO	$0.900\pm 0.008$	$0.408\pm 0.015$	$0.021\pm 0.007$	$0.022\pm 0.005$
CEBRA	$0.909$	$0.444$	$0.027$	$0.015$

Appendix

Table 17: Cross-session decoding on the 53 held-out IBL sessions when each representation is given the one-hot first-movement bin (Table 1 probe with z-scored features; mean

\pm

SD over 5 model seeds; CEBRA is a single run). PerceiverIO uses the same CNN backbone as TWS.

Figure 5: Session means of every tokenizer

\times

backbone

\times

training combination, drawn as in Figure 2 A.

Figure 6: What the trial-embedding variance of each combination is made of, drawn as in Figure 2 B.

Figure 7: Linear CKA between the trial embeddings of every combination (first seed, 20,000 shared trials).

Tokenizer (backbone)	Training	$R$	$\Delta R$ (movement)	$\Delta R$ (session)
TWS (CNN)	random init	$11.65\pm 1.27$	$0.049\pm 0.004$	$3.00\pm 0.32$
	from scratch	$39.71\pm 6.27$	$0.836\pm 0.108$	$12.67\pm 2.29$
	pretrained + finetuned	$28.27\pm 4.16$	$0.607\pm 0.079$	$6.99\pm 1.51$
PerceiverIO (Transformer)	random init	$10.13\pm 0.59$	$0.030\pm 0.005$	$3.03\pm 0.17$
PerceiverIO (Transformer)	from scratch	$56.07\pm 4.97$	$0.382\pm 0.029$	$14.26\pm 1.06$

Appendix

Table 18: Coding rate

R

and its reduction

\Delta R

(nats) when IBL trial embeddings are grouped by movement or by session (243 sessions, all trials with a movement label;

\epsilon^{2}{=}0.5

). Models trained without the 53 held-out sessions and with the clipped boundary. Mean

\pm

SD over 5 model seeds. Other backbones follow with Table 3 .

Dataset	Target	Level	Pattern	Level over time	TWS
Steinmetz	movement	$0.419$	$0.441$	$0.475$	$\mathbf{0.581}$
	feedback	$0.160$	$0.174$	$0.030$	$\mathbf{0.188}$
	choice	$0.012$	$0.027$	$0.126$	$\mathbf{0.151}$
NLB	movement	$0.354$	$0.421$	$\mathbf{0.476}$	$0.392$
Gallego et al.	reach direction	$0.057$	$0.083$	$0.116$	$\mathbf{0.232}$

Appendix

Table 19: Cross-dataset MCC of model-free population summaries, against TWS (Table 4 ).

	Steinmetz 2019			Gallego et al.	NLB MC_Maze
Method	Choice	Feedback	Movement	Reach direction	Movement
TWS from scratch	$0.151\pm 0.015$	$\mathbf{0.188\pm 0.008}$	$0.581\pm 0.003$	$\mathbf{0.232\pm 0.010}$	$0.392\pm 0.010$
TWS (random init)	$0.036\pm 0.016$	$0.077\pm 0.017$	$0.550\pm 0.007$	$0.128\pm 0.002$	$0.247\pm 0.010$
TWS (random init, random bias)	$0.114\pm 0.009$	$0.178\pm 0.005$	$\mathbf{0.592\pm 0.002}$	$0.173\pm 0.004$	$0.361\pm 0.005$
PerceiverIO, trained on IBL	$\mathbf{0.190\pm 0.016}$	$0.176\pm 0.004$	$0.572\pm 0.004$	$0.197\pm 0.013$	$\mathbf{0.443\pm 0.020}$
PerceiverIO (random init)	$0.169\pm 0.024$	$0.158\pm 0.014$	$0.570\pm 0.009$	$0.160\pm 0.009$	$0.419\pm 0.017$

Appendix

Table 20: Cross-dataset MCC of two frozen tokenizers without neuron or session embeddings, both with the CNN backbone (protocol of Table 4 ; mean

\pm

SD over 5 seeds). The two TWS rows are the runs of Table 4 .

Task	Intact pretraining	Time-shuffled pretraining	TWS (random init)
NLB movement	$0.385\pm 0.018$	$0.385\pm 0.007$	$0.254\pm 0.004$
Gallego et al. reach direction	$0.215\pm 0.014$	$0.232\pm 0.009$	$0.126\pm 0.004$
Steinmetz choice	$0.109\pm 0.016$	$0.125\pm 0.015$	$0.031\pm 0.014$
Steinmetz feedback	$0.142\pm 0.011$	$0.163\pm 0.007$	$0.085\pm 0.027$
Steinmetz movement	$0.563\pm 0.003$	$0.568\pm 0.005$	$0.552\pm 0.004$

Appendix

Table 21: Pretrained-only TWS (no behavioral finetuning), cross-dataset MCC, mean

\pm

SD over 5 seeds.

Dataset	Target	Definition	Event-time information
IBL	movement	a first movement occurs within $2$ s of stimulus onset	$0.908$
IBL	RT	stimulus onset to the IBL response time	$0.443$
Steinmetz	movement	mean wheel speed of the trial above the session median	$0.319$
	feedback	reward or no reward	$0.149$
	choice	wheel turned left, against right or no-go	$-0.009$
NLB	movement	mean cursor speed of the trial above the median	$0.260$

Appendix

Table 22: How each target is defined and what the boundary position alone decodes (MCC; RT:

R^{2}

Dataset	Target	Event-time information	PCA(50)	TWS (random init)	TWS from scratch
Steinmetz	choice	$-0.009$	$0.011$	$0.036$	$\mathbf{0.151\pm 0.015}$
	feedback	$0.149$	$0.012$	$0.077$	$\mathbf{0.188\pm 0.008}$
	movement	$0.319$	$0.390$	$0.550$	$\mathbf{0.581\pm 0.003}$
Gallego et al.	reach direction	$0.010$	$0.090$	$0.128$	$\mathbf{0.232\pm 0.010}$
NLB	movement	$0.260$	$0.144$	$0.247$	$\mathbf{0.392\pm 0.010}$

Appendix

Table 23: Cross-dataset MCC of the boundary position alone versus neural representations (TWS: the model of Table 4 , trained without the 53 held-out IBL sessions; mean

\pm

SD over 5 seeds; PCA and TWS (random init): mean over 5 seeds).

Method	All trials	Moved	Did not move	Go trials
TWS from scratch	$\mathbf{0.151\pm 0.015}$	$\mathbf{0.122\pm 0.010}$	$\mathbf{0.091\pm 0.011}$	$\mathbf{0.105\pm 0.007}$
TWS (random init)	$0.032\pm 0.012$	$0.086\pm 0.015$	$0.004\pm 0.007$	$0.076\pm 0.005$
PCA (per session)	$0.011\pm 0.010$	$0.054\pm 0.015$	$-0.001\pm 0.003$	$0.043\pm 0.009$
Event-time information	$-0.009$	$0.005$	$-0.005$	$0.031$

Appendix

Table 24: Steinmetz choice MCC within movement groups and on go trials, left against right (cross-session linear probe as in Table 4 ; mean

\pm

SD over 5 seeds; Event-time information as in Table 23 ).

Configuration	Movement	Feedback
Full (MSE + diversity + geodesic + state masking)	0.873	0.817
MSE + diversity (no geodesic)	0.875	0.822
MSE + geodesic (no diversity)	0.876	0.821
MSE only (no geometric losses)	0.883	0.823
No pretraining (from scratch)	0.889	0.824
+ Behavioral supervision during pretraining	0.887	0.825

Appendix

Table 25: Pretraining loss ablation (IBL, 243 sessions, finetuned). Movement and feedback accuracy.

Model	Original ( $\mathbf{h}$ )	Residual ( $\mathbf{h}-\bar{\mathbf{h}}_{s}$ )
TWS pretrained + finetuned	0.555	0.561
TWS from scratch	0.564	0.584
TWS (random init)	0.301	0.294
PerceiverIO	0.310	0.324
POYO	0.000	0.001
CEBRA	0.000	0.000

Appendix

Table 26: Cross-session movement MCC before (Original) and after (Residual) removing each session’s mean embedding. CNN backbone for TWS and PerceiverIO; models trained without the 53 held-out sessions, scored on them (first model seed).

	Movement MCC			RT $R^{2}$
Comparison	Difference [95% CI]	$p$	Sessions	Difference [95% CI]	$p$	Sessions
TWS vs. PerceiverIO	$+0.261$ [ $+0.210$ , $+0.313$ ]	$3.3\times 10^{-9}$	$91\%$	$+0.233$ [ $+0.170$ , $+0.299$ ]	$2.4\times 10^{-8}$	$91\%$
TWS vs. aligned PCA	$+0.216$ [ $+0.159$ , $+0.273$ ]	$1.2\times 10^{-7}$	$83\%$	$+0.176$ [ $+0.103$ , $+0.254$ ]	$4.0\times 10^{-7}$	$85\%$
TWS + boundary vs. event-time information	$+0.001$ [ $-0.000$ , $+0.003$ ]	$0.23$	$51\%$	$+0.006$ [ $-0.004$ , $+0.016$ ]	$0.56$	$49\%$
TWS + boundary vs. PerceiverIO + boundary	$+0.001$ [ $-0.001$ , $+0.002$ ]	$0.35$	$57\%$	$-0.001$ [ $-0.015$ , $+0.014$ ]	$0.72$	$51\%$
PerceiverIO + boundary vs. event-time information	$+0.001$ [ $-0.001$ , $+0.003$ ]	$0.57$	$49\%$	$+0.007$ [ $-0.010$ , $+0.027$ ]	$0.54$	$57\%$

Appendix

Table 27: Paired comparisons over the 53 held-out IBL sessions (cross-session protocol of Table 1 ; TWS is the from-scratch model with the convolutional backbone unless named). Difference: first minus second, mean over sessions with a bootstrap

95\%

interval;

p

: Wilcoxon signed-rank; Sessions: fraction of sessions where the first is higher. Each session’s score is averaged over 5 model seeds; PerceiverIO uses the same CNN backbone.

		Remove session directions		Session identification from
Model	$k$	Movement MCC	RT $R^{2}$	behavior 2D	random 2D
TWS from scratch	3	$0.564\to 0.546$	$0.401\to 0.252$	$0.050$	$0.061$
TWS pretrained + finetuned	3	$0.550\to 0.526$	$0.398\to 0.251$	$0.054$	$0.072$
TWS (random init)	1	$0.308\to 0.302$	$0.071\to 0.106$	$0.080$	$0.087$
PerceiverIO	2	$0.312\to 0.303$	$0.170\to 0.141$	$0.043$	$0.065$

Appendix

Table 28: Session directions (after removing what behavior explains) and behavior decoding directions on the 53 held-out IBL sessions, first model seed. Arrows: cross-session decoding before and after projecting out the

k

session directions; projecting out

k

random directions leaves it unchanged. Session identification: nearest-centroid accuracy over the 53 sessions (chance

0.019

		Session directions (90% of the variance of session means)
Model	All	$k$	Only these	All but these	$k$ random	Behavior directions (2D)
TWS from scratch	$0.232$	8	$0.187$	$0.248$	$0.145$	$0.050$
TWS pretrained + finetuned	$0.212$	7	$0.185$	$0.200$	$0.132$	$0.054$
TWS (random init)	$0.126$	3	$0.102$	$0.098$	$0.107$	$0.080$
PerceiverIO	$0.417$	12	$0.332$	$0.425$	$0.220$	$0.043$

Appendix

Table 29: Nearest-centroid session identification on the 53 held-out IBL sessions from subsets of embedding dimensions (chance

0.019

; two random directions give

0.05

–

0.10

). First model seed.

Factor	TWS pretrained + finetuned	TWS from scratch	TWS (random init)	PerceiverIO	POYO	CEBRA
Lab	0.174	0.209	0.078	0.281	0.426	0.024
Mouse	0.334	0.361	0.111	0.570	0.809	0.113
Session	0.420	0.440	0.137	0.733	0.959	0.220

Appendix

Table 30: Nuisance information retention: MI

(\mathbf{h};\,Y)/H(Y)

at each hierarchical level. Lower values indicate greater invariance to the corresponding factor. TWS and PerceiverIO are trained without the 53 held-out sessions, POYO pretrained on all sessions and CEBRA fits one model per session; PerceiverIO uses the same CNN backbone as TWS.

Figure 8: State tokens of TWS pretrained (the 53 held-out sessions excluded from training) for all 243 IBL sessions. (A) t-SNE of 2,400 state tokens (800 per state) colored by state. (B) The same coordinates colored by recording lab. (C) Silhouette scores of the t-SNE coordinates for state, lab, subject and session.

Panel of Figure 9 — Figure 9: 2D t-SNE of the trial embeddings of 10 held-out IBL sessions. Top: one color per session. Bottom: the same maps of TWS pretrained + finetuned, TWS (random init) and CEBRA, colored by whether the mouse moved within 2 s. POYO and PCA separate the sessions, while TWS mixes them and still separates the two behaviors. Nearby trials from other sessions share the movement label above chance for TWS pretrained + finetuned ( $+0.13$ ), but hardly for TWS (random init) ( $+0.03$ ) or CEBRA ( $+0.02$ ), whose separate per-session models overlap only by chance (15 nearest neighbors from other sessions).

Figure 10: Session identifiability (normalized MI of a 39-class probe) against cross-session movement accuracy on the 39 Steinmetz sessions. Mean

\pm

SD over 5 model seeds.

Model	Consistency	Five-session MCC	Consistency	Five-session $R^{2}$	Session identification
	Movement		Reaction time
TWS from scratch	$0.82$	$0.513$	$0.93$	$0.313$	$0.272$
TWS pretrained + finetuned	$0.82$	$0.507$	$0.91$	$0.330$	$0.228$
TWS pretrained	$0.55$	$0.383$	$0.56$	$0.009$	$0.132$
TWS (random init)	$0.50$	$0.182$	$0.62$	$0.021$	$0.123$
PerceiverIO	$0.64$	$0.254$	$0.73$	$0.073$	$0.406$

Appendix

Table 31: Direction consistency, five-session decoding and session identification on the 53 held-out IBL sessions (mean over 5 model seeds). PerceiverIO uses the same CNN backbone as TWS.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 179 papers

Feb 18, 2026

A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models

Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.

SungJun Cho, Chetan Gohil, Rukuang Huang +2

Oxford Centre for Human Brain Activity, University of Oxford, Oxford OX3 7JX, U.K. · Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford OX3 9DU, U.K. · Department of Psychiatry, University of Oxford, Oxford OX3 7JX, U.K. +1

Time Series Tokenization Time-Series Foundation Models

Ask PDF

Jul 15, 2026

Leveraging unlabelled data for generalizable neural population decoding

Robust and accurate neural decoders are integral to neurotechnologies such as brain-computer interfaces and closed-loop experiments. Recent work has shown that tokenizing neural data at the spike level facilitates multi-session pretraining and delivers state-of-the-art decoding performance. However, current spike-based models are restricted to supervised learning (SL), limiting training to datasets with paired behavioural labels. To address this limitation, we introduce MOJO (Masked autOencoder-based JOint training), a training framework for spike-tokenizing models that jointly leverages self-supervised learning (SSL) via masked autoencoding and SL objectives. We evaluate MOJO on three spiking datasets spanning monkey motor cortex during reaching tasks and multi-regional mouse recordings during vision and decision making tasks, demonstrating superior performance over purely SL-trained models. This improvement is especially pronounced when training with limited labelled data, particularly in few-shot finetuning, where only a small amount of labelled data from a new session is available. Incorporating SSL also yields more interpretable neuronal representations, improving performance on brain region classification and spike-statistics prediction without explicit optimization for these tasks. We further show that MOJO generalizes beyond spiking data to human electrocorticography during speech, where it continues to outperform purely SL-trained models and achieves performance comparable to neuro-foundation models (NFMs) designed specifically for continuous signals. Overall, augmenting spike-tokenizing models with SSL improves performance in label-impoverished settings and enables the use of unlabelled data across various tasks and species, while generalizing to other neural modalities. These results suggest a path towards more flexible and scalable data usage when training NFMs.

Ximeng Mao, Nanda H. Krishna, Avery Hee-Woon Ryoo +2

Mila – Quebec AI Institute · Université de Montréal · Canada CIFAR AI Chair

Masked Autoencoders Self-Supervised Pre-Training Neural Decoding

Ask PDF

Jun 9, 2026

GRAFT: Gain-Recalibrated Adapters for Transformer-Based Neural Population Activity Modeling

Neural population activity models can recover rich temporal structure from binned spikes, but their read-in and readout layers often remain tied to a fixed set of recorded neurons. This coupling limits reuse in long-term brain-computer interfaces, where recorded neuron identities, counts, and response statistics can change across days. We introduce GRAFT, a Transformer-based neural population activity model that separates reusable temporal dynamics from a recalibratable neuron interface. The neuron interface controls how recorded neurons enter and leave the shared backbone, and auxiliary gain and positional mechanisms support neural activity modeling inside the Transformer. On MC Maze under the standard NLB'21 protocol, GRAFT reaches 0.3866 co-bps as an ensemble, setting a new state of the art on the primary co-bps metric among public and reported NLB'21 results. In a cross-day protocol constructed from the NLB'21 MC Maze dataset series, GRAFT recalibrates from MC Maze to the scaled MC Maze datasets (Large/Medium/Small) by updating only 9.21% of parameters, reaching 0.3749, 0.3112, and 0.3152 co-bps with restricted target-day support sets. These results show that the same interface-backbone separation supports both strong Transformer-based neural population activity modeling and data-efficient cross-day recalibration.

Xiangsheng Ge, Yang Xie

Shanghai Jiao Tong University

Brain-Computer Interfaces Transformer Neural Encoding Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Neural Decoding

Explore similar work

A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models

Leveraging unlabelled data for generalizable neural population decoding

GRAFT: Gain-Recalibrated Adapters for Transformer-Based Neural Population Activity Modeling