Recent neural decoding approaches achieve strong performance by training on large collections of paired neural-behavioral recordings, but such datasets are expensive, invasive, and difficult to scale. In contrast, behavioral data is abundant and easy to collect, raising the question of whether neural decoding can benefit from decoupling behavioral and neural representation learning. We introduce BEEMO, a flexible multi-session framework that enables training with arbitrary mixtures of paired neural-behavioral recordings and unpaired behavioral data. A behavior decoder learns movement dynamics directly from behavioral trajectories, while neural activity is incorporated through cross-attention layers when available, allowing behavioral and neural supervision to scale independently. Across multiple intracortical motor datasets and tasks in nonhuman primates, we show that incorporating unpaired behavioral data improves decoding performance, particularly when transferring to new tasks, and that adding even a small amount of behavioral data from the target task further improves transfer. By decoupling neural and behavioral data, we unlock a new dimension for scaling: unpaired behavioral data, which is substantially cheaper and easier to collect than paired neural recordings, can be used directly to improve decoding performance. We further show that behavior-only pretraining, without any neural pretraining, achieves performance comparable to models pretrained on substantially larger neural datasets. Together, our results suggest that scaling behavioral data offers a practical and cost-effective path toward neural decoding models that generalize across subjects and tasks. Code is available at https://github.com/nerdslab/beemo.
Figures & tables
Figure 1: Overview of BeeMO : (A) Architecture: Behavioral context and query tokens pass through alternating self-attention and cross-attention layers. The behavioral and neural components are decoupled: the behavior decoder operates independently on unpaired behavioral data, and neural latents are incorporated via cross-attention when available. (B) Neural Encoders: POYO (left) represents neural activity as spike-level tokens, where each spike is embedded with its timing and unit identity. NDT* (right) represents neural activity as bin-level tokens, aggregating spikes into fixed time bins across units.
Within-Task
Cross-Task
Method
CO
RT
Isometric Wrist
Single-Session
Linear
0.6816±0.0011
0.6146±0.0003
0.5826±0.0006
MLP
0.8995±0.0014
0.7477±0.0014
0.7392±0.0035
GRU
0.9100±0.0015
0.7839±0.0016
0.7888±0.0011
POYO-SS
0.8438±0.0030
0.7219±0.0101
0.6967±0.0063
Table 1: Asymmetric training results. All multi-session models are pretrained on center-out (CO) reaching data and evaluated on within-task (CO) and cross-task (RT, Isometric Wrist) transfer. BeeMO variants incorporate ∼ 6.5 hours of additional unpaired behavioral data during pretraining, while base models use paired neural-behavioral data only. Performance is reported as mean ± SEM over three seeds. Paired Δ reports the mean paired difference in R2 between each BeeMO variant and its matched backbone across held-out sessions. Bold indicates a significant improvement ( p<0.05 , two-sided paired t -test). Session-wise mean ± SD and exact p -values are reported in Table A7 .
Figure 2: Scaling Analysis: Average cross-task decoding performance ( R2 averaged across RT and IW tasks) as a function of the number of paired neural sessions used during pretraining (models pretrained on CO). Top: BEEMO-POYO versus POYO. Bottom: BEEMO-NDT* versus NDT*. BeeMO outperforms both POYO and NDT* at matched neural data budgets by leveraging unpaired behavioral data during pretraining. Notably, BeeMO matches the performance of POYO and NDT* using approximately 3 × less neural data. Shaded regions denote SEM over three seeds.
Figure 3: Behavioral pretraining improves neural decoding. Error bars denote SEM over three seeds. (A) Task-specific behavioral data: Adding task-matched behavioral data (IW) to the pretraining corpus consistently improves downstream decoding performance on the IW task on a new animal for both BeeMO -POYO and BeeMO -NDT*. (B) Behavior-only pretraining: Decoding performance across three tasks (CO, RT, IW) for single-session models and multi-session neural-pretrained models, compared against BeeMO with no neural pretraining (behavior-only).
Figure 4: Behavioral context improves neural decoding. Predicted velocity trajectories ( vx , vy ) under neural-only (blue) and joint neural-behavior (orange) context, vs. ground truth (black), across 1 sec decoding windows sorted by ΔR2 (largest to smallest improvement). Additional trajectory examples are provided in Figure A2 in the Appendix.
Cross-Task
Model Variant
RT
IW
BeeMO -POYO
0.7609±0.0014
0.7361±0.0020
w/o dropout
0.6925±0.0133
0.6898±0.0035
w/o unpaired
0.7476±0.0012
0.6849±0.0017
BeeMO -NDT*
0.7734±0.0018
0.7802±0.0001
w/o dropout
0.7101±0.0083
0.7266±0.0019
Table 2: Ablation study: Values are reported as mean ± SEM across three seeds.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Motor tasks and representative behavioral trajectories. Center-out reaching (CO) and random-target reaching (RT) are performed with a planar manipulandum; isometric wrist (IW) uses a force transducer with no arm movement.
Hyperparameter
Search Space
Sampling
Model
Bin size (ms)
{1,5,10,20,40}
Categorical
Training
Number of epochs
{100,300,500}
Categorical
Batch size
[16,128]
Discrete ( log2 )
Weight decay
[10−6,10−1]
Loguniform
Appendix
Table A1: Hyperparameter search space for the linear baseline, tuned for 100 trials.
Hyperparameter
Search Space
Sampling
Model
Bin size (ms)
{5,10,20,40}
Categorical
Depth
{1,2,3}
Categorical
Hidden dim
[32,512]
Discrete ( log2 )
Dropout
{0.0,0.2,0.4,0.6}
Categorical
Batch norm
{True,False}
Categorical
Appendix
Table A2: Hyperparameter search space for the MLP baseline, tuned for 100 trials.
Hyperparameter
Search Space
Sampling
Model
Bin size (ms)
{1,5,10,20,40}
Categorical
Depth
{1,2,3}
Categorical
Hidden dim
[32,512]
Discrete ( log2 )
Dropout
{0.0,0.2,0.4,0.6}
Categorical
Bidirectional
{True,False}
Categorical
Appendix
Table A3: Hyperparameter search space for the GRU baseline, tuned for 100 trials.
Hyperparameter
Search Space
Sampling
Model
Embedding dim
{64,96,128,192,256}
Categorical
Number of heads
{1,2,4,8}
Discrete ( log2 )
Depth
{0,1,…,8}
Discrete
Dropout
[0.0,0.9]
Discrete (step 0.1)
Training
Appendix
Table A4: Hyperparameter search space for the POYO baseline, tuned for 100 trials.
Hyperparameter
Search Space
Sampling
Model
Embedding dim
{64,96,128,192,256}
Categorical
Number of heads
{1,2,4,8}
Discrete ( log2 )
Depth
{0,1,…,8}
Discrete
Dropout
[0.0,0.9]
Discrete (step 0.1)
Bin size
{1,5,10,20,40} ms
Categorical
Appendix
Table A5: Hyperparameter search space for the NDT* baseline, tuned for 100 trials.
Stage
Runs
GPU-time / run
Total GPU-hours
Single-session tuning (per model)
Linear
2,300
59.9±102.7 min (22.3)
∼ 2,300
MLP
2,300
53.7±122.5 min (20.8)
∼ 2,100
GRU
2,300
46.8±80.9 min (16.9)
∼ 1,800
POYO-SS
2,300
55.8±142.4 min (17.0)
∼ 2,100
NDT*-SS
2,300
23.6±19.2 min (19.0)
∼ 900
Appendix
Table A6: Estimated computational cost. For single-session baselines, run times are reported as mean ± SD across tuning runs, with the median in parentheses, and total GPU-hours are estimated over all 2,300 runs. For pretrained models, we report the one-time pretraining cost and the cost of fine-tuning across the 23 evaluation sessions.
Figure A2: Behavioral context improves neural decoding. Predicted velocity trajectories ( vx , vy ) under neural-only (blue) and joint neural-behavior (orange) context, vs. ground truth (black), across 1 sec decoding windows sorted by ΔR2 (largest to smallest improvement). The gray trace shows the preceding behavioral context provided to the joint model. R2 values are reported as Rjoint2−Rneural2=ΔR2 . Behavioral context consistently produces smoother, better-calibrated trajectories, with the largest gains occurring when neural-only decoding fails to constrain trajectory amplitude and timing
Backbone
Task
BeeMO
Baseline
Paired ΔR2
Paired t -test ( p )
NDT*
CO
0.9027±0.0138
0.9010±0.0175
+0.0017
0.4240
RT
0.7734±0.0716
0.7600±0.0789
+0.0134
0.0123
IW
0.7802±0.0460
0.7767±0.0435
+0.0035
0.1920
POYO
CO
0.9031±0.0178
0.8973±0.0185
+0.0058
0.1875
RT
0.7609±0.0726
0.7498±0.0742
+0.0112
0.0001
IW
0.7361±0.0644
0.7237±0.0680
+0.0124
0.0089
Appendix
Table A7: Session-wise paired comparisons of BeeMO and matched backbones. Values report mean R2± standard deviation across held-out sessions (each session averaged over three seeds). Paired t -tests (two-sided) compare each BeeMO variant with its matched multi-session backbone and were conducted separately for each task ( n=6 for CO and RT, n=11 for IW). Significant results ( p<0.05 ) are bolded.
Method
CO
RT
Kalman Filter (SS)
0.4000±0.0381
0.4066±0.0399
POYO + Kalman Smoother (SS)
0.8982±0.0062
0.7515±0.0018
POYO + Kalman Smoother (MS)
0.9004±0.0015
0.7508±0.0017
POYO
0.8973±0.0013
0.7498±0.0015
BeeMO -POYO
0.9031±0.0003
0.7609±0.0014
Appendix
Table A8: Kalman-based baselines. Models pretrained on CO and evaluated on CO and RT. Values are reported as mean ± SEM across three seeds.
Unpaired behavioral data
RT
None
0.7511
+ 6.5 h CO
0.7637
+ 6.5 h CO + 2 h RT
0.7705
Appendix
Table A9: Task-specific behavioral data on RT. BeeMO -POYO evaluated on held-out RT sessions with increasing amounts of unpaired behavioral data during pretraining (single seed).
Within-Task
Cross-Task
Model Variant
CO
RT
IW
BeeMO -POYO
0.9031±0.0003
0.7609±0.0014
0.7361±0.0020
w/o dropout
0.8179±0.0035
0.6925±0.0133
0.6898±0.0035
w/o unpaired
0.8992±0.0012
0.7476±0.0012
0.6849±0.0017
BeeMO -NDT
0.9027±0.0016
0.7734±0.0018
0.7802±0.0001
w/o dropout
0.8110±0.0062
0.7101±0.0083
0.7266±0.0019
Appendix
Table A10: Ablation study: Values are reported as mean ± SEM across three seeds.
Decoding continuous motor trajectories from neural activity is essential for developing practical brain-computer interfaces (BCIs). However, current neural decoders are constrained by the limited scale and heterogeneity of neural recordings. In contrast, behavioral data can be collected more readily and at substantially larger scale from humans, animals, simulations, and robotic systems. Here, we introduce NeuroPB, a framework that scales neural decoding by transferring knowledge from pretrained behavioral representations. NeuroPB first pretrains a motor encoder on large-scale motor behavior data and then aligns neural activity with the resulting behavioral representation space using a limited set of paired neural-behavioral recordings. A neural encoder and lightweight motor decoder are subsequently optimized to reconstruct continuous movement from the aligned neural representations. Across multiple macaque motor datasets, behavioral pretraining improves trajectory decoding, including an 11% R2 increase on center-out and 8% on random-target compared with training the motor encoder from scratch. Notably, pretraining on robotic trajectories achieves performance comparable to pretraining on macaque trajectories, demonstrating that transferable kinematic structure is shared across biological and artificial models. Moreover, decoding performance improves as the scale and diversity of robotic pretraining data increase, when the amount of neural data is fixed. Pretraining also enhances generalization across recording sessions, subjects, and motor tasks, with only 10% calibration needed to match training from scratch. Overall, these results establish behavioral pretraining as a scalable source for neural decoding and provide a promising route toward high-performance and calibration-efficient BCIs under limited neural data.
Luyao Jin, Yonghao Song, Huan Zhao +2
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong · Department of Biomedical Engineering, School of Medicine, Tsinghua University · School of Biomedical Sciences, The Chinese University of Hong Kong
Understanding how cortical activity represents natural whole-body behaviors in primates remains challenging. Limited by the diversity of movements and inaccessibility of large-scale neural representation of whole-body kinematics, previous motor decoding studies focused on constrained tasks and limited limb movements. Here, we present a neural-behavioral recording and modeling framework for freely moving monkeys, combining large-scale epidural cortical signals from distributed sensory- and motor-related areas with synchronized multi-view motion capture through a custom-made data collection platform. We reconstructed whole-body monkey kinematics and learned a compact behavior prior using an autoregressive encoder-decoder model. Conditioned on neural signals, the model decoded accurate and realistic whole-body movement without explicit physical constraints. Our results provide a novel proof-of-concept approach for decoding natural whole-body movements in primates using large-scale intracranial neural activity.
Jieshi He, Puzhe Li, Yanan Sui +1
Center for Excellence in Brain Science and Intelligence Technology, CAS · 2Tsinghua University
Robust and accurate neural decoders are integral to neurotechnologies such as brain-computer interfaces and closed-loop experiments. Recent work has shown that tokenizing neural data at the spike level facilitates multi-session pretraining and delivers state-of-the-art decoding performance. However, current spike-based models are restricted to supervised learning (SL), limiting training to datasets with paired behavioural labels. To address this limitation, we introduce MOJO (Masked autOencoder-based JOint training), a training framework for spike-tokenizing models that jointly leverages self-supervised learning (SSL) via masked autoencoding and SL objectives. We evaluate MOJO on three spiking datasets spanning monkey motor cortex during reaching tasks and multi-regional mouse recordings during vision and decision making tasks, demonstrating superior performance over purely SL-trained models. This improvement is especially pronounced when training with limited labelled data, particularly in few-shot finetuning, where only a small amount of labelled data from a new session is available. Incorporating SSL also yields more interpretable neuronal representations, improving performance on brain region classification and spike-statistics prediction without explicit optimization for these tasks. We further show that MOJO generalizes beyond spiking data to human electrocorticography during speech, where it continues to outperform purely SL-trained models and achieves performance comparable to neuro-foundation models (NFMs) designed specifically for continuous signals. Overall, augmenting spike-tokenizing models with SSL improves performance in label-impoverished settings and enables the use of unlabelled data across various tasks and species, while generalizing to other neural modalities. These results suggest a path towards more flexible and scalable data usage when training NFMs.
Ximeng Mao, Nanda H. Krishna, Avery Hee-Woon Ryoo +2
Mila – Quebec AI Institute · Université de Montréal · Canada CIFAR AI Chair