Brain activity encodes motor information across multiple temporal scales, enabling continuous neural control beyond discrete action recognition. However, existing methods are typically tailored to specific subjects and tasks, with direct neural-to-kinematic mappings that can entangle subject-specific neural variability with task-dependent motion patterns, limiting generalization. To address these limitations, we introduce AMRT, a shared neural decoding framework for heterogeneous electrode configurations and movement paradigms. Motivated by cortical encoding of movement direction and speed, AMRT adopts a two-stage design that anchors local motion through velocity estimation and refines the resulting trajectories over longer horizons, encouraging reuse of local motion tendencies across subjects and tasks. Across three datasets, AMRT generalizes to unseen trials and subjects and demonstrates transfer potential across neural recordings and movement paradigms, highlighting the potential of shared neural representations and anchor-and-refine decoding for generalizable continuous neural control.
Figures & tables
Figure 1: Overview of the Anchoring Motion, Refining Trajectories (AMRT) framework.
ctri-mix
ctri
WAY-EEG-GAL
JT-common
Joystick-Track
U-Limb
WAY-EEG-GAL
Joystick-Track
M3T-Attention
0.813 ± 0.036
0.464 ± 0.097
–
0.256 ± 0.041
0.869 ± 0.058
0.575 ± 0.128
Copilot-Assisted
0.674 ± 0.022
0.378 ± 0.084
–
0.314 ± 0.025
0.785 ± 0.083
0.465 ± 0.303
TBR
0.248 ± 0.053
–
0.012 ± 0.051
0.033 ± 0.028
0.464 ± 0.096
0.037 ± 0.035
AMRT
0.837 ± 0.045
0.707 ± 0.037 ***
0.787 ± 0.064 ***
0.530 ± 0.029 ***
0.914 ± 0.028 *
0.834 ± 0.107 **
Table 1: Comparison of baselines and AMRT. Results are trajectory PCC. For Joystick-Track under ctri-mix , JT-common uses the channel subset shared across subjects, enabling matched comparison with fixed-channel baselines, while Joystick-Track uses all available electrodes.
WAY-EEG-GAL
U-Limb
Joystick-Track
JT-common
M3T-Attention
0.785 ± 0.128
0.160 ± 0.061
–
0.334 ± 0.110
Copilot-Assisted
0.539 ± 0.159
0.085 ± 0.078
–
0.094 ± 0.079
TBR
0.263 ± 0.100
0.033 ± 0.028
0.013 ± 0.041
–
AMRT w/o CA
0.789 ± 0.103
0.360 ± 0.105
0.646 ± 0.138
0.673 ± 0.107
AMRT w/o CA-res/si
0.824 ± 0.068
0.372 ± 0.114
0.680 ± 0.116
0.679 ± 0.092
AMRT
0.841 ± 0.055
0.430 ± 0.073 ***
0.719 ± 0.109 ***
0.687 ± 0.125 ***
Table 2: Comparison of baselines, the full AMRT, and the channel-adapter ablation (AMRT w/o CA), while AMRT w/o CA-res/si removes the learnable adapter residual for WAY-EEG-GAL and U-Limb, and spatial information for Joystick-Track. Best results are shown in bold. Significance markers are computed against the best baseline.
Figure 2: Summary of the main experimental results. (a) Few-shot adaptation across calibration sizes. (b) Cross-dataset transfer across target-data fractions. (c) Stage-1-only gains over matched scratch training. (d) Stage-1 input ablation with Full, Coarse-only, and Neural-only variants. (e) Representative trajectory reconstructions from Vel-Int and AMRT.
Setting
Dataset
Direct-Pos
Pos-Vel
Vel-Int
AMRT
ctri
W
0.934 ± 0.022
0.935 ± 0.024
0.886 ± 0.040
0.914 ± 0.028
J
0.849 ± 0.065
0.860 ± 0.050
0.823 ± 0.114
0.834 ± 0.107
ctri-mix
W
0.831 ± 0.020
0.831 ± 0.020
0.785 ± 0.038
0.837 ± 0.045
U
0.259 ± 0.053
0.187 ± 0.061
0.471 ± 0.020
0.530 ± 0.029 ***
J
0.702 ± 0.018
0.694 ± 0.020
0.770 ± 0.076
0.787 ± 0.064
csub
W
0.825 ± 0.066
0.836 ± 0.062
0.800 ± 0.076
0.841 ± 0.055
Table 3: Trajectory PCC under different motion-to-trajectory reconstruction strategies. The best results are highlighted in bold.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Modality
Task
Subjects
Channels
Trials
Mean duration
WAY-EEG-GAL
EEG
6 conditions grasp-and-lift
12
32
3,528
9.2 s
U-Limb
EEG
30 movement tasks
20
128
764
12.6 s
Joystick-Track
ECoG
Circular joystick tracking
4
48-64
150
6.3s
Appendix
Table 4: Summary of the datasets used in our experiments.
Component
Configuration
Hidden dimension
128
Dropout
0.15
Spatial queries
4
Attention heads
8
BiLSTM hidden size
64 per direction
Temporal blocks
4
Appendix
Table 5: Architecture hyperparameters of AMRT.
Hyperparameters
Settings
Seed
42
Stage-0 optimizer
AdamW
Stage-0 learning rate
2×10−4
Stage-0 weight decay
10−4
Stage-0 batch size
32(J), 128(others)
Stage-0 epochs
100
Appendix
Table 6: Default training hyperparameters for cross-trial and cross-subject.
ctri-mix
ctri
WAY-EEG-GAL
Joystick-Track
U-Limb
WAY-EEG-GAL
Joystick-Track
M3T-Attention
4.924 ± 1.137
–
108.410 ± 6.499
3.088 ± 0.659
8745.303 ± 418.109
Copilot-Assisted
5.292 ± 0.161
–
96.721 ± 3.891
4.344 ± 0.902
9521.424 ± 2389.939
TBR
7.364 ± 2.139
8719.876 ± 1873.751
98.770 ± 18.371
6.515 ± 0.591
8502.913 ± 1823.509
AMRT
5.237 ± 1.159
7692.757 ± 616.446
80.926 ± 3.760
4.047 ± 0.616
7422.698 ± 604.229
Appendix
Table 7: Comparison of baselines and AMRT. Results are trajectory RMSE.
WAY-EEG-GAL
U-Limb
Joystick-Track
M3T-Attention
5.088 ± 1.049
173.312 ± 52.440
–
Copilot-Assisted
6.254 ± 0.900
196.135 ± 58.474
–
TBR
7.574 ± 0.634
117.822 ± 24.034
9559.497 ± 393.377
AMRT
5.809 ± 0.565
79.106 ± 13.723
8494.897 ± 1492.847
Appendix
Table 8: Comparison of baselines and AMRT. Results are trajectory RMSE.
Setting
Dataset
Direct-Pos
Pos-Vel
Vel-Int
AMRT
ctri
W
2.330 ± 0.357
2.298 ± 0.374
16.764 ± 13.366
4.047 ± 0.616
J
6075.707 ± 326.454
5815.055 ± 366.050
9028.251 ± 1842.146
7422.698 ± 604.229
ctri-mix
W
4.184 ± 0.262
4.156 ± 0.297
12.260 ± 4.502
5.237 ± 1.159
U
115.118 ± 1.935
111.472 ± 1.297
85.979 ± 4.331
80.926 ± 3.760
J
6686.755 ± 253.153
6766.083 ± 199.182
8591.401 ± 1070.737
7692.757 ± 616.446
csub
W
4.008 ± 0.633
3.654 ± 0.677
20.814 ± 15.911
5.809 ± 0.565
Appendix
Table 9: Trajectory RMSE under different motion-to-trajectory reconstruction strategies. The best results are highlighted in bold.
Dataset
Adaptation
zero-shot
1-shot
3-shot
5-shot
10-shot
WAY-EEG-GAL
Full-FT
0.837 ± 0.056
0.819 ± 0.078
0.838 ± 0.073
0.853 ± 0.061
0.873 ± 0.030
Stage-1-only
0.822 ± 0.073
0.838 ± 0.060
0.841 ± 0.058
0.849 ± 0.057
U-Limb
Full-FT
0.427 ± 0.073
0.381 ± 0.117
0.413 ± 0.113
0.476 ± 0.105
0.525 ± 0.100
Stage-1-only
0.400 ± 0.090
0.440 ± 0.084
0.457 ± 0.086
0.476 ± 0.085
Joystick-Track
Full-FT
0.710 ± 0.117
0.579 ± 0.208
0.653 ± 0.121
0.633 ± 0.172
0.679 ± 0.132
Stage-1-only
0.714 ± 0.123
0.710 ± 0.122
0.710 ± 0.119
0.722 ± 0.120
Appendix
Table 10: Few-shot calibration on unseen subjects. Results are trajectory PCC (mean ± std) across held-out subjects. The zero-shot result denotes performance without target-subject calibration. Bold indicates the better adaptation strategy at each shot count.
Transfer
Adaptation
10%
25%
50%
100%
J→U
Scratch
0.137 ± 0.049
0.152 ± 0.080
0.163 ± 0.120
0.192 ± 0.146
Full-FT
0.337 ± 0.135
0.311 ± 0.139
0.411 ± 0.132
0.301 ± 0.051
Stage-1-only
0.454 ± 0.056
0.434 ± 0.048
0.426 ± 0.077
0.413 ± 0.063
W→U
Scratch
0.137 ± 0.049
0.152 ± 0.080
0.163 ± 0.120
0.192 ± 0.146
Full-FT
0.264 ± 0.136
0.261 ± 0.183
0.376 ± 0.037
0.301 ± 0.148
Stage-1-only
0.371 ± 0.072
0.387 ± 0.067
0.419 ± 0.076
0.427 ± 0.043
Appendix
Table 11: Complete single-source cross-dataset transfer results. Results are trajectory PCC. Bold indicates the best result for each transfer direction and target-data fraction. J , U , and W denote Joystick-Track, U-Limb, and WAY-EEG-GAL.
Transfer
Stage-1 input
10%
25%
50%
100%
U→U
Scratch
0.137 ± 0.049
0.152 ± 0.080
0.163 ± 0.120
0.192 ± 0.146
J→U
Full
0.454 ± 0.056
0.434 ± 0.048
0.426 ± 0.077
0.413 ± 0.063
Coarse-only
0.324 ± 0.121
0.325 ± 0.085
0.357 ± 0.109
0.362 ± 0.065
Neural-only
0.310 ± 0.038
0.299 ± 0.060
0.315 ± 0.042
0.290 ± 0.040
W→U
Full
0.371 ± 0.072
0.387 ± 0.067
0.419 ± 0.076
0.427 ± 0.043
Coarse-only
0.374 ± 0.066
0.390 ± 0.063
0.395 ± 0.081
0.399 ± 0.065
Appendix
Table 12: Stage-1 input ablation for single-source transfer to U-Limb under Stage-1-only adaptation. Results are trajectory PCC (mean ± std); bold indicates the best result at each target-data fraction.
Dataset
Adaptation
10%
25%
50%
100%
Joystick-Track
Template, Zero-shot PCC: -0.262 ± 0.113
Scratch
0.730 ± 0.087
0.702 ± 0.104
0.733 ± 0.096
0.742 ± 0.109
Full-FT
0.717 ± 0.101
0.674 ± 0.118
0.721 ± 0.097
0.729 ± 0.110
Stage-1-only
-0.064 ± 0.140
0.035 ± 0.248
0.073 ± 0.241
0.245 ± 0.243
WAY-EEG-GAL
Template, Zero-shot PCC: 0.062 ± 0.180
Scratch
0.757 ± 0.094
0.738 ± 0.107
0.787 ± 0.083
0.787 ± 0.112
Appendix
Table 13: Dual-source cross-dataset transfer under different target-data fractions. Template denotes direct cross-dataset trajectory transfer without EEG. Results are trajectory PCC (mean ± std); bold indicates the best result at each target-data fraction.
Training trials
U-Limb
WAY-EEG-GAL
Joystick
Macro
90
0.359 ± 0.055
0.657 ± 0.172
0.459 ± 0.160
0.492 ± 0.082
180
0.306 ± 0.130
0.728 ± 0.051
0.629 ± 0.067
0.554 ± 0.069
270
0.344 ± 0.074
0.747 ± 0.006
0.650 ± 0.037
0.580 ± 0.033
360
0.380 ± 0.042
0.731 ± 0.030
0.673 ± 0.043
0.595 ± 0.029
Appendix
Table 14: Trajectory reconstruction performance under balanced trial scaling. Values report the mean ± standard deviation of trajectory PCC over five common random seeds. The Macro column is the unweighted average across the three datasets.
Transfer
Data
AMRT
Vel-Int
Direct-Pos
Pos-Res
Pos-Vel
J → U
10%
0.34 ± 0.14 [-1pt] 93.06 ± 15.22
0.01 ± 0.07 [-1pt]101.74 ± 11.67
0.07 ± 0.10 [-1pt]97.50 ± 11.31
0.12 ± 0.09 [-1pt]98.52 ± 10.43
0.11 ± 0.11 [-1pt] 94.41 ± 12.05
25%
0.31 ± 0.14 [-1pt] 91.92 ± 16.41
0.06 ± 0.10 [-1pt]99.36 ± 11.98
0.06 ± 0.05 [-1pt]97.54 ± 11.26
0.07 ± 0.05 [-1pt]98.59 ± 10.34
0.09 ± 0.10 [-1pt] 95.65 ± 10.56
50%
0.41 ± 0.13 [-1pt] 87.05 ± 15.37
0.08 ± 0.05 [-1pt]100.42 ± 11.71
0.05 ± 0.07 [-1pt] 96.16 ± 13.94
0.06 ± 0.03 [-1pt]98.67 ± 10.77
0.12 ± 0.10 [-1pt]96.96 ± 10.82
100%
0.30 ± 0.05 [-1pt] 89.99 ± 11.78
0.06 ± 0.07 [-1pt]101.65 ± 17.16
0.15 ± 0.11 [-1pt] 93.23 ± 14.69
0.19 ± 0.16 [-1pt]93.49 ± 14.48
0.19 ± 0.09 [-1pt]96.20 ± 10.42
W → U
10%
0.26 ± 0.14 [-1pt] 92.80 ± 9.84
0.02 ± 0.11 [-1pt]102.39 ± 12.00
0.08 ± 0.06 [-1pt]97.61 ± 11.27
0.10 ± 0.05 [-1pt]98.69 ± 10.33
0.03 ± 0.08 [-1pt] 96.55 ± 13.16
25%
0.26 ± 0.18 [-1pt] 91.24 ± 10.79
0.05 ± 0.08 [-1pt]99.49 ± 9.17
0.07 ± 0.07 [-1pt] 97.24 ± 10.94
0.08 ± 0.07 [-1pt]98.68 ± 10.41
0.09 ± 0.12 [-1pt]97.57 ± 11.33
Appendix
Table 15: Cross-dataset reconstruction under Full-FT. Each cell reports PCC ( ↑ ) / RMSE ( ↓ ), as mean ± std. Best and second-best results are bolded and underlined.
Model
U-Limb
WAY-EEG-GAL
Joystick-Track
AMRT
0.530 ± 0.029
0.837 ± 0.045
0.787 ± 0.064
AMRT-Stage0
0.471 ± 0.020
0.785 ± 0.038
0.770 ± 0.076
w/o BiLSTM
0.364 ± 0.038
0.831 ± 0.018
0.725 ± 0.024
w/o Multi-Horizon
0.330 ± 0.037
0.768 ± 0.095
0.720 ± 0.031
w/o Gate
0.502 ± 0.022
0.802 ± 0.041
0.727 ± 0.036
w/o Gate-Increment
0.497 ± 0.021
0.793 ± 0.042
0.705 ± 0.022
Appendix
Table 16: Model ablations under the ctri-mix protocol. Results are trajectory PCC (mean ± std); bold indicates the best result.
Figure 3: Summary of supplementary experimental results. (a) Dual-source cross-dataset transfer across target-data fractions. (b) Example of simultaneous trajectory and grasp-state decoding on WAY-EEG-GAL: ground-truth and predicted grasp state (top), and the corresponding z -axis trajectory (bottom). (c) Three-dimensional reconstruction of a representative WAY-EEG-GAL trajectory, comparing the prediction with the ground truth in the original movement space.
Protocol
Accuracy
F1
ctri
0.905 ± 0.030
0.848 ± 0.066
ctri-mix
0.868 ± 0.040
0.794 ± 0.056
csub
0.883 ± 0.031
0.844 ± 0.047
Appendix
Table 17: Grasp-state decoding performance on WAY-EEG-GAL under different evaluation protocols. We report accuracy and F1 score as mean ± std.
Decoding continuous motor trajectories from neural activity is essential for developing practical brain-computer interfaces (BCIs). However, current neural decoders are constrained by the limited scale and heterogeneity of neural recordings. In contrast, behavioral data can be collected more readily and at substantially larger scale from humans, animals, simulations, and robotic systems. Here, we introduce NeuroPB, a framework that scales neural decoding by transferring knowledge from pretrained behavioral representations. NeuroPB first pretrains a motor encoder on large-scale motor behavior data and then aligns neural activity with the resulting behavioral representation space using a limited set of paired neural-behavioral recordings. A neural encoder and lightweight motor decoder are subsequently optimized to reconstruct continuous movement from the aligned neural representations. Across multiple macaque motor datasets, behavioral pretraining improves trajectory decoding, including an 11% R2 increase on center-out and 8% on random-target compared with training the motor encoder from scratch. Notably, pretraining on robotic trajectories achieves performance comparable to pretraining on macaque trajectories, demonstrating that transferable kinematic structure is shared across biological and artificial models. Moreover, decoding performance improves as the scale and diversity of robotic pretraining data increase, when the amount of neural data is fixed. Pretraining also enhances generalization across recording sessions, subjects, and motor tasks, with only 10% calibration needed to match training from scratch. Overall, these results establish behavioral pretraining as a scalable source for neural decoding and provide a promising route toward high-performance and calibration-efficient BCIs under limited neural data.
Luyao Jin, Yonghao Song, Huan Zhao +2
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong · Department of Biomedical Engineering, School of Medicine, Tsinghua University · School of Biomedical Sciences, The Chinese University of Hong Kong
Recent neural decoding approaches achieve strong performance by training on large collections of paired neural-behavioral recordings, but such datasets are expensive, invasive, and difficult to scale. In contrast, behavioral data is abundant and easy to collect, raising the question of whether neural decoding can benefit from decoupling behavioral and neural representation learning. We introduce BEEMO, a flexible multi-session framework that enables training with arbitrary mixtures of paired neural-behavioral recordings and unpaired behavioral data. A behavior decoder learns movement dynamics directly from behavioral trajectories, while neural activity is incorporated through cross-attention layers when available, allowing behavioral and neural supervision to scale independently. Across multiple intracortical motor datasets and tasks in nonhuman primates, we show that incorporating unpaired behavioral data improves decoding performance, particularly when transferring to new tasks, and that adding even a small amount of behavioral data from the target task further improves transfer. By decoupling neural and behavioral data, we unlock a new dimension for scaling: unpaired behavioral data, which is substantially cheaper and easier to collect than paired neural recordings, can be used directly to improve decoding performance. We further show that behavior-only pretraining, without any neural pretraining, achieves performance comparable to models pretrained on substantially larger neural datasets. Together, our results suggest that scaling behavioral data offers a practical and cost-effective path toward neural decoding models that generalize across subjects and tasks. Code is available at https://github.com/nerdslab/beemo.
Divyansha Lachi, Vinam Arora, Shivashriganesh P. Mahato +2
Understanding how cortical activity represents natural whole-body behaviors in primates remains challenging. Limited by the diversity of movements and inaccessibility of large-scale neural representation of whole-body kinematics, previous motor decoding studies focused on constrained tasks and limited limb movements. Here, we present a neural-behavioral recording and modeling framework for freely moving monkeys, combining large-scale epidural cortical signals from distributed sensory- and motor-related areas with synchronized multi-view motion capture through a custom-made data collection platform. We reconstructed whole-body monkey kinematics and learned a compact behavior prior using an autoregressive encoder-decoder model. Conditioned on neural signals, the model decoded accurate and realistic whole-body movement without explicit physical constraints. Our results provide a novel proof-of-concept approach for decoding natural whole-body movements in primates using large-scale intracranial neural activity.
Jieshi He, Puzhe Li, Yanan Sui +1
Center for Excellence in Brain Science and Intelligence Technology, CAS · 2Tsinghua University