Brain activity encodes motor information across multiple temporal scales, enabling continuous neural control beyond discrete action recognition. However, existing methods are typically tailored to specific subjects and tasks, with direct neural-to-kinematic mappings that can entangle subject-specific neural variability with task-dependent motion patterns, limiting generalization. To address these limitations, we introduce AMRT, a shared neural decoding framework for heterogeneous electrode configurations and movement paradigms. Motivated by cortical encoding of movement direction and speed, AMRT adopts a two-stage design that anchors local motion through velocity estimation and refines the resulting trajectories over longer horizons, encouraging reuse of local motion tendencies across subjects and tasks. Across three datasets, AMRT generalizes to unseen trials and subjects and demonstrates transfer potential across neural recordings and movement paradigms, highlighting the potential of shared neural representations and anchor-and-refine decoding for generalizable continuous neural control.
Figures & tables
Figure 1: Overview of the Anchoring Motion, Refining Trajectories (AMRT) framework.
ctri-mix
ctri
WAY-EEG-GAL
JT-common
Joystick-Track
U-Limb
WAY-EEG-GAL
Joystick-Track
M3T-Attention
0.813 ± 0.036
0.464 ± 0.097
–
0.256 ± 0.041
0.869 ± 0.058
0.575 ± 0.128
Copilot-Assisted
0.674 ± 0.022
0.378 ± 0.084
–
0.314 ± 0.025
0.785 ± 0.083
0.465 ± 0.303
TBR
0.248 ± 0.053
–
0.012 ± 0.051
0.033 ± 0.028
0.464 ± 0.096
0.037 ± 0.035
AMRT
0.837 ± 0.045
0.707 ± 0.037 ***
0.787 ± 0.064 ***
0.530 ± 0.029 ***
0.914 ± 0.028 *
0.834 ± 0.107 **
Table 1: Comparison of baselines and AMRT. Results are trajectory PCC. For Joystick-Track under ctri-mix , JT-common uses the channel subset shared across subjects, enabling matched comparison with fixed-channel baselines, while Joystick-Track uses all available electrodes.
WAY-EEG-GAL
U-Limb
Joystick-Track
JT-common
M3T-Attention
0.785 ± 0.128
0.160 ± 0.061
–
0.334 ± 0.110
Copilot-Assisted
0.539 ± 0.159
0.085 ± 0.078
–
0.094 ± 0.079
TBR
0.263 ± 0.100
0.033 ± 0.028
0.013 ± 0.041
–
AMRT w/o CA
0.789 ± 0.103
0.360 ± 0.105
0.646 ± 0.138
0.673 ± 0.107
AMRT w/o CA-res/si
0.824 ± 0.068
0.372 ± 0.114
0.680 ± 0.116
0.679 ± 0.092
AMRT
0.841 ± 0.055
0.430 ± 0.073 ***
0.719 ± 0.109 ***
0.687 ± 0.125 ***
Table 2: Comparison of baselines, the full AMRT, and the channel-adapter ablation (AMRT w/o CA), while AMRT w/o CA-res/si removes the learnable adapter residual for WAY-EEG-GAL and U-Limb, and spatial information for Joystick-Track. Best results are shown in bold. Significance markers are computed against the best baseline.
Figure 2: Summary of the main experimental results. (a) Few-shot adaptation across calibration sizes. (b) Cross-dataset transfer across target-data fractions. (c) Stage-1-only gains over matched scratch training. (d) Stage-1 input ablation with Full, Coarse-only, and Neural-only variants. (e) Representative trajectory reconstructions from Vel-Int and AMRT.
Setting
Dataset
Direct-Pos
Pos-Vel
Vel-Int
AMRT
ctri
W
0.934 ± 0.022
0.935 ± 0.024
0.886 ± 0.040
0.914 ± 0.028
J
0.849 ± 0.065
0.860 ± 0.050
0.823 ± 0.114
0.834 ± 0.107
ctri-mix
W
0.831 ± 0.020
0.831 ± 0.020
0.785 ± 0.038
0.837 ± 0.045
U
0.259 ± 0.053
0.187 ± 0.061
0.471 ± 0.020
0.530 ± 0.029 ***
J
0.702 ± 0.018
0.694 ± 0.020
0.770 ± 0.076
0.787 ± 0.064
csub
W
0.825 ± 0.066
0.836 ± 0.062
0.800 ± 0.076
0.841 ± 0.055
Table 3: Trajectory PCC under different motion-to-trajectory reconstruction strategies. The best results are highlighted in bold.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Modality
Task
Subjects
Channels
Trials
Mean duration
WAY-EEG-GAL
EEG
6 conditions grasp-and-lift
12
32
3,528
9.2 s
U-Limb
EEG
30 movement tasks
20
128
764
12.6 s
Joystick-Track
ECoG
Circular joystick tracking
4
48-64
150
6.3s
Appendix
Table 4: Summary of the datasets used in our experiments.
Component
Configuration
Hidden dimension
128
Dropout
0.15
Spatial queries
4
Attention heads
8
BiLSTM hidden size
64 per direction
Temporal blocks
4
Appendix
Table 5: Architecture hyperparameters of AMRT.
Hyperparameters
Settings
Seed
42
Stage-0 optimizer
AdamW
Stage-0 learning rate
2×10−4
Stage-0 weight decay
10−4
Stage-0 batch size
32(J), 128(others)
Stage-0 epochs
100
Appendix
Table 6: Default training hyperparameters for cross-trial and cross-subject.
ctri-mix
ctri
WAY-EEG-GAL
Joystick-Track
U-Limb
WAY-EEG-GAL
Joystick-Track
M3T-Attention
4.924 ± 1.137
–
108.410 ± 6.499
3.088 ± 0.659
8745.303 ± 418.109
Copilot-Assisted
5.292 ± 0.161
–
96.721 ± 3.891
4.344 ± 0.902
9521.424 ± 2389.939
TBR
7.364 ± 2.139
8719.876 ± 1873.751
98.770 ± 18.371
6.515 ± 0.591
8502.913 ± 1823.509
AMRT
5.237 ± 1.159
7692.757 ± 616.446
80.926 ± 3.760
4.047 ± 0.616
7422.698 ± 604.229
Appendix
Table 7: Comparison of baselines and AMRT. Results are trajectory RMSE.
WAY-EEG-GAL
U-Limb
Joystick-Track
M3T-Attention
5.088 ± 1.049
173.312 ± 52.440
–
Copilot-Assisted
6.254 ± 0.900
196.135 ± 58.474
–
TBR
7.574 ± 0.634
117.822 ± 24.034
9559.497 ± 393.377
AMRT
5.809 ± 0.565
79.106 ± 13.723
8494.897 ± 1492.847
Appendix
Table 8: Comparison of baselines and AMRT. Results are trajectory RMSE.
Setting
Dataset
Direct-Pos
Pos-Vel
Vel-Int
AMRT
ctri
W
2.330 ± 0.357
2.298 ± 0.374
16.764 ± 13.366
4.047 ± 0.616
J
6075.707 ± 326.454
5815.055 ± 366.050
9028.251 ± 1842.146
7422.698 ± 604.229
ctri-mix
W
4.184 ± 0.262
4.156 ± 0.297
12.260 ± 4.502
5.237 ± 1.159
U
115.118 ± 1.935
111.472 ± 1.297
85.979 ± 4.331
80.926 ± 3.760
J
6686.755 ± 253.153
6766.083 ± 199.182
8591.401 ± 1070.737
7692.757 ± 616.446
csub
W
4.008 ± 0.633
3.654 ± 0.677
20.814 ± 15.911
5.809 ± 0.565
Appendix
Table 9: Trajectory RMSE under different motion-to-trajectory reconstruction strategies. The best results are highlighted in bold.
Dataset
Adaptation
zero-shot
1-shot
3-shot
5-shot
10-shot
WAY-EEG-GAL
Full-FT
0.837 ± 0.056
0.819 ± 0.078
0.838 ± 0.073
0.853 ± 0.061
0.873 ± 0.030
Stage-1-only
0.822 ± 0.073
0.838 ± 0.060
0.841 ± 0.058
0.849 ± 0.057
U-Limb
Full-FT
0.427 ± 0.073
0.381 ± 0.117
0.413 ± 0.113
0.476 ± 0.105
0.525 ± 0.100
Stage-1-only
0.400 ± 0.090
0.440 ± 0.084
0.457 ± 0.086
0.476 ± 0.085
Joystick-Track
Full-FT
0.710 ± 0.117
0.579 ± 0.208
0.653 ± 0.121
0.633 ± 0.172
0.679 ± 0.132
Stage-1-only
0.714 ± 0.123
0.710 ± 0.122
0.710 ± 0.119
0.722 ± 0.120
Appendix
Table 10: Few-shot calibration on unseen subjects. Results are trajectory PCC (mean ± std) across held-out subjects. The zero-shot result denotes performance without target-subject calibration. Bold indicates the better adaptation strategy at each shot count.
Transfer
Adaptation
10%
25%
50%
100%
J→U
Scratch
0.137 ± 0.049
0.152 ± 0.080
0.163 ± 0.120
0.192 ± 0.146
Full-FT
0.337 ± 0.135
0.311 ± 0.139
0.411 ± 0.132
0.301 ± 0.051
Stage-1-only
0.454 ± 0.056
0.434 ± 0.048
0.426 ± 0.077
0.413 ± 0.063
W→U
Scratch
0.137 ± 0.049
0.152 ± 0.080
0.163 ± 0.120
0.192 ± 0.146
Full-FT
0.264 ± 0.136
0.261 ± 0.183
0.376 ± 0.037
0.301 ± 0.148
Stage-1-only
0.371 ± 0.072
0.387 ± 0.067
0.419 ± 0.076
0.427 ± 0.043
Appendix
Table 11: Complete single-source cross-dataset transfer results. Results are trajectory PCC. Bold indicates the best result for each transfer direction and target-data fraction. J , U , and W denote Joystick-Track, U-Limb, and WAY-EEG-GAL.
Transfer
Stage-1 input
10%
25%
50%
100%
U→U
Scratch
0.137 ± 0.049
0.152 ± 0.080
0.163 ± 0.120
0.192 ± 0.146
J→U
Full
0.454 ± 0.056
0.434 ± 0.048
0.426 ± 0.077
0.413 ± 0.063
Coarse-only
0.324 ± 0.121
0.325 ± 0.085
0.357 ± 0.109
0.362 ± 0.065
Neural-only
0.310 ± 0.038
0.299 ± 0.060
0.315 ± 0.042
0.290 ± 0.040
W→U
Full
0.371 ± 0.072
0.387 ± 0.067
0.419 ± 0.076
0.427 ± 0.043
Coarse-only
0.374 ± 0.066
0.390 ± 0.063
0.395 ± 0.081
0.399 ± 0.065
Appendix
Table 12: Stage-1 input ablation for single-source transfer to U-Limb under Stage-1-only adaptation. Results are trajectory PCC (mean ± std); bold indicates the best result at each target-data fraction.
Dataset
Adaptation
10%
25%
50%
100%
Joystick-Track
Template, Zero-shot PCC: -0.262 ± 0.113
Scratch
0.730 ± 0.087
0.702 ± 0.104
0.733 ± 0.096
0.742 ± 0.109
Full-FT
0.717 ± 0.101
0.674 ± 0.118
0.721 ± 0.097
0.729 ± 0.110
Stage-1-only
-0.064 ± 0.140
0.035 ± 0.248
0.073 ± 0.241
0.245 ± 0.243
WAY-EEG-GAL
Template, Zero-shot PCC: 0.062 ± 0.180
Scratch
0.757 ± 0.094
0.738 ± 0.107
0.787 ± 0.083
0.787 ± 0.112
Appendix
Table 13: Dual-source cross-dataset transfer under different target-data fractions. Template denotes direct cross-dataset trajectory transfer without EEG. Results are trajectory PCC (mean ± std); bold indicates the best result at each target-data fraction.
Training trials
U-Limb
WAY-EEG-GAL
Joystick
Macro
90
0.359 ± 0.055
0.657 ± 0.172
0.459 ± 0.160
0.492 ± 0.082
180
0.306 ± 0.130
0.728 ± 0.051
0.629 ± 0.067
0.554 ± 0.069
270
0.344 ± 0.074
0.747 ± 0.006
0.650 ± 0.037
0.580 ± 0.033
360
0.380 ± 0.042
0.731 ± 0.030
0.673 ± 0.043
0.595 ± 0.029
Appendix
Table 14: Trajectory reconstruction performance under balanced trial scaling. Values report the mean ± standard deviation of trajectory PCC over five common random seeds. The Macro column is the unweighted average across the three datasets.
Transfer
Data
AMRT
Vel-Int
Direct-Pos
Pos-Res
Pos-Vel
J → U
10%
0.34 ± 0.14 [-1pt] 93.06 ± 15.22
0.01 ± 0.07 [-1pt]101.74 ± 11.67
0.07 ± 0.10 [-1pt]97.50 ± 11.31
0.12 ± 0.09 [-1pt]98.52 ± 10.43
0.11 ± 0.11 [-1pt] 94.41 ± 12.05
25%
0.31 ± 0.14 [-1pt] 91.92 ± 16.41
0.06 ± 0.10 [-1pt]99.36 ± 11.98
0.06 ± 0.05 [-1pt]97.54 ± 11.26
0.07 ± 0.05 [-1pt]98.59 ± 10.34
0.09 ± 0.10 [-1pt] 95.65 ± 10.56
50%
0.41 ± 0.13 [-1pt] 87.05 ± 15.37
0.08 ± 0.05 [-1pt]100.42 ± 11.71
0.05 ± 0.07 [-1pt] 96.16 ± 13.94
0.06 ± 0.03 [-1pt]98.67 ± 10.77
0.12 ± 0.10 [-1pt]96.96 ± 10.82
100%
0.30 ± 0.05 [-1pt] 89.99 ± 11.78
0.06 ± 0.07 [-1pt]101.65 ± 17.16
0.15 ± 0.11 [-1pt] 93.23 ± 14.69
0.19 ± 0.16 [-1pt]93.49 ± 14.48
0.19 ± 0.09 [-1pt]96.20 ± 10.42
W → U
10%
0.26 ± 0.14 [-1pt] 92.80 ± 9.84
0.02 ± 0.11 [-1pt]102.39 ± 12.00
0.08 ± 0.06 [-1pt]97.61 ± 11.27
0.10 ± 0.05 [-1pt]98.69 ± 10.33
0.03 ± 0.08 [-1pt] 96.55 ± 13.16
25%
0.26 ± 0.18 [-1pt] 91.24 ± 10.79
0.05 ± 0.08 [-1pt]99.49 ± 9.17
0.07 ± 0.07 [-1pt] 97.24 ± 10.94
0.08 ± 0.07 [-1pt]98.68 ± 10.41
0.09 ± 0.12 [-1pt]97.57 ± 11.33
Appendix
Table 15: Cross-dataset reconstruction under Full-FT. Each cell reports PCC ( ↑ ) / RMSE ( ↓ ), as mean ± std. Best and second-best results are bolded and underlined.
Model
U-Limb
WAY-EEG-GAL
Joystick-Track
AMRT
0.530 ± 0.029
0.837 ± 0.045
0.787 ± 0.064
AMRT-Stage0
0.471 ± 0.020
0.785 ± 0.038
0.770 ± 0.076
w/o BiLSTM
0.364 ± 0.038
0.831 ± 0.018
0.725 ± 0.024
w/o Multi-Horizon
0.330 ± 0.037
0.768 ± 0.095
0.720 ± 0.031
w/o Gate
0.502 ± 0.022
0.802 ± 0.041
0.727 ± 0.036
w/o Gate-Increment
0.497 ± 0.021
0.793 ± 0.042
0.705 ± 0.022
Appendix
Table 16: Model ablations under the ctri-mix protocol. Results are trajectory PCC (mean ± std); bold indicates the best result.
Figure 3: Summary of supplementary experimental results. (a) Dual-source cross-dataset transfer across target-data fractions. (b) Example of simultaneous trajectory and grasp-state decoding on WAY-EEG-GAL: ground-truth and predicted grasp state (top), and the corresponding z -axis trajectory (bottom). (c) Three-dimensional reconstruction of a representative WAY-EEG-GAL trajectory, comparing the prediction with the ground truth in the original movement space.
Protocol
Accuracy
F1
ctri
0.905 ± 0.030
0.848 ± 0.066
ctri-mix
0.868 ± 0.040
0.794 ± 0.056
csub
0.883 ± 0.031
0.844 ± 0.047
Appendix
Table 17: Grasp-state decoding performance on WAY-EEG-GAL under different evaluation protocols. We report accuracy and F1 score as mean ± std.
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong · Department of Biomedical Engineering, School of Medicine, Tsinghua University · School of Biomedical Sciences, The Chinese University of Hong Kong