NI-ACT: A Neural Intent Interface for Brain-Embodied Manipulation
Authors: Yinghao Li, WeiHsien Lee, Zeyu Zhang, Canyang Zhao, Ke Fang, Xing Li, Bing Liu
Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · MindTrix · University of Chinese Academy of Sciences, Beijing, China
Intracortical brain-computer interfaces can drive robotic arms, but they keep the user inside the fast sensorimotor loop: every approach, correction, and grasp must be neurally specified. Whether intracortical population activity can instead serve as a task-level condition for a learned embodied policy, without symbolic commands or hand-designed skill libraries, remains unknown. NI-ACT addresses this question with a hierarchical brain-embodied framework: a brief motor-cortical signal is read once as a persistent intent, and a visuomotor policy then closes the perception-action loop from vision and proprioception. In a counterfactual grasping task where the scene cannot specify the target, the neural intent alone determined which object the robot grasped; replacing the intent changed the behavior accordingly. Passing the full population latent as a dedicated intent token, rather than a one-hot decoded direction, raised grasp completion from 69% to 81% over 132 held-out scenes, whereas a continuous direction alone did not improve on a one-hot label. Where the signal was read also mattered: in one dual-array session, with the readout window matched, premotor (PMd) intent completed more grasps than primary motor (M1) intent (77% vs. 53%), and the conventional post-cue M1 readout reached 42%. Preparatory activity was itself sufficient: in a center-out task, intent read only from PMd before the go cue (that is, before any movement) drove complete grasps, and in one recording it completed more grasps than the same animal's post-cue readout. These results show that how a neural signal is represented, and where and when it is read, jointly determine how effectively it conveys movement goals to an embodied agent.
Figures & tables
Figure 1: The NI-ACT framework. (A) Control division. Conventional BCIs decode continuous velocity commands at every time step and keep the user inside the fast control loop. In NI-ACT, cortical activity supplies a task-level intent token once per task, sent at task start, and an embodied policy closes the fast perception–action loop with the robot and its environment. (B) Network architecture. A temporal convolutional network (TCN) maps binned PMd and M1 spikes to a 128-D population latent, a 2-D goal direction, and 7-class direction logits; an intent interface (Hard-7, Soft-2D, or Full-Latent) passes one of these to an action-chunking transformer (ACT) that also observes two cameras and the joint state, and the policy predicts 32 future joint targets. Fig. 3 a gives how each interface enters the policy.
Interface
Dim.
Continuous
Preserves geometry
Decoder needed
Hard-7
7
No
No
Yes (argmax)
Soft-2D
2
Yes
Partial (direction only)
Yes (regression)
Full-Latent
128
Yes
Yes †
No
Table 1: Design space of neural intent interfaces.
Figure 2: What the three interfaces pass to the policy, read from one frozen encoder (monkey J, PMd, movement-onset window) on all 426 held-out trials. (a) Hard-7: the one-hot vector of argmaxℓ , shown for one example trial per direction; each row has a single nonzero element. (b) Soft-2D: the unit direction v^ of every held-out trial on the unit circle; dotted lines mark the true directions. (c) Full-Latent: the 128-D latent z of the same example trials as in (a); each direction is a distinct pattern across all dimensions.
Figure 3: Architecture and data protocol of NI-ACT. (a) The frozen temporal convolutional encoder maps a window of binned spikes to a 128-D population latent z , from which a direction head reads a unit goal direction v^ and a class head reads 7-way logits ℓ . The interface G selects one output: Hard-7 (one-hot of argmaxℓ ) and Soft-2D ( v^ ) are appended to the 9-D joint state, giving 16-D and 11-D state inputs; Full-Latent ( z ) is kept separate from the joint state, projected from 128 to 512 dimensions, and enters the transformer encoder as its own intent token. The interfaces therefore differ in content, dimensionality, and injection pathway. Both camera images pass through one shared ImageNet-initialized ResNet-18, giving 7×7 feature maps (49 tokens per camera). With the VAE disabled, the latent token is constant. The encoder (4 layers) attends over all tokens; the decoder (1 layer, 32 learned queries) and a linear head predict 32 future 9-D joint targets. (b) Neural trials are split into training and held-out sets before any pairing. Each of the 207 counterfactual demonstrations is paired with a training trial of the same target direction, and the resulting intent is held constant over every frame; ACT is trained by behavior cloning, one policy per interface and neural source. At deployment, each of the 132 held-out scenes receives a held-out trial of its target direction; the intent is read once and held fixed while the policy replans every 32 steps. No neural trial is shared between training and evaluation. Trial counts refer to the monkey-J recording.
Monkey
Dataset (DANDI)
Units (PMd/M1)
Split (Tr/Val/Te)
Readout Windows (ms)
J
MC_Maze (000128)
182 (–/–)
1505/300/449
−560 to +440
J
Dual-array (000070)
96 / 96
1433/286/426
pre-go, post-go, move
M
Session A (000688)
31 / 18
80% / 20%
go 0–1000
M
Session B (000688)
60 / 30
80% / 20%
go 0–1000
T
5 sessions (000688)
(PMd only)
80% / 20%
early, late, combined
Table 2: Summary of neural datasets and readout configurations.
Figure 4: Motor-cortical populations encode the reach goal. (a) Held-out seven-way accuracy of the frozen encoders used for closed-loop control; dots show encoder seeds. (b,c) Monkey M session A, accuracy by area and bin width around movement (b) and in the delay period before the go cue (c), mean ± s.d. over three seeds. Dotted line: chance.
Figure 5: Neural intent determines which object the robot grasps. (a) Grasp success on 30 held-out scenes with decoded intent (colored) and substituted intent (orange), with Wilson 95% intervals. (b) Wrong-object grasps. Every condition uses a separately trained policy and the same test scenes.
Figure 6: A held-out NI-ACT rollout (MC_Maze, Full-Latent, target: red cube). The neural intent is read once from the scene’s held-out trial; the policy then executes the grasp from front-camera (top) and wrist-camera (bottom) observations.
Figure 7: The neural signal selects the target; the policy executes it. (a) Fraction of held-out scenes in which the robot approached the intended cube, with decoded (filled) and substituted (open) intent; error bars: Wilson 95% CI; dashed line: chance (1/3). (b) Grasp completion among scenes with correct selection (counts shown). A and B denote the two monkey-M sessions; B PMd and B pre-go use the PMd-only and pre-go-window inputs of session B.
Figure 8: Full population latents versus decoded directions on identical scenes and held-out neural trials. (a) Grasp success on the original 30 held-out scenes, on 102 new scenes collected after training, and pooled (Wilson 95% intervals). (b) Discordant scenes over the 132 pooled scenes (23 grasped only with Full-Latent, 7 only with Hard-7; 18 failed by both; exact McNemar p=0.005 ). (c) Pooled grasp success by target direction; n is the number of scenes per direction; see Table 5 for direction-wise tests.
Condition
Content
Dim.
Pathway
Success
Full-Latent
trial latent
128
own token
80 (78.4%, 69.5–85.3)
Class prototype
class only
128
own token
83 (81.4%, 72.7–87.7)
Hard-7
class only
7
appended to state
70 (68.6%, 59.1–76.8)
Hard-7-token
class only
7
own token
55 (53.9%, 44.3–63.3)
Table 3: Packaging controls on the 102 scenes of the second test set (decoded intent; one policy per condition, each trained once). Class prototype: for each decoded direction, the mean 128-D latent of the training trials that the frozen encoder assigns to it. Hard-7-token: the Hard-7 one-hot delivered through the dedicated-token pathway of Full-Latent. Success: intended cube lifted, with Wilson 95% CI. No condition lifted a wrong cube.
Held-out-trial draw
Full-Latent
Hard-7
Difference
Primary (used in pooling)
27/30
21/30
6
Draw 2
21/30
19/30
2
Draw 3
23/30
21/30
2
All three draws combined
71/90
61/90
10
Table 4: Robustness of the representation comparison to the assignment of held-out neural trials (MC_Maze, the 30 scenes of test set 1, both policies fixed). Each row independently re-assigns a held-out trial of the correct direction to every scene. Entries: grasped / scenes.
Direction
n
Full-Latent
Hard-7
Discordant
p
pHolm
down-left
30
23 (59–88)
16 (36–70)
8/1
0.039
0.273
left
12
11 (65–99)
11 (65–99)
0/0
1.000
1.000
up-left
16
9 (33–77)
4 (10–49)
6/1
0.125
0.625
up
18
14 (55–91)
7 (20–61)
9/2
0.065
0.393
up-right
18
15 (61–94)
15 (61–94)
0/0
1.000
1.000
right
15
12 (55–93)
15 (80–100)
0/3
0.250
1.000
Table 5: Direction-wise comparison of Full-Latent and Hard-7 on the 132 held-out scenes (exploratory). Entries in the Full-Latent and Hard-7 columns: grasped scenes, with the Wilson 95% CI of the success rate in percent. Discordant: scenes grasped only with Full-Latent / only with Hard-7. p : exact McNemar test per direction; pHolm : Holm-corrected over the seven directions.
Figure 9: The reach goal resides predominantly in PMd (monkey J, maze task, DANDI 000070). Colors: PMd blue, M1 orange-red, PMd+M1 gray. (a) Held-out seven-way accuracy of the frozen encoder for each area and readout window; windows are aligned to the go cue (go) or to movement onset (move), in ms. (b) Accuracy in 100 ms bins around movement onset (linear decoder, five-fold cross-validation). (c) After movement onset, PMd and M1 decoding of the goal (all trials, straight reaches only) and of the initial hand direction. Dotted lines: chance.
Window (ms)
PMd
M1
PMd+M1
go −400 to 0
68.9 / 59.2
41.7 / 31.4
70.0 / 62.0
go 0 to 500
86.7 / 79.8
69.2 / 67.6
85.0 / 79.6
go 500 to 1000
86.4 / 84.3
72.7 / 70.9
85.7 / 85.9
go 0 to 1000
93.4 / 88.5
81.1 / 77.0
92.0 / 91.3
move −500 to 500
99.3 / 97.2
91.6 / 90.1
99.3 / 97.7
Table 6: Encoder accuracy by area and readout window (monkey J, maze). Validation / held-out test, %.
M1
PMd
PMd+M1
go 0–1000
move ± 500
move ± 500
move ± 500
(conventional)
(window-matched)
Encoder test accuracy
77.0%
90.1%
97.2%
97.7%
Scene direction decoded
101/132
120/132
126/132
129/132
Set 1 (30)
14/30
14/30
22/30
21/30
Set 2 (102)
42/102
56/102
80/102
87/102
Table 7: Closed-loop grasping with intent from PMd or M1 (monkey J, Full-Latent, 132 held-out scenes). Entries: grasped / scenes. The window-matched M1 policy reads M1 around movement onset, the same window as PMd, so that it differs from PMd only in cortical area.
Figure 10: Goal information accumulates in PMd (monkey T, five sessions). Eight-direction accuracy from early, dimension-matched early (split), late, and combined windows during preparation (a, target-aligned, before the go cue) and execution (b, go-aligned). Gray lines: sessions; markers: mean ± s.e.m.; dashed line: chance.
Motor brain-computer interfaces (BCIs) aim to decode motor intention, enabling people with paralysis to control external devices. Neural motor decoding typically learns task-specific mappings from neural activity to kinematics, yet remains constrained by scarce paired neural-action data. We propose BrainVLA, a framework that enables neural motor decoding by drawing on a pretrained vision-language-action (VLA) model through language-mediated alignment. BrainVLA mitigates reliance on scarce paired neural-action data by leveraging VLA policies. We first construct VLA-compatible datasets including paired neural activity, action signals, language instructions, and rendered visual observations. Then, we adapt the OpenVLA-OFT policy to the target action spaces through LoRA fine-tuning. To establish an effective interface through which neural activity can convey motor intention to adapted VLA policies and guide action generation, we train a neural encoder via neural-language alignment, using language representations as semantic targets to capture latent motor intent from neural activity. The resulting neural representations serve as an endogenous intention signal to guide VLA policies to generate executable actions, while visual observations provide complementary information about the evolving task state. BrainVLA is evaluated on two neural motor datasets with different action dimensionalities using causal rollout decoding. It outperforms the evaluated baselines in cross-session decoding R2 and task success rate, while demonstrating high training data efficiency. These results establish a route for neural motor decoding to draw on large-scale robotic priors through brain-conditioned VLA policies.
Luyao Jin, Running Zhao, Huan Zhao +2
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong · Department of Electrical and Computer Engineering, University of Hong Kong · School of Biomedical Sciences, The Chinese University of Hong Kong
Robot policies still fail at everyday manipulation: an unfamiliar task, an object never held before, a scene that changes mid-motion, a skill that should improve with use. They fail despite pretraining on over a million trajectories, pooled from dozens of robots at great expense, while a child acquires all of these abilities within a few years of learning from a single body. This survey argues that the gap is one of design, and proposes designs for the next generation of robot models, drawn from how the nervous system solves the same problems. From the body we take compliance: the policy commands stiffness alongside position, so the controller yields where the task allows and presses where it must, which greatly lowers the demand for precise position output. From the visual system, where a fast pathway for action runs in parallel with the slow pathway for recognition, we take a dedicated fast vision system for motor behavior: it precomputes the quantities the policy needs and drives a reactive policy at camera rate. From the cerebellum we take a predictor beside the policy that forecasts the consequence of each command and recalibrates from its own error. From motor cortex we take a single model that composes action units from primitives, verifies each at a sensory boundary, practices the weakest unit with a hypothesis in mind, and reasons each attempt closer to the goal. Proven primitives are then consolidated from code into weights, so they run fluently and respond quickly the next time. Together these describe a robot that acts on its own initiative, the basis of deployment and recursive self-improvement, and we offer each proposal as a hypothesis to be tested.
Decoding continuous motor trajectories from neural activity is essential for developing practical brain-computer interfaces (BCIs). However, current neural decoders are constrained by the limited scale and heterogeneity of neural recordings. In contrast, behavioral data can be collected more readily and at substantially larger scale from humans, animals, simulations, and robotic systems. Here, we introduce NeuroPB, a framework that scales neural decoding by transferring knowledge from pretrained behavioral representations. NeuroPB first pretrains a motor encoder on large-scale motor behavior data and then aligns neural activity with the resulting behavioral representation space using a limited set of paired neural-behavioral recordings. A neural encoder and lightweight motor decoder are subsequently optimized to reconstruct continuous movement from the aligned neural representations. Across multiple macaque motor datasets, behavioral pretraining improves trajectory decoding, including an 11% R2 increase on center-out and 8% on random-target compared with training the motor encoder from scratch. Notably, pretraining on robotic trajectories achieves performance comparable to pretraining on macaque trajectories, demonstrating that transferable kinematic structure is shared across biological and artificial models. Moreover, decoding performance improves as the scale and diversity of robotic pretraining data increase, when the amount of neural data is fixed. Pretraining also enhances generalization across recording sessions, subjects, and motor tasks, with only 10% calibration needed to match training from scratch. Overall, these results establish behavioral pretraining as a scalable source for neural decoding and provide a promising route toward high-performance and calibration-efficient BCIs under limited neural data.
Luyao Jin, Yonghao Song, Huan Zhao +2
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong · Department of Biomedical Engineering, School of Medicine, Tsinghua University · School of Biomedical Sciences, The Chinese University of Hong Kong