Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery
Authors: Yichang Liu, Tianyu Wang, Ziyi Ye, Yawei Li, Yu-Gang Jiang, Shouyan Wang, Yanwei Fu
Organizations: Institute of Science and Technology for Brain-inspired Intelligence (ISTBI), Fudan University · School of Data Science, Fudan University · Shanghai Innovation Institute · Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · ETH Zürich
We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement. Motivated by the promise of BCI-driven robotics to enhance human-robot interaction, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner within an online streaming pipeline. This establishes a dual-channel intent interface that translates visual intent into robotic actions, with VI identifying objects for grasping and MI determining placement poses, enabling intuitive control over both what to grasp and where to place. The system operates solely on EEG via a cue-free imagery protocol, achieving integration and online validation. Implemented on a Base robotic platform and evaluated across diverse scenarios, including occluded targets or varying participant postures, the system achieves online decoding accuracies of 40.23% (VI) and 62.59% (MI), with an end-to-end task success rate of 20.88%. These results demonstrate that high-level visual cognition can be decoded in real time and translated into executable robot commands, bridging the gap between neural signals and physical interaction, and validating the flexibility of a purely imagery-based BCI paradigm for practical human-robot collaboration.
Figures & tables
Fig. 1 : High-level cognitive control pipeline for EEG-based robotic manipulation. The framework consists of three main components: (1) Offline EEG data collection : visual imagery (VI) and motor imagery (MI) EEG data are collected and labeled to train visual and motor decoders. (2) Online EEG acquisition : A dual-channel system is deployed in which VI-EEG is decoded into grasping intentions and MI-EEG determines placement positions, enabling real-time task-level command generation. (3) Robotic control : the trained decoders drive a robotic arm to perform the grasp and place task in real-world, demonstrating the seamless mapping from high-level visual cognition to physical manipulation.
Fig. 2 : Visual Imagery and Motor Imagery Paradigms.
Subject
EEGNet
MLP
RGNN
SVM
DT
Visual Perception
00
0.4844
0.4767
0.4622
0.4067
0.4033
01
0.4911
0.4667
0.4278
0.4211
0.3711
02
0.4678
0.4522
0.4155
0.3833
0.3667
03
0.4978
0.4533
0.4322
0.4011
0.3911
04
0.4911
0.4889
0.4633
0.4633
0.3978
TABLE I : Offline classification results for different models in VP, VI and MI tasks. The mean decoding accuracy across all subjects is reported for each model in different frequency conditions across three tasks. Bolded values indicate the highest accuracy in each row, while underlined values denote the second highest accuracy to report competitive alternatives for model selection.
Fig. 3 : Offline results for three tasks: (a) Visual Perception, (b) Visual Imagery, and (c) Motor Imagery. Points show per-subject accuracies across models and frequency settings; the dashed line denotes chance level (VP/VI: 33.3%, MI: 50%). MI consistently outperforms VP/VI, and all tasks exceed chance.
Fig. 4 : Comparison of decoding accuracies for VI and MI. Motor imagery achieves higher accuracy than the visual EEG tasks overall, and within the visual paradigm, VP slightly outperforms VI.
Fig. 5 : Evoked EEG responses to fruit visual imagery. Brain activity patterns during VI for three stimuli in Sub00 (59 channels): (a) Apple, (b) Banana, (c) Orange. Responses vary over time yet remain broadly similar across categories, reflecting the fine-grained nature of VI decoding.
Online Base Demonstration results
Subject
VI-online Task
MI-online Task
Top1 Acc. (%)
Top2 Acc. (%)
Overall Avg. (%)
Top1 Acc. (%)
Top2 Acc. (%)
Overall Avg. (%)
01
63.64 (RGNN/40)
44.44 (RGNN/60)
47.14
75.00 (RGNN/60)
60.00 (MLP/40)
61.67
02
50.00 (EEGNet/100)
39.13 (MLP/100)
40.82
65.22 (MLP/100)
60.00 (MLP/40)
58.41
03
50.00 (MLP/60)
33.67 (RGNN/60)
38.89
60.00 (RGNN/100)
53.85 (RGNN/60)
54.62
04
37.50 (RGNN/100)
33.67 (MLP/60)
34.72
66.67 (MLP/60)
60.00 (MLP/40)
60.00
TABLE II : Online system results. Top: Base demo performance per subject on online VI/MI tasks across model/frequency settings (e.g., 63.64 (RGNN/40) denotes accuracy using the RGNN model at 40 Hz) . For each subject, we report the top-2 online accuracies and the overall mean. Bolded values indicate the row maximum, while underlined values mark the second highest. Middle: Overall online system accuracy, with online VI/MI accuracy representing the mean top-2 accuracies across all subjects. Bottom: Overall system runtime statistics.
Fruit Catalogue
Success Rate (%)
Apple
89.83
Banana
55.84
Orange
84.44
Overall Success Rate
76.11
TABLE III : Grasp Success Rates for Different Fruits
Fig. 6 : Online Demos for extended application scenarios. (a) Hidden Object Interaction: Using only VI-EEG, the subject guides the system to identify and reveal a hidden object, revealing an ability to decode mental imagery without visual cues. (b) Direct Human-Robot Interaction: By imagining the object and controlling hand movements through Visual Imagery and Motor Imagery, the subject interacts seamlessly with the robot, which places the object directly into their hand.
Department of Computer Science, Imam Mohammad Ibn Saud Islamic University, Saudi Arabia · Department of Computer Science, Durham University, UK · Department of Computer Science, King Saud University, Saudi Arabia +1
Department of Computer Science City University of Hong Kong Hong Kong, China · Department of Computer Science Florida State University Tallahassee, United States