A Multi-Layer Sim-to-Real Framework for Gaze-Driven Assistive Neck Exoskeletons
Authors: Colin Rubow, Eric Brewer, Ian Bales, Haohan Zhang, Daniel S. Brown
Organizations: Dept. of Mechanical Engineering, University of Utah, UT 84112, USA. · Robotics Center, University of Utah, UT 84112, USA. · Kahlert School of Computing, University of Utah, UT 84112, USA.
Dropped head syndrome, caused by neck muscle weakness from neurological diseases, severely impairs an individual's ability to support and move their head, causing pain and making everyday tasks challenging. Our long-term goal is to develop an assistive powered neck exoskeleton that restores natural movement. However, predicting a user's intended head movement remains a key challenge. We leverage virtual reality (VR) to collect coupled eye and head movement data from healthy individuals to train models capable of predicting head movement based solely on eye gaze. We also propose a novel multi-layer controller selection framework, where head control strategies are evaluated across decreasing levels of abstraction -- from simulation and VR to a physical neck exoskeleton. This pipeline effectively rejects poor-performing controllers early, identifying two novel gaze-driven models that achieve strong performance when deployed on the physical exoskeleton. Our results reveal that no single controller is universally preferred, highlighting the necessity for personalization in gaze-driven assistive control. Our work demonstrates the utility of VR-based evaluation for accelerating the development of intuitive, safe, and personalized assistive robots.
Figures & tables
Fig. 1: We propose to leverage multiple levels of abstraction and emulation to assist in efficiently finding useful gaze-driven neck exoskeleton controllers. As the initial controller pool moves down through each layer, poor controllers are rejected from further study. Our goal is to arrive at a final set of controller designs suitable for restoring head-neck motions.
Fig. 2: Two examples of the Vector Parameterized control law for either pitch or yaw directions. The v and c values are equal for both examples. The control law transitions from a deadzone with zero velocity to a line with proportional velocity. Gray dotted lines mark inflection point locations of the plots. Blue dashed lines show what the control law transitions to and from. The Purple curve on the right has a larger but stricter deadzone.
Fig. 3: Trajectories of an autoregressive rollout for one MLP and two LSTM variations. The y-axis shows the x (blue), y (orange), and z (green) values of the normalized direction of the head where (0,0,1) represents the forward direction. The x-axis shows the time step t throughout the trajectory. Solid lines represent the direction of the simulated, rotated “virtual head” during the experiment. Dashed lines represent the ground-truth direction of the head as seen in the dataset. ( Top ) An MLP with two intermediate hidden layers each of size 16 . ( Middle ) An LSTM with a context vector of size 2 . ( Bottom ) An LSTM with a context vector of size 16 .
Fig. 4: Distribution of mean squared error (lower is better) between imputed head direction and ground-truth head direction for various models over trajectories from the held-out test set. Results are grouped by task.
Model
Training
Autoregressive
MLP
0.00011273
0.370922
MLP-H16_16
0.00011108
0.499707
LSTM
0.00032809
0.248416
LSTM-H2
0.00032678
0.261365
LSTM-H16
0.00037722
1.608792
LSTM-H128
0.00190842
2.125073
TABLE I: MSE Loss - Training and Autoregressive
Fig. 5: Example target trajectories. (a) The target in the Linear Smooth Pursuit task moves in straight lines of random direction and distance. (b) The target in the Arc Smooth Pursuit task moves in circular lines of random direction, distance, and curvature. Both pursuit tasks feature only a single target. (c) Each of the three targets in the Rapid Visual Search task starts some distance from the participant and moves towards the participant. (d) In the Rapid Visual Search Avoidance task there are three targets to fixate on (blue) and three blocking targets to not fixate on (yellow). Each target and blocking cube starts from a plane some distance from the participant and moves towards the participant. The dashed line represents a horizon.
Fig. 6: Stacked bar plot showing aggregate controller preference for each VR task. Since there were 30 subjects, the total number of points distributed across each controller was 30. A bar of greater length, or higher score, correlates to a more preferred controller for that task.
Fig. 7: The Columbia Brace [ 30 ] neck exoskeleton. Links are manufactured from PLA 3D printed parts. Note that in this paper’s implementation, the eye tracking glasses are not attached to the robot, but are worn directly by the user on the Neon glasses frame from Pupil Labs.
Fig. 8: A subject provides preference regarding eye tracking controllers applied to the Columbia Brace [ 30 ] . The eye tracking frame is placed over the subject’s glasses. The subject evaluated each controller by searching for a specific symbol and color combination on the wall. The clutter on the wall provides distraction to make the task more difficult and realistic.
Quadrant
Vector
MLP
subject
1st comparison
2nd comparison
0
11
17
14
11
1
13
18
19
15
2
10
4
13
16
3
6
8
8
8
4
16
19
18
20
Fig. 9: Task score and preference rankings for neck exoskeleton experiment. The colored cells represent a trial using a particular controller; green for Baseline Quadrant, blue for Vector Parameterized model, and orange for MLP. Reading from left to right corresponds to the order each controller was given to the subjects. The score (number of found symbols) is printed on each cell which is colored with a shading that corresponds to the normalized score for that subject. Thus, for a row, a darker shaded cell corresponds to a higher score. A bold score indicates the controller was preferred in that comparison. If both scores are bold then equal preference was reported. Overall, three subjects preferred Vector Parameterized, four preferred MLP, and five preferred Baseline Quadrant.
Fig. 10: User preferences per user and averaged over all users. Darker shading indicates a more preferred controller. Linear and Arc Pursuit were grouped together into Pursuit Tasks because they are similar. Rapid Search and Rapid Avoidance were also grouped together due to their similar natures. The preference over all tasks is also present in the third column and the overall controller preference during the physical neck exoskeleton experiment is present in the fourth column. Preference averaged over all subjects is presented in the bottom row.
Powered neck exoskeletons are developed to restore head-neck motions for those with neck muscle weakness. Current methods using hand-held input devices to control these robotic systems are unintuitive or non-viable for these populations. Alternatively, the eyes coordinate with the head to accomplish many daily tasks. Therefore, movements of the eyes can be a useful signal to control the neck exoskeletons without needing to use the hands. However, previous control models lack biobehavioral basis as well as means to personalize the control behavior, causing unnatural head-neck movements. This paper presents a novel eye-tracking controller design leveraging the vestibulo-ocular reflex to emulate natural eyehead coordination. Personalization tools were also designed, consisting of sliders to allow users to adjust the parameters of this controller. Here, we present the technical design and validation of this system in human users. Compared to the previous eye-tracking model, the new controller provides natural head motions personalized to the user through more natural velocity profiles and a 24 percent average increase of accurate motion intent detection. This system provides a powerful platform for studying eye-tracking controllers in the context of personalization and preference, and ultimately restoring the lost head-neck mobility in patient populations.
Colin Rubow, Haohan Zhang
Department of Mechanical Engineering and Robotics Center, University of Utah, Salt Lake City, UT 84112, USA. · Rocky Mountain Center for Occupational and Environmental Health, University of Utah, Salt Lake City, UT 84112, USA.
Wearable exosuits assist human movement in tasks ranging from rehabilitation to daily activities; specifically, head-neck support is necessary for patients with certain neurological disorders. Rigid-link exoskeletons have shown to enable head-neck mobility compared to static braces, but their bulkiness and restrictive structure inspire designs using "soft" actuation methods. In this paper, we propose a fabric pneumatic artificial muscle-based exosuit design for head-neck support. We describe the design of our prototype and physics-based model, enabling us to derive actuator pressures required to compensate for gravitational load. Our modeled range of motion and workspace analysis indicate that the limited actuator lengths impose slight limitations (83% workspace coverage), and gravity compensation imposes a more significant limitation (43% workspace coverage). We introduce compression force along the neck as a novel, potentially comfort-related metric. We further apply our model to compare the torque output of various actuator placement configurations, allowing us to select a design with stability in lateral deviation and high axial rotation torques. The model correctly predicts trends in measured data where wrapping the actuators around the neck is not a significant factor. Our test dummy and human user demonstration confirm that the exosuit can provide functional head support and trajectory tracking, underscoring the potential of artificial muscle-based soft actuation for head-neck mobility assistance.
Katalin Schäffer, Ian Bales, Haohan Zhang +1
Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN 46556, USA. · Faculty of Information Technology and Bionics, Pázmány Péter Catholic University, 1083 Budapest, Hungary. · Department of Mechanical Engineering and Robotics Center, University of Utah, Salt Lake City, UT 84112 USA.
Gaze coordinates attention in human interaction through coupled eye, head, and eyelid motion. A humanoid robot must choose where to look while generating a suitable movement toward that target; fixed tracking rules alone do not model the temporal structure of human gaze. We separate these tasks with an asynchronous vision-language model that selects a scene referent, a reactive perception module that maintains fixation, and a Realistic Gaze Transformer (RGT) that generates target-conditioned motion. RGT is a Transformer-VAE trained on markerless human recordings retargeted to the robot's 11-DoF joint space: self-supervised pretraining encodes recent full-head history, and autoregressive fine-tuning exposes the model to its own predictions during rollout. At inference, a four-dimensional binocular-eye trajectory conditions coordinated neck and eye motion; an auxiliary blink predictor triggers eyelid templates during playback. On a two-second benchmark in which all controllers receive the same ground-truth target, RGT-Base obtains the lowest reported overall seven-joint MSE, and physical demonstrations show the integrated targeting and control pipeline. The evidence establishes robot-space kinematic fidelity and real-time RGT inference; perceived naturalness and open-vocabulary target-selection accuracy remain questions for controlled evaluation.
Yuhang Hu, Junlin Lyu, Yuan Fang +6
Creative Machines Lab, Department of Mechanical Engineering, Columbia University