cs.ROMar 6, 2026

A Multi-Layer Sim-to-Real Framework for Gaze-Driven Assistive Neck Exoskeletons

Authors: Colin Rubow, Eric Brewer, Ian Bales, Haohan Zhang, Daniel S. Brown

Organizations: Dept. of Mechanical Engineering, University of Utah, UT 84112, USA. · Robotics Center, University of Utah, UT 84112, USA. · Kahlert School of Computing, University of Utah, UT 84112, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dropped head syndrome, caused by neck muscle weakness from neurological diseases, severely impairs an individual's ability to support and move their head, causing pain and making everyday tasks challenging. Our long-term goal is to develop an assistive powered neck exoskeleton that restores natural movement. However, predicting a user's intended head movement remains a key challenge. We leverage virtual reality (VR) to collect coupled eye and head movement data from healthy individuals to train models capable of predicting head movement based solely on eye gaze. We also propose a novel multi-layer controller selection framework, where head control strategies are evaluated across decreasing levels of abstraction -- from simulation and VR to a physical neck exoskeleton. This pipeline effectively rejects poor-performing controllers early, identifying two novel gaze-driven models that achieve strong performance when deployed on the physical exoskeleton. Our results reveal that no single controller is universally preferred, highlighting the necessity for personalization in gaze-driven assistive control. Our work demonstrates the utility of VR-based evaluation for accelerating the development of intuitive, safe, and personalized assistive robots.

Figures & tables

Fig. 1: We propose to leverage multiple levels of abstraction and emulation to assist in efficiently finding useful gaze-driven neck exoskeleton controllers. As the initial controller pool moves down through each layer, poor controllers are rejected from further study. Our goal is to arrive at a final set of controller designs suitable for restoring head-neck motions.
Fig. 2: Two examples of the Vector Parameterized control law for either pitch or yaw directions. The vv and cc values are equal for both examples. The control law transitions from a deadzone with zero velocity to a line with proportional velocity. Gray dotted lines mark inflection point locations of the plots. Blue dashed lines show what the control law transitions to and from. The Purple curve on the right has a larger but stricter deadzone.
Fig. 3: Trajectories of an autoregressive rollout for one MLP and two LSTM variations. The y-axis shows the xx (blue), yy (orange), and zz (green) values of the normalized direction of the head where (0,0,1)(0,0,1) represents the forward direction. The x-axis shows the time step tt throughout the trajectory. Solid lines represent the direction of the simulated, rotated “virtual head” during the experiment. Dashed lines represent the ground-truth direction of the head as seen in the dataset. ( Top ) An MLP with two intermediate hidden layers each of size 16 . ( Middle ) An LSTM with a context vector of size 2 . ( Bottom ) An LSTM with a context vector of size 16 .
Fig. 4: Distribution of mean squared error (lower is better) between imputed head direction and ground-truth head direction for various models over trajectories from the held-out test set. Results are grouped by task.
ModelTrainingAutoregressive
MLP0.000112730.370922
MLP-H16_160.000111080.499707
LSTM0.000328090.248416
LSTM-H20.000326780.261365
LSTM-H160.000377221.608792
LSTM-H1280.001908422.125073
TABLE I: MSE Loss - Training and Autoregressive
Fig. 5: Example target trajectories. (a) The target in the Linear Smooth Pursuit task moves in straight lines of random direction and distance. (b) The target in the Arc Smooth Pursuit task moves in circular lines of random direction, distance, and curvature. Both pursuit tasks feature only a single target. (c) Each of the three targets in the Rapid Visual Search task starts some distance from the participant and moves towards the participant. (d) In the Rapid Visual Search Avoidance task there are three targets to fixate on (blue) and three blocking targets to not fixate on (yellow). Each target and blocking cube starts from a plane some distance from the participant and moves towards the participant. The dashed line represents a horizon.
Fig. 6: Stacked bar plot showing aggregate controller preference for each VR task. Since there were 30 subjects, the total number of points distributed across each controller was 30. A bar of greater length, or higher score, correlates to a more preferred controller for that task.
Fig. 7: The Columbia Brace [ 30 ] neck exoskeleton. Links are manufactured from PLA 3D printed parts. Note that in this paper’s implementation, the eye tracking glasses are not attached to the robot, but are worn directly by the user on the Neon glasses frame from Pupil Labs.
Fig. 8: A subject provides preference regarding eye tracking controllers applied to the Columbia Brace [ 30 ] . The eye tracking frame is placed over the subject’s glasses. The subject evaluated each controller by searching for a specific symbol and color combination on the wall. The clutter on the wall provides distraction to make the task more difficult and realistic.
Quadrant Vector MLP
subject 1st comparison 2nd comparison
0 1117 1411
1 1318 1915
2 104 1316
3 68 88
4 1619 1820
Fig. 9: Task score and preference rankings for neck exoskeleton experiment. The colored cells represent a trial using a particular controller; green for Baseline Quadrant, blue for Vector Parameterized model, and orange for MLP. Reading from left to right corresponds to the order each controller was given to the subjects. The score (number of found symbols) is printed on each cell which is colored with a shading that corresponds to the normalized score for that subject. Thus, for a row, a darker shaded cell corresponds to a higher score. A bold score indicates the controller was preferred in that comparison. If both scores are bold then equal preference was reported. Overall, three subjects preferred Vector Parameterized, four preferred MLP, and five preferred Baseline Quadrant.
Fig. 10: User preferences per user and averaged over all users. Darker shading indicates a more preferred controller. Linear and Arc Pursuit were grouped together into Pursuit Tasks because they are similar. Rapid Search and Rapid Avoidance were also grouped together due to their similar natures. The preference over all tasks is also present in the third column and the overall controller preference during the physical neck exoskeleton experiment is present in the fourth column. Preference averaged over all subjects is presented in the bottom row.
More in this topic

Assistive Robotics

Assistive robotics develops robots and wearable devices that support mobility, daily activities, rehabilitation, and care.

See all 199 papers

Explore similar work

CardsList

  1. A Biomimetic Gaze Control to Restore Head-Neck Movements

    Oct 2, 2026Colin Rubow, Haohan ZhangAssistive RoboticsEye Tracking

    Department of Mechanical Engineering and Robotics Center, University of Utah, Salt Lake City, UT 84112, USA. · Rocky Mountain Center for Occupational and Environmental Health, University of Utah, Salt Lake City, UT 84112, USA.

  2. Fabric Pneumatic Artificial Muscle-Based Head-Neck Exosuit: Design, Modeling, and Evaluation

    Mar 13, 2026Katalin Schäffer, Ian Bales, Haohan Zhang +1Soft Pneumatic ActuatorsAssistive Robotics

    Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN 46556, USA. · Faculty of Information Technology and Bionics, Pázmány Péter Catholic University, 1083 Budapest, Hungary. · Department of Mechanical Engineering and Robotics Center, University of Utah, Salt Lake City, UT 84112 USA.

  3. Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE

    Oct 14, 2026Yuhang Hu, Junlin Lyu, Yuan Fang +6Robot Motion GenerationHumanoid Robot Control

    Creative Machines Lab, Department of Mechanical Engineering, Columbia University