Gaze-Assisted Supervisory Control for Lunar Robotic Arm Teleoperation Under Concurrent Monitoring
Authors: Jiayu He, Yifan Li, Oliver R. Runswick, Ernest N. Kamavuako, Yunjie Jia, Peter D. Hodkinson, Yang Gao
Organizations: Department of Engineering, King’s College London, London, UK · Department of Psychology, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK · Spacesuit Physiology Laboratory, Centre for Human and Applied Physiological Sciences, King’s College London, London, UK · Centre for AI Robotics in Space Sustainability, The Hong Kong University of Science and Technology, Hong Kong, China
Rover-mounted robotic arms can extend human reach during lunar sample handling, but continuous joystick teleoperation requires repeated visual-motor alignment while operators may also need to monitor system states. This paper presents a gaze-assisted supervisory control framework in which gaze nominates a rock or placement target, explicit confirmation authorizes the action, and the robot performs bounded autonomous grasp-and-place execution. The framework was evaluated against manual joystick teleoperation in a mission-analog virtual reality study with 16 participants. The study used a 2 x 2 x 3 within-subject design varying control mode, placement difficulty, and concurrent monitoring load. Mean recorded task completion time decreased from 47.65 to 18.43 s per rock, with 72.5% of this difference arising during grasping. Under the tested auditory monitoring conditions, gaze-assisted supervisory control showed less average proportional slowing, with no detected control-mode difference in monitoring-response accuracy or speed. The rate of gaze shifts between core areas of interest decreased from 49.98 to 27.73 shifts/min, while the raw NASA Task Load Index score decreased from 32.54 to 18.26. However, the benefit was task dependent. In precision placement, manual control yielded lower mean placement error and a higher placement success rate. These findings support a mixed-initiative allocation in which gaze enables rapid task-level intention specification and autonomous execution, while precision-critical stages retain manual refinement.
Figures & tables
Figure 1: Conceptual illustration of manual robotic teleoperation and concurrent operator demands in a spaceflight setting. (a) An astronaut operating a robotic system using a virtual reality (VR) headset and haptic joystick, while attending to additional information, with visual, auditory, and manual demands highlighted in red, blue, and purple, respectively. (b) The teleoperated robotic arm. Source images: European Space Agency (ESA)/National Aeronautics and Space Administration (NASA), adapted from Pilotes and Pilote timelapse under the ESA Standard Licence.
Figure 2: Overview of the proposed gaze-assisted supervisory control framework and the relationship between its interface and execution components. (a) System-level allocation of task-level intention specification and low-level robot execution. (b) Expansion of the gaze-assisted interface, showing the transformation from gaze geometry to an explicitly confirmed task target. (c) Expansion of the autonomous execution module, showing the finite-state grasping and placement sequences.
Figure 3: Binocular VR views, controller mappings, and task sequence used in the mission-analog study. (a) Paired left- and right-eye images presented through the VR headset under manual joystick teleoperation and gaze-assisted supervisory control, together with the Xbox controller mappings. In the gaze-assisted view, the red point indicates the real-time gaze position, while the identified candidate target is highlighted using a visual mask. The controller supports end-effector translation and rotation, gripper operation, explicit target confirmation, and concurrent monitoring responses. (b) Task-completion process, with gaze-assisted supervisory control used as an illustrative example. The sequence progresses through grasp-target selection, robotic-arm grasping, placement-target selection, and robotic-arm placement.
Figure 4: Experimental apparatus and operator-station configuration used in the controlled VR study. (a) Schematic of the principal hardware components, comprising a PICO 4 Enterprise VR headset with integrated eye tracking, an Xbox controller, and a Linux simulation workstation. (b) Staged demonstration of the operator-station configuration.
Figure 5: Placement-platform configurations used to manipulate placement difficulty in the VR study. The highlighted grid cell indicates the designated target region in the precision-placement condition.
Figure 6: Per-rock TCT under manual joystick teleoperation and gaze-assisted supervisory control. Coarse and precision placement are shown separately across no (L0), routine (L1), and emergency (L2) concurrent monitoring. Each observation represents one participant’s mean completion time across the three rocks in that condition; boxplots and half-violin plots show the distributions across participants. Bracketed p -values are from unadjusted two-sided paired t -tests ( n=16 ); all six Holm-adjusted p -values were <.001 . Cohen’s dz values for L0, L1, and L2 were 5.258, 3.618, and 3.214 for coarse placement, and 3.681, 4.813, and 3.992 for precision placement, respectively.
Control mode
Grasping time (s)
Placement time (s)
Total TCT(s)
Manual joystick
27.8 (4.6)
19.9 (3.4)
47.6 (7.3)
Gaze-assisted
6.6 (0.5)
11.8 (3.8)
18.4 (3.7)
Table 1: Per-rock TCT by control mode and task phase. Values are means across 16 participants, averaged over placement difficulties, monitoring-load conditions, and rocks; standard deviations (SDs) are shown in parentheses.
Figure 7: Per-rock TCT across coarse and precision placement under (a) manual joystick teleoperation and (b) gaze-assisted supervisory control. Each point represents one participant’s mean across the L0–L2 monitoring conditions, and connecting lines link the same participant’s two difficulty conditions. Diamonds and thicker lines indicate group means; boxes show the interquartile ranges. The connecting lines describe paired changes between the two difficulty levels, not a continuous difficulty-response relationship.
Figure 8: Primary-task DTC under manual joystick teleoperation and gaze-assisted supervisory control during coarse and precision placement. DTC represents the percentage change in TCT under concurrent monitoring relative to the corresponding no-monitoring condition (L0). Positive values indicate slower completion under concurrent monitoring, whereas negative values indicate faster completion than in L0; the dashed horizontal line denotes zero DTC. Points show individual participant values, white diamonds indicate means, and boxplots and half-violin profiles summarize the distributions. Bracketed p -values are from unadjusted two-sided paired t -tests ( n=16 ); asterisks indicate unadjusted p<.05 . For coarse L1, coarse L2, precision L1, and precision L2, respectively, Holm-adjusted p -values are .118, .118, .101, and .632, and Cohen’s dz values are 0.536, 0.565, 0.622, and 0.122. These condition-specific comparisons supplement the model-based analysis of monitoring interference.
Figure 9: Secondary-task performance under manual joystick teleoperation and gaze-assisted supervisory control. (a) Monitoring-response accuracy and (b) response time for correct positive responses. L0 was excluded because no monitoring stimuli were presented in that condition. Points show participant means collapsed across placement difficulty and the L1 and L2 monitoring conditions; white diamonds indicate overall means, while boxplots and half-violin profiles summarize the distributions. Bracketed p -values are from unadjusted two-sided paired t -tests of participant-level summaries ( n=16 ). For accuracy and response time, respectively, Cohen’s dz values are -0.007 and 0.081; both Holm-adjusted p -values are 1.000.
Figure 10: Normalized horizontal placement error under manual joystick teleoperation and gaze-assisted supervisory control for coarse and precision placement. Points show participant means across monitoring-load conditions and rocks; boxplots and half-violin profiles show their distributions. Lower values indicate placement closer to the target-region centre. Bracketed values are unadjusted p -values for contrasts from the placement-error mixed model; Holm-adjusted p -values are reported in the text.
Figure 11: Mode-level raw NASA-TLX subscale ratings for manual joystick teleoperation and gaze-assisted supervisory control ( n=16 per mode). Crosses show individual participant ratings, white circles indicate means, and boxplots show the distributions. Lower scores on mental, physical, and temporal demand, effort, and frustration indicate lower perceived demand. The performance subscale is scored such that lower values indicate better perceived performance. Accordingly, lower overall raw NASA-TLX scores indicate lower self-reported workload.
Figure 12: Participant-level distributions of visual transitions and continuous dwell within core AOIs under manual joystick teleoperation and gaze-assisted supervisory control. (a) Core-AOI gaze-shift rate, calculated as transitions between consecutive valid gaze samples assigned to different core AOIs per minute of valid gaze. (b) Mean duration of consecutive samples assigned to the same core AOI. Each point represents one participant averaged across the six placement-difficulty-by-monitoring-load conditions; diamonds indicate group means, and boxplots summarize the distributions. Bracketed p -values are from unadjusted, two-sided paired t -tests ( n=16 ). AOI dwell duration is a measure of consecutive AOI-labelled samples, not fixation duration.
Figure 13: Distribution of valid gaze time across the target rock, end effector, placement platform, and other scene regions under manual joystick teleoperation and gaze-assisted supervisory control. T1 and T2 denote coarse and precision placement, respectively; None, Routine, and Emergency denote monitoring loads L0, L1, and L2. Each stacked bar represents the mean AOI allocation across 16 participants for one placement-difficulty and monitoring-load condition. The four AOI categories are normalized to sum to 100% within each condition. Comparisons between categories and control modes are descriptive.
Figure 14: Participant preferences for gaze-assisted supervisory control and manual joystick teleoperation under coarse and precision placement, low and high monitoring workload, and in the overall final choice. Percentages indicate the proportion of participants preferring each mode in each context ( N=16 ).
Figure 15: Exploratory within-mode Pearson correlations among NASA-TLX ratings under manual joystick teleoperation and gaze-assisted supervisory control ( n=16 per mode). Values and colours indicate the magnitude and direction of Pearson’s r , while crossed cells indicate uncorrected two-sided p≥.05 .
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
Goal-directed eye movements are a fundamental component of visuomotor control, enabling humans to anticipate and guide their actions. Whether this anticipatory and task-driven behavior is preserved when actions are executed through a robot rather than through one's own body remains unclear. Here we address this question by investigating gaze behavior during goal-directed telemanipulation to determine how visuomotor control adapts to altered embodiment. Our findings show that gaze remains strongly aligned with task goals, preserving its predictive role even during robot-mediated manipulation. At the same time, teleoperation systematically redistributes visual attention toward the robotic end-effector and manipulated objects, increasing online monitoring. These findings show that predictive gaze is not lost under altered embodiment, but reorganized in response to changes in sensory feedback and control demands. More broadly, they reveal the flexibility of the human visuomotor system when the natural sensorimotor coupling is disrupted and identify gaze as an informative signal for inferring action intentions in human-robot interaction.
Manuela Uliano, Silvia Fattorini, Marco Controzzi
The BioRobotics Institute, Scuola Superiore Sant’Anna; Pisa, Italy · Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna; Pisa, Italy
Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)
Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce eye-tracker-supervised gaze prompting, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with π0, gaze prompting increases mean success from 26.3% to 56.0% across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release GazeMani, a dataset of 1,200 teleoperated trajectories with synchronized gaze.
Yihan Zhou, Rui Yan, Mingcong Li +4
Department of Automation, Tsinghua University · Beijing Key Laboratory of Embodied Intelligence Systems · Institute for Embodied Intelligence and Robotics, Tsinghua University +1