cs.ROOct 29, 2026
SaveGaze-Assisted Supervisory Control for Lunar Robotic Arm Teleoperation Under Concurrent Monitoring
Organizations: Department of Engineering, King’s College London, London, UK · Department of Psychology, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK · Spacesuit Physiology Laboratory, Centre for Human and Applied Physiological Sciences, King’s College London, London, UK · Centre for AI Robotics in Space Sustainability, The Hong Kong University of Science and Technology, Hong Kong, China
Abstract
Rover-mounted robotic arms can extend human reach during lunar sample handling, but continuous joystick teleoperation requires repeated visual-motor alignment while operators may also need to monitor system states. This paper presents a gaze-assisted supervisory control framework in which gaze nominates a rock or placement target, explicit confirmation authorizes the action, and the robot performs bounded autonomous grasp-and-place execution. The framework was evaluated against manual joystick teleoperation in a mission-analog virtual reality study with 16 participants. The study used a 2 x 2 x 3 within-subject design varying control mode, placement difficulty, and concurrent monitoring load. Mean recorded task completion time decreased from 47.65 to 18.43 s per rock, with 72.5% of this difference arising during grasping. Under the tested auditory monitoring conditions, gaze-assisted supervisory control showed less average proportional slowing, with no detected control-mode difference in monitoring-response accuracy or speed. The rate of gaze shifts between core areas of interest decreased from 49.98 to 27.73 shifts/min, while the raw NASA Task Load Index score decreased from 32.54 to 18.26. However, the benefit was task dependent. In precision placement, manual control yielded lower mean placement error and a higher placement success rate. These findings support a mixed-initiative allocation in which gaze enables rapid task-level intention specification and autonomous execution, while precision-critical stages retain manual refinement.
Figures & tables
Figure 1: Conceptual illustration of manual robotic teleoperation and concurrent operator demands in a spaceflight setting. (a) An astronaut operating a robotic system using a virtual reality (VR) headset and haptic joystick, while attending to additional information, with visual, auditory, and manual demands highlighted in red, blue, and purple, respectively. (b) The teleoperated robotic arm. Source images: European Space Agency (ESA)/National Aeronautics and Space Administration (NASA), adapted from Pilotes and Pilote timelapse under the ESA Standard Licence.
Figure 2: Overview of the proposed gaze-assisted supervisory control framework and the relationship between its interface and execution components. (a) System-level allocation of task-level intention specification and low-level robot execution. (b) Expansion of the gaze-assisted interface, showing the transformation from gaze geometry to an explicitly confirmed task target. (c) Expansion of the autonomous execution module, showing the finite-state grasping and placement sequences.
Figure 3: Binocular VR views, controller mappings, and task sequence used in the mission-analog study. (a) Paired left- and right-eye images presented through the VR headset under manual joystick teleoperation and gaze-assisted supervisory control, together with the Xbox controller mappings. In the gaze-assisted view, the red point indicates the real-time gaze position, while the identified candidate target is highlighted using a visual mask. The controller supports end-effector translation and rotation, gripper operation, explicit target confirmation, and concurrent monitoring responses. (b) Task-completion process, with gaze-assisted supervisory control used as an illustrative example. The sequence progresses through grasp-target selection, robotic-arm grasping, placement-target selection, and robotic-arm placement.
Figure 4: Experimental apparatus and operator-station configuration used in the controlled VR study. (a) Schematic of the principal hardware components, comprising a PICO 4 Enterprise VR headset with integrated eye tracking, an Xbox controller, and a Linux simulation workstation. (b) Staged demonstration of the operator-station configuration.
Figure 5: Placement-platform configurations used to manipulate placement difficulty in the VR study. The highlighted grid cell indicates the designated target region in the precision-placement condition.
Figure 6: Per-rock TCT under manual joystick teleoperation and gaze-assisted supervisory control. Coarse and precision placement are shown separately across no (L0), routine (L1), and emergency (L2) concurrent monitoring. Each observation represents one participant’s mean completion time across the three rocks in that condition; boxplots and half-violin plots show the distributions across participants. Bracketed -values are from unadjusted two-sided paired -tests ( ); all six Holm-adjusted -values were . Cohen’s values for L0, L1, and L2 were 5.258, 3.618, and 3.214 for coarse placement, and 3.681, 4.813, and 3.992 for precision placement, respectively.
| Control mode | Grasping time (s) | Placement time (s) | Total TCT(s) |
|---|---|---|---|
| Manual joystick | 27.8 (4.6) | 19.9 (3.4) | 47.6 (7.3) |
| Gaze-assisted | 6.6 (0.5) | 11.8 (3.8) | 18.4 (3.7) |
Table 1: Per-rock TCT by control mode and task phase. Values are means across 16 participants, averaged over placement difficulties, monitoring-load conditions, and rocks; standard deviations (SDs) are shown in parentheses.
Figure 7: Per-rock TCT across coarse and precision placement under (a) manual joystick teleoperation and (b) gaze-assisted supervisory control. Each point represents one participant’s mean across the L0–L2 monitoring conditions, and connecting lines link the same participant’s two difficulty conditions. Diamonds and thicker lines indicate group means; boxes show the interquartile ranges. The connecting lines describe paired changes between the two difficulty levels, not a continuous difficulty-response relationship.
Figure 8: Primary-task DTC under manual joystick teleoperation and gaze-assisted supervisory control during coarse and precision placement. DTC represents the percentage change in TCT under concurrent monitoring relative to the corresponding no-monitoring condition (L0). Positive values indicate slower completion under concurrent monitoring, whereas negative values indicate faster completion than in L0; the dashed horizontal line denotes zero DTC. Points show individual participant values, white diamonds indicate means, and boxplots and half-violin profiles summarize the distributions. Bracketed -values are from unadjusted two-sided paired -tests ( ); asterisks indicate unadjusted . For coarse L1, coarse L2, precision L1, and precision L2, respectively, Holm-adjusted -values are .118, .118, .101, and .632, and Cohen’s values are 0.536, 0.565, 0.622, and 0.122. These condition-specific comparisons supplement the model-based analysis of monitoring interference.
Figure 9: Secondary-task performance under manual joystick teleoperation and gaze-assisted supervisory control. (a) Monitoring-response accuracy and (b) response time for correct positive responses. L0 was excluded because no monitoring stimuli were presented in that condition. Points show participant means collapsed across placement difficulty and the L1 and L2 monitoring conditions; white diamonds indicate overall means, while boxplots and half-violin profiles summarize the distributions. Bracketed -values are from unadjusted two-sided paired -tests of participant-level summaries ( ). For accuracy and response time, respectively, Cohen’s values are -0.007 and 0.081; both Holm-adjusted -values are 1.000.
Figure 10: Normalized horizontal placement error under manual joystick teleoperation and gaze-assisted supervisory control for coarse and precision placement. Points show participant means across monitoring-load conditions and rocks; boxplots and half-violin profiles show their distributions. Lower values indicate placement closer to the target-region centre. Bracketed values are unadjusted -values for contrasts from the placement-error mixed model; Holm-adjusted -values are reported in the text.
Figure 11: Mode-level raw NASA-TLX subscale ratings for manual joystick teleoperation and gaze-assisted supervisory control ( per mode). Crosses show individual participant ratings, white circles indicate means, and boxplots show the distributions. Lower scores on mental, physical, and temporal demand, effort, and frustration indicate lower perceived demand. The performance subscale is scored such that lower values indicate better perceived performance. Accordingly, lower overall raw NASA-TLX scores indicate lower self-reported workload.
Figure 12: Participant-level distributions of visual transitions and continuous dwell within core AOIs under manual joystick teleoperation and gaze-assisted supervisory control. (a) Core-AOI gaze-shift rate, calculated as transitions between consecutive valid gaze samples assigned to different core AOIs per minute of valid gaze. (b) Mean duration of consecutive samples assigned to the same core AOI. Each point represents one participant averaged across the six placement-difficulty-by-monitoring-load conditions; diamonds indicate group means, and boxplots summarize the distributions. Bracketed -values are from unadjusted, two-sided paired -tests ( ). AOI dwell duration is a measure of consecutive AOI-labelled samples, not fixation duration.
Figure 13: Distribution of valid gaze time across the target rock, end effector, placement platform, and other scene regions under manual joystick teleoperation and gaze-assisted supervisory control. T1 and T2 denote coarse and precision placement, respectively; None, Routine, and Emergency denote monitoring loads L0, L1, and L2. Each stacked bar represents the mean AOI allocation across 16 participants for one placement-difficulty and monitoring-load condition. The four AOI categories are normalized to sum to 100% within each condition. Comparisons between categories and control modes are descriptive.
Figure 14: Participant preferences for gaze-assisted supervisory control and manual joystick teleoperation under coarse and precision placement, low and high monitoring workload, and in the overall final choice. Percentages indicate the proportion of participants preferring each mode in each context ( ).
Figure 15: Exploratory within-mode Pearson correlations among NASA-TLX ratings under manual joystick teleoperation and gaze-assisted supervisory control ( per mode). Values and colours indicate the magnitude and direction of Pearson’s , while crossed cells indicate uncorrected two-sided .
More in this topic
Robot Teleoperation
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
See all 207 papers