Visualizing Impedance Control in Augmented Reality for Teleoperation: Design and User Evaluation

This paper presents an immersive bilateral teleoperation framework for beyond-human-scale manipulators, combining motion/force transparency with enhanced operator embodiment through virtual reality (VR) and distributed haptic feedback. The platform integrates a full-scale industrial hydraulic manipulator, a 7-DoF haptic exoskeleton, and head-tracked visual feedback to reinforce operator agency and self-location. A force-sensorless adaptive controller incorporates an augmented human-robot dynamic model to address master-surrogate asymmetries, input nonlinearities, model uncertainties, and communication delays. Rigorous analysis establishes semi-global uniform ultimate boundedness of the closed-loop system. Extensive full-scale experiments demonstrate precise motion and force tracking with scaling up to 1:13 and 1:1000, respectively, and fixed or time-varying delays up to 150 ms. Coordinated 6-DoF pick-and-place and multi-axis contact tasks further validate system performance. A ten-participant user study supports system usability and indicates approximately 50% higher normalized embodiment scores with VR than with monitor-based feedback. These results advance intuitive, force-reflected remote manipulation at industrial scale.

Fixed Cartesian impedance makes contact-rich teleoperation demonstrations practical, but gains that secure progress and contact support also determine impact and force variability. We study single-demonstration controller-to-controller impedance retargeting. Given one fixed Cartesian impedance command sequence {K0, D0, xcmd}, Manifold-Decomposed Impedance Retargeting (MDIR) deterministically reparameterizes the recorded controller into an executable task-channel variable-impedance command. MDIR targets this local retargeting problem by preserving projected task-channel responses near the demonstrated trajectory. It represents the source response in operational work, exertion, and support channels with a passive residual complement under a control-chain metric, computes an executable Cartesian-to-Manifold Retargeting (C2M) baseline, and applies Manifold-Constrained Parameter Optimization (MPO) to select a feasible representative with lower wrist-force peaks, impulse, force variability, and nominal controller power. Across planar wiping, pick-and-place, and pushing on a Franka Panda, the full MDIR controller passes Task Check in all 15 closed-loop executions and reduces all four aggressiveness metrics relative to the fixed-impedance demonstrations.

Direct hand-driven teleoperation maps an operator's hand motion to robot end-effector commands at every frame, enabling precise control, but it requires constant monitoring and correction during approach, grasp, and placement, which can be slow and fatiguing. For repetitive pick-and-place tasks, supervisory (goal-based) teleoperation simplifies this process: the operator specifies goals/waypoints, and the robot executes the motion using planning algorithms. Yet, this introduces latency, as the robot must wait for the next command before it can plan and act. "How can we reduce robot reaction time while lowering operator workload?" To tackle this question, we present AHEAD, a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control. In a digital twin, the operator performs pick-and-place naturally, using hand motion to convey high-level commands rather than a continuous robot trajectory. AHEAD processes a short window of 3D hand and head signals together with scene context through an attention-based classifier to predict the intended grasp object and placement slot. A state machine converts intent predictions into stable robot goals, enabling early motion while remaining stable under noisy predictions and corrective hand movements. AHEAD's intent prediction module achieves Top1 accuracy: 76% for grasp objects and 76% for target slots. Moreover, our user study shows AHEAD reduces robot reaction latency by 0.6 s (object) and 1.4 s (slot) relative to baselines. Participants also reported lower operator load, indicating faster robot responses while maintaining low operator effort in practice.