cs.ROFeb 19, 2026

Physical Human-Robot Interaction for Grasping in Augmented Reality via Rigid-Soft Robot Synergy

Authors: Huishi Huang, Jack Klusmann, Haozhe Wang, Shuchen Ji, Fengkang Ying, Yiyuan Zhang, John Nassour, Gordon Cheng, +4 more

Organizations: Advanced Robotics Centre (ARC), National University of Singapore (NUS), 117608, Singapore. · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), 138632, Singapore. · Chair of Cognitive Systems, Technical University of Munich (TUM), 80333, Munich. · Singapore-MIT Alliance for Research and Technology (SMART) Centre, 138602, Singapore. · Computer Science and Artificial Intelligence Lab (CSAIL), Massachusetts Institute of Technology, Cambridge, MA 02139 USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Hybrid rigid-soft robots combine the precision of rigid manipulators with the compliance and adaptability of soft arms, offering a promising approach for versatile grasping in unstructured environments. However, coordinating hybrid robots remains challenging, due to difficulties in modeling, perception, and cross-domain kinematics. In this work, we present a novel augmented reality (AR)-based physical human-robot interaction framework that enables direct teleoperation of a hybrid rigid-soft robot for simple reaching and grasping tasks. Using an AR headset, users can interact with a simulated model of the robotic system integrated into a general-purpose physics engine, which is superimposed on the real system, allowing simulated execution prior to real-world deployment. To ensure consistent behavior between the virtual and physical robots, we introduce a real-to-simulation parameter identification pipeline that leverages the inherent geometric properties of the soft robot, enabling accurate modeling of its static and dynamic behavior as well as the control system's response.

Figures & tables

Fig. 1: Schematic diagram of the proposed AR interaction method.
Fig. 2: (a) Restricted workspace of the rigid robot. (b) Reachability map of the spiral-shaped soft arm with its backbone aligned with the gravity direction. (c) Reachability maps of our soft arm at 0∘0^{\circ} , 60∘60^{\circ} , and 120∘120^{\circ} relative to the gravity direction. (d) Visualization of the restricted reachability map overlaid on the AR robot model.
Fig. 3: (a). Experimental setup of our soft arm and the motion capture system with six infrared cameras. (b). Static tilting experiment to identify bending stiffness for each section, corresponds to Table I Exp.1. (c) Curling and uncurling in different actuation statuses to identify damping properties for the robot and the actuator properties for the control system, corresponds to Table I Exp.2 and 3.
Exp.ObjectiveImplementationData
1Identify the joint stiffness for each section.Tilt the base by 0∘0^{\circ} , 30∘30^{\circ} , 60∘60^{\circ} , and 90∘90^{\circ} relative to gravity and record the equilibrium section poses.Static
2Identify the joint damping for each section and tendon friction.Release the robot from a fully bent state without actuation and record the oscillatory motion.Dynamic
3Identify actuator control gains and force range of the control system.Perform curling and uncurling motions of the robot at actuation levels of [20,40,60,80,100][20,40,60,80,100] mm, and record the resulting trajectories during each actuation cycle.Dynamic
TABLE I: Parameter Identification Experiments for our Soft Arm
Fig. 4: Additional details are provided in the supplementary video. (a) Adjustment of the ray length projected from the joystick; (b) Tilting the joystick to change the orientation of the rigid robot for alignment with the joystick. (c) Control of the soft robot’s curling and uncurling motions by specifying the end-effector position using the ray. (d) Control of the rigid robot’s end-effector position using the ray. (e) Overview of the joystick functions.
Fig. 5: (a) Uncurling motion of the physical robot with motion-capture markers illustrating the trajectory over time. (b) Simulated uncurling motion after parameter identification. (c) Simulated uncurling motion using initial, unoptimized parameters. (d) Positional error between the optimised and unoptimized models. (e) Velocity error between the optimized and unoptimized models.
Fig. 6: Supplementary videos provide additional details of the demonstration. (a) The user moves the virtual robot freely in 3D space, while the physical robot follows with a designed delay. (b) The user teleoperates the robot to reach a hanging target point. (c) The user teleoperates the robot to grasp an object being handed over in mid-air. (d) The user teleoperates the robot to grasp a toy placed on a chair.
Exp.ObjectiveImplementationSuccess Criteria
1Object ReachingReach a static object by using the rigid robot to position and orient the soft robot and by curling the soft robot to bring its tip into contact with the target.Successful contact between the soft robot’s tip and the target object.
2Object FollowingFollow a dynamic trajectory by coordinating rigid-arm positioning and orientation with soft-robot curling and uncurling.Successful trajectory completion without interruption.
3Object GraspingGrasp an object either placed on a flat surface or suspended in mid-air.Object securely grasped and lifted without slipping or dropping.
4Object HandlingGrasp an object from one surface, move it in mid-air, and release it onto another surface.Object grasped, transferred, and released onto the target surface.
TABLE II: Human–Robot Interaction Experiments

Explore similar work

CardsList

  1. Observer Design for Augmented Reality-based Teleoperation of Soft Robots

    Mar 5, 2026Jorge Francisco García-Samartín, Iago López Pérez, Emirhan Yolcu +2Robot TeleoperationSoft Robotics

    Centro de Automática y Robótica, Universidad Politécnica de Madrid, c/ José Gutiérrez Abascal, nº 2, 28006, Madrid, España. · Middle East Technical University, Dumlupinar Blv No 1, 06800 Cankaya, Ankara, Türkiye.

  2. Learning to infer and manipulate through distributed whole-arm interaction in a soft robot

    Aug 31, 2026Chuhan Zhang, Ebrahim Shahabi, Kseniia Khomenko +2Robotic GraspingRobotic RL

    Cognitive Robotics Department, Faculty of Mechanical Engineering, Delft University of Technology, Delft 2628 CD, The Netherlands. · School of Engineering, Newcastle University, UK.

  3. Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies

    Apr 1, 2026Uksang Yoo, Mengjia Zhu, Evan Pezent +8Motion RetargetingRobot Imitation Learning

    The Robotics Institute, Carnegie Mellon University · Meta Reality Labs