Recent advancements in virtual reality and robotic teleoperation have greatly increased the variety of haptic information that must be conveyed to users. While existing haptic devices typically provide unimodal feedback to enhance situational awareness, a gap remains in their ability to deliver rich, multimodal sensory feedback encompassing force, pressure, and thermal sensations. To address this limitation, we present the Multimodal Feedback Exoskeleton (MFE), a hand exoskeleton designed to deliver hybrid haptic feedback. The MFE features 20 degrees of freedom for capturing hand pose. For force feedback, it employs an active mechanism capable of generating 3.5-8.1 N of pushing and pulling forces at the fingers' resting pose, enabling realistic interaction with deformable objects. The fingertips are equipped with flat actuators based on the electro-osmotic principle, providing pressure and vibration stimuli and achieving up to 2.47 kPa of contact pressure to render tactile sensations. For thermal feedback, the MFE integrates thermoelectric heat pumps capable of rendering temperatures from 10 to 55 degrees Celsius. We validated the MFE by integrating it into a robotic teleoperation system using the X-Arm 6 and Inspire Hand manipulator. In user studies, participants successfully recognized and manipulated deformable objects and differentiated remote objects with varying temperatures. These results demonstrate that the MFE enhances situational awareness, as well as the usability and transparency of robotic teleoperation systems.
Figures & tables
Fig. 1: Our exoskeleton is capable of capturing finger positions and rendering remote contact force , fingertip pressure , and palm temperature .
Interface
Motion Cap
Feedback
Cost
SenseGlove DK1 [ 17 ]
20 DoFs
Resistant Force Fingertip Vibration
$3524
HaptX Gloves G1 [ 34 ]
36 DoFs
Active Force Fingertip Pressure
$5495
LucidGloves [ 37 ]
5 DoFs
No
$60
DOGlove [ 38 ]
21 DoFs
Active Force Fingertip Vibration
$600
MFE (Ours)
20 DoFs
Active Force Fingertip Pressure Temperature
$350
TABLE I: Comparison of different hand’s haptic interfaces.
Fig. 2: Technical pipeline for the MFE multimodal teleoperation system, encompassing the exoskeleton for finger’s positional and force feedback, microfludic actuators for fingertip pressure rendering, thermoelectric module for temperature feedback.
Fig. 3: MFE Exoskeleton: (A) structures, (B) canonical design, and (C) motion capture with additional joint encoders.
Fig. 5: (A) Characterization of pulling force at finger’s resting pose. Characterization of (B) maximum force, and (C) deformation distance generated by microfluid actuator (at 50, 100, 150, and 200 V, respectively). (D) Characterization of surface temperature of thermoelectric module (at +1, +3, +5, -1, -3, -5 V, respectively).
Fig. 6: Experimental setup for (A) Task 1: distinguishing objects by shape and material stiffness, (B) Task 2: grasping a deformable cup filled with granular objects, and (C) Task 3: distinguishing temperatures of three cups of water.
Force
Fingertip
Multimodal
Obj. Shape
8/10
6/10
10/10
Obj. Stiffness
6/10
4/10
9/10
TABLE II: Performance of 10 human subjects in object recognition task (Task 1). Numbers are successful trails.
Fig. 7: Performance of 10 human subjects in manipulating cups with granular objects (Task 2), and statistical test results.
In robot teleoperation, haptic feedback can be used to help human operators accomplish dexterous manipulation tasks. However, existing haptic feedback methods try to replicate high-fidelity sensory haptics that are felt in real world interactions, which are constrained by the sensing and feedback hardware capability and may lead to higher workload. To addresses these limitations, this work introduces semantic haptics for teleoperation, which uses abstract haptic patterns to convey critical information about robot states. We categorize robot states into "Confirmations" and "Exceptions", implement a modular haptic rendering pipeline in robot simulation, and deliver semantic haptic feedback to operators through pneumatic and vibrotactile wristbands. This simplifies hardware requirements and enables one-to-many mappings between haptic patterns and robot states. Through three evaluation studies, we identify the most effective semantic haptic design for a common pick and place teleoperation task and compare semantic haptics to other teleoperation feedback approaches including sensory haptics and visual feedback. Results suggest that while semantic haptics performs similarly as other feedback in unimanual tasks, it achieves superior performance in bimanual tasks, with reduced task workload, increased situational awareness, and overall preference.
This paper addresses the challenge of providing kinesthetic feedback in bilateral teleoperation by designing a wearable, lightweight (20 g), and compact haptic device, the HJ-Haptic, utilizing a honeycomb jamming mechanism for object stiffness rendering. The HJ-Haptic device can vary its stiffness, from 1.15 N/mm to 2.64 N/mm, using a 30 kPa vacuum pressure. We demonstrate its implementation in a teleoperation framework, enabling operators to adjust grip force based on a reliable haptic feedback on object stiffness. A three-point flexural test on the honeycomb jamming mechanism and teleoperated object-grasping tasks were conducted to evaluate the device's functionality. Our experiments demonstrated a small RMSE and strong correlations in teleoperated motion, stiffness rendering, and interaction force feedback. The HJ-Haptic effectively adjusts its stiffness in response to real-time gripper feedback, mimicking the sensation of direct object grasping with hands. The device's use of vacuum pressure ensures operator safety by preventing dangerous outcomes in case of gas leakage or material failure. Incorporating the HJ-Haptic into the teleoperation framework provided the reliable perception of object stiffness and stable teleoperation. This study highlights the potential of the honeycomb jamming mechanism for enhancing haptic feedback in various applications, including teleoperation scenarios, as well as interactions with extended-reality environments.
Thomas M. Kwok, Bohan Zhang, Wai Tuck Chow
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, Singapore
Recent advances in teleoperation have enabled sophisticated manipulation of dexterous robotic hands, with most systems concentrating on guiding finger positions to achieve desired grasp configurations. However, while accurate finger positioning is essential, it often overlooks the equally critical task of grasp force modulation, vital for handling objects of diverse hardness, texture, and shape. This limitation poses a significant challenge for users, especially individuals with upper limb disabilities who lack natural tactile feedback and rely on indirect cues to infer appropriate force levels. To address this gap, We present the tactile enhanced grasping assistant (TEGA), a closed loop assistive teleoperation framework that fuses EMG based intent2force inference with visuotactile sensing mapped into real time vibrotactile feedback via a wearable haptic vest, enabling intuitive, proportional force adjustment during manipulation. A wearable haptic vest delivers real time tactile feedback, allowing users to dynamically refine grasp force during manipulation. User studies confirm that the system substantially improves grasp stability and task success, underscoring its potential for assistive robotic applications.
Hengxu You, Tianyu Zhou, Fang Xu +2
All authors are with the Department of Civil and Coastal Engineering, University of Florida, Gainesville, FL 32611, USA. · All authors are with Nvidia