TEGA: A Tactile-Enhanced Grasping Assistant for Assistive Robotics via Sensor Fusion and Closed-Loop Haptic Feedback
Authors: Hengxu You, Tianyu Zhou, Fang Xu, Kaleb Smith, Eric Jing Du
Organizations: All authors are with the Department of Civil and Coastal Engineering, University of Florida, Gainesville, FL 32611, USA. · All authors are with Nvidia
Recent advances in teleoperation have enabled sophisticated manipulation of dexterous robotic hands, with most systems concentrating on guiding finger positions to achieve desired grasp configurations. However, while accurate finger positioning is essential, it often overlooks the equally critical task of grasp force modulation, vital for handling objects of diverse hardness, texture, and shape. This limitation poses a significant challenge for users, especially individuals with upper limb disabilities who lack natural tactile feedback and rely on indirect cues to infer appropriate force levels. To address this gap, We present the tactile enhanced grasping assistant (TEGA), a closed loop assistive teleoperation framework that fuses EMG based intent2force inference with visuotactile sensing mapped into real time vibrotactile feedback via a wearable haptic vest, enabling intuitive, proportional force adjustment during manipulation. A wearable haptic vest delivers real time tactile feedback, allowing users to dynamically refine grasp force during manipulation. User studies confirm that the system substantially improves grasp stability and task success, underscoring its potential for assistive robotic applications.
Figures & tables
Fig. 1: Overview of the Tactile-Enhanced Grasping Assistant (TEGA) system. The system integrates EMG-based force inference, visuo-tactile sensing, and haptic feedback to enable intuitive, precise, and adaptive grasping force control through real-time closed-loop interaction. EMG signals from the user’s upper arm muscles are processed to infer grasp intent, while tactile sensor data from the robotic fingertips is converted into real-time vibrotactile feedback delivered via a wearable haptic vest.
Fig. 2: Hardware devices setup. (a) robot arm-hand system with mounted DIGIT sensors and (b) human operator with haptic vest, motion trackers and EMG sensors.
Fig. 3: Visualization of CCI and EDA. The grayscale image is extracted from the raw RGB tactile image to estimate the depth map of the deformation. The green circle marks the pixel point with the highest Contact CCI, representing the location with the most concentrated pressure. The blue circle identifies the sharpest pressure point, encompassing EDA, which represents the spatial extent of the deformation. Both CCI and EDA values are normalized within the range [0,1] and transmitted to the haptic vest, where they modulate the intensity of the corresponding vibration unit.
Fig. 4: Target objects to be tested. Object 1 (water bottle, rigid and heavy), Object 2 (plastic wet wipes container, medium stiffness), and Object 3 (bag of bread, soft and deformable).
Fig. 5: Experiment setup Experimental Setup for the TEGA System Evaluation. The setup consists of a 7-degree-of-freedom (7-DOF) Franka Emika Panda robotic arm with an Allegro Hand as the end-effector, equipped with DIGIT tactile sensors on each fingertip to capture real-time contact data. The human operator wears a motion tracker, three EMG sensors, and a bHaptics vest to facilitate teleoperation with haptic feedback. The operator’s forearm muscle contractions, detected by the EMG sensors, control the robotic hand’s grasping force, while the haptic vest provides vibrotactile feedback based on the Contact Concentration Index (CCI) and Effective Deformation Area (EDA) extracted from the tactile sensor data. The pick-and-place task is performed in a structured workspace.
Fig. 6: EMG sensors and motion trackers mounted on the arm. Placement of EMG Sensors Corresponding to Forearm Muscles. This figure illustrates the placement of three EMG sensors on the forearm, each corresponding to a specific muscle involved in grasp force modulation. Sensor 1 (S1) is placed over the Extensor Carpi Radialis (ECR), responsible for wrist extension and stabilization during grasping. Sensor 2 (S2) is placed over the Flexor Carpi Ulnaris (FCU), which assists in wrist flexion and contributes to grip force generation. Sensor 3 (S3) is placed over the Flexor Digitorum Superficialis (FDS), a key muscle responsible for flexing the fingers during grasp execution.
Fig. 7: Comparison of task performance metrics across Haptic and Non-Haptic condition. Top row: completion time for Objects 1–3; bottom row: slippage counts. Plots show trial data (black dots), distributions (violin), and group means (red diamonds). Mann–Whitney U test p-values are indicated.
Fig. 8: Real-Time Correlation Between Tactile Feedback and Grasping Pose Selection. This figure illustrates the relationship between grasping pose selection and tactile feedback signals, including CCI and EDA, across different object types. The red dashed line represents the grasping pose index, indicating the level of applied force, where a higher index corresponds to a stronger grip. The blue solid line represents CCI, reflecting the sharpness of pressure concentration at the contact point. A higher CCI value indicates a more localized, intense contact. The green solid line represents EDA, which quantifies the spatial extent of the deformation. A larger EDA value suggests a broader contact area, while a smaller value indicates localized force application. Each column represents a specific object (Object 1, Object 2, Object 3), while each row corresponds to individual trials.
Condition
Metric
Object 1
Object 2
Object 3
Mean
Mean
Mean
Haptic
Completion Time
152.8
177.4
151.5
Slippage
2.8
0.7
0.3
Deformation
0.0
0.0
0.5
Success Rate
5/5
5/5
5/5
Non-Haptic
Completion Time
168.5
171.4
160.1
TABLE I: Performance across Haptic and Non-Haptic Conditions
Rochester Institute of Technology, Rochester, NY · TikTok Pico Lab, San Jose, CA · The Hong Kong University of Science and Technology, Guangzhou, China
Division of Microsystems Technology, Department of Electrical Engineering · Cyber-Physical Systems Lab, Department of Information Technology, Uppsala University, Uppsala, Sweden