Fiber Optic Sensing Glove for High Performance Dexterous Manipulation Capture
Authors: J. D. Peiffer, Taylor Niehues, Li Guan, Ziyi Kou, Ergys Ristani
Organizations: Meta, Redmond, Washington, 98052, USA · Department of Biomedical Engineering, Northwestern University, Evanston, Illinois, 60208, USA · Work done at Meta
Capturing hand pose during dexterous manipulation remains difficult: vision-based methods degrade under occlusion and challenging lighting, while sensorized gloves, though occlusion-free, are prone to drift and magnetic interference and rarely match motion-capture accuracy. We introduce a fiber optic sensing glove for full hand pose tracking that targets these failure modes, using multi-core shape-sensing fibers that capture each fiber's full 3D shape rather than curvature alone. A novel pipeline registers each reconstructed fiber shape to a common hand reference frame, and a new inverse-kinematics solver reconstructs full hand pose at 60 Hz using curve constraints. Benchmarked on a 2-hour dataset of dexterous object manipulation tasks across 5 subjects, the glove achieves 7.2 mm mean fingertip position error against motion capture ground truth, reduced to 4.9 mm by a one-time factory calibration of the fiber routing hub that transfers across users and sessions. These capabilities enable high-fidelity data capture and bimanual virtual teleoperation - both essential to advancing the robotics field.
Figures & tables
Fig. 1 : We demonstrate the utility of our gloves for occlusion-free, high accuracy dexterous manipulation data capture and for virtual teleoperation. Top left : Our sensing gloves instrumented with multi-core optical fibers and a dense array of Fiber Bragg Gratings. Bottom left : The system enables whole-hand pose estimation and is not vulnerable to drift, magnetic interference, or occlusion. Right : The user successfully opens a jar in virtual reality with minimal effort.
Fig. 2 : Our sensing glove routes multicore optical fibers—which reconstruct 3D shape—through fabric loops along each finger. The fibers are fixed at the fingertip but otherwise allowed to move freely along the fabric loops and through the routing hub. The routing hub uses known tunnel geometry to register 3D fiber shape reconstructions to a common reference frame.
Fig. 3 : Optic fibers are first reconstructed in their own reference frame. The shape registration process matches routing hub tunnels to the section of the fiber with same curvature. This is done for every fiber at every time sample to put the fibers in the routing hub frame. Next, inverse kinematic solutions uses spline position and direction costs to pose the hand model.
Fig. 4 : Our system allows for tracking of hand pose in situations where self-occlusion or object occlusion prevent marker-based systems from accurately tracking the whole hand. Left : Self occlusion causes mocap to cross fingers incorrectly Right : Marker occlusion on index, ring, and pinky fingers results in incorrect fingertip position.
Fig. 5 : Our method tracks fingertip landmarks closely across a variety of manipulation tasks (Left: 9 Hole Peg Test, Right: in hand object rotation). Notably, our glove tracks during periods of mocap marker occlusion.
Fig. 6 : Un-aligned glove pose estimates (red) are corrected with a one-time routing tunnel alignment to repose fibers and hand mesh (blue) to better align with ground truth (green). First row: a bias in the rotation of the fiber is corrected to align the pinky. Second row: alignment brings the ring finger closer to the thumb in a pinch.
Metric
Fiber-to-Marker Residual (mm)
MKPE (mm)
F.MKPE (mm)
Phase
Range of Motion
Manipulation
Range of Motion
Manipulation
Range of Motion
Manipulation
Base
3.3
3.4
5.3
5.9
6.5
7.2
Tunnel Aligned
1.7
2.1
4.4
4.4
4.5
4.9
TABLE I : Pose Benchmarking Errors
Fig. 7 : Histogram of landmark errors across subject and sessions. Routing tunnel alignment decreases fingertip error in all cases. After alignment, errors at the fingertips are generally below 5 mm.