Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics

Hendrik Chiche; Antoine Jamme; Trevor Rigoberto Martinez; Gabriel Gomes

cs.ROMar 11, 2026

Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics

Authors: Hendrik Chiche, Antoine Jamme, Trevor Rigoberto Martinez, Gabriel Gomes

Organizations: OMGrab Inc. and affiliated with the University of California, Berkeley, CA 94720 USA, through the Capstone Project program of the Fung Institute for Engineering Leadership · Department of Mechanical Engineering, University of California, Berkeley, CA 94720 USA

Abstract

Teleoperation of low-cost robotic manipulators remains challenging due to the difficulty of retargeting human hand motion to robot joint commands. We present an offline hand-shadowing inverse-kinematics (IK) retargeting pipeline driven by a single egocentric RGB-D camera mounted on 3D-printed glasses. The pipeline detects 21 hand landmarks per hand using MediaPipe Hands, deprojects them into 3D via depth sensing, transforms them into the robot coordinate frame, and solves a damped-least-squares IK problem to produce joint commands for the SO-ARM101 robot (5 arm + 1 gripper joints). A gripper controller maps thumb-index finger geometry to grasp aperture with a multi-level fallback hierarchy. Actions are previewed in a physics simulation before replay on the physical robot. We evaluate the pipeline on a structured pick-and-place benchmark (5-tile grid, 10 grasps per tile, 3 independent runs) achieving an 86.7% +/- 4.2% success rate, and compare it against four vision-language-action (VLA) policies (ACT, SmolVLA, pi_0.5, GR00T N1.5) trained on leader-follower teleoperation data. We provide a quantitative error analysis of the pipeline, reporting a mean IK position error of 36.4 mm, trajectory smoothness metrics showing 57-68% jerk reduction from EMA smoothing, and an ablation study over the smoothing parameter. We also test the pipeline in unstructured real-world environments (grocery store, pharmacy) and find that success is reduced to 9.3% due to hand occlusion by surrounding objects. To mitigate this, we integrate WiLoR as an alternative hand detector, achieving an 8% improvement in hand detection rate over MediaPipe, highlighting both the promise and current limitations of marker-free analytical retargeting.

Figures & tables

Panel of Figure 1 — Fig. 1: Pipeline architecture. Each stage (A–E) transforms the data from egocentric RGB-D frames to robot joint commands. Intermediate representations are shown between stages. The output feeds both a simulation preview (F) and the physical robot (G).

Panel of Figure 2 — Fig. 2: Hardware setup. Left: annotated photo showing the RealSense D435i camera, angled mount, camera stand, 6-DOF arm, and 12V battery for in-the-wild deployment. Right: schematic showing the camera tilt angle $\theta=50^{\circ}$ below horizontal and height offset $t_{z}=0.48$ m relative to the robot base (see Section IV-D ).

Panel of Figure 3 — Fig. 3: Gripper orientation vectors. $\mathbf{e}_{1}$ is the normalised gripper width direction ( $\mathbf{P}_{1}\to\mathbf{P}_{2}$ ). $\mathbf{u}_{\text{thumb}}$ and $\mathbf{u}_{\text{index}}$ are unit vectors from each MCP to its fingertip; their normalised average $\mathbf{d}$ (Eq. 8 ) serves as a reference pointing direction. Note that $\mathbf{d}$ is generally not orthogonal to $\mathbf{e}_{1}$ ; the orthonormal frame is obtained by computing $\mathbf{e}_{3}=\mathbf{e}_{1}\times\mathbf{d}$ (normalised, out of page) and $\mathbf{e}_{2}=\mathbf{e}_{3}\times\mathbf{e}_{1}$ (Eqs. 9 – 10 ).

Fig. 4: PyBullet simulation preview. Top-left: RGB frame from the egocentric camera showing the operator’s hand. Top-right: depth colour map. Bottom: robot arm in PyBullet with debug joint labels and IK target markers (green/red spheres), tracking the hand-derived trajectory.

Fig. 5: Hand shadowing. Left: the operator wearing the RealSense glasses performs a grasp. Right: the SO-ARM101 robot mirrors the hand pose through the IK pipeline.

Joint	Min	Max	Range ( ∘ )
Shoulder pan	$-1.920$	$+1.920$	$220.0$
Shoulder lift	$-1.745$	$+1.745$	$200.0$
Elbow flex	$-1.745$	$+1.571$	$190.0$
Wrist flex	$-1.658$	$+1.658$	$190.0$
Wrist roll	$-2.793$	$+2.793$	$320.0$
Gripper	$-0.175$	$+1.745$	$110.0$

TABLE I: SO-ARM101 Joint Angle Ranges (Radians)

Parameter	Value
P coefficient	12
I coefficient	0
D coefficient	24
Arm acceleration limit	50 / 254
Gripper acceleration limit	100 / 254
Arm velocity limit	1500 RPM

TABLE II: Physical Robot Servo Parameters

Panel of Figure 8 — Fig. 6: Pick-and-place benchmark layout. Left: $3\times 3$ tile grid numbered #1–#9 from the robot’s perspective, with the drop box to the left. Right: camera point-of-view showing the grid with the purple cube on tile #5. Tiles #1–#5 are used for evaluation; #6–#9 are excluded due to hand self-occlusion.

Fig. 7: Per-stage latency breakdown. The RGB/depth frame overlay (110 ms) and PyBullet IK solver (80 ms) dominate, yielding a total of 213 ms per frame ( ${\sim}$ 5 FPS). Processing is performed offline on recorded .bag files.

Fig. 8: IK retargeting per-tile success (out of 10 grasps each, averaged across 3 runs). Tile #1 (farthest from robot, natural hand pose) achieves a perfect score. Performance degrades toward tile #5 (closest to robot base) where the operator’s hand orientation causes partial thumb/index self-occlusion from the egocentric camera. Mean total: 130/150 = 86.7%.

Approach	Training	Inference	Success
	Steps	Hz	Rate
IK Retargeting (ours)	—	${\sim}$ 5 (offline)	$86.7\pm 4.2$ %
ACT [ 4 ]	50 k	${\sim}$ 10	92%
SmolVLA [ 5 ]	20 k	${\sim}$ 10	50%
$\pi_{0.5}$ [ 6 ]	3 k	${\sim}$ 10	40%
GR00T N1.5	3 k	${\sim}$ 10	35%

TABLE III: Pick-and-Place Success Rates (tiles #1–#5, 10 grasps per tile). The IK pipeline requires no training and is evaluated across 3 runs (150 episodes); VLA policies are trained on leader–follower teleoperation demonstrations (50 episodes each).

Property	IK (ours)	Learned Policies
Training data	None	50 episodes (33 min)
Training time	0	2–8 h on T4 GPU
Inference rate	${\sim}$ 5 Hz (offline)	${\sim}$ 10 Hz
Generalisation	Task-agnostic	Task-specific (ACT) or language-conditioned
Failure mode	Hand occlusion	Gripper self-occlusion

TABLE IV: Comparison of Teleoperation Approaches

Panel of Figure 13 — Fig. 9: In-the-wild deployment in a grocery store. The robot and supporting compute setup are placed on a shopping basket for repeatable height. Left: the operator reaches for shelf items while wearing the glasses-mounted camera. Right: the robot mirrors the motion; the PyBullet preview is visible on the laptop screen.

Fig. 10: Mosaic of successful in-the-wild grasps (7 out of 75 attempts). Each pair shows the human hand capture (left) and the corresponding robot IK retargeting (right) for various store items across grocery and pharmacy environments. Despite an overall 9.3% success rate, successful transfers demonstrate the pipeline’s potential in unstructured scenes.

Fig. 11: IK solver accuracy from real pipeline data ( $N=125$ frames). Top-left: position error histogram. Top-right: orientation error histogram. Bottom-left: per-axis position error. Bottom-right: position error over time.

Joint	Raw	Smoothed	Reduction
Shoulder pan	10.4	3.3	$-68\%$
Shoulder lift	16.9	6.5	$-62\%$
Elbow flex	25.8	9.5	$-63\%$
Wrist flex	35.9	13.6	$-62\%$
Wrist roll	47.3	20.4	$-57\%$
Gripper	66.2	24.2	$-63\%$

TABLE V: Trajectory Smoothness: RMS Jerk by Joint (rad/s 3 ). EMA smoothing (

\alpha_{\text{IK}}=0.5

) reduces jerk by 57–68%.

Fig. 12: EMA smoothing ablation. Joint-space jerk (left), Cartesian position jitter (centre), and orientation jitter (right) as a function of $\alpha_{\text{IK}}$ (with $\alpha_{\text{lm}}=0.8$ fixed). The dashed line marks $\alpha_{\text{IK}}=0.5$ (used in the pipeline).

Fig. 13: End-effector trajectory analysis. Top: position and orientation jitter distributions (raw vs. EMA-smoothed). Bottom-left: 3D workspace scatter. Bottom-right: jitter reduction summary.

Joint	At Lower	At Upper	Total
Shoulder pan	0.1%	0.0%	0.1%
Shoulder lift	0.2%	0.0%	0.2%
Elbow flex	1.9%	0.0%	1.9%
Wrist flex	0.0%	4.7%	4.7%
Wrist roll	0.0%	2.1%	2.1%
Gripper	0.0%	0.0%	0.0%

TABLE VI: Joint Saturation (% of 855 frames within 0.05 rad of URDF joint limits).

Fig. 14: Sim-to-real tracking error. Per-joint tracking error (commanded minus achieved position, in motor-space units $[-100,100]$ ) during physical robot replay. The error oscillates around zero, confirming unbiased tracking. The gripper exhibits the largest transient errors.

Metric	MediaPipe	WiLoR
Hand detected	218 (62.8%)	235 (67.7%)
Valid IK target	208 (59.9%)	225 (64.8%)
No detection	129 (37.2%)	112 (32.3%)
Recovery (WiLoR found, MP missed)	12 frames (3.5%)
Detection improvement	+7.8%
Valid target improvement	+8.2%

TABLE VII: Hand detector comparison on 347 egocentric frames (left hand, structured lab environment).

Motion Retargeting

Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.

See all 104 papers

Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting
Jul 8, 2026Robert Jomar Malate, Erik Bauer, Norica Bacuieti +4Robot Teleoperation Motion Retargeting
ETH Zurich, Switzerland · mimic robotics, Switzerland · Stanford University, United States
AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance
Jul 9, 2026Chenxi Wang, Ying Feng, Hongjie Fang +4Dexterous Manipulation Robot Teleoperation
1Noematrix · 2Shanghai Jiao Tong University · 3Shanghai Innovation Institute
ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling
Jul 4, 2026Yuanchuan Lai, Qing Gao, Ziyan Liang +2Contact-Rich Robotic Manipulation Motion Retargeting
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China. · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK.

Jul 8, 2026J/K move · Enter open · S save

Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting

Robert Jomar Malate, Erik Bauer, Norica Bacuieti, Stefanos Charalambous+3

ETH Zurich, Switzerland · mimic robotics, Switzerland · Stanford University, United States

Robot Teleoperation Motion Retargeting Robotic Manipulation

Open paper AskPDF arXiv

Advances in learning-based robotic manipulation, such as Vision-Language-Action (VLA) models and Video Action Models (VAMs), heavily rely on high-quality teleoperation data. Their capabilities are strictly upper-bounded by the quality of the underlying human demonstrations. Current gradient-based retargeting algorithms often converge to different local minima, resulting in jitter that affects data quality and teleoperation experience. To address this, we introduce the Sampling-Based Retargeter (SBR), a novel gradient-free retargeting method drawn from the rich literature of sampling-based control and explicitly designed for low-jitter, real-time kinematic retargeting. We evaluate SBR both in simulation and through a rigorous real-world user study involving 18 participants performing 3 complex manipulation tasks. Compared to gradient-based baselines, SBR achieved the highest overall task success rate (54.1%) while significantly reducing operator cognitive fatigue, recording the lowest NASA-TLX workload score (36.4 out of 100). Ultimately, we establish SBR as a highly effective, intuitive retargeter for dexterous manipulation, providing the community with a rigorous benchmarking methodology to guide future retargeting research.

Abstract

Motion Retargeting

Explore similar work

Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting

AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling