DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

Amber Xie; Jiawei Fu; Xiaofeng Guo; Carmelo Sferrazza; Guanya Shi; Haozhi Qi

cs.ROOct 23, 2026

DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

Authors: Amber Xie, Jiawei Fu, Xiaofeng Guo, Carmelo Sferrazza, Guanya Shi, Haozhi Qi

Organizations: Amazon FAR (Frontier AI & Robotics) · Stanford University · UC San Diego

Abstract

Collecting high-quality demonstrations remains a central bottleneck for learning dexterous manipulation. Teleoperation records executable robot actions, but it requires a complete robot setup and places the burden of coordinating many finger joints on the operator. Human demonstrations are easier to collect, but their motions must be estimated and retargeted to the robot, and they do not directly provide robot actions. In this paper, we introduce DexPort, a portable data-collection interface for high-DoF robot hands. It uses the same hand hardware for collection and deployment, avoiding a hand embodiment gap, and records both commanded joint targets and measured joint states. Our key observation is that a small set of skills can support a wide range of dexterous tasks. During collection, the operator moves the hand-held interface to specify the robot wrist motion and selects or adjusts skills that coordinate the finger motion. This removes the need to control each joint independently while retaining behaviors such as precise grasping and in-hand manipulation. We collect demonstrations across several real-world tasks and in unstructured environments without a robot arm. We then use behavior cloning to train visuomotor policies directly on the recorded wrist motions and hand commands, without skill labels. We evaluate across 4 tasks: pick-and-place with diverse objects, screwing with in-hand tool use, drawer stowage which requires simultaneous grasp and pulling, and a bimanual scanning task. We collect over 2,500 demonstrations with our portable device, in both laboratory and in-the-wild settings. Across our tasks, we see an improvement from 11% to 64% when using recorded robot commands instead of states, and an improvement from 36% to 78% for task success when using in-the-wild data.

Figures & tables

System	DoF	Weight	Same Emb.	Action Data
UMI [ 9 ]	1	780 g	✓	✗
DexViTac [ 13 ]	6	$<$ 600 g	✗	✗
DEXOP [ 11 ]	12/9/7	$>$ 1000 g	✓	✗
DexEXO [ 14 ]	6	$>$ 1000 g	✓	✗
DEX-Mouse [ 15 ]	6	$>$ 1000 g	✓	✓
Exo-ViHa [ 16 ]	6	1100 g	✗	✓

TABLE I: Comparison of dexterous demonstration interfaces. Systems are compared by controllable hand degrees of freedom (DoF), device weight, use of the deployed robot’s hand embodiment (Same Emb.), and recording of commanded joint targets (Action Data). DexPort controls a 20-DoF hand and records the commands sent to its joints.

Fig. 1: Transferable Hardware Interface and Control Mapping. (a) Transferable Module: The Wuji hand (A) and fisheye camera (B) form a transferable hand–camera module that mounts to the Quest 2 controller for data collection and to the robot wrist for deployment. Reusing the same module preserves hand embodiment and camera-to-hand extrinsics. (b) Ergonomic Redesign: The final hardware interface places the operator’s grasp near the assembly’s center of mass and adds a support strap, shortening the moment arm and reducing wrist torque. (c) Skill Control Mapping: Quest 2 inputs start and stop demonstration episodes, select hand or screw mode, and modulate thumb, grasp, and screwing skills. The joystick adjusts thumb flexion and abduction in Thumb Mode and incrementally tightens or loosens the grasp in Finger Mode.

Fig. 2: Data Visualization. We show fisheye camera views recorded during data collection. For our pick-and-place task (PnP), we collect in-domain and in-the-wild data. For drawer stowing, we add a third-person camera because the grasped object occludes much of the scene. We show both camera views at three stages of the trajectory. For bimanual scanning and drawer stowing, the operator is occasionally visible, but we do not post-process the data to remove the operator.

Fig. 3: Task Rollouts. We show successful rollouts annotated with the phases of each task. We use these phases to score partial success. We also annotate grasp types that emerge in the rollouts, such as a pinch grasp for the cherry tomato and a power grasp for the cucumber. These annotations describe observed behavior. The policy predicts raw joint targets without skill labels.

Fig. 4: Policy Learning Results. We show imitation learning performance across four tasks. Policies trained on recorded hand commands achieve higher success rates and faster screwdriver rotation than the state-as-action baseline. Partial success for pick-and-place denotes grasping and placing one object. For drawer stowing and bimanual scanning, we report success rates for the task phases shown in Fig. 3 . A phase counts as successful only if all preceding phases also succeed. Each policy is evaluated for 15 episodes per task.

Fig. 5: In-the-Wild Co-Training. One benefit of DexPort ’s design is its portability. For our pick-and-place task, we collect 1,000 in-the-wild demonstrations (visualized in Fig. 2 ) and co-train with the 700 laboratory demonstrations. We find that increasing the number of cotrained trajectories increases the success rate of the policy in every condition. In the distractor setting, full success rises from 40% to 87%.

	DexPort	Skill-Teleop	Glove-Teleop	Human hand
Traj. / 30 min	122	68	86	275
Portable	✓	✗	✗	✓
Same hand emb.	✓	✓	✓	✗
Action data	✓	✓	✓	✗

TABLE II: Data Collection Throughput. Trajectories collected during 30 minutes of continuous operation on the unimanual pick-and-place task. For Glove Teleoperation, we use the Wuji Glove for hand tracking and the VR controller for end-effector control. For Skill Teleoperation, we use the same skill interface as DexPort to command the hand and VR tracking to control the end effector. DexPort is the only method compared here that combines portable collection with the deployed hand’s embodiment and its commanded actions.

Fig. 6: State-as-Action Failure. Policies trained with measured joint states as action labels produce weak grasps and fail to hold onto objects. Top: the baseline drops the package when presenting it to the scanner. Bottom: the baseline cannot retain the laundry once it begins to pull the drawer open.

		Success (%)
# Demos	# In-the-Wild	Seen	Unseen	Distractors
500	0	0	20	0
500	500	33	13	53
500	1,000	47	7	40
700	0	47	20	40
700	500	53	53	53

TABLE III: Scaling In-the-Wild Data. Pick-and-place success rates (%) of policies trained on 500 or 700 in-scene laboratory demonstrations (# Demos) and 0, 500, or 1,000 in-the-wild demonstrations (# In-the-Wild). We evaluate on seen objects, unseen objects, and scenes containing distractors. In-the-wild data improves full success on seen objects and with distractors in both data regimes, but improves success on unseen objects only with 700 laboratory demonstrations.

Hyperparameter	Value
RL library	RSL-RL [ 35 ]
Parallel environments	$4096$
Rollout length	$24$ steps
Iterations	$1200$ ( $1.18\times 10^{8}$ steps)
Learning rate	$5\times 10^{-3}$ , adaptive (KL target $0.01$ )
Discount $\gamma$ / GAE $\lambda$	$0.99$ / $0.95$

TABLE IV: PPO Hyperparameters. Settings for training the in-hand rotation skill in Isaac Lab [ 33 ] . The simulation runs at 200 Hz with 10 physics steps per policy step.

Term	Weight	Definition
Handle rotation	$6.0$	$\mathrm{clip}(\dot{\theta},-4,4)$
Fingertip proximity	$2.0$	$\mathrm{clip}(1-\bar{d}/0.035,0,1)$
Action smoothness	$1.0$	$\exp\!\big(-\overline{(a_{t}-a_{t-1})^{2}}/0.5^{2}\big)$
Pose deviation	$-6.0$	$\lVert q-q_{0}\rVert^{2}$
Over-speed	$-0.3$	$\max(\dot{\theta}-10,0)$
Joint torque	$-0.1$	$\lVert\tau\rVert^{2}$

TABLE V: Rotation Skill Rewards.

\dot{\theta}

is the handle’s angular velocity in rad/s,

\bar{d}

is the mean horizontal distance in meters from the thumb, index, and middle fingertips to the handle axis,

q

and

q_{0}

are the current and initial joint position vectors, and

\tau

is the joint torque vector.

a_{t}

is the policy action at step

t

\dot{q}_{j}

is the velocity of joint

j

, and the overline in the action smoothness term denotes the mean over joints.

Hyperparameter	Value
Vision encoder	DINOv2 ViT-S/14 [ 37 ] , fine-tuned
Denoiser	1D conditional U-Net [ 34 ] , FiLM [ 38 ]
U-Net channels per level
Pick-and-place in-the-wild	$(512,1024,2048)$
Pick-and-place baseline	$(256,512,1024)$
Screwdriver	$(512,1024,2048)$

TABLE VI: Diffusion Policy Hyperparameters. Settings are shared across tasks except for the task-specific U-Net channel sizes listed above. Color jitter lists the brightness, contrast, saturation, and hue ranges. At inference, images are center-cropped to 212 px and resized to 224 px.

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 175 papers

TeleDex: Accessible Dexterous Teleoperation
Mar 17, 2026Omar Rayyan, Maximilian Gilles, Yuchen CuiRobot Teleoperation Robotic Data Collection
University of California, Los Angeles
Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations
Oct 9, 2026Jiachen Zhang, Yuanyuan Li, Shaqi LuoRobotic Manipulation Cross-Embodiment Robot Learning
Beijing Academy of Artificial Intelligence, Beijing, China.
DEX-Mouse: A Low-cost Portable and Universal Interface with Force Feedback for Data Collection of Dexterous Robotic Hands
Apr 16, 2026Joonho Koh, Haechan Jung, Nayoung Kim +2Dexterous Manipulation Robotic Data Collection
Dept. of Artificial Intelligence, Sogang University, Korea · Dept. of Electronic Engineering, Sogang University, Korea · Dept. of Mechanical Engineering, Sogang University, Korea +1

Mar 17, 2026J/K move · Enter open · S save

TeleDex: Accessible Dexterous Teleoperation

Omar Rayyan, Maximilian Gilles, Yuchen Cui

University of California, Los Angeles

Robot Teleoperation Robotic Data Collection Dexterous Manipulation

Open paper AskPDF arXiv

Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.

Abstract

Robotic Data Collection

Explore similar work

TeleDex: Accessible Dexterous Teleoperation

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

DEX-Mouse: A Low-cost Portable and Universal Interface with Force Feedback for Data Collection of Dexterous Robotic Hands