DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

Amber Xie; Jiawei Fu; Xiaofeng Guo; Carmelo Sferrazza; Guanya Shi; Haozhi Qi

cs.ROOct 23, 2026

DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

Authors: Amber Xie, Jiawei Fu, Xiaofeng Guo, Carmelo Sferrazza, Guanya Shi, Haozhi Qi

Organizations: Amazon FAR (Frontier AI & Robotics) · Stanford University · UC San Diego

Abstract

Collecting high-quality demonstrations remains a central bottleneck for learning dexterous manipulation. Teleoperation records executable robot actions, but it requires a complete robot setup and places the burden of coordinating many finger joints on the operator. Human demonstrations are easier to collect, but their motions must be estimated and retargeted to the robot, and they do not directly provide robot actions. In this paper, we introduce DexPort, a portable data-collection interface for high-DoF robot hands. It uses the same hand hardware for collection and deployment, avoiding a hand embodiment gap, and records both commanded joint targets and measured joint states. Our key observation is that a small set of skills can support a wide range of dexterous tasks. During collection, the operator moves the hand-held interface to specify the robot wrist motion and selects or adjusts skills that coordinate the finger motion. This removes the need to control each joint independently while retaining behaviors such as precise grasping and in-hand manipulation. We collect demonstrations across several real-world tasks and in unstructured environments without a robot arm. We then use behavior cloning to train visuomotor policies directly on the recorded wrist motions and hand commands, without skill labels. We evaluate across 4 tasks: pick-and-place with diverse objects, screwing with in-hand tool use, drawer stowage which requires simultaneous grasp and pulling, and a bimanual scanning task. We collect over 2,500 demonstrations with our portable device, in both laboratory and in-the-wild settings. Across our tasks, we see an improvement from 11% to 64% when using recorded robot commands instead of states, and an improvement from 36% to 78% for task success when using in-the-wild data.

Figures & tables

System	DoF	Weight	Same Emb.	Action Data
UMI [ 9 ]	1	780 g	✓	✗
DexViTac [ 13 ]	6	$<$ 600 g	✗	✗
DEXOP [ 11 ]	12/9/7	$>$ 1000 g	✓	✗
DexEXO [ 14 ]	6	$>$ 1000 g	✓	✗
DEX-Mouse [ 15 ]	6	$>$ 1000 g	✓	✓
Exo-ViHa [ 16 ]	6	1100 g	✗	✓

TABLE I: Comparison of dexterous demonstration interfaces. Systems are compared by controllable hand degrees of freedom (DoF), device weight, use of the deployed robot’s hand embodiment (Same Emb.), and recording of commanded joint targets (Action Data). DexPort controls a 20-DoF hand and records the commands sent to its joints.

Fig. 1: Transferable Hardware Interface and Control Mapping. (a) Transferable Module: The Wuji hand (A) and fisheye camera (B) form a transferable hand–camera module that mounts to the Quest 2 controller for data collection and to the robot wrist for deployment. Reusing the same module preserves hand embodiment and camera-to-hand extrinsics. (b) Ergonomic Redesign: The final hardware interface places the operator’s grasp near the assembly’s center of mass and adds a support strap, shortening the moment arm and reducing wrist torque. (c) Skill Control Mapping: Quest 2 inputs start and stop demonstration episodes, select hand or screw mode, and modulate thumb, grasp, and screwing skills. The joystick adjusts thumb flexion and abduction in Thumb Mode and incrementally tightens or loosens the grasp in Finger Mode.

Fig. 2: Data Visualization. We show fisheye camera views recorded during data collection. For our pick-and-place task (PnP), we collect in-domain and in-the-wild data. For drawer stowing, we add a third-person camera because the grasped object occludes much of the scene. We show both camera views at three stages of the trajectory. For bimanual scanning and drawer stowing, the operator is occasionally visible, but we do not post-process the data to remove the operator.

Fig. 3: Task Rollouts. We show successful rollouts annotated with the phases of each task. We use these phases to score partial success. We also annotate grasp types that emerge in the rollouts, such as a pinch grasp for the cherry tomato and a power grasp for the cucumber. These annotations describe observed behavior. The policy predicts raw joint targets without skill labels.

Fig. 4: Policy Learning Results. We show imitation learning performance across four tasks. Policies trained on recorded hand commands achieve higher success rates and faster screwdriver rotation than the state-as-action baseline. Partial success for pick-and-place denotes grasping and placing one object. For drawer stowing and bimanual scanning, we report success rates for the task phases shown in Fig. 3 . A phase counts as successful only if all preceding phases also succeed. Each policy is evaluated for 15 episodes per task.

Fig. 5: In-the-Wild Co-Training. One benefit of DexPort ’s design is its portability. For our pick-and-place task, we collect 1,000 in-the-wild demonstrations (visualized in Fig. 2 ) and co-train with the 700 laboratory demonstrations. We find that increasing the number of cotrained trajectories increases the success rate of the policy in every condition. In the distractor setting, full success rises from 40% to 87%.

	DexPort	Skill-Teleop	Glove-Teleop	Human hand
Traj. / 30 min	122	68	86	275
Portable	✓	✗	✗	✓
Same hand emb.	✓	✓	✓	✗
Action data	✓	✓	✓	✗

TABLE II: Data Collection Throughput. Trajectories collected during 30 minutes of continuous operation on the unimanual pick-and-place task. For Glove Teleoperation, we use the Wuji Glove for hand tracking and the VR controller for end-effector control. For Skill Teleoperation, we use the same skill interface as DexPort to command the hand and VR tracking to control the end effector. DexPort is the only method compared here that combines portable collection with the deployed hand’s embodiment and its commanded actions.

Fig. 6: State-as-Action Failure. Policies trained with measured joint states as action labels produce weak grasps and fail to hold onto objects. Top: the baseline drops the package when presenting it to the scanner. Bottom: the baseline cannot retain the laundry once it begins to pull the drawer open.

		Success (%)
# Demos	# In-the-Wild	Seen	Unseen	Distractors
500	0	0	20	0
500	500	33	13	53
500	1,000	47	7	40
700	0	47	20	40
700	500	53	53	53

TABLE III: Scaling In-the-Wild Data. Pick-and-place success rates (%) of policies trained on 500 or 700 in-scene laboratory demonstrations (# Demos) and 0, 500, or 1,000 in-the-wild demonstrations (# In-the-Wild). We evaluate on seen objects, unseen objects, and scenes containing distractors. In-the-wild data improves full success on seen objects and with distractors in both data regimes, but improves success on unseen objects only with 700 laboratory demonstrations.

Hyperparameter	Value
RL library	RSL-RL [ 35 ]
Parallel environments	$4096$
Rollout length	$24$ steps
Iterations	$1200$ ( $1.18\times 10^{8}$ steps)
Learning rate	$5\times 10^{-3}$ , adaptive (KL target $0.01$ )
Discount $\gamma$ / GAE $\lambda$	$0.99$ / $0.95$

TABLE IV: PPO Hyperparameters. Settings for training the in-hand rotation skill in Isaac Lab [ 33 ] . The simulation runs at 200 Hz with 10 physics steps per policy step.

Term	Weight	Definition
Handle rotation	$6.0$	$\mathrm{clip}(\dot{\theta},-4,4)$
Fingertip proximity	$2.0$	$\mathrm{clip}(1-\bar{d}/0.035,0,1)$
Action smoothness	$1.0$	$\exp\!\big(-\overline{(a_{t}-a_{t-1})^{2}}/0.5^{2}\big)$
Pose deviation	$-6.0$	$\lVert q-q_{0}\rVert^{2}$
Over-speed	$-0.3$	$\max(\dot{\theta}-10,0)$
Joint torque	$-0.1$	$\lVert\tau\rVert^{2}$

TABLE V: Rotation Skill Rewards.

\dot{\theta}

is the handle’s angular velocity in rad/s,

\bar{d}

is the mean horizontal distance in meters from the thumb, index, and middle fingertips to the handle axis,

q

and

q_{0}

are the current and initial joint position vectors, and

\tau

is the joint torque vector.

a_{t}

is the policy action at step

t

\dot{q}_{j}

is the velocity of joint

j

, and the overline in the action smoothness term denotes the mean over joints.

Hyperparameter	Value
Vision encoder	DINOv2 ViT-S/14 [ 37 ] , fine-tuned
Denoiser	1D conditional U-Net [ 34 ] , FiLM [ 38 ]
U-Net channels per level
Pick-and-place in-the-wild	$(512,1024,2048)$
Pick-and-place baseline	$(256,512,1024)$
Screwdriver	$(512,1024,2048)$

TABLE VI: Diffusion Policy Hyperparameters. Settings are shared across tasks except for the task-specific U-Net channel sizes listed above. Color jitter lists the brightness, contrast, saturation, and hue ranges. At inference, images are center-cropped to 212 px and resized to 224 px.

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 175 papers

Mar 17, 2026

TeleDex: Accessible Dexterous Teleoperation

Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.

Omar Rayyan, Maximilian Gilles, Yuchen Cui

University of California, Los Angeles

Robot Teleoperation Robotic Data Collection Dexterous Manipulation

Ask PDF

Oct 9, 2026

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

Collecting dexterous manipulation demonstrations for each robot hand requires repeated task-specific data acquisition. We ask whether one shared corpus of natural human demonstrations can instead serve as a reusable data interface for different dexterous hands, without collecting task demonstrations on each target robot. We present Human2Dex, a wrist-centric human demonstration interface that separates shared human observation and motion semantics from embodiment-specific execution. It aligns observations through local hand-appearance adaptation and a shared 21-point hand representation with consistent structural semantics across human and robot observations. It constructs executable supervision by retargeting fused human hand motion to each target hand, and introduces the Grasp-Pocket Interaction Representation (GPIR) to express tracked objects relative to a hand-centered Functional Grasp Pocket. The corpus is converted into separate training sets for Linker O6 and Wuji, while action labels, normalization, training, and diffusion-policy checkpoints remain hand-specific. Across six real-world tasks, Human2Dex achieves 109/120 (90.8%) successes on Linker O6 and 97/120 (80.8%) on Wuji. Across both hands, Human-Raw achieves 63/240 (26.3%) successes, whereas the complete Human2Dex configuration achieves 206/240 (85.8%). These results support the use of wrist-centric human demonstrations as a reusable interface across calibrated dexterous hands while retaining embodiment-specific policy learning. The project website is publicly available at https://human2dex.pages.dev/.

Jiachen Zhang, Yuanyuan Li, Shaqi Luo

Beijing Academy of Artificial Intelligence, Beijing, China.

Robotic Manipulation Cross-Embodiment Robot Learning Robot Imitation Learning Data-Efficient Robot Learning Hand-Object Interaction Motion Retargeting

Ask PDF

Apr 16, 2026

DEX-Mouse: A Low-cost Portable and Universal Interface with Force Feedback for Data Collection of Dexterous Robotic Hands

Data-driven dexterous hand manipulation requires large-scale, physically consistent demonstration data. Simulation and video-based methods suffer from sim-to-real gaps and retargeting problems, while MoCap glove-based teleoperation systems require per-operator calibration and lack portability, as the robot hand is typically fixed to a stationary arm. Portable alternatives improve mobility but lack cross-platform and cross-operator compatibility. We present DEX-Mouse, a portable, calibration-free hand-held teleoperation interface with integrated kinesthetic force feedback, built from commercial off-the-shelf components under USD 150. The operator-agnostic design requires no calibration or structural modification, enabling immediate deployment across diverse environments and platforms. The interface supports a configuration in which the target robot hand is mounted directly on the forearm of an operator, producing robot-aligned data. In a comparative user study across various dexterous manipulation tasks, operators using the proposed system achieved an 86.67% task completion rate under the attached configuration. Also, we found that the attached configuration reduced the perceived workload of the operators compared to spatially separated teleoperation setups across all compared interfaces. The complete hardware and software stack, including bill of materials, CAD models, and firmware, is open-sourced at https://dex-mouse.github.io/ to facilitate replication and adoption.

Joonho Koh, Haechan Jung, Nayoung Kim +2

Dept. of Artificial Intelligence, Sogang University, Korea · Dept. of Electronic Engineering, Sogang University, Korea · Dept. of Mechanical Engineering, Sogang University, Korea +1

Dexterous Manipulation Robotic Data Collection Robot Teleoperation

Ask PDF

Abstract

Robotic Data Collection

Explore similar work

TeleDex: Accessible Dexterous Teleoperation

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

DEX-Mouse: A Low-cost Portable and Universal Interface with Force Feedback for Data Collection of Dexterous Robotic Hands