DEX-PHD: A Dexterous Portable Handheld Device for Multimodal Data Collection

Jooyoung Hong; Noboru Myers; Obin Kwon; Yixian Yang; Joohyung Kim

cs.ROOct 26, 2026

DEX-PHD: A Dexterous Portable Handheld Device for Multimodal Data Collection

Authors: Jooyoung Hong, Noboru Myers, Obin Kwon, Yixian Yang, Joohyung Kim

Organizations: All authors are with the KIMLAB (Kinetic Intelligent Machine LAB), University of Illinois, Urbana-Champaign, IL 61801, USA

Abstract

Existing manipulation data collection methods typically offer either accurate robot-state acquisition through kinematic-chain-based teleoperation or convenient, portable operation through vision-based approaches. We present DEX-PHD, a portable handheld device that combines these advantages for dexterous manipulation by integrating a dexterous robot hand, continuous per-finger control, and multimodal logging. The system records proprioceptive information, fingertip contact data, and wrist-camera video, while the operator directly controls the robot hand through joint-level mapping. Its untethered design enables data collection across diverse environments. We demonstrate DEX-PHD on contact-rich tasks requiring coordinated multi-finger manipulation and deploy the collected demonstrations through open-loop replay on a physical humanoid robot. The results show that open-loop replay preserves the main temporal patterns of finger motion and task-relevant contact. Therefore, DEX-PHD provides a practical platform for portable, multimodal dexterous manipulation data collection.

Figures & tables

Fig. 1 : DEX-PHD: A dexterous portable handheld device for collecting multimodal demonstrations with a dexterous robot hand. By placing the handle alongside the robot hand, DEX-PHD enables intuitive, per-finger control and collects time-synchronized multimodal data from the dexterous robot hand and the wrist camera.

Fig. 2 : DEX-PHD hardware overview. (a) Continuous joint mapping via trigger buttons and a thumb knob, with tactile sensing from six FSRs per finger and SD card logging. (b) Handle design enabling side-by-side operation with the human hand. (c) Aligned device-hand bottom planes for table-level reach during planar manipulation.

Microcontroller	Teensy 4.0
Robot hand	Ability Hand (Psyonic)
Control fingers	Index, middle, ring, pinkie, thumb, thumb rotation
Finger range	$0\sim 110^{\circ}$
Wrist camera	Hero 11 Black (GoPro)
Data format	CSV (Hand) & MP4 (Camera)
Weight	1.11 kg (0.66 kg without the hand)

TABLE I : Specification of DEX-PHD

Panel of Figure 4 — Fig. 3 : Data collected with DEX-PHD across diverse indoor and outdoor environments and subsequently replayed on a physical humanoid robot.

Panel of Figure 5 — Fig. 4 : Wrist-trajectory generation and cross-embodiment deployment of data collected with DEX-PHD.

Fig. 5 : Hardware setup for the humanoid robot.

Panel of Figure 7 — Fig. 6 : Force and position trajectories of fingers during tasks.

Task	Joint position MAE [deg]	Joint position correlation	Task-relevant contact correlation
Peg-in-hole	2.66	0.560	0.197
Multi-grasping	5.48	0.691	0.691
Drilling	7.54	0.803	0.761

TABLE II : Quantitative analysis

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 178 papers

DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection
Oct 23, 2026Amber Xie, Jiawei Fu, Xiaofeng Guo +3Robotic Data Collection Robotic Manipulation
Amazon FAR (Frontier AI & Robotics) · Stanford University · UC San Diego
DEX-Mouse: A Low-cost Portable and Universal Interface with Force Feedback for Data Collection of Dexterous Robotic Hands
Apr 16, 2026Joonho Koh, Haechan Jung, Nayoung Kim +2Dexterous Manipulation Robotic Data Collection
Dept. of Artificial Intelligence, Sogang University, Korea · Dept. of Electronic Engineering, Sogang University, Korea · Dept. of Mechanical Engineering, Sogang University, Korea +1
TeleDex: Accessible Dexterous Teleoperation
Mar 17, 2026Omar Rayyan, Maximilian Gilles, Yuchen CuiRobot Teleoperation Robotic Data Collection
University of California, Los Angeles

Oct 23, 2026J/K move · Enter open · S save

DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

Amber Xie, Jiawei Fu, Xiaofeng Guo, Carmelo Sferrazza+2

Amazon FAR (Frontier AI & Robotics) · Stanford University · UC San Diego

Robotic Data Collection Robotic Manipulation Robot Manipulation Datasets Robot Imitation Learning

Open paper AskPDF arXiv

Collecting high-quality demonstrations remains a central bottleneck for learning dexterous manipulation. Teleoperation records executable robot actions, but it requires a complete robot setup and places the burden of coordinating many finger joints on the operator. Human demonstrations are easier to collect, but their motions must be estimated and retargeted to the robot, and they do not directly provide robot actions. In this paper, we introduce DexPort, a portable data-collection interface for high-DoF robot hands. It uses the same hand hardware for collection and deployment, avoiding a hand embodiment gap, and records both commanded joint targets and measured joint states. Our key observation is that a small set of skills can support a wide range of dexterous tasks. During collection, the operator moves the hand-held interface to specify the robot wrist motion and selects or adjusts skills that coordinate the finger motion. This removes the need to control each joint independently while retaining behaviors such as precise grasping and in-hand manipulation. We collect demonstrations across several real-world tasks and in unstructured environments without a robot arm. We then use behavior cloning to train visuomotor policies directly on the recorded wrist motions and hand commands, without skill labels. We evaluate across 4 tasks: pick-and-place with diverse objects, screwing with in-hand tool use, drawer stowage which requires simultaneous grasp and pulling, and a bimanual scanning task. We collect over 2,500 demonstrations with our portable device, in both laboratory and in-the-wild settings. Across our tasks, we see an improvement from 11% to 64% when using recorded robot commands instead of states, and an improvement from 36% to 78% for task success when using in-the-wild data.

Abstract

Robotic Data Collection

Explore similar work

DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection

DEX-Mouse: A Low-cost Portable and Universal Interface with Force Feedback for Data Collection of Dexterous Robotic Hands

TeleDex: Accessible Dexterous Teleoperation