cs.RONov 18, 2026

SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data

Authors: Yaping Li, Zhihong Liu, Jiacheng Li, Yi Yang, Wenchao Yue, Mingzhe Ye, Xiangcheng Liu, Yiyang Chen, 

Organizations: Sharpa Robotics · The Chinese University of Hong Kong · SJTU-Sharpa Joint Lab for Embodied AI, SJTU

Abstract

Egocentric human recordings are far cheaper than dexterous robot demonstrations, promising stronger pretraining and direct skill learning without task-specific robot data. Yet humans and robots differ in morphology and kinematics, so human recordings lack the joint positions and actuator commands that supervise robot policies, providing only retargeted end-effector (EEF) actions. This raises two questions: how human data should supervise the policy, and how a dexterous robot should read out actions at deployment. We address both in SHAPE (Supervision from Human Action tokens with Parallel Executable streams), a vision-language-action (VLA) model whose backbone predicts discrete action tokens of EEF chunks as an auxiliary task, while a shared flow-matching expert outputs two executable streams, EEF and joint. On a bimanual robot with 22-DoF hands, a matched study shows that discrete supervision helps learn from human data, driving backbone gradients and shaping action-informative representations across domains. Under standard control pipelines at deployment, direct joint readout excels at fine-grained control with robot demonstrations, scoring 0.80 across six tasks against 0.63 for π0.5, whereas IK-based EEF readout excels at human-to-robot transfer. Pretrained with human data and discrete tokens, SHAPE learns new tasks from human demonstrations without task-specific robot data.

Figures & tables

Explore similar work

CardsList
  1. EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

    Feb 18, 2026Ruijie Zheng, Dantong Niu, Yuqi Xie +12Robotic ManipulationRobot Learning

  2. AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning

    Aug 14, 2026Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5Robotic ManipulationLearning from Demonstration

  3. METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model

    Nov 21, 2025Yankai Fu, Ning Chen, Junkai Zhao +5Egocentric Video Learning for RoboticsRobotic Manipulation