Scaling Dexterous Manipulation via Action Representation Pretraining

Yanjinghao Xu; Jiexin Zhang; Huan Zhao; Jianjia Mo; Zhenghao Yin; Dingkang Liang; Xiang Bai; Han Ding

cs.ROOct 25, 2026

Scaling Dexterous Manipulation via Action Representation Pretraining

Authors: Yanjinghao Xu, Jiexin Zhang, Huan Zhao, Jianjia Mo, Zhenghao Yin, Dingkang Liang, Xiang Bai, Han Ding

Organizations: Huazhong University of Science and Technology

Abstract

Large-scale vision, language, and video pretraining has substantially improved the perception and generalization capabilities of robotic manipulation models, but action generation for complex dexterous manipulation remains constrained by the scarcity of robot demonstrations. Limited robot demonstrations must be used to learn both action representations and task-conditioned action distributions, making it difficult to adequately learn action representations and, in turn, limiting the ability of downstream generative models to generate complex dexterous actions. To address this limitation, we propose ARPG (Action Representation Pretraining and Generation), which explicitly separates action representation learning from action generation: we first pretrain an action representation encoder on large-scale human manipulation data. We then freeze the encoder and train a compact generative backbone (130M parameters) to learn task-conditioned action distributions in the encoder's representation space. Experimental results show that ARPG achieves state-of-the-art performance on the DexJoCo benchmark and improves the average success rate on bimanual dexterous manipulation from 10.1% for $π_{0.5}$ to 44.7%. Scaling experiments further show that broader coverage of human manipulation behaviors effectively improves downstream performance. These results indicate that action representation pretraining provides an independent scaling dimension for improving complex dexterous manipulation capabilities.

Figures & tables

Figure 1: ARPG framework. Stage 1a pretrains the action encoder through vision–action alignment on human manipulation data; Stage 1b trains the action decoder using demonstrations from the target robot; Stage 2 learns conditional action generation in the frozen action representation space. The action encoder is used only to construct training targets and does not participate in inference.

Single-task training
Method	Inference cost			Success rate (%)
	Params (B) $\downarrow$	VRAM (GB) $\downarrow$	Latency (ms) $\downarrow$	Single arm $\uparrow$	Bimanual $\uparrow$	Overall $\uparrow$
$\pi_{0.5}$	3.35	9.05	84.9	69.7	35.3	54.1
GR00T N1.5	2.72	6.21	93.6	56.2	21.1	40.2
ARPG	0.46	2.63	50.8	75.2	44.5	61.3
Multi-task training

Table 1: Dexterous manipulation performance and deployment efficiency on the DexJoCo benchmark. We report macro-average success rates (%) over single-arm, bimanual, and all tasks under single-task and multi-task training, together with inference costs. Complete per-task results are in Appendix H . Bold indicates the best result within each setting.

Figure 2: Real-world task success rates. We compare ARPG and $\pi_{0.5}$ on Hammer Handover, Pouring Coffee Beans, and Cabinet Retrieval. Each result is the proportion of successful rollouts over 30 trials; photographs show the task setups.

Figure 3: Scaling action representation pretraining. DexJoCo average success rate as a function of (a) human pretraining data, (b) action encoder parameter count, and (c) pretraining steps. Panel (b) uses a base-2 logarithmic x-axis. In (a), the solid line connects the 10–500-hour results, the dashed blue segment connects 500 and 3,000 hours, and the dashed gray line marks the Robot-only reference.

(a) Action representation		(b) Aggregation depth		(c) Auxiliary objectives
Representation	Overall $\uparrow$	Last- $K$	Overall $\uparrow$	Objective	Overall $\uparrow$
Raw Action	13.3	1	52.5	w/o $\mathcal{L}_{\mathrm{REPA}}$	54.9
Action-MAE	36.4	2	57.8	w/o $\mathcal{L}_{\mathrm{dec}}$	56.0
Action-iBOT	30.5	3	51.6	Full	57.8
Visual Alignment + iBOT	45.6
Visual Alignment	57.8

Table 2: Ablations of action representation and generative design. We report the overall success rate (%) on DexJoCo.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hand	Action component	Index range	Dim.	Decoder output
Left	15 local joint rotations (6D)	$[0,90)$	90	Yes
Left	Palm rotation (6D)	$[90,96)$	6	Yes
Left	Absolute wrist position	$[96,99)$	3	Yes
Left	21 wrist-relative keypoints	$[99,162)$	63	No
Right	15 local joint rotations (6D)	$[162,252)$	90	Yes
Right	Palm rotation (6D)	$[252,258)$	6	Yes

Appendix

Table 3: Detailed layout of the unified action format. Indices are zero-based and half-open.

Metric	500 h	3,000 h
Summed clip duration	500 h 20 m 16 s	3,088 h 16 m 59 s
Total samples	1,801,216	11,117,819
Number of tasks	17,382	23,316
Largest single-task share	0.09%	9.85%
Top-10 task share	0.77%	23.87%
Mean samples per task	103.63	476.83

Appendix

Table 4: Scale and task distribution of human-manipulation pretraining data.

Task	Task description	Demonstrations	Duration
Hammer Handover	Pick up, hand over, and place a hammer	40	3:19:31
Pouring Coffee Beans	Pick up a container and pour coffee beans	40	2:15:03
Cabinet Retrieval	Open a cabinet, retrieve the target, and close it	40	1:47:31
Total	—	120	7:22:05

Appendix

Table 5: Data scale for the three real-robot tasks. Durations are estimated original recording times (h:mm:ss).

Pretraining objective	Visual Alignment
Optimizer	AdamW
Batch size	1,536
Learning rate	$2\times 10^{-4}$
Minimum learning rate	$2\times 10^{-6}$
Weight decay	0.05
Warmup steps	1,000

Appendix

Table 9

Task	Stage 1	Stage 2	Stage 3
Hammer Handover	Pick up the hammer	Complete the bimanual handover	Place the hammer at the target
Pouring Coffee Beans	Pick up the bean container	Successfully pour out the coffee beans	Completely pour out the coffee beans
Cabinet Retrieval	Open the cabinet door	Retrieve the target object	Close the cabinet door

Appendix

Table 8: Stage definitions for the real-world evaluation tasks.

Figure 4: Hammer Handover: Detailed Procedure. The right hand picks up the hammer, hands it to the left hand, and the left hand puts it down.

Figure 5: Pouring Coffee Beans: Detailed Procedure. Both hands pick up measuring cups, and the coffee beans are poured from the left-hand cup into the right-hand cup.

Figure 6: Cabinet Retrieval: Detailed Procedure. The right hand opens the cabinet, the left hand retrieves the object, and the right hand closes the cabinet.

Single-arm
Task	Single-task training			Multi-task training
	$\pi_{0.5}$	GR00T N1.5	ARPG	$\pi_{0.5}$	ARPG
Hammer Nail	84.7 $\pm$ 5.0	67.3 $\pm$ 4.2	72.0 $\pm$ 9.2	74.7 $\pm$ 2.3	52.7 $\pm$ 5.0
Click Mouse	64.7 $\pm$ 8.1	85.3 $\pm$ 3.1	73.3 $\pm$ 8.1	48.0 $\pm$ 6.9	96.0 $\pm$ 3.3
Pick Bucket	84.0 $\pm$ 7.2	72.0 $\pm$ 6.0	80.0 $\pm$ 2.0	60.7 $\pm$ 2.3	78.0 $\pm$ 1.6
Pinch Tongs	24.0 $\pm$ 6.9	12.7 $\pm$ 2.3	72.7 $\pm$ 13.0	44.0 $\pm$ 15.9	50.0 $\pm$ 1.6

Appendix

Table 9: Full task-level results on the DexJoCo benchmark.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
Aug 14, 2026Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5Robotic Manipulation Learning from Demonstration
Zhejiang University · Shanghai Innovation Institute · Fudan University +2
METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model
Nov 21, 2025Yankai Fu, Ning Chen, Junkai Zhao +5Robotic Manipulation VLMs for Robotics
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence
IronMind: Scaling Humanoid Dexterous Manipulation via Camera-Space Ego-Centric Pretraining
Sep 30, 2026Huimin Pan, Yufan Ren, Kunpeng Song +16Cross-Embodiment Robot Learning Vision-Language-Action Models
World Model Team, XPENG Robotics

Aug 14, 2026J/K move · Enter open · S save

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning

Zhiyue Zhao, Jingyi Wu, Hairuo Liu, Mingyu Liu+4

Zhejiang University · Shanghai Innovation Institute · Fudan University+2

Robotic Manipulation Learning from Demonstration Cross-Embodiment Robot Learning Zero-Shot Robotic Manipulation

Open paper AskPDF arXiv

Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an

\mathrm{SE}(3)

wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.

Abstract

Robotic Manipulation

Explore similar work

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning

METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model

IronMind: Scaling Humanoid Dexterous Manipulation via Camera-Space Ego-Centric Pretraining