Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, also with the School of Artificial Intelligence, University of Chinese Academy of Science, Beijing 100049, China, also with the CAS Engineering Laboratory for Intelligent Industrial Vision, Beijing 100190, China. · School of Mechanical and Automotive Engineering, Shanghai University of Engineering Science, Shanghai 201620 · Beijing Zhongke Huiling Robot Technology Co., Beijing 100192, China.
Achieving human-like dexterous manipulation through the collaboration of multi-fingered hands with robotic arms remains a longstanding challenge in robotics, primarily due to the scarcity of high-quality demonstrations and the complexity of high-dimensional action spaces. To address these challenges, we propose FAR-Dex, a hierarchical framework that integrates few-shot data augmentation with adaptive residual refinement to enable robust and precise arm-hand coordination in dexterous tasks. First, FAR-DexGen leverages the IsaacLab simulator to generate diverse and physically constrained trajectories from a few demonstrations, providing a data foundation for policy training. Second, FAR-DexRes introduces an adaptive residual module that refines policies by combining multi-step trajectory segments with observation features, thereby enhancing accuracy and robustness in manipulation scenarios. Experiments in both simulation and real-world demonstrate that FAR-Dex improves data quality by 13.4% and task success rates by 7% over state-of-the-art methods. It further achieves over 80% success in real-world tasks, enabling fine-grained dexterous manipulation with strong positional generalization.
Figures & tables
Fig. 1: Overview of the proposed FAR-Dex. (a) FAR-DexGen: demonstration trajectories are decomposed and transformed to generate large-scale synthetic data in the simulator. (b) FAR-DexRes: spatio-temporal residual fine-tuning is applied to four dexterous manipulation tasks. (c) Real-world validation: the trained policy is directly deployed in physical environments.
Fig. 2: FAR-Dex framework pipeline. (a) Few demonstrations Dh are segmented and augmented via spatial transformations in IsaacLab to form a large-scale dataset Dg . (b) The combined data are encoded with a pyramid convolutional network, where a consistency model distills the denoising network of πbase to accelerate inference (c) Building on abase from πbase , FAR-DexRes integrates multi-step trajectory embedding and observation features to generate adaptive weights σ for spatio-temporal residual refinement in dexterous manipulation.
Fig. 3: Data generation pipeline. Demonstration trajectories are illustrated using a top view, where transformations of the initial object pose Δci induce corresponding variations across different trajectory segments. These trajectories are then deployed in the simulation environment to collect data.
Fig. 4: Demonstration Data Collection System in the Real World.
Method
Generation Time ( Tg )
Generation Quality ( ηg )
MimiGen [ 10 ]
8.3ms
68.3%
DemoGen [ 13 ]
9.1ms
74.5%
FAR-DexGen (Ours)
10.3ms
87.9%
TABLE I: Comparison of different data generation methods
Methods
Task Success Rate ηp↑
Per-step Interference time Ts↓
Insert Cylinder
Pinch Pen
Grasp Handle
Move Card
Insert Cylinder
Pinch Pen
Grasp Handle
Move Card
ACT+3D [ 1 ]
23%
25%
46%
34%
4.5ms
4.6ms
4.5ms
4.3ms
DP3 [ 3 ]
83%
77%
80%
53%
29.1ms
31.5ms
29.8ms
29.6ms
ManiCM [ 30 ]
24%
22%
54%
45%
4.4ms
4.8ms
5.2ms
5.5ms
IDP3 [ 33 ]
80%
68%
73%
86%
31.6ms
32ms
31.7ms
32.4ms
Flow Policy [ 34 ]
25%
48%
43%
47%
3.2ms
3.7ms
3.3ms
3.6ms
TABLE II: Overall Performance Comparison with Baseline Methods in Simulation.
Fig. 5: Visualization of object interactions in simulation, compared with typical failure cases from existing methods
Ablation
Insert
Pinch
Grasp
Move
w/o RL
86%
65%
72%
73%
w/o Traj-Embed.
82%
58%
69%
80%
w/o Obs-Feat.
86%
78%
74%
82%
w/o Time-steps
87%
75%
70%
86%
FAR-DexRes (Ours)
93%
83%
88%
95%
TABLE III: Ablation Study of the RL Components.
Fig. 6: Residual Weight Variation Curves, where the Hand and Arm Weights Dynamically Change across Different Trajectory Segments.
Fig. 7: Position Generalization Success Rate Curves. As the degree of randomization increases, we record the task success rates of FAR-DexRes(Ours), DP3 [ 3 ] , and ResiP [ 16 ] .
Method
Insert
Pinch
Grasp
Move
ACT+3D [ 1 ]
20%
0%
40%
25%
DP3 [ 3 ]
65%
70%
70%
45%
ManiCM [ 30 ]
20%
0%
35%
40%
IDP3 [ 33 ]
75%
65%
65%
80%
FlowPolicy [ 34 ]
25%
10%
35%
40%
ResiP [ 16 ]
80%
70%
75%
80%
TABLE IV: Performance of FAR-DexRes in the Real World.
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Jusuk Lee, Sungha Kim, Yeonsoo Park +8
Seoul National University · University of Maryland, College Park · All Purpose AI +1
Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an SE(3) wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5
Zhejiang University · Shanghai Innovation Institute · Fudan University +2
Dexterous manipulation policies can solve individual skills, but composing them to perform multiple tasks with a single hand remains challenging. Adding a new task on top of an existing manipulation skill often imposes conflicting demands on overlapping fingers and contact modes, causing destructive interference between preserving an existing manipulation outcome and executing a new one. We propose DexCompose, a role-aware residual composition framework that reuses pretrained dexterous policies for multi-task manipulation through explicit finger-level action ownership. Given two pretrained full-hand policies, DexCompose first collects successful post-task states from the first skill and performs release tests over candidate finger masks to identify which fingers are necessary for maintaining the established skill state. It then trains two asymmetric residual modules: a bounded residual stabilizer for task preservation, and a context-aware residual that adapts the frozen downstream policy only within the action subspace assigned to the new task. We evaluate the framework on 16 composite dexterous manipulation tasks spanning four object-retention skills and four downstream interactions. DexCompose achieves a 77.4% average composite success rate, demonstrating that structural action ownership with dual residuals offers a promising direction for composing dexterous skills beyond conventional policy chaining.