FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulation

Yushan Bai; Fulin Chen; Hongzheng Sun; Yuchuang Tong; En Li; Zhengtao Zhang

cs.ROMar 11, 2026

FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulation

Authors: Yushan Bai, Fulin Chen, Hongzheng Sun, Yuchuang Tong, En Li, Zhengtao Zhang

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, also with the School of Artificial Intelligence, University of Chinese Academy of Science, Beijing 100049, China, also with the CAS Engineering Laboratory for Intelligent Industrial Vision, Beijing 100190, China. · School of Mechanical and Automotive Engineering, Shanghai University of Engineering Science, Shanghai 201620 · Beijing Zhongke Huiling Robot Technology Co., Beijing 100192, China.

Abstract

Achieving human-like dexterous manipulation through the collaboration of multi-fingered hands with robotic arms remains a longstanding challenge in robotics, primarily due to the scarcity of high-quality demonstrations and the complexity of high-dimensional action spaces. To address these challenges, we propose FAR-Dex, a hierarchical framework that integrates few-shot data augmentation with adaptive residual refinement to enable robust and precise arm-hand coordination in dexterous tasks. First, FAR-DexGen leverages the IsaacLab simulator to generate diverse and physically constrained trajectories from a few demonstrations, providing a data foundation for policy training. Second, FAR-DexRes introduces an adaptive residual module that refines policies by combining multi-step trajectory segments with observation features, thereby enhancing accuracy and robustness in manipulation scenarios. Experiments in both simulation and real-world demonstrate that FAR-Dex improves data quality by 13.4% and task success rates by 7% over state-of-the-art methods. It further achieves over 80% success in real-world tasks, enabling fine-grained dexterous manipulation with strong positional generalization.

Figures & tables

Fig. 1: Overview of the proposed FAR-Dex. (a) FAR-DexGen: demonstration trajectories are decomposed and transformed to generate large-scale synthetic data in the simulator. (b) FAR-DexRes: spatio-temporal residual fine-tuning is applied to four dexterous manipulation tasks. (c) Real-world validation: the trained policy is directly deployed in physical environments.

Fig. 2: FAR-Dex framework pipeline. (a) Few demonstrations $D_{h}$ are segmented and augmented via spatial transformations in IsaacLab to form a large-scale dataset $D_{g}$ . (b) The combined data are encoded with a pyramid convolutional network, where a consistency model distills the denoising network of $\pi_{\text{base}}$ to accelerate inference (c) Building on $a_{\text{base}}$ from $\pi_{\text{base}}$ , FAR-DexRes integrates multi-step trajectory embedding and observation features to generate adaptive weights $\sigma$ for spatio-temporal residual refinement in dexterous manipulation.

Fig. 3: Data generation pipeline. Demonstration trajectories are illustrated using a top view, where transformations of the initial object pose $\Delta c_{i}$ induce corresponding variations across different trajectory segments. These trajectories are then deployed in the simulation environment to collect data.

Fig. 4: Demonstration Data Collection System in the Real World.

Method	Generation Time ( $T_{g}$ )	Generation Quality ( $\eta_{g}$ )
MimiGen [ 10 ]	8.3ms	68.3%
DemoGen [ 13 ]	9.1ms	74.5%
FAR-DexGen (Ours)	10.3ms	87.9%

TABLE I: Comparison of different data generation methods

Methods	Task Success Rate $\eta_{p}$ $\uparrow$				Per-step Interference time $T_{s}$ $\downarrow$
Methods	Insert Cylinder	Pinch Pen	Grasp Handle	Move Card	Insert Cylinder	Pinch Pen	Grasp Handle	Move Card
ACT+3D [ 1 ]	23%	25%	46%	34%	4.5ms	4.6ms	4.5ms	4.3ms
DP3 [ 3 ]	83%	77%	80%	53%	29.1ms	31.5ms	29.8ms	29.6ms
ManiCM [ 30 ]	24%	22%	54%	45%	4.4ms	4.8ms	5.2ms	5.5ms
IDP3 [ 33 ]	80%	68%	73%	86%	31.6ms	32ms	31.7ms	32.4ms
Flow Policy [ 34 ]	25%	48%	43%	47%	3.2ms	3.7ms	3.3ms	3.6ms

TABLE II: Overall Performance Comparison with Baseline Methods in Simulation.

Fig. 5: Visualization of object interactions in simulation, compared with typical failure cases from existing methods

Ablation	Insert	Pinch	Grasp	Move
w/o RL	86%	65%	72%	73%
w/o Traj-Embed.	82%	58%	69%	80%
w/o Obs-Feat.	86%	78%	74%	82%
w/o Time-steps	87%	75%	70%	86%
FAR-DexRes (Ours)	93%	83%	88%	95%

TABLE III: Ablation Study of the RL Components.

Fig. 6: Residual Weight Variation Curves, where the Hand and Arm Weights Dynamically Change across Different Trajectory Segments.

Fig. 7: Position Generalization Success Rate Curves. As the degree of randomization increases, we record the task success rates of FAR-DexRes(Ours), DP3 [ 3 ] , and ResiP [ 16 ] .

Method	Insert	Pinch	Grasp	Move
ACT+3D [ 1 ]	20%	0%	40%	25%
DP3 [ 3 ]	65%	70%	70%	45%
ManiCM [ 30 ]	20%	0%	35%	40%
IDP3 [ 33 ]	75%	65%	65%	80%
FlowPolicy [ 34 ]	25%	10%	35%	40%
ResiP [ 16 ]	80%	70%	75%	80%

TABLE IV: Performance of FAR-DexRes in the Real World.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Oct 8, 2026Jusuk Lee, Sungha Kim, Yeonsoo Park +8Robotic Manipulation Learning from Demonstration
Seoul National University · University of Maryland, College Park · All Purpose AI +1
AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
Aug 14, 2026Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5Robotic Manipulation Learning from Demonstration
Zhejiang University · Shanghai Innovation Institute · Fudan University +2
DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand
Jun 26, 2026Dihong Huang, Zhenyu Wei, Zhuxiu Xu +3Dexterous Manipulation Robotic Manipulation
University of North Carolina at Chapel Hill

Oct 8, 2026J/K move · Enter open · S save

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon+7

Seoul National University · University of Maryland, College Park · All Purpose AI+1

Robotic Manipulation Learning from Demonstration Zero-Shot Robotic Manipulation RL for Robotics Reinforcement Learning Sim-to-Real Transfer

Open paper AskPDF arXiv

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Abstract

Robotic Manipulation

Explore similar work

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning

DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand