DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine
Authors: Haozhe Lou, Runyi Yang, Wanqi Zhong, Caiyun Liu, Yicheng Liu, Wenhao Liu, Wenlong Ma, Jiawei Xia, Kanzhi Wu, Danda Pani Paudel, Luc Van Gool, Hang Zhao, Yiming Li
Organizations: College of AI, Tsinghua University · INSAIT, Sofia University “St. Kliment Ohridski” · Peking University · IIIS, Tsinghua University · University of Southern California · Beijing Institute of Technology · University of California, Berkeley · vivo Mobile Communication Co., Ltd.
Cutting a piece of food looks effortless in human hands. For a robot, however, every cut is a high-contact interaction with a deformable, elasto-plastic object whose behavior changes with pose, contact mode, and cutting stage. Collecting real-world data is expensive and wasteful: each trial typically contaminates or damages the food, so obtaining scalable real world data for dexterous cutting is expensive and difficult. Simulation is an appealing alternative, but a large sim-to-real gap exists in both physics and perception. This gap is especially severe for topology-changing interactions like cutting, where small errors in contact, friction, or material and elasticity responses quickly compound. We propose DexNinja, a real2sim2real framework that turns a handful of real demonstrations into a dense and realistic training distribution. DexNinja (i) reconstructs object instances from real captures, (ii) randomizes physically meaningful parameters under category-level constraints, and (iii) produces augmented dexterous manipulation episodes using a custom differentiable simulator that couples robot dynamics, tactile contact, and deformable cutting. We evaluate DexNinja on a dexterous food-slicing task and show that (1) with tens of real demonstrations and hundreds of augmented simulator data, the success rate could reach more than 60%. (2) the augmented data consistently improves success rates when only using real data with more than 20% via sim-to-real, and (3) the resulting policies generalize to out-of-distribution objects with varying shapes and sizes.
Figures & tables
Figure 2: Tactile perception visualization of representative food-cutting examples.
Figure 3: Learning policy to control arm and dexterous hand separately via adaptive action chunking. We give an real-world inference result at bottom.
Figure 4: On-manifold approximation of real-data neighborhood for data augmentation pipeline.
Training regime
Food
Progress ↑
Success ↑
Integrity ↑
Sim-only
Strawberry
10/25
6/25
10/25
Apple
5/25
4/25
5/25
Banana
16/25
12/25
14/25
Potato
15/25
12/25
11/25
Avg.
46/100
34/100
40/100
Real-only
Strawberry
12/25
8/25
10/25
Table 1: Sim-to-real gap and the effect of mixing simulation and real demonstrations, with 25 trials per food.
Figure 5: Real-world lemon cutting results under different cutting requirements. The success rate is declining as the manipulation complexity increase.
Variant
P/S/I
Δ P
Δ S
Δ I
Full DexNinja (Table 1 )
76/50/60
–
–
–
w/o Quality Control
72/46/55
−4
−4
−5
w/o Physical Randomization
72/44/51
−4
−6
−9
w/o Trajectory Refinement
65/39/53
−11
−11
−7
w/o Tactile Observation
73/43/45
−3
−7
−15
w/o Generated Assets (Reconstruction Only)
67/42/50
−9
−8
−10
Table 2: Component ablations. Δ is relative to Full DexNinja.
Table 7
Component
Embodiment A: primary setup
Embodiment B: transfer setup
Robot arms
DR3/DR5 bimanual arm configuration
Dual Franka arm configuration
Dexterous hands
LinkerHand L10 dexterous hands
LinkerBot L20 dexterous hands
Tactile sensing
Pressure-based tactile sensors on both hands
Pressure-based tactile sensors on both hands
Vision
Fixed RGB-D camera (Orbbec 435Le)
Fixed RGB-D camera with recalibrated extrinsics
Role in evaluation
Main quantitative real-world experiments
Cross-embodiment qualitative verification
Table 1: Two-embodiment hardware verification. Embodiment A is the primary DexNinja platform used for the main experiments. Embodiment B replaces the arms and hands while preserving the object-centric real2sim2real pipeline.
Figure 1: Two-embodiment setup verification. (a) Primary DexNinja setup with DR3/DR5 arms and LinkerHand L10 dexterous hands. (b) Embodiment-shift setup with Franka arms and LinkerBot L20 dexterous hands. The real2sim2real pipeline is kept unchanged at the algorithmic level; only robot-specific calibration, kinematic models, IK solvers, joint limits, and safety projections are updated.
Figure 2: Generated instance samples from the knowledge-guided augmentation pipeline.
Method
Strawberry
Potato
Banana
Apple
Avg.
RoboNinja (baseline)
1/10
2/10
2/10
7/10
12/40
Ours
6/10
5/10
9/10
3/10
23/40
Table 2: Baseline comparison between RoboNinja and our method on four objects. We report successes over 10 trials per object; Avg. aggregates over all 40 trials.
+Sim episodes N
Strawberry
Banana
Potato
Apple
Avg.
10
9/25
14/25
8/25
5/25
36/100
50
12/25
18/25
12/25
8/25
50/100
100
14/25
20/25
13/25
9/25
56/100
300
16/25
22/25
15/25
10/25
63/100
Table 3: Scaling law of simulation augmentation for real-world cutting. We train with a fixed real dataset and add N simulated episodes; values are real-world success (25 trials per food), and Avg. aggregates 100 trials.
Object
Universal Head
Separate Heads
Fixed Action Chunk
Full System
Apple
2/10
4/10
1/10
5/10
Banana
4/10
7/10
6/10
7/10
Strawberry
2/10
5/10
2/10
7/10
Potato
3/10
5/10
4/10
5/10
Average
11/40
21/40
13/40
24/40
Table 4: System-level ablation study on head design and action chunking across objects. Metric: task success rate (successes / 10 trials).
Figure 3: Tactile responses under nominal and high-force contact. (a,b) Thumb and index tactile maps under nominal contact with an undeformed guava. (c,d) Thumb and index tactile maps under high-force contact with a compressed guava. The comparison illustrates why tactile feedback is needed to regulate contact within the slip–over-compression regime.
Figure 4: Qualitative real-world deployment results with 2 different embodiment settings. Representative executions across food categories. Background colors denote the object type: orange (banana), pink (strawberry), red (apple), and yellow (potato).