Dexterous robot hands can receive useful motion intent from teleoperation or learned policies, but successful execution also depends on contact decisions that these commands rarely specify. We introduce DexFLEX, a contact-aware foundation controller that turns upstream fingertip-motion drafts into contact-consistent joint commands. Instead of treating a draft as a trajectory to copy, DexFLEX treats it as evidence about intent: from the current tactile-proprioceptive state, it proposes feasible short-horizon motion chunks, predicts their contact consequences, and selects the candidate that best follows the command while preserving future contact stability. At inference time, a simple Draft-Dream-Select procedure combines pure-prior proposals for recovery with draft-seeded proposals for responsiveness, then decodes the selected motion into executable joint-space targets. Across degraded-command simulation, real-world shared control, and visuomotor policy learning, the same trained controller improves robustness without retraining, increasing real-world teleoperation success from 29.2% to 78.3% and reaching 46.7% policy-learning success, 3.5 times direct joint-action prediction, and 1.56 times prior-only correction. Videos are available at dex-flex.github.io.
Figures & tables
Figure 1: DexFLEX converts upstream motion drafts into contact-stable dexterous actions. Human operators or learned policies provide fingertip-motion drafts, and DexFLEX supplies tactile-guided low-level correction for contact-consistent execution. With the same draft, DexFLEX preserves contact where direct execution can fail.
Figure 2: DexFLEX pipeline. (1) A state encoder converts proprioceptive and tactile histories into a contact–proprioceptive latent zt , whose temporal history is summarized as a slow phase token gt . (2) At inference, Draft–Dream–Select proposes pure-prior and draft-seeded motion chunks, predicts their contact consequences, scores them against the upstream draft, and decodes the selected chunk into joint-space actions.
Figure 3: Robustness under command degradation. Expert commands are corrupted with additive noise, command scaling, delay, and support-finger removal. We evaluate whether each controller preserves contact stability while recovering the expert intent rather than merely damping motion.
Variant
Dur. ↑
Prog. ↑
Intent ↓
DexFLEX
0.86
0.78
0.18
w/o predictive score
0.52
0.46
0.45
w/o tactile/contact
0.62
0.55
0.36
w/o consequence pred.
0.64
0.57
0.34
w/o pure-prior prop.
0.66
0.61
0.25
w/o draft-seeded prop.
0.74
0.66
0.38
Table 1: Component ablations. Metrics are averaged over the four command-degradation settings. w/o predictive score replaces the learned score with nearest-to-draft selection over the same proposal set.
Method
Brush
Nut
Hammer
Screwdrv.
Spatula
Toast
Avg SR ↑
Avg CT ↓
Avg NHT ↑
Raw Teleop
50
30
40
30
15
10
29.2
27.5
0.42
Filter
60
40
50
40
25
20
39.2
33.8
0.55
DexGen
70
55
60
55
40
35
52.5
31.2
0.65
DexFLEX w/o Tactile
75
60
65
60
50
45
59.2
30.8
0.71
DexFLEX
90
80
85
80
70
65
78.3
32.0
0.85
Table 2: Real-world teleoperation results. We report per-task success rate (SR, %) and average SR / CT / NHT across tasks. CT is completion time in seconds over successful trials. NHT is normalized holding time. Each method is evaluated for 20 trials per task across two operators. Averages are computed over task-level means.
Figure 4: Selective intervention during real-world tool use. We visualize intervention magnitude It=∥u^t−ut∥2/(∥ut∥2+ϵ) along representative rollouts. DexFLEX keeps intervention low during benign user motion and produces peaks near contact-critical moments such as release, impact, unstable tool contact, and regrasp.
Figure 5: Policy learning with different action interfaces. All methods use the same image-based Flow-matching policy and 50 demos per task. Direct joint-action prediction is compared with fingertip-motion prediction executed by a prior-only DexGen controller and by DexFLEX .
Teleoperation is widely used to collect high-quality demonstrations for robot learning. Dexterous manipulation, however, requires coordinated control of motion and contact forces. Operators can specify motion, but struggle to reproduce the subtle force adjustments of human manipulation without feeling the robot's contacts. We introduce Force-Modulated Dexterity (FOMO-Dex), a unified compliance formulation for contact-rich manipulation. FOMO-Dex expresses nominal actions in coordinates defined by task geometry and contact structure. It applies force-driven motion corrections along selected directions while leaving the remaining command components unchanged. Across card picking, paper cutting, screwdriving, and nut-and-bolt assembly, FOMO-Dex reduces excessive contact forces compared with stiff control. A user study shows that operators collect more successful demonstration data per hour after brief practice. For policy learning, imitating the corrected actions improves card-picking success over imitating raw operator commands. Policies trained on FOMO-Dex demonstrations also achieve more rotation in nut-and-bolt manipulation and screwdriving than those trained on demonstrations collected with the stiff controller. Videos are shown on https://yingyuan0414.github.io/fomo-dex/ .
Dexterous manipulation policies trained in simulation often fail to transfer to the real world because of errors in contact timing and force regulation. Yet these policies can retain useful multi-finger coordination for task progression. We propose ReDex, a framework for adapting a simulation-trained base policy to the real world by correcting local contact failures and incorporating tactile feedback. Starting from a proprioception-only base policy, ReDex allows a human operator to physically correct contact failures at selected fingers under compliant control during real-world rollouts, while the frozen base policy continues to control the remaining fingers. These rollouts combine base policy execution, human-corrected finger motion, and fingertip force observations. We reconstruct force-informed targets from these rollouts to train a standalone force-conditioned policy via behavior cloning. This design reduces human correction effort, enables learning of contact regulation from real-world interaction, and introduces force feedback into a proprioception-only policy without tactile simulation or complex full-hand teleoperation. We evaluate ReDex on two challenging, contact-rich dexterous manipulation tasks on real hardware. Compared with sim-to-real transferred base policies, ReDex increases Object Flipping success rate from 14% to 86% across two objects and average Screwdriver Rotation progress from 26.0% to 95.3% across three objects.
Jinzhou Li, Hadi Tabatabaee, Kelin Yu +6
Amazon · Duke University · University of Maryland, College Park +2
Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.
Puhao Li, Zeyuan Chen, Yingying Wu +9
1Tsinghua University · 2State Key Lab of General Artificial Intelligence, BIGAI · 3Peking University