This paper presents Grasp2Twist, a bimanual dexterous manipulation system that learns to grasp and twist open jar lids using reinforcement learning. Learning this task raises three challenges: learning a unified policy for a multi-stage task, sustaining lid twisting, and sim-to-real transfer. To address the first challenge, we introduce a continuous enclosure measure to guide grasp formation and a binary enclosure indicator to guide the grasp-to-twist transition for unified policy learning. We derive both from the geometric relationship between the object center and the convex hull formed by the hand's palm and fingertips. Kinematic constraints limit how far the hand can rotate the lid with fixed contacts, so sustained twisting requires finger contact reconfiguration. We use a three-stage curriculum to facilitate exploration of these contact changes and also improve robustness for sim-to-real transfer. With our approach, the learned policy demonstrates finger gaiting, reconfiguring hand-object contacts to sustain lid rotation. It transfers zero-shot to the physical system and achieves an 88% task success rate across six household containers, including peanut-butter, vitamin, and instant-coffee jars. Ablations further validate the roles of the geometric enclosure in grasp formation and the curriculum in contact-reconfiguration exploration.
Figures & tables
Fig. 1: Real-world and simulated bimanual dexterous jar opening with two robot arms and dexterous hands.
Fig. 2: Overview of the real-world system. The vision node estimates jar-body and lid poses and dimensions from segmented depth images. The policy node combines these estimates with joint states, joint position targets, and end-effector poses to generate actions. The driver node tracks the resulting joint position targets and returns joint states for policy observations and forward kinematics.
Fig. 3: Geometric hand–object enclosure representation. (a) The palm position ppalm and three fingertip positions ptip,i define a tetrahedral convex hull; the displacement from the palm ppalm to the object center pobj is decomposed along the three palm-to-fingertip edge vectors ei . (b) The binary enclosure indicator Ienclose identifies whether the object center lies inside the tetrahedron convex hull. (c) The geometric enclosure reward derived from the enclosure measure menclose .
Parameter
Mode
Final range
Jar-body mass (kg)
Value
[0.05,0.30]
Jar friction
Value
[0.20,2.50]
Lid-joint damping
Value
[0.10,0.50]
Jar body radius (m)
Value
[0.03,0.05]
Jar body height (m)
Value
[0.10,0.20]
Lid radius (m)
Value
[0.02,0.06]
TABLE I: Final domain randomization ranges.
Fig. 4: Real-world test instances. The leftmost object is the ID 3D-printed jar. The remaining objects are OOD-Geom household containers, referred to as Blue Peanut, Yellow Mayo, Pink Glass, Green Vitamin, Brown Peanut, and Instant Coffee.
Fig. 5: Real-world execution sequence from pre-grasp to lid removal. After grasp acquisition, the holding hand stabilizes the jar body while the twisting hand rotates the lid. Individual fingers asynchronously twist, release, reposition, and re-establish contact. Twisting continues until complete thread disengagement, after which the lid is removed open loop.
Metric
Blue Peanut
Yellow Mayo
Pink Glass
Green Vitamin
Brown Peanut
Instant Coffee
Approx. Rotation to Disengagement
2π
4π
π
1.5π
4π
π
Jar-Opening Success (Replay)
1/10
2/10
0/10
3/10
6/10
6/10
Jar-Opening Success (Ours)
8/10
9/10
6/10
10/10
10/10
10/10
Lid Angular Velocity (Ours, Mean ± Std)
(0.19±0.01)π /s
(0.26±0.02)π /s
(0.06±0.01)π /s
(0.22±0.02)π /s
(0.19±0.05)π /s
(0.21±0.01)π /s
TABLE II: Real-World Evaluation on OOD-Geom Household Containers
Error
Unoccluded(mm)
Occluded(mm)
ID 3D position
24.2±10.3
22.9±6.4
OOD-Geom 3D position
26.9±10.7
29.3±12.5
ID vertical position
1.9±0.9
3.3±3.0
OOD-Geom vertical position
7.5±6.0
14.6±11.9
ID dimensions
4.3±2.6
5.5±3.4
TABLE III: Static State Estimation Errors (mean ± std).
Fig. 6: Ablation studies of the geometric enclosure representation and the curriculum for contact reconfiguration. (a)–(b) DFC residuals ∥Gc∥2 of the holding hand and twisting hand during evaluation rollouts for the contact-reward-only baseline and full method with geometric enclosure; the lower subpanels show the corresponding number of hand–object contacts. (c)–(d) Representative grasps from the contact-reward-only baseline and the full method, respectively, showing the different spatial arrangements of the fingers around the object. (e) Lid rotation during training under two curriculum settings, Stage 1 only and Stages 1 and 2. (f) Lid rotation during evaluation rollouts for policies trained with Stage 1 only and with Stages 1 and 2. The dashed curve shows the single Stage-1-only seed that discovers contact reconfiguration, separated from the remaining four seeds. (g)–(h) Representative contact-reconfiguration behaviors observed in simulation: synchronous contact changes in twist-and-regrasp behavior and asynchronous contact changes in finger gaiting; percentages give each finger’s contact time fraction. Curves show the mean over seeds; shaded regions denote one standard deviation across seeds; each seed’s curve is itself averaged over 4096 parallel environments.
Dexterous object rotation is a sequential contact problem: each support, release, and re-contact decision must both produce the desired object motion, and prepare the hand configuration for continued rotation. Existing reinforcement learning methods discover such movement patterns through trial and error on specific robotic hand embodiments, without explicitly accounting for how each contact transition affects the hand's ability to sustain object rotation in subsequent steps. We introduce DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution. This prior captures how successful human contact transitions reshape the object-rotation directions available to the hand. DexMani then learns this manipulability evolution and uses it to guide downstream reinforcement learning, enabling rotation skills to be acquired across robot embodiments with distinct kinematics and active-contact configurations. Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates in every evaluated setting for both seen and unseen objects. DexMani reaches an average success rate of 57.5% on LEAP Hand, outperforming other baselines and producing smoother rotatory motions. Project site: https://dexmani.github.io
Xiaoyang Chen, Shengcheng Luo, Haoran Guo +4
Shanghai Jiao Tong University · ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) +1
Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.
Dexterous manipulation requires planning a grasp configuration suited to the object and task, which is then executed through coordinated multi-finger control. However, specifying grasp plans with dense pose or contact targets for every object and task is impractical. Meanwhile, end-to-end reinforcement learning from task rewards alone lacks controllability, making it difficult for users to intervene when failures occur. To this end, we present GRIT, a two-stage framework that learns dexterous control from sparse taxonomy guidance. GRIT first predicts a taxonomy-based grasp specification from the scene and task context. Conditioned on this sparse command, a policy generates continuous finger motions that accomplish the task while preserving the intended grasp structure. Our result shows that certain grasp taxonomies are more effective for specific object geometries. By leveraging this relationship, GRIT improves generalization to novel objects over baselines and achieves an overall success rate of 87.9%. Moreover, real-world experiments demonstrate controllability, enabling grasp strategies to be adjusted through high-level taxonomy selection based on object geometry and task intent.
Juhan Park, Taerim Yoon, Seungmin Kim +10
Department of Artificial Intelligence, Korea University, Seoul, Korea · Korea University of Technology and Education, Cheonan, Korea · Naver AI Lab, Korea +4