Learning In-Hand Object Reaching to General 6D Poses

Junxiao Lin; Tianyue Wu; Jie Yin; Jia Pan; Kaifeng Zhang; Weiming Zhi

cs.ROSep 12, 2026

Learning In-Hand Object Reaching to General 6D Poses

Authors: Junxiao Lin, Tianyue Wu, Jie Yin, Jia Pan, Kaifeng Zhang, Weiming Zhi

Organizations: Sharpa. · School of Computer Science, The University of Sydney, Australia. · School of Computing and Data Science, The University of Hong Kong, HKSAR.

Abstract

In-hand manipulation allows multi-fingered dexterous hands to reconfigure grasped objects without releasing and regrasping them. This improves manipulation efficiency by reducing repeated grasp acquisition and large arm motions. However, most learning-based methods focus on reorientation, continuous rotation, or translation, whereas many tasks require joint control of object position and orientation. We formulate this capability as in-hand 6D object pose reaching: starting from an existing grasp, coordinated finger motions move the object to a palm-relative target pose. We present POISE (Palm-relative Object reaching In SE(3)), a sim-to-real reinforcement learning framework for this task. POISE combines diverse stable-grasp initialization, goal- and geometry-conditioned control, an adaptive 6D goal curriculum, and a compact reward scheme for pose reaching and grasp preservation. In simulation, diverse initialization raises held-out-grasp success from 40.1% to 51.5% and post-drop recovery from 33.8% to 72.9%; the curriculum raises full-range success from 6.2% to 59.5%. On hardware, the grasp-maintenance reward improves three-target sequence success from 20% to 80%. In real-world experiments, POISE reaches successive 6D targets without manual reset across multiple object geometries and wrist orientations, and recovers from external disturbances. To support further research in dexterous manipulation, we will release our code at https://junxiaolin.github.io/poise-website/.

Figures & tables

Fig. 2: Overview of POISE. Diverse stable grasps initialize simulation training, while FoundationPose supplies object-pose feedback during real-world deployment.

Quantity	Range or distribution
Object scale	$\mathcal{U}[0.975,1.025]$
Object mass	$10$ – $100$ g
Inertia multiplier	$\mathrm{LogU}[1,4]$
CoM offset	Up to $5$ mm per axis
Friction multiplier	$\mathcal{U}[0.5,2.0]$
PD-gain multipliers	$\mathcal{U}[0.5,2.0]$

TABLE I: Sim-to-real randomization used during training.

\mathcal{U}

and

\mathrm{LogU}

denote uniform and log-uniform sampling.

Fig. 3: First-target success of the diverse policy across independently generated held-out initial grasps under paired $30$ -mm/ $180^{\circ}$ targets. Sphere positions indicate the initial object positions in the palm frame, and color indicates success rate.

Fig. 4: First-target success of the narrow and diverse policies, marginalized along each palm-frame position axis. Palm-frame $X$ is approximately normal to the palm, while $Y$ and $Z$ lie in the palm plane.

Fig. 5: Simulation recovery from the same post-drop state and target. The diverse policy (top) rebuilds contact, lifts the object from the palm, and reaches the target. The narrow policy (bottom) attempts to regrasp the object but stalls before reaching the target.

	Full range			Maximum change
Training	Strict	Near	Goals/ep.	Strict	Near	Goals/ep.
Without curriculum	6.2%	18.8%	0.08	3.4%	13.9%	0.04
Curriculum	59.5%	77.0%	1.55	55.3%	72.8%	1.08

TABLE II: Ablation of the adaptive goal curriculum on the hexagonal prism. Strict and near denote first-target success under the

10^{\circ}

/10-mm and

20^{\circ}

/20-mm criteria, respectively.

Metric	Without grasp reward	With grasp reward
Grasp-wrench quality $Q$	$0.032\!\pm\!0.002$	$\mathbf{0.045\!\pm\!0.005}$
Minimum force margin	$0.077\!\pm\!0.003$	$\mathbf{0.107\!\pm\!0.010}$
Minimum torque margin	$0.036\!\pm\!0.002$	$\mathbf{0.047\!\pm\!0.004}$
Episode success rate (%)	$53.7\!\pm\!2.5$	$\mathbf{59.1\!\pm\!2.5}$

TABLE III: Grasp-maintenance reward ablation.

Fig. 6: Representative snapshots after successful 6D pose reaching for the Hexagonal Prism, Cube, Square Bifrustum, and Hammer. The cyan outline denotes the commanded object pose, and the coordinate frames indicate the current and target poses.

Fig. 7: Closed-loop recovery from two consecutive disturbances on hardware. After reaching the target, the object is first perturbed in orientation and then displaced more substantially, changing its contacts. The policy returns it to the unchanged target after each disturbance.

Fig. 8: Representative real-world reaches under three wrist orientations with different gravity directions in the palm frame.

Metric	Without grasp reward	With grasp reward
Complete sequences	$2/10$	$\mathbf{8/10}$
Planned targets completed	$8/30$	$\mathbf{27/30}$
Mean targets reached	$0.8$	$\mathbf{2.7}$
Reach time (s)	$6.8$	$\mathbf{4.9}$
Position error (mm)	$4.8$	$\mathbf{6.4}$
Orientation error ( ∘ )	$4.7$	$\mathbf{5.9}$

TABLE IV: Real-world reaching success with and without the grasp reward.

In-Hand Manipulation

In-hand manipulation studies how robot hands reposition and reorient objects without releasing them.

See all 34 papers

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Oct 9, 2026Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3Motion Retargeting Robotic Manipulation
Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping
Jun 3, 2026Haofei Lu, Hongjia Liu, Yifei Dong +3Robotic Grasping Robotic RL
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden. · Robotics and Autonomous Systems at University of Turku, Finland.
ArtManip: Category-Level Articulated In-Hand Manipulation
Sep 11, 2026Yang Yang, Tengyu Liu, Puhao Li +6RL for Robotics Dexterous Manipulation
State Key Lab of CAD&CG, Zhejiang University · State Key Lab of General Artificial Intelligence, BIGAI · Tsinghua University +1

Oct 9, 2026J/K move · Enter open · S save

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon, Joel Burdick+2

Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Motion Retargeting Robotic Manipulation Generalization in Robotic Manipulation Hand-Object Interaction RL for Robotics Residual Policy Learning

Open paper AskPDF arXiv

Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.

Abstract

In-Hand Manipulation

Explore similar work

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

ArtManip: Category-Level Articulated In-Hand Manipulation