Learning In-Hand Object Reaching to General 6D Poses

Junxiao Lin; Tianyue Wu; Jie Yin; Jia Pan; Kaifeng Zhang; Weiming Zhi

cs.ROSep 12, 2026

Learning In-Hand Object Reaching to General 6D Poses

Authors: Junxiao Lin, Tianyue Wu, Jie Yin, Jia Pan, Kaifeng Zhang, Weiming Zhi

Organizations: Sharpa. · School of Computer Science, The University of Sydney, Australia. · School of Computing and Data Science, The University of Hong Kong, HKSAR.

Abstract

In-hand manipulation allows multi-fingered dexterous hands to reconfigure grasped objects without releasing and regrasping them. This improves manipulation efficiency by reducing repeated grasp acquisition and large arm motions. However, most learning-based methods focus on reorientation, continuous rotation, or translation, whereas many tasks require joint control of object position and orientation. We formulate this capability as in-hand 6D object pose reaching: starting from an existing grasp, coordinated finger motions move the object to a palm-relative target pose. We present POISE (Palm-relative Object reaching In SE(3)), a sim-to-real reinforcement learning framework for this task. POISE combines diverse stable-grasp initialization, goal- and geometry-conditioned control, an adaptive 6D goal curriculum, and a compact reward scheme for pose reaching and grasp preservation. In simulation, diverse initialization raises held-out-grasp success from 40.1% to 51.5% and post-drop recovery from 33.8% to 72.9%; the curriculum raises full-range success from 6.2% to 59.5%. On hardware, the grasp-maintenance reward improves three-target sequence success from 20% to 80%. In real-world experiments, POISE reaches successive 6D targets without manual reset across multiple object geometries and wrist orientations, and recovers from external disturbances. To support further research in dexterous manipulation, we will release our code at https://junxiaolin.github.io/poise-website/.

Figures & tables

Fig. 2: Overview of POISE. Diverse stable grasps initialize simulation training, while FoundationPose supplies object-pose feedback during real-world deployment.

Quantity	Range or distribution
Object scale	$\mathcal{U}[0.975,1.025]$
Object mass	$10$ – $100$ g
Inertia multiplier	$\mathrm{LogU}[1,4]$
CoM offset	Up to $5$ mm per axis
Friction multiplier	$\mathcal{U}[0.5,2.0]$
PD-gain multipliers	$\mathcal{U}[0.5,2.0]$

TABLE I: Sim-to-real randomization used during training.

\mathcal{U}

and

\mathrm{LogU}

denote uniform and log-uniform sampling.

Fig. 3: First-target success of the diverse policy across independently generated held-out initial grasps under paired $30$ -mm/ $180^{\circ}$ targets. Sphere positions indicate the initial object positions in the palm frame, and color indicates success rate.

Fig. 4: First-target success of the narrow and diverse policies, marginalized along each palm-frame position axis. Palm-frame $X$ is approximately normal to the palm, while $Y$ and $Z$ lie in the palm plane.

Fig. 5: Simulation recovery from the same post-drop state and target. The diverse policy (top) rebuilds contact, lifts the object from the palm, and reaches the target. The narrow policy (bottom) attempts to regrasp the object but stalls before reaching the target.

	Full range			Maximum change
Training	Strict	Near	Goals/ep.	Strict	Near	Goals/ep.
Without curriculum	6.2%	18.8%	0.08	3.4%	13.9%	0.04
Curriculum	59.5%	77.0%	1.55	55.3%	72.8%	1.08

TABLE II: Ablation of the adaptive goal curriculum on the hexagonal prism. Strict and near denote first-target success under the

10^{\circ}

/10-mm and

20^{\circ}

/20-mm criteria, respectively.

Metric	Without grasp reward	With grasp reward
Grasp-wrench quality $Q$	$0.032\!\pm\!0.002$	$\mathbf{0.045\!\pm\!0.005}$
Minimum force margin	$0.077\!\pm\!0.003$	$\mathbf{0.107\!\pm\!0.010}$
Minimum torque margin	$0.036\!\pm\!0.002$	$\mathbf{0.047\!\pm\!0.004}$
Episode success rate (%)	$53.7\!\pm\!2.5$	$\mathbf{59.1\!\pm\!2.5}$

TABLE III: Grasp-maintenance reward ablation.

Fig. 6: Representative snapshots after successful 6D pose reaching for the Hexagonal Prism, Cube, Square Bifrustum, and Hammer. The cyan outline denotes the commanded object pose, and the coordinate frames indicate the current and target poses.

Fig. 7: Closed-loop recovery from two consecutive disturbances on hardware. After reaching the target, the object is first perturbed in orientation and then displaced more substantially, changing its contacts. The policy returns it to the unchanged target after each disturbance.

Fig. 8: Representative real-world reaches under three wrist orientations with different gravity directions in the palm frame.

Metric	Without grasp reward	With grasp reward
Complete sequences	$2/10$	$\mathbf{8/10}$
Planned targets completed	$8/30$	$\mathbf{27/30}$
Mean targets reached	$0.8$	$\mathbf{2.7}$
Reach time (s)	$6.8$	$\mathbf{4.9}$
Position error (mm)	$4.8$	$\mathbf{6.4}$
Orientation error ( ∘ )	$4.7$	$\mathbf{5.9}$

TABLE IV: Real-world reaching success with and without the grasp reward.

In-Hand Manipulation

In-hand manipulation studies how robot hands reposition and reorient objects without releasing them.

See all 33 papers

Oct 9, 2026

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.

Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3

Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Motion Retargeting Robotic Manipulation Generalization in Robotic Manipulation Hand-Object Interaction RL for Robotics Residual Policy Learning

Ask PDF

Jun 3, 2026

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

This work addresses sequentially grasping multiple objects with a single dexterous hand without releasing those already held. Most dexterous grasping methods commit all of the hand's degrees of freedom to a single object, underutilizing its dexterity and leaving no redundancy for subsequent grasps. The proposed solution, MoDex, is a diffusion policy that predicts the next gripper pose directly from observations, conditioned on an opposition space and point cloud. The opposition space condition specifies which fingers participate in the current grasp, enabling the gripper to use only a subset of its available degrees of freedom while reserving the remaining degrees of freedom for subsequent grasps. To facilitate sim-to-real transfer, MoDex is trained in two stages: first through imitation learning on expert demonstrations, and subsequently through reinforcement learning fine-tuning, which consistently improves success rates over the pre-trained policy. We evaluate MoDex in simulation on a MuJoCo-based Franka Emika Panda robot equipped with an Allegro Hand and on the corresponding real-world hardware platform. Across both simulation and real-world experiments, MoDex achieves higher success rates than the evaluated learning-based baselines, improving performance by 2.92-17.92% and 6.67-17.78%, respectively. Project page: https://modex2026.github.io/.

Haofei Lu, Hongjia Liu, Yifei Dong +3

Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden. · Robotics and Autonomous Systems at University of Turku, Finland.

Robotic Grasping Robotic RL Diffusion Policy

Ask PDF

Sep 11, 2026

ArtManip: Category-Level Articulated In-Hand Manipulation

Category-level in-hand manipulation of articulated objects is a formidable yet underexplored challenge for dexterous robotic hands. This difficulty stems from two core bottlenecks: first, controlling an object's internal degrees of freedom is tightly coupled with maintaining grasp stability on a free-floating base; second, acquiring diverse object models and functional grasps at scale is highly labor-intensive, yet vital for generalization given the system's sensitivity to initial configurations. In this work, we present ArtManip, the first category-level articulated in-hand manipulation method that generalizes across object instances and diverse initial grasps. For initial configuration construction, we develop an automated pipeline that procedurally generates diverse articulated objects and synthesizes task-oriented functional grasps. For policy learning, we propose a robust two-stage training strategy that incorporates articulation physics randomization, reward curriculum, and latent representation distillation to handle complex contact and joint dynamics during deployment. Extensive experiments across four object categories demonstrate that our policy generalizes to unseen instances and varied configurations in simulation, and achieves zero-shot transfer to 12 real-world objects featuring diverse shapes and joint mechanics.

Yang Yang, Tengyu Liu, Puhao Li +6

State Key Lab of CAD&CG, Zhejiang University · State Key Lab of General Artificial Intelligence, BIGAI · Tsinghua University +1

RL for Robotics Dexterous Manipulation In-Hand Manipulation Generalization in Robotic Manipulation Articulated Object Manipulation

Ask PDF

Abstract

In-Hand Manipulation

Explore similar work

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

ArtManip: Category-Level Articulated In-Hand Manipulation