Pushing Dexterity and Safety for In-Hand Object Pose Reaching That Works Remarkably Well

Ian Chuang; Hiya Shah; Vamsi Chilamkur; Iman Soltani; Claire Tomlin

cs.ROOct 27, 2026

Pushing Dexterity and Safety for In-Hand Object Pose Reaching That Works Remarkably Well

Authors: Ian Chuang, Hiya Shah, Vamsi Chilamkur, Iman Soltani, Claire Tomlin

Organizations: University of California, Berkeley. · University of California, Davis.

Abstract

Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.

Figures & tables

Fig. 2: Hardware setup. (a) The UR5 arm with the LEAP hand inside the motion-capture cage. (b) Close-up of the hand and the wrist-mounted stereo camera. (c) Workspace camera view. (d) Wrist camera view.

Fig. 3: Objects used in our experiments. For each object we show its simulation mesh, its collision meshes from convex decomposition, the real object, and the real object with motion-capture markers.

Fig. 4: Training pipeline. A1–A4: The task policy: A1 PPO with privileged observations and an asymmetric critic, A2 DAgger into a deployable state student, A3 PPO fine-tuning of that student, and A4 DAgger into a depth student. B1–B2: The safety filter: B1 learns a residual safety filter on top of the frozen A3 policy; B2 distills it into a depth student filter on top of A4. C: This gives a task policy and a safety filter that use either motion capture or the wrist camera, and both transfer zero-shot to the real robot.

Fig. 5: Effect of the correction weight

w

in simulation. Coarse goal success vs. time to the first drop, per object, for the state (left) and depth (right) policies, from no filter (hollow) through

w=2

to 0.

		Goals reached (%) $\uparrow$			Time to 1st drop or	Avg. action change (%)
Obs.	Filter	coarse	precise	Drops (%) $\downarrow$	end (s) $\uparrow$
State	None	92.1 $\pm$ 0.3	88.4 $\pm$ 0.3	9.1 $\pm$ 0.3	154 $\pm$ 5	–
	ResSafe, $w{=}0.5$	91.3 $\pm$ 0.3	80.1 $\pm$ 0.4	4.5 $\pm$ 0.3	266 $\pm$ 6	15.3
	HCSF $\alpha{=}0.3,\beta{=}0.9$	87.4 $\pm$ 0.5	80.4 $\pm$ 0.5	6.6 $\pm$ 0.2	185 $\pm$ 5	10.8
	HCSF $\alpha{=}0.2,\beta{=}0.9$	84.7 $\pm$ 0.4	75.3 $\pm$ 0.5	6.0 $\pm$ 0.2	201 $\pm$ 5	14.4
	HCSF $\alpha{=}0.3,\beta{=}0.95$	82.0 $\pm$ 0.4	70.8 $\pm$ 0.5	4.8 $\pm$ 0.1	230 $\pm$ 6	17.2

TABLE I: ResSafe vs. the HJ filters in simulation (rubber duck, state policy, 1,024 environments

\times

50 goals): three HCSF settings near ResSafe’s drop rate (full sweep in Table IX ) and LRSF at

\alpha{=}0.1

. Mean

\pm

standard error of the mean over the 1,024 environments.

Fig. 6: Knife curriculum ablation over 3 seeds. Goals per episode during A1 for each curriculum variant, including a gravity curriculum (0

\to

9.81 m/s 2 ). Mean with min–max band; dots mark full difficulty. Gravity runs are faded, since most do not reach full gravity within 10k iterations.

Fig. 7: The six evaluation wrist orientations , which point gravity along each axis of the hand.

			Goals reached (/300) $\uparrow$		Drops (/300) $\downarrow$	Time to 1st drop or end (s) $\uparrow$	Reached before 1st drop (/50) $\uparrow$	Avg. action change (%)
Obs.	Object	Safety filter	precise	coarse
State	Lightbulb	$\times$	277	297	2	427	42.3	–
	Lightbulb	$\checkmark$	268	292	0	500	48.7	8.5
	Rubber duck	$\times$	255	291	7	298	29.3	–
	Rubber duck	$\checkmark$	194	262	2	441	39.2	15.4
	Knife	$\times$	234	288	4	257	24.7	–

TABLE II: Real-robot results. Each row pools 6 runs of 50 consecutive goals (one per wrist orientation), 300 goals. Precise and coarse: goals reached within 1 cm, 0.2 rad and 2 cm, 0.4 rad. Drops: goals in which the object fell. Time to the first drop (or 500 s, the end of the run) and coarse goals reached before the first drop: means over the 6 runs. Last column: RMS filter correction (% of the action limit). Bold: better of each pair. The filter cuts drops at a small cost in success.

Fig. 8: ResSafe prevents a drop. The same wineglass goal (palm down, depth policy) without the filter (top), where the glass slips and falls, and with it (bottom). The red line traces the falling glass.

Fig. 9: Spinning the lightbulb about its own axis (depth policy). Palm up (top) and down (bottom), counter-clockwise (left) and clockwise (right) goals.

Fig. 10: Difficult out-of-distribution goals reached on the real robot (palm down, state policy; dashed green outline: goal).

Fig. 11: Rotating object during large arm motion. During a 30 s arm trajectory, the depth policy keeps the lightbulb and the duck spinning. Four poses are shown, with the object path colored by time; the orange arrow is the commanded rotation.

Fig. 12: Robustness to physical disturbances (depth policy, no filter). Left: a finger is perturbed while the hand holds the object, and the hand keeps the object on goal. Right: the object is pushed into the palm, and the hand recovers and returns it to the goal about 4 s after release.

Fig. 13: Multi-step arm and hand tasks with the depth policy, using only the wrist camera (no motion capture or external cameras).

Fig. 14: Wrist-camera input during the tasks of Fig. 13 , with clutter in the background. Camera image (top) and the depth the policy receives (bottom).

Parameter	Value	Unit	Randomization
BAM m5 model, identified on a test bench
Torque constant $k_{t}$	0.571	N m/A	$\times\mathcal{U}(0.9,1.1)$
Winding resistance $R$	2.327	$\Omega$	$\times\mathcal{U}(0.9,1.1)$
Armature	$1.83{\times}10^{-3}$	kg m 2	$\times\mathcal{U}(0.9,1.1)$
Base friction	$4.85{\times}10^{-3}$	N m	$\times\mathcal{U}(0.9,1.1)$
Viscous friction	$3.97{\times}10^{-3}$	N m s/rad	$\times\mathcal{U}(0.9,1.1)$

TABLE III: Identified XC330 actuator model and its randomization.

\pm x

\mathcal{U}(-x,x)

; drawn independently for each joint at every reset, except the armature, supply voltage and bus resistance (once per environment) and the command channel (every step).

Fig. 15: Actuator identification of the XC330 servo. Left: pendulum test bench (mass $m$ , arm length $L$ ). Right: commanded, real and identified-model angles on three test trajectories.

Fig. 16: Sim-to-real check of the identified hand. The same chirp on all 16 actions, played on the real hand and replayed in simulation. Top: hand and wrist depth at the same instant. Bottom: all 16 joints, real (solid) vs. simulated (dashed).

	Knife	Lightbulb	Rubber duck	Wineglass
Mass, nominal [g]	40	27	46.7	62
Mass DR, $\log\mathcal{U}$ [g]	20–90	10–50	20–70	30–90
Bounding box [mm]	39 $\times$ 47 $\times$ 210	57 $\times$ 57 $\times$ 103	59 $\times$ 81 $\times$ 85	80 $\times$ 80 $\times$ 137
Goal a	pose	axis	pose	axis
Grasps in dataset	451,681	453,131	439,212	421,949
Training goal tolerance (A1 final, A2, A3, B1)

TABLE IV: Per-object settings.

Quantity	Range or distribution	Drawn
Contact friction (A1 curriculum start $\to$ final; final after A1)
Fingertip sliding $\mu$	$\mathcal{U}(2.5,3.0)\to\mathcal{U}(0.3,1.2)$	ep.
Non-fingertip sliding $\mu$	$\mathcal{U}(1.0,1.5)\to\mathcal{U}(0,0.6)$	ep.
Torsional $\mu$ ( $\times 10^{-3}$ )	$\log\mathcal{U}(7,20)\to\log\mathcal{U}(0.1,20)$	ep.
Rolling $\mu$ ( $\times 10^{-3}$ )	$\log\mathcal{U}(1.7,5)\to\log\mathcal{U}(0.01,5)$	ep.
Inertial a

TABLE V: Domain randomization, shared by every training stage (actuator randomization in Table III ).

\pm x

\mathcal{U}(-x,x)

; drawn per environment (env), episode (ep.) or control step (step).

Fig. 17: Example grasps from the grasp datasets generated with LightningGrasp [ 37 ] and expanded by RRT.

Term	Dim	Hist.
$o^{\mathrm{prop}}$ proprioception
Projected gravity	3	1
Last smoothed action $s_{t-1}$	16	1
Command residual $(q^{\mathrm{cmd}}{-}q)/0.1$	16	1
Joint position a	16	10
$x$ object pose (motion capture)

TABLE VI: Observations, grouped by the symbols of Fig. 4 , all in the hand frame. Dim: per-frame size; Hist.: frames stacked.

Fig. 18: Simulated depth observations. Scene with distractors, wrist camera view, rendered depth and augmented depth.

Fig. 19: Curriculum schedules as a function of progress

p

. Left: goal curriculum (candidate subset, tolerances, required hold). Right: friction curriculum (sampled range per coefficient).

Fig. 20: Adaptive curriculum progress during A1 : goal (top) and friction (middle) curriculum progress, and the rolling goal success that drives them (bottom).

	A1	A3	B1
Environments	2 $\times$ 8192 a	8192	8192
Max. iterations b	30k	5k	4k
Wall-clock e	15–17 h	2.5–2.8 h	2.5–2.6 h
Steps per env. per iteration	24	24	24
Epochs / mini-batches	5 / 4	5 / 4	5 / 4
Learning rate c	$5{\times}10^{-4}$	$5{\times}10^{-4}$	$5{\times}10^{-4}$

TABLE VII: PPO hyperparameters (A1, A3, B1).

\mathcal{M}_{5}=[1024,512,512,256,256]

hidden units, ELU.

	A2	A4	B2
Environments	4096	512	512
Iterations a	2000	20000	3000
Wall-clock b	0.6 h	10 h	1.8 h
Steps per env. per iteration	24	24	24
Epochs / mini-batches	5 / 4	5 / 4	5 / 4
Learning rate	$5{\times}10^{-4}$	$5{\times}10^{-4}$	$5{\times}10^{-4}$

TABLE VIII: DAgger hyperparameters (A2, A4, B2).

\mathcal{M}_{5}=[1024,512,512,256,256]

\mathcal{M}_{3}=[1024,512,256]

hidden units, ELU.

Fig. 21: Training curves of the four task stages (goals per episode). Hollow circles: deployed checkpoints.

Gate	$\alpha$	$\beta$	coarse	precise	Drops	TTD	Corr.
None			92.1	88.4	9.1	154	–
ResSafe, $w{=}0.5$			91.3	80.1	4.5	266	15.3
LRSF	$0.05$	–	60.6	50.4	5.7	195	30.2
LRSF	$0.1$	–	79.3	71.3	8.2	154	16.5
LRSF	$0.2$	–	86.1	81.0	9.4	142	9.1
LRSF	$0.3$	–	87.8	83.4	9.3	147	6.7

TABLE IX: Every swept setting of the HJ filter (simulation, rubber duck, state policy). TTD: time to the first drop or end (s); Corr.: average action change (%).

Fig. 22: Outcome of every goal in the main real-robot experiment (96 runs, 4,800 goals). Each row of cells is one run of 50 goals. Dark blue: precise; light blue: coarse only; grey: not reached; orange: drop.

Fig. 23: Goal diversity, knife, palm down. Goal poses of one run from the main (left) and wide (right) evaluation sets, marked reached (dot), not reached (open circle) or dropped (diamond).

		Goals reached (/50) $\uparrow$		Drops (/50) $\downarrow$	Avg. action change (%)
Object	Safety filter	precise	coarse
Knife	$\times$	24	37	0	–
Knife	$\checkmark$	21	39	0	12.1
Lightbulb	$\times$	30	42	0	–
Lightbulb	$\checkmark$	23	38	0	11.9
Rubber duck	$\times$	25	42	0	–

TABLE X: Wide goals from a 1.5

\times

larger grasp box: state policy, palm down, 50 goals per setting, without (

\times

) and with (

\checkmark

) the filter. Avg. action change: RMS change of the action by the filter (% of the action limit).

	Knife				Lightbulb				Rubber duck				Wineglass
Filter $w$	Coarse	Precise	Drops	$T_{\text{drop}}$	Coarse	Precise	Drops	$T_{\text{drop}}$	Coarse	Precise	Drops	$T_{\text{drop}}$	Coarse	Precise	Drops	$T_{\text{drop}}$
State policy (A3) and state filter (B1)
None	94.8	91.2	3.7	255	93.8	92.1	6.5	266	92.1	88.4	9.1	154	88.6	67.7	6.1	287
2	94.8	90.1	3.0	296	96.9	94.8	2.9	317	92.8	88.0	7.2	205	91.5	67.8	3.0	312
1.5	94.5	89.4	2.9	304	97.0	95.0	2.9	315	92.6	87.0	6.6	208	91.3	67.2	2.6	323
1	94.4	88.6	2.6	304	97.1	95.0	2.5	333	92.6	85.4	5.9	232	91.0	65.6	2.6	317

TABLE XI: Simulated results behind Fig. 5 for every correction weight

w

(1,024 environments

\times

50 goals). Coarse and precise: goals reached (%) within 2 cm, 0.4 rad and 1 cm, 0.2 rad; drops: % of goals;

T_{\text{drop}}

: mean time to the first drop (s; 500 if none). † Deployed on the real robot.

Fig. 24: Goal success versus drop rate on the real robot , per object, from without (open) to with (filled) the safety filter, for the state (left) and depth (right) policies.

Fig. 25: More drops prevented by ResSafe. The same goal without (top) and with (bottom) the filter; the red line traces the fall path.

Fig. 26: Object pose perceived by the depth policy. Error of its auxiliary pose head in simulation (blue) and on the real robot (orange); dashed lines are medians.

				Turns $\uparrow$
Wrist	Safety filter	Reached $\uparrow$	Drops $\downarrow$	$\geq\!\tfrac{1}{2}$	$\geq$ 1	mean
Palm up	$\times$	46	0	34	28	1.24
Palm up	$\checkmark$	43	0	25	20	0.74
Palm down	$\times$	44	0	19	12	0.63
Palm down	$\checkmark$	41	0	14	7	0.36

TABLE XII: Lightbulb spinning: depth policy, 50 goals of 20 s per row. Turns about the bulb axis in the commanded direction, counted within the coarse tolerance;

\geq\!\tfrac{1}{2}

\geq

1: goals with at least half / one full turn.

			Held (/10) $\uparrow$
	Goal height (cm)	Spin (rad/s)	No safety filter ( $\times$ )	Safety filter ( $\checkmark$ )
Lightbulb	11.5	$-1.5$	10/10	10/10
Lightbulb	15.6	$-1.0$	7/10	10/10
Rubber duck	13.0	0.5	8/10	9/10
Rubber duck	16.5	0.5	7/10	9/10
All			32/40	38/40

TABLE XIII: Trials (of 10) with the object still in hand after 30 s of arm motion. Depth policy, filter at

w{=}0.5

; goal height above the palm.

Safety filter	Trials that passed each stage (cumulative) $\uparrow$
Rubber duck : grasp, reorient & place
	Grasp + lift	Reorient (base down)	Place upright
$\times$	8/10	8/10	7/10
$\checkmark$	7/10	6/10	3/10
Lightbulb : screw in
	Upright from palm	Into socket	Screwed in (lights up)

TABLE XIV: Downstream tasks: cumulative stage success (a trial passes a stage only if it passed all earlier ones). Depth policy without (

\times

) and with (

\checkmark

w{=}1.5

) the filter.

Fig. 27: Curriculum ablation for A1 on all four objects (one seed). Curves are faded while a curriculum is still ramping; dots mark full difficulty.

Fig. 28: ResSafe penalty ablation (rubber duck, 3 seeds): L2 penalty (ours), L1 penalty, and L2 without nominal-action noise. Labels give

w

; error bars: s.e.m. over seeds.

Fig. 29: A1 needs privileged actor observations. Knife goals per episode with the privileged actor (ours) and with the student’s observations (3 seeds).

Fig. 30: ResSafe B1 training (rubber duck, 3 seeds): unsafe-episode rate (left) and goals per 100 steps (right); mean with min–max band over seeds.

Dexterous Manipulation

Dexterous manipulation studies how robots use multi-fingered hands to grasp, reorient, and interact with objects through contact.

See all 143 papers

Sep 12, 2026

Learning In-Hand Object Reaching to General 6D Poses

In-hand manipulation allows multi-fingered dexterous hands to reconfigure grasped objects without releasing and regrasping them. This improves manipulation efficiency by reducing repeated grasp acquisition and large arm motions. However, most learning-based methods focus on reorientation, continuous rotation, or translation, whereas many tasks require joint control of object position and orientation. We formulate this capability as in-hand 6D object pose reaching: starting from an existing grasp, coordinated finger motions move the object to a palm-relative target pose. We present POISE (Palm-relative Object reaching In SE(3)), a sim-to-real reinforcement learning framework for this task. POISE combines diverse stable-grasp initialization, goal- and geometry-conditioned control, an adaptive 6D goal curriculum, and a compact reward scheme for pose reaching and grasp preservation. In simulation, diverse initialization raises held-out-grasp success from 40.1% to 51.5% and post-drop recovery from 33.8% to 72.9%; the curriculum raises full-range success from 6.2% to 59.5%. On hardware, the grasp-maintenance reward improves three-target sequence success from 20% to 80%. In real-world experiments, POISE reaches successive 6D targets without manual reset across multiple object geometries and wrist orientations, and recovers from external disturbances. To support further research in dexterous manipulation, we will release our code at https://junxiaolin.github.io/poise-website/.

Junxiao Lin, Tianyue Wu, Jie Yin +3

Sharpa. · School of Computer Science, The University of Sydney, Australia. · School of Computing and Data Science, The University of Hong Kong, HKSAR.

In-Hand Manipulation Robotic Manipulation Hand-Object Interaction RL for Robotics

Ask PDF

Oct 24, 2026

Learning Dexterous Manipulation: Hands, Simulators, Training, and Evaluation

A dexterous hand can change an object's pose without setting it down. This survey covers 113 learning-based methods that attempt it, the hands they run on, the simulators they train in, and how they report results. The field has largely converged on one recipe: reinforcement learning on privileged state in a GPU simulator, distilled into a vision-only student for transfer to hardware. Almost half the corpus trains that way, nearly all of it in one simulator, whose own successors are barely used. A second and faster-growing branch learns from teleoperated demonstration instead, through vision-language-action models, diffusion policies, and systems whose contribution is the collection rig itself. Published work has tripled since 2022. Grasping remains the largest task family and bimanual coordination the fastest-rising, and almost every experiment runs on one of four hands. What the survey adds comes from reading released code and vendor specifications beside the papers. Eight hands that can be bought or built from published designs appear in no method paper here. Where the depth a hand sinks into the object it holds is reported at all, it is not measured in the physics that produced the motion. Section VII proposes a protocol for the unreported quantities; the body gives the counts behind every claim above.

Luai Abuelsamen

Independent Researcher

Robotic Manipulation RL for Robotics Robot Manipulation Benchmarks Sim-to-Real Transfer Robot Imitation Learning

Ask PDF

Feb 25, 2026

DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations

Robotic dexterous manipulation is a challenging problem due to high degrees of freedom (DoFs) and complex contacts of multi-fingered robotic hands. Many existing deep reinforcement learning (DRL) based methods aim at improving sample efficiency in high-dimensional output action spaces. However, existing works often overlook the role of representations in achieving generalization of a manipulation policy in the complex input space during the hand-object interaction. In this paper, we propose DexRep, a novel hand-object interaction representation to capture object surface features and spatial relations between hands and objects for dexterous manipulation skill learning. Based on DexRep, policies are learned for three dexterous manipulation tasks, i.e. grasping, in-hand reorientation, bimanual handover, and extensive experiments are conducted to verify the effectiveness. In simulation, for grasping, the policy learned with 40 objects achieves a success rate of 87.9% on more than 5000 unseen objects of diverse categories, significantly surpassing existing work trained with thousands of objects; for the in-hand reorientation and handover tasks, the policies also boost the success rates and other metrics of existing hand-object representations by 20% to 40%. The grasp policies with DexRep are deployed to the real world under multi-camera and single-camera setups and demonstrate a small sim-to-real gap.

Qingtao Liu, Zhengnan Sun, Yu Cui +5

the College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · the College of Control Science and Engineering and the State Key Laboratory of Industrial Control Technology, Zhejiang University, China · the Department of Computer Science, National University of Singapore, 15 Computing Dr, 117418, Singapore

Dexterous Manipulation Representation Learning Robotic Grasping Generalization in Robotic Manipulation Robotic RL Bimanual Robotic Manipulation

Ask PDF

Abstract

Dexterous Manipulation

Explore similar work

Learning In-Hand Object Reaching to General 6D Poses

Learning Dexterous Manipulation: Hands, Simulators, Training, and Evaluation

DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations