Learning Dexterous Manipulation: Hands, Simulators, Training, and Evaluation

Luai Abuelsamen

cs.ROOct 24, 2026

Learning Dexterous Manipulation: Hands, Simulators, Training, and Evaluation

Authors: Luai Abuelsamen

Organizations: Independent Researcher

Abstract

A dexterous hand can change an object's pose without setting it down. This survey covers 113 learning-based methods that attempt it, the hands they run on, the simulators they train in, and how they report results. The field has largely converged on one recipe: reinforcement learning on privileged state in a GPU simulator, distilled into a vision-only student for transfer to hardware. Almost half the corpus trains that way, nearly all of it in one simulator, whose own successors are barely used. A second and faster-growing branch learns from teleoperated demonstration instead, through vision-language-action models, diffusion policies, and systems whose contribution is the collection rig itself. Published work has tripled since 2022. Grasping remains the largest task family and bimanual coordination the fastest-rising, and almost every experiment runs on one of four hands. What the survey adds comes from reading released code and vendor specifications beside the papers. Eight hands that can be bought or built from published designs appear in no method paper here. Where the depth a hand sinks into the object it holds is reported at all, it is not measured in the physics that produced the motion. Section VII proposes a protocol for the unreported quantities; the body gives the counts behind every claim above.

Figures & tables

Panel of Figure 1 — Fig. 1 : The field on one page, and the map of this survey: nodes sized by corpus papers, edges drawn for the routes that carry at least two fifths of the smaller endpoint, so the dominant path reads as a path, each column head naming the section that covers it.

task family	what is held	what moves	what gravity does	what counts as failure	shortest success criterion as stated
In-hand reorientation	one rigid object, held by the fingers for the whole episode	the object’s orientation in the palm frame, while contacts break and remake	a disturbance whose direction is set by hand orientation, and a load to counteract with the palm down, [ 16 ]	the object leaves the hand, e.g. the fall reset at goal distance 0.24 m in [ 14 ]	orientation error below a stated tolerance under a stated stopping rule, 0.4 rad in [ 2 ] and [ 14 ] , 0.1 rad in [ 15 ]
Grasping	nothing at the start, the object rests on a support	the hand closes, then the object is lifted clear	the test itself, since the grasp exists to resist it	no lift, or a drop inside the hold window	a stated lift height held for a stated duration, 10 cm for 20 s in [ 10 ] , 10 cm to episode end in [ 47 ] , 0.5 s in [ 11 ]
Functional and tool use	a tool, held in the pose its function requires rather than the most stable one	the tool’s own joint, or the tool acting on a second object	secondary, the function dominates	a stable grasp in the wrong place, [ 13 ]	no agreed criterion. Criteria range from never dropping the object, [ 13 ] , to a four-point rubric, [ 48 ] . The one mechanical threshold is holding the tool joint past a commanded angle for the episode, [ 49 ]
Tracking a human reference	whatever the human held in the captured clip	hand and object follow a recorded trajectory under physics	full, and the reference never obeyed it, so the reference is not dynamically feasible	per-frame error leaves the band, or the object drops	all stated per-frame errors below threshold for the whole clip, 30 deg and 3 cm and 8 cm joint and 6 cm fingertip in [ 17 ] . Thresholds are not comparable across papers, and DexMachina [ 18 ] calls the results “highly sensitive to the threshold values”
Bimanual coordination	one object by two hands, or one object per hand	both hands, the object, and the relative pose of the hands	can be carried by one hand while the other acts, [ 40 ]	the object drops, the arms collide, or the hands work against each other	no agreed criterion. Task-specific thresholds dominate, e.g. 0.035 m block-to-cup distance in [ 40 ] . The one criterion with a physical-plausibility gate is 2.0 s of hold with total penetration under 1.5 mm, [ 41 ]
Handover and in-hand transfer	by the giver at the start, by the receiver at the end	the object crosses between hands, released only after it is regained	unopposed during the exchange, and total in a thrown transfer, [ 19 ]	a drop at the moment of release, and the catcher’s grasp phase, [ 19 ] Sec. 5.4	the receiver holds the object and moves it clear, more than 10 cm from the first hand in [ 50 ] , or holds it to episode end in [ 42 ]

TABLE I : Task families against the properties that define success.

Panel of Figure 3 — Fig. 2 : Three answers to where the motor goes, and what each one costs. A coupling bar saves an actuator by tying two joints, so the fingertip reaches a curve through the reachable region rather than the region itself.

Panel of Figure 4 — Fig. 3 : Which hands the literature runs on, banded by how the hand can be got, with every tabulated hand shown and those no paper uses as named slots at zero. A paper naming two hands counts under both.

Panel of Figure 5 — Fig. 4 : The geometry the solver uses against the geometry the renderer draws. A single convex hull fills the concavity with material that exists only in the collision model, and holds the fingertip out of a contact the visual mesh allows.

Panel of Figure 6 — Fig. 5 : One simulation step, and where engines differ: the stages every rigid-body engine runs, with the choices that separate the engines used for hands under each; dashed arrows mark the three stages that overlap.

Panel of Figure 7 — Fig. 6 : A compliant tactile sensor reads the overlap depth. The tactile image, the distributed force and the binary contact flag are all functions of the one quantity every other engine in this section works to drive to zero.

Panel of Figure 8 — Fig. 7 : What supervises a dexterous policy, subdivided to where the 113 method papers actually differ; each leaf names two of its corpus keys and counts the rest, bars scaled by paper count, dashed curves marking overlap.

Panel of Figure 9 — Fig. 8 : A retargeted pose can be scored good and still be inside the object. The joint-angle error is a sum over joints with no term for the object; a few degrees at each joint compose along the finger into a fingertip displacement larger than a thin object is thick.

Panel of Figure 10 — Fig. 9 : Papers whose released code was readable, and the 8 where a value in it differs from the value the paper prints. Table II names them, with the file and both values.

method	conf.	repository	file, linked at the revision read	what the paper prints	what that file contains
PDDM [ 108 ]	high	google-research/pddm	cube_env.py _get_obs	observation dimension 46 for in-hand reorientation, Table 2	the six fields _get_obs concatenates sum to 39 on the code’s own inline comments
DeXtreme [ 14 ]	high	isaac-sim/IsaacGymEnvs	AllegroHandDextremeADR.yaml actionDeltaPenaltyScale	action-delta penalty weight -0.25, Table 2	actionDeltaPenaltyScale: -0.2 , and -0.01 in the ManualDR yaml beside it
Visual Dexterity [ 16 ]	high	Improbable-AI/dexenv	dclaw.yaml alg.num_envs	32,000 teacher training environments, Table S1	num_envs: 8000 , and 16384 in the parent hand_default.yaml
DexPBT [ 110 ]	high	NVIDIA-Omniverse/IsaacGymEnvs	allegro_kuka_base.py compute_kuka_reward	four staged reward terms, Sec. III-C	eight summed components, hand_delta_penalty multiplied by 0 with the comment currently disabled
PhysHOI [ 143 ]	high	wyhuai/PhysHOI	physhoi.py compute_humanoid_reward	object-rotation weights 0.1 and 0.01 for GRAB, Table 4	eor and eorv set to torch.zeros_like(ep) , the computation commented out on the same lines
UniDexGrasp [ 151 ]	high	PKU-EPIC/UniDexGrasp	shadow_hand_grasp.py compute_hand_reward, goal_cond branch	seven named weights from 0.1 to 10, Table 7	a torch.where cascade on literals -0.5, -1.0, 0.9, 0.1, 0.2, none of the seven present

TABLE II : The 8 papers whose repository states a different value from the paper. Each file links to the revision read.

Panel of Figure 12 — Fig. 10 : Four mechanisms for controlling two dexterous hands, counted over the 28 papers in this corpus with a learned closed-loop controller on two multi-fingered hands, each read from its note. The panels are not a partition: the fourth is the same single paper as the third, and the two papers whose notes never say which architecture they use appear in neither.

Panel of Figure 13 — Fig. 11 : Where a capture set’s contact annotation comes from. Nothing in the chain is measured at a surface, and a vertex that has gone through the surface is written down as a contact rather than as an error.

Fig. 12 : Two failures of synthesised two-hand grasps, each enlarged at the contact: (A) a finger inside the object, (C) a finger inside the other hand. Both look plausible whole [ 41 ] .

Panel of Figure 15 — Fig. 13 : What gets reported, and what does not. Each bar is a count of the 113 corpus method papers; the pale track behind it is the denominator that statistic is quoted against, printed beside the count, because four of the six are not 113. A faint tail past a track is the papers whose notes left that question unsettled. The two rarest reporting practices are marked in color.

Panel of Figure 16 — Fig. 14 : Where a denominator is stated it is small: the 39 per-cell real-trial counts, binned by fives, median 15 marked. The 24 grand totals are a separate quantity, counted beneath the axis and not pooled in.

axis	what is measured	minimum trials	reported with it
Task success	fraction of episodes meeting a criterion written before the run, and a graded rubric score in $[0,1]$	100 real and 200 sim per policy, task and condition for an absolute rate at a 10-point interval; a comparison alone needs 57 matched pairs per arm	raw counts, the criterion verbatim, the rubric, the initial-condition protocol, and an interval whose kind is named
Robustness	success under each perturbation axis, and the ratio to the unperturbed anchor	40 per axis per policy to rank the axes; 101 per axis and 101 at the anchor before any claim that a named axis hurt	both absolute rates with their Wilson intervals, the axis ranges, and the ratio without an interval unless the certifying count was run
Unseen objects	mean over held-out objects of per-object success	20 objects at 5 trials to screen; 93 objects for a 10-point claim	the object list, the per-object rates, a bootstrap interval over objects, and the within-object trial count
Physical plausibility	maximum and mean hand-object penetration depth per rollout, and the fraction of frames above 2 mm	100 rollouts, simulation only	an interval over rollouts, the sample density, the threshold, the solver’s depenetration settings, and at least one rendered rollout
Sample and wall-clock cost	environment steps and wall-clock to the checkpoint that produced the headline number	3 training seeds	the range over seeds, and the statement that 3 seeds is a range and not an interval
Real-robot transfer	paired real and simulated outcomes on matched initial conditions, their correlation, and the rectifier variance against the real variance	the 100 real trials paired to 100 sim; about 457 pairs to separate two correlations 0.1 apart	the correlation with its Fisher-z interval, the variance ratio with its bootstrap interval, and which decision the count supports

TABLE III : The proposed evaluation protocol, and this survey’s own contribution. No cell is empty: every cell is a proposal.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

hand	maker	DoF	act. DoF	actuation	mass g	force N	force kind	tactile	price USD	open HW	status	uses
Shadow Dexterous Hand [ 51 ]	Shadow Robot Co.	24	20	tendon	4 300	–	–	optional	–	no	sold	21
Proception ProHand 1.0 [ 87 ]	Proception Inc	22	–	tendon	–	–	–	unstated	–	no	sold	0
SharpaWave 22-DoF [ 80 ]	Sharpa Robotics	22	22	–	1 300	20 / 12	grip	unstated	–	no	sold	6
BiDexHand [ 59 ]	Northwestern Univ.	21	16	tendon	–	2.14	fingertip normal	none	–	yes	open-source	0
Ruka-v2 [ 57 ]	NYU and NYU Shanghai	20	16	tendon	–	–	–	optional	1 500	yes	open-source	0
TESOLLO DG-5F [ 86 ]	Tesollo Inc.	20	20	direct drive	1 763	–	–	none	–	no	sold	0

Appendix

TABLE IV : Hands that can be obtained, sorted by joint count. 68 of 160 specification cells are empty.

hand	maker	DoF	act. DoF	actuation	mass g	force N	force kind	tactile	price USD	open HW	status	uses	claim date	evidence
Clone Robotics [ 78 ]	Clone Robotics	27	–	fluidic muscle	<910	–	–	array	–	no	internal-only	0	2025-11-24	video
1X NEO [ 76 ]	1X Tech.	25	25	tendon	–	45	vendor, no protocol	unstated	–	no	announced	0	2026-07-09	datasheet
LEAP Hand V2 [ 187 ]	CMU	21	17	tendon	–	–	–	none	3 000	yes	announced	2	undated	datasheet
Sanctuary AI Phoenix [ 188 ]	Sanctuary AI	21	–	fluidic muscle	–	–	–	unstated	–	no	internal-only	0	2024-12-17	press
ILDA hand [ 55 ]	Korea Univ.	20	15	linkage	1 100	34	fingertip normal	array	–	no	prototype	0	2021-11-05	datasheet
Pisa/IIT SoftHand [ 54 ]	Centro E. Piaggio, Univ. Pisa and IIT	19	1	tendon	–	–	–	none	–	no	prototype	0	2014	datasheet

Appendix

TABLE V : Hands announced, prototyped or held internally, which cannot be obtained. 60 of 104 specification cells are empty.

engine	contact model	solver	diff.	GPU	dt (s)	penetration exposed	hands
Brax [ 192 ]	convex relaxation	explicit/spring	yes	yes	–	no	1
ComFree-Sim [ 96 ]	complementarity-free	closed form	–	yes	0.002	yes	–
Dojo [ 95 ]	hard NCP	interior point	yes	no	–	yes	–
Genesis [ 99 ]	soft	Newton	yes	yes	–	–	1
Isaac Gym [ 92 ]	convex relaxation	PGS	no	yes	0.008333	no	3
Isaac Lab [ 97 ]	rigid, model unstated	Gauss-Seidel	no	yes	–	no	4

Appendix

TABLE VI : 15 simulators and physics engines, by contact formulation and penetration exposure. 40 of 105 property cells are empty.

survey	yr	dexterous scope	taxonomy	bimanual	hardware	evaluation	gaps named
Surveys of the field
Zhao et al. [ 202 ]	2026	$\checkmark$	$\checkmark$	$\circ$	$\checkmark$	$\circ$	$\checkmark$
An et al. [ 198 ]	2025	$\checkmark$	$\circ$	$\circ$	$\circ$	$\circ$	$\checkmark$
Bai et al. [ 36 ]	2025	$\circ$	$\checkmark$	$\circ$	$\circ$	$\circ$	$\checkmark$
Welte et al. [ 199 ]	2025	$\circ$	$\checkmark$	$\circ$	$\checkmark$	$\times$	$\checkmark$
Firoozi et al. [ 201 ]	2023	$\times$	$\circ$	$\times$	$\times$	$\circ$	$\checkmark$

Appendix

TABLE VII : What 14 predecessor surveys cover:

\checkmark

covered,

\circ

partly,

\times

not. 24 of 84 coverage cells are empty.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

May 13, 2026

Towards Robotic Dexterous Hand Intelligence: A Survey

Robotic dexterous hands are central to contact-rich manipulation, with rapid progress driven by advances in hardware, sensing, control, simulation, and data generation. However, existing studies are often developed under different assumptions regarding hand embodiments, sensory configurations, task settings, training data, and evaluation protocols, making systematic comparison difficult and obscuring the developmental trajectory of the field. This survey provides a holistic review of dexterous hand research from four complementary aspects. First, we present a hardware-level analysis covering actuation, transmission, perception, and representative hand designs, highlighting the key trade-offs in force capability, compliance, bandwidth, integration, and system complexity. Furthermore, we review control and learning methods for dexterous manipulation from a methodological perspective, grouping representative works by major paradigms and tracing their evolution in chronological order. In addition, we consolidate datasets, modality design, and evaluation practices, which enables methodological progress to be interpreted together with the ways in which it is trained, benchmarked, and assessed. Finally, we discuss the major limitations of current dexterous hand research and summarize the corresponding future directions. By connecting hardware analysis, methodological development, data resources, and evaluation, this survey aims to provide a structured understanding of dexterous hand research and to clarify the most important open challenges for future study.

Weiguang Zhao, Tian Liang, Xihao Guo +3

University of Liverpool · Xi’an Jiaotong-Liverpool University · Duke Kunshan University +1

Robotic Control Robotic Hand Design Dexterous Manipulation Robot Manipulation Benchmarks

Ask PDF

Oct 19, 2026

HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

General-purpose manipulation requires reliable behavior across embodiments, objects, and task demands. We introduce HandsOn, a composable simulation platform and benchmark for cross-embodiment dexterous manipulation. It supports five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets in randomized scenes. The benchmark pairs 18,000 curated human demonstrations with 66 human-verified tasks: Core-50 supports demonstration-based learning and execution tests, while Transfer-16 evaluates asset and relation transfer, skill composition, and deformable manipulation without target-task training demonstrations. Qualitative real-robot rollouts illustrate sim-to-real transfer. Across diffusion, vision-language-action, world-action, and prompted VLM-based methods, the best trained multi-task policy achieves 29.1% Core-50 success. Without benchmark finetuning, GPT-6 using move_eef for palm-and-finger control achieves 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. Trained-policy stage diagnostics expose failures in contact establishment, action coordination, and maintaining achieved goals, with additional sequencing errors in composed tasks. HandsOn provides a shared basis for investigating manipulation generalization and the limits of reliable physical interaction.

Tianwei Luo, Yiming He, Hongjin Wu +14

Tsinghua University · NatureWill · Beijing University of Posts and Telecommunications +1

Robotic Manipulation Robot Manipulation Benchmarks Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Robot Manipulation Datasets

Ask PDF

Jul 6, 2026

Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation

Human-like dexterous hands with multiple fingers offer human-level manipulation capabilities but remain difficult to train the control policies that can deploy on real hardware due to contact-rich physics and imperfect actuation. We present a sim-to-real reinforcement learning method that leverages dense tactile feedback combined with joint torque sensing to explicitly regulate physical interactions. To enable effective sim-to-real transfer, we introduce (i) a computationally fast tactile simulation that computes distances between dense virtual tactile units and the object via parallel forward kinematics, providing high-rate, high-resolution touch signals needed by RL; (ii) a current-to-torque calibration that eliminates the need for torque sensors on dexterous hands by mapping motor current to joint torque; and (iii) actuator dynamics modeling with randomization to account for non-ideal torque-speed effects and bridge the actuation gaps. Using an asymmetric actor-critic PPO pipeline, we train policies entirely in simulation and deploy them directly to a five-finger hand. The resulting policies demonstrate two essential human-hand skills: (1) command-based controllable grasp force tracking and (2) reorientation of objects in the hand, both of which are robustly executed without fine-tuning on the robot. By combining tactile and torque in the observation space with scalable sensing and actuation modeling, our system provides a practical solution to achieve reliable dexterous manipulation. To our knowledge, this is the first demonstration of controllable grasping on a multi-finger dexterous hand trained entirely in simulation and transferred zero-shot on real hardware.

Zhe Zhao, Zhibin Li, Yilin Ou +1

State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom

Robotic Grasping Dexterous Manipulation In-Hand Manipulation Sim-to-Real Transfer Robotic RL Force-Controlled Robotic Manipulation

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

Towards Robotic Dexterous Hand Intelligence: A Survey

HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation