HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

Tianwei Luo; Yiming He; Hongjin Wu; Ting Pan; Debin Ren; Haorui Guo; Junlin Yang; Yuxin Zuo

cs.ROOct 19, 2026

HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

Authors: Tianwei Luo, Yiming He, Hongjin Wu, Ting Pan, Debin Ren, Haorui Guo, Junlin Yang, Yuxin Zuo,

Organizations: Tsinghua University · NatureWill · Beijing University of Posts and Telecommunications · Peking University

Abstract

General-purpose manipulation requires reliable behavior across embodiments, objects, and task demands. We introduce HandsOn, a composable simulation platform and benchmark for cross-embodiment dexterous manipulation. It supports five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets in randomized scenes. The benchmark pairs 18,000 curated human demonstrations with 66 human-verified tasks: Core-50 supports demonstration-based learning and execution tests, while Transfer-16 evaluates asset and relation transfer, skill composition, and deformable manipulation without target-task training demonstrations. Qualitative real-robot rollouts illustrate sim-to-real transfer. Across diffusion, vision-language-action, world-action, and prompted VLM-based methods, the best trained multi-task policy achieves 29.1% Core-50 success. Without benchmark finetuning, GPT-6 using move_eef for palm-and-finger control achieves 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. Trained-policy stage diagnostics expose failures in contact establishment, action coordination, and maintaining achieved goals, with additional sequencing errors in composed tasks. HandsOn provides a shared basis for investigating manipulation generalization and the limits of reliable physical interaction.

Figures & tables

	DexJoCo	DexVerse	Bench2Dex	DMG	SPD	HandsOn
Task catalog	11	100	26	9	6 scenes	50 + 16
Human demos	1,100	3,180	$\sim$ 1,300	60	$\sim$ 2,000	18,000
Generated demos	—	—	—	21,000	—	—
Language variation	Digits / words	NR	NR	NR	No input	32 / 8 per goal a
Test-only tasks	NR	NR	NR	NR	NR	16 tasks
Transfer setting	Visual; language	Visual; embodiment	Visual / geometric	Initial states; objects	Real-task finetuning	Assets; skills; soft objects

Table 1: Dexterous learning resources (DMG: DexMimicGen; NR: not reported). Human and generated trajectories are counted separately; catalogs and evaluation scopes differ; HandsOn shaded.

Figure 3: Three-dimensional task taxonomy of HandsOn. Tasks are classified by interaction type (primitive, functional, contact-rich), task structure (single-stage, repetitive, long-horizon), and asset type (rigid bodies, assets with revolute joints, assets with prismatic joints, deformable objects). A task may have multiple asset-type labels. Each cell gives the number of tasks at the intersection of the indicated categories. The right panel illustrates the nine interaction–structure combinations in Core-50; badges show the corresponding task counts. Evaluation-only Transfer-16 tasks are shown separately in orange, with deformable manipulation reserved for this suite. Table 10 provides the full per-task taxonomy.

	Core-50	Transfer-16
Purpose	Policy learning and execution evaluation	Asset, relation, and skill-composition generalization
Tasks	50	16 (6 EVAL-Comb, 7 EVAL-Gen, 3 EVAL-Flex)
Demonstrations	46–463 training trajectories per task (median 268)	None used for training; 13 tasks use a described duration proxy
Scope	Functional tool use, numerical and spatial grounding, articulation, bimanual manipulation	New asset relations, skill composition, three deformable interactions
Scoring	Task-macro success	Task-macro success and ordered progress (trained multi-task methods); success only for GPT-6

Table 2: The two HandsOn suites. Core-50 carries demonstrations; Transfer-16 is evaluation-only.

Figure 4: Human demonstration collection. Gloves record finger articulation and wrist trackers record hand motion. Retargeted commands drive the simulated arm–hand system while the operator observes the scene.

Unit	Count
Curated demonstration trajectories	18,000
Training / validation trajectories	15,178 / 2,822
Recorded time steps	6,053,555
Camera videos (three per trajectory)	54,000
Native task identities / quota groups	54 / 52
Core-50 / Transfer-16 tasks	50 / 16

Table 3: Distinct counting units in the dataset and catalog. The video count is three views of the same interactions, not three times the data.

Factor	Examples	Must preserve
Appearance	Material, lighting, background, ceiling	Preserve recorded interaction and task-relevant visual semantics.
Camera	Extrinsic perturbation about calibration	Preserve timing; transform camera-frame labels if affected.
Language	Canonical instruction and paraphrases	Preserve object, numerical goal, direction, order, and constraints.
Task geometry	Object pose, goal relation, native reset	Retain each source task’s joint sampling distribution and compatible actions.
Dynamics	Mass, friction, compliance, actuation	Requires valid re-execution; not established by frozen-state re-rendering.

Table 4: Visual, language, and physical variation and the labels each must preserve. Geometry and dynamics changes require evidence beyond visual state replay.

Model	Steps/plan	SR	Tasks $\geq$ 50%	P	F	CR	@1 $\times$	Early	Stall
OpenWAM	16	29.1	11	40.8	23.8	22.1	8.3	10.9	65
FastWAM	16	26.4	9	42.7	20.2	15.3	10.2	13.0	72
$\pi_{0.5}$	30	24.9	10	40.4	14.6	19.7	14.6	14.1	78
DM0.5	30	21.4	9	37.7	10.0	16.7	10.9	13.9	80
LingBot-VLA	30	19.7	8	26.9	14.3	17.9	12.0	12.7	83
HY-Embodied	30	18.9	8	27.1	13.3	16.4	11.0	14.4	81

Table 5: Core-50 task-macro success (%), 100 episodes per task and learned checkpoint (GPT-6: ten); 95% intervals in Table 15 . Steps/plan: control steps (30 Hz) per policy query. Tasks

\geq

50%: number of the 50 tasks with at least 50% success. P/F/CR: primitive (17), functional (18), contact-rich (15) tasks. @1

\times

: success within the expert p75 duration. Early: episodes ended by an irrecoverable failure. Stall: the percentage of instrumented time-outs with no new checker state after the expert p75 duration (Section 5.3.3 ). Shaded: best learned checkpoint (lowest for Early, Stall).

Figure 5: Core-50 diagnostics. (a) Task-macro success by hand-skill group for eight multi-task checkpoints and GPT-6 ( move_eef ); ten trials/task for GPT-6 versus 100 for trained checkpoints (one invalid HY-Embodied episode). Task groups are listed in Table 9 . (b)–(d) Episodes at the three transitions of Section 5.3.3 , measured for the eight trained checkpoints; grey: transition never reached, orange: failed at it, light blue: failed after it. (b) Mouse moved 8 cm onto its pad before any click. (c) Both knobs within $5^{\circ}$ of target at once (grey: at most one knob). (d) Laptop display ever at the requested angle, and still there at the end.

Figure 6: Task-macro success counting only episodes completed within a given multiple of the expert 75th-percentile duration. Shorter limits are prefixes of the evaluated

3\times

rollouts, not reruns, so the curves do not estimate what a longer limit would add. Solid: world-action models; dashed: VLA checkpoints. Success at

1\times

1.5\times

2\times

3\times

is OpenWAM 8.3/19.1/24.0/29.1; FastWAM 10.2/19.1/22.8/26.4;

\pi_{0.5}

14.6/19.6/22.2/24.9; DM0.5 10.9/16.9/19.3/21.4; LingBot-VLA 12.0/16.1/17.8/19.7; HY-Embodied 11.0/15.1/16.8/18.9; Motus 5.4/11.5/14.6/18.1; Spatial Forcing 5.6/8.7/10.2/12.4. Twelve of the 28 checkpoint pairs change order between

1\times

and

3\times

Model	Asset/relation		Skill composition		Deformable		All 16
	SR	Prog.	SR	Prog.	SR	Prog.	SR	Prog.
$\pi_{0.5}$	0.0	14.0	0.3	11.7	0.0	8.3	0.1	11.9
DM0.5	1.4	9.2	0.7	14.4	0.0	8.8	0.9	11.2
LingBot-VLA	0.0	13.3	0.0	4.2	0.0	11.3	0.0	9.3
HY-Embodied	0.0	9.6	0.0	4.6	0.0	6.2	0.0	6.9
OpenWAM	0.9	14.8	0.0	12.5	0.0	10.2	0.4	12.9

Table 6: Transfer-16 results (%): task-macro success (SR) and ordered progress (Prog.) within the declared time limit over the 7 Gen, 6 Comb, and 3 Flex tasks; 50 episodes per task and checkpoint (Stack Boxes 37 and Spray Shoe 43, whose missing resets fail for every checkpoint; at most two invalid episodes elsewhere). Gen02 logs no graded hinge progress and is excluded from Prog. DP is not run (Section 5.1 ). Shaded: best trained checkpoint per column. GPT-6 is a prompted reference with ten trials/task and different inputs/call budgets; graded progress was not evaluated for this condition.

Suite	Tasks	Successes / trials	SR (%)	95% CI	Cap (%)
Core-50	50	26 / 500	5.2	[3.6, 6.8]	39.8
Transfer-16	16	3 / 160	1.9	[0.0, 3.8]	38.1
All	66	29 / 660	4.4	[3.2, 5.6]	39.4

Table 7: GPT-6 with move_eef : ten scored trials per task. SR is the unweighted task mean; equal trial counts make it identical to pooled success. Intervals are conditional 95% within-task episode bootstrap intervals (20,000 resamples), with the task catalog fixed. Cap: unsuccessful outcomes ended by the applicable model-call budget. Recorded budget settings are described in Appendix C.3 .

Figure 7: GPT-6 with move_eef on all 660 scored trials. (a) Success by the declared task families; labels give successes/trials. Each task contributes ten trials. (b) Trial outcomes within each suite. A call-budget stop is an unsuccessful outcome under the recorded budget, not a native physical failure. Native/rule failures include the ten forbidden direct-hand-contact endings on Line Up Blocks. Physical time does not advance while the model is queried.

Figure 8: (a) Single-task data scaling for

\pi_{0.5}

, 20,000 updates per run (all: 438 and 439 trajectories); each point is one checkpoint on 100 closed-loop trials, with 95% Wilson bands over rollouts, not over training seeds or subsets. (b) How Core-50 episodes end: the task-macro mean of eight trained checkpoints and GPT-6 ( move_eef , ten trials/task). Rule failures and physical time-outs are distinct from call-budget stops; detailed measured categories are in Figure 9 . (c) Matched hand transitions in Core-50 sources and Transfer-16, pooled over the eight checkpoints (800 source and 296–400 transfer episodes per task). Panels (a) and (c) have no corresponding GPT-6 experiment.

Group	Demos	P75 (s)	Raw	With covariates
GTR (reference)	392	8.2	34.9 [26.9, 43.1]	—
Nonprehensile	362	10.5	$+16.4$ [ $-19.3$ , 52.2]	$+18.7$ [ $-16.1$ , 52.9]
Articulated part	259	6.7	$+8.0$ [ $-4.6$ , 21.9]	$+7.4$ [ $-3.0$ , 19.8]
Finger actuation	272	12.9	$-20.9$ [ $-31.2$ , $-10.8$ ]	$-19.9$ [ $-27.8$ , $-12.2$ ]
Pour or fill	208	17.7	$-21.5$ [ $-34.6$ , $-6.7$ ]	$-13.8$ [ $-24.8$ , $-3.3$ ]
In-hand reorientation	89	12.3	$-23.5$ [ $-35.2$ , $-11.2$ ]	$-22.1$ [ $-31.6$ , $-11.7$ ]

Table 8: Task success (mean of the eight multi-task checkpoints, %) regressed on hand-skill group over the 50 Core-50 tasks, with grasp–transport–release (GTR) as the reference; the GTR row gives the intercept of the raw model. Covariates are natural logs of the task’s training trajectories and expert P75 duration. Brackets: 95% intervals from 4,000 task bootstrap draws stratified by group. Demos and P75: group medians over tasks.

Figure 9: How Core-50 episodes end (task-macro share over 50 tasks). Top: the eight trained checkpoints and their mean. Early endings are native terminations before the time limit, including forbidden contact, premature events, or drops. A time-out is stalled if no new checker state follows the expert P75 duration, still progressing otherwise, or not instrumented if fewer than two progress states are logged. Bottom: GPT-6 ( move_eef ), 500 trials: 26 successes, 210 native/rule failures, 65 physical time-outs, and 199 call-budget stops. Its progress-based time-out subtypes were not evaluated.

Figure 10: Core-50 success (%) for eight trained multi-task checkpoints and GPT-6 ( move_eef ), grouped by hand skill (Table 9 ). Task order is fixed by the trained-checkpoint mean. Trained cells have 100 episodes (99 for HY-Embodied on Place Wine Glass); GPT-6 cells have ten, so its 10-point increments reflect coarser sampling. DP, task IDs, and the same GPT-6 results are in Table 16 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task ID	Task Name	Interaction Type	Task Structure	Asset Type
01	Align Blocks in Midair	CR	SS	RB
02	Connect Charger and Cable	CR	LH	RB
03	Cover Box	CR	SS	RB
04	Cover Pitcher	CR	SS	RB
05	Fill Cup	F	LH	RB, AR
06	Fit Device Cover	CR	SS	RB

Appendix

Table 10: Complete task inventory and three-axis taxonomy of HandsOn; shaded rows are the evaluation-only Transfer-16 tasks. Abbreviations are defined below the table.

ID	Upstream task or asset	Adaptation or reuse in HandsOn
08 J	DexJoCo: Hammer nail	Adds a supporting hand holding the wood block and a task-specific nail-depth condition.
12 J	DexJoCo: Bimanual microwave	Adapts the loading, door-closing, and activation sequence to the collection platform.
14 J	DexJoCo: Click mouse	Requires moving the mouse onto its pad before a permitted click.
23 J	DexJoCo: Bimanual photograph	Retains camera aiming and shutter activation with platform-specific contact and execution logic.
50 J	DexJoCo: Water plant	Adds grasp, trigger-contact, and aim-sensitive spray conditions.
10 V	DexVerse: BimanualLiftCarton	Adapts bilateral lifting to lower-side support and a three-second airborne hold.

Appendix

Table 11: Acknowledged task-design and asset sources. IDs follow Table 10 . All listed tasks use our platform’s execution and observation pipeline; source attribution does not imply identical reset distributions or success predicates.

Task ID	Supporting Core-50 tasks	P75 frames	Seconds	Actions
Comb01	Cover Pitcher + Cover Box	$382+210$	59.2	1776
Comb02	Open Laptop + Move and Click Mouse	$196+125$	32.1	963
Comb03	Pass Block + Cover Box	$195+210$	40.5	1215
Comb04	Place Fruit in Bucket + Photograph Logo	$110+158$	26.8	804
Comb05	Scrub Bottle + Fill Cup	$259+530$	78.9	2367
Comb06	Pour Seasonings + Stir Pot	$236+547$	78.3	2349

Appendix

Table 12: Transfer-16 physical-time budgets. Flex budgets (shaded) are native exceptions and are not described as P75 estimates.

Dimension	Training support	Evaluation question
Asset/relation	Object interaction, covering, orienting, passing	Can a skill operate on a new target or asset relation?
Skill composition	Opening a laptop; moving and clicking a mouse; scrubbing; filling	Can trained behaviors be connected in a new sequence with valid stage transitions?
Deformable tasks	Rigid/articulated object manipulation	Can the policy adapt to compression, folding, and packing?

Appendix

Table 13: Training support and evaluation changes. Relations shown here describe task semantics; the source/asset mapping specifies actual exposure. Training support refers to the learned methods; GPT-6 receives no HandsOn training.

ID	Task	Successes	SR (%)	Cap
Core-50
01	Align Blocks in Midair	0 / 10	0	5
02	Connect Charger and Cable	0 / 10	0	10
03	Cover Box	0 / 10	0	0
04	Cover Pitcher	0 / 10	0	6
05	Fill Cup	0 / 10	0	10

Appendix

Table 14: GPT-6 with move_eef : per-task results. All rows have ten trials; Cap gives the number stopped by the applicable call budget. Positive success entries are shaded.

Model	Successes / episodes	SR (%)	Episodes CI (%)	Tasks+episodes CI (%)
OpenWAM	1,453 / 5,000	29.1	[28.0, 30.1]	[22.8, 36.0]
FastWAM	1,319 / 5,000	26.4	[25.4, 27.4]	[20.0, 33.8]
$\pi_{0.5}$	1,243 / 5,000	24.9	[23.8, 25.9]	[18.6, 31.7]
DM0.5	1,072 / 5,000	21.4	[20.5, 22.4]	[15.2, 28.1]
LingBot-VLA	984 / 5,000	19.7	[18.8, 20.6]	[13.6, 26.2]
HY-Embodied	947 / 4,999	18.9	[18.0, 19.8]	[13.2, 25.4]

Appendix

Table 15: Core-50 task-macro success on all 50 tasks with 95% bootstrap intervals over episodes only and over tasks and episodes. GPT-6 uses ten trials/task; its task-and-episode interval uses the same 2,000-resample procedure. These intervals describe sampling uncertainty, not equivalence of model inputs or query budgets. Shaded: best trained checkpoint.

ID	Task	OW	FW	$\pi_{0.5}$	DM	LB	HY	Motus	SF	DP
01	Align Blocks in Midair	2	3	0	0	1	0	0	1	0
02	Connect Charger and Cable	1	0	0	0	0	0	0	0	0
03	Cover Box	61	0	34	34	57	38	32	0	0
04	Cover Pitcher	12	3	5	5	5	1	10	0	2
05	Fill Cup	4	12	6	0	2	0	0	0	0
06	Fit Device Cover	4	3	0	0	2	1	1	1	0

Appendix

Table 16: Per-task success rates (%) on the 50 Core-50 tasks. Each cell has 100 episodes for trained checkpoints, so those cells also give success counts; GPT denotes GPT-6 ( move_eef ), with ten trials/cell, and its values are percentages, not counts. ∗ HY-Embodied has 99 valid episodes on Place Wine Glass. Columns use the selected checkpoints from Section 5.1 ; OW, FW, DM, LB, HY, and SF denote OpenWAM, FastWAM, DM0.5, LingBot-VLA, HY-Embodied, and Spatial Forcing. The DP sampler caveat above applies to every DP cell. Shaded: best trained checkpoint on each task, ties included. GPT-6 is a descriptive comparison with different inputs and query budgets.

Trajectories	Task 24 $s/n$	95% CI	Task 43 $s/n$	95% CI
10	7/100	[3.4, 13.7]	1/100	[0.2, 5.4]
50	23/100	[15.8, 32.2]	11/100	[6.3, 18.6]
100	31/100	[22.8, 40.6]	13/100	[7.8, 21.0]
200	65/100	[55.3, 73.6]	18/100	[11.7, 26.7]
Full (438 / 439)	83/100	[74.5, 89.1]	44/100	[34.7, 53.8]

Appendix

Table 17: Single-task scaling results. Success counts are out of 100 trials per cell; intervals are pointwise 95% Wilson intervals in percent. Full-pool sizes are given for Tasks 24 and 43, respectively. Shaded: best.

Task	Screenshot	Task Instruction
01 Align Blocks in Midair		Lift both assigned blocks with their assigned hands, then mate their marked faces in midair and hold the physical contact stable for 0.4 seconds.
02 Connect Charger and Cable		Insert the charger plug into the power strip first, then insert the charging cable connector into the charger head.
03 Cover Box		Align the box lid and place it onto the clear box.
04 Cover Pitcher		Align the pitcher lid and place it onto the pitcher body.
05 Fill Cup		Hold the mug upright under the faucet with the left hand. Open the right lever outward past 70 percent, fill for half a second, place the mug on the marked pad, and close the faucet.
06 Fit Device Cover		Use both hands on opposite sides of the device cover to carry it together over the device body, then align it with its matching interface and seat it.

Appendix

Table 18: Task profiles of HandsOn.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,615 papers

BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks
Oct 18, 2026Yong Wang, Kecheng Zheng, Xing Zhu +3Bimanual Robotic Manipulation Robotic Manipulation
Zhejiang University · Ant Group
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
Jul 9, 2026Yunchao Yao, Zhuxiu Xu, Tianqi Zhang +12Dexterous Manipulation Long-Horizon Robotic Manipulation
UNC-Chapel Hill · The University of Hong Kong · UC Berkeley
DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
May 3, 2026Zijian Zeng, Fei Ding, Huiming Yang +2Domain Randomization Dexterous Manipulation
Tsinghua University · Alibaba Group · Bengbu University +1

Oct 18, 2026J/K move · Enter open · S save

BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks

Yong Wang, Kecheng Zheng, Xing Zhu, Chao Xu+2

Zhejiang University · Ant Group

Bimanual Robotic Manipulation Robotic Manipulation Robot Manipulation Benchmarks Generalization in Robotic Manipulation Long-Horizon Robotic Manipulation

Open paper AskPDF arXiv

Bimanual manipulation with dexterous hands has attracted growing attention in research on robotic action models. However, few studies have developed benchmarks specifically tailored for dexterous manipulation, leaving the dexterous manipulation capabilities of existing models insufficiently characterized. In this paper, we present BiDex-100, a simulation benchmark for bimanual dexterous manipulation that combines broad task coverage, multi-dimensional evaluation, and high-DoF hand embodiments. Concretely, the benchmark comprises 106 tasks organized along five capability axes: generalization, instruction following, limited demonstrations, long horizon, and memory. Each task is paired with an out-of-domain variant featuring controlled visual and spatial changes, allowing systematic assessment of robustness. In addition, the benchmark supports two embodiments and provides 8,080 success-validated teleoperated demonstrations, totaling 6.25 million frames. We evaluate seven recent Vision-Language-Action models under a unified protocol, establishing baselines for capability-specific performance, robustness, and data efficiency. The website is available at https://BiDex-100.github.io/BiDex-100/.

Abstract

Robotic Manipulation

Explore similar work

BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation