OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes

Naiyu Fang; Zhongjin Luo; Yuxin Mo; Siyuan Huang; Jianbo Liu; Yufei Liu; Zheyuan Zhou; Chenkai Jin

cs.ROOct 8, 2026

OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes

Authors: Naiyu Fang, Zhongjin Luo, Yuxin Mo, Siyuan Huang, Jianbo Liu, Yufei Liu, Zheyuan Zhou, Chenkai Jin, +2 more

Organizations: ACE Robotics · CUHK, MMLab · CPII under InnoHK · Tongji University · Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University

Abstract

Dexterous grasping is the foundational primitive in embodied AI, demanding massive data to train robust models. As real-world data collection is expensive, simulation has become the mainstream paradigm. Yet, while cluttered scenes best reflect real-world applications, learning to grasp within them is bottlenecked by a critical scarcity of large-scale data. To resolve this, we curate high-quality 3D objects and supporting bases, proposing a scalable seed-and-filter strategy that bypasses sluggish scene-level optimization. This yields an unprecedented benchmark comprising over 2.6 million scenes and 0.4B scene-specific grasp ground truths, featuring diverse realistic layouts paired with rich semantic and geometric observations. Furthermore, we introduce the OmniDex model to overcome the grasp multimodality and last-millimeter precision errors plaguing current generative models. By coupling Soft Winner-Takes-All learning with human-inspired physical constraints during training, and utilizing physics-driven ranking, our approach achieves robust dexterous grasping without the latency of post-optimization. Experimental results show that OmniDex model achieves state-of-the-art performance and strong generalization across diverse scenes, views, and unseen objects.

Figures & tables

Figure 1 : Overview of OmniDex , the largest benchmark for dexterous grasping in cluttered scenes to date. Left : Multi-view RGB-D and instruction masks paired with scene-specific grasp poses. Right : A scalable seed-and-filter strategy enables unprecedented scale (2.6M scenes, 0.4B grasp ground truths), encompassing diverse objects, supporting bases, and realistic layouts.

Method	# Scenes	Setting	# Objects	Workspace	Modality
Dex1B [ 31 ]	-	Single	6K	-	Depth
AffordDexGrasp [ 27 ]	2K	Single	1K	Table	RGB-D
DDGC [ 16 ]	0.4K	Cluttered	0.3K	Table	RGB-D
DexGraspNet 2.0 [ 32 ]	8K	Cluttered	1K	Table	Depth
ClutterDexGrasp [ 5 ]	1K	Cluttered	2K	Table	Depth
Ours	2.6M	Cluttered	6K	Table, Box, Shelf	RGB-D

Table 1 : Quantitative comparison of simulation-based dexterous grasping benchmarks. Our dataset features the large scale of scenes and objects, emphasizing diverse layouts in clutter.

Figure 2 : Seed-and-filter generation pipeline . Our two-stage framework generates scene-valid grasps by: (1) constructing a reusable seed grasp library in free space, and (2) simulating cluttered scenes in Isaac Gym for collision filtering, followed by multi-modal rendering in Blender.

Figure 3 : Dataset statistics and visualization of 3D assets . (a): Instance counts and category-wise distributions of the curated graspable objects, highlighting the top 50 semantic classes. (b): Visual examples of the assets, which range from diverse grasped objects (medicine bottles, boxes, apples) to realistic supporting bases (tables, open-top boxes, shelves).

Subset	Obj. Div.	Scene Obj. Comp.	Sup. Base Type	Sup. Base Div.	Density	Scene Num.	Generalization Split
Subset	Obj. Div.	Scene Obj. Comp.	Sup. Base Type	Sup. Base Div.	Density	Scene Num.	Train	Seen	Unseen
2 ${{{\cal D}_{{\rm{std}}}}}$	✓	Multi-category	Table	✗	0.1 / 0.2 / 0.3	1.14M	919K	62K	163K
${{{\cal D}_{{\rm{mix}}}}}$	✓	Multi-category	Table	✓	0.1 / 0.2 / 0.3	0.94M	706K	80K	152K
${{{\cal D}_{{\rm{box}}}}}$	✓	Multi-category	Box	✓	0.9	0.14M	112K	12K	16K
${{{\cal D}_{{\rm{grid}}}}}$	✓	Single-category	Table	✗	-	0.28M	246K	11K	24K
${{{\cal D}_{{\rm{shelf}}}}}$	✓	Multi-category	Shelf	✓	-	0.14M	107K	11K	28K

Table 2 : Benchmark diversity and evaluation protocol across OmniDex subsets. Each subset is organized with a large training split and two decoupled test tracks to evaluate generalization.

Figure 4 : Architecture of the OmniDex Model . Condition embeddings are transformed from RGB-D inputs, instruction masks, and scene parameters. During training, a denoising network adds identical noise to $M$ grasp poses, optimized jointly via a Soft Winner-Takes-All loss (for multimodal distributions) and a physics-aware loss (for stable contact). During inference, the network generates $M$ candidates, followed by a physics-based ranking module to select the optimal grasp.

Method	Subset	Input View	Seen Objects							Unseen Objects
Method	Subset	Input View	SR ${}_{\text{scene}}$	SR ${}_{\text{scene}}^{*}$	SR ${}_{\text{any}}$	SR ${}_{\text{any}}^{*}$	SR ${}_{\text{one}}$	SR ${}_{\text{one}}^{*}$	CFR	SR ${}_{\text{scene}}$	SR ${}_{\text{scene}}^{*}$	SR ${}_{\text{any}}$	SR ${}_{\text{any}}^{*}$	SR ${}_{\text{one}}$	SR ${}_{\text{one}}^{*}$	CFR
Ours	$\mathcal{D}_{\text{std}}$	Ego	33.60	55.80	18.87	36.79	65.15	79.35	84.12	40.15	59.74	27.18	40.77	65.54	81.77	84.61
	$\mathcal{D}_{\text{mix}}$	Ego	32.05	52.21	17.12	34.04	58.51	74.52	73.12	37.90	55.10	24.13	35.19	62.41	78.02	83.22
	$\mathcal{D}_{\text{box}}$	Ego	32.34	35.83	17.06	19.79	55.15	56.52	99.12	33.48	36.50	17.80	20.17	56.77	58.11	95.72
	$\mathcal{D}_{\text{grid}}$	Ego	55.01	79.60	33.68	55.02	91.91	93.70	95.46	43.11	68.05	33.97	52.67	87.50	92.09	94.86
	$\mathcal{D}_{\text{shelf}}$	Ego	29.49	39.46	18.31	25.25	55.17	60.57	81.95	14.96	24.08	11.58	15.08	34.26	37.89	63.57

Table 3 : Main Results on the OmniDex Benchmark . We evaluate our method against baselines across diverse scene subsets and compare with other methods on

\mathcal{D}_{\text{std}}

. * denotes the use of the finger-force trick [ 24 ]

Metrics	20%	40%	60%	80%	100%
$\text{SR}_{\text{scene}}$	40.50	41.04	43.84	44.32	44.41
CFR	81.54	82.64	82.13	84.15	83.66

Table 4 : Quantitative results when trained on incrementally larger fractions of training dataset.

RGB	Depth	Cam. Para.	Table Height	Multi-view	$\text{SR}_{\text{scene}}$	CFR
✓		✓	✓		31.89	79.97
	✓	✓	✓		36.80	82.81
✓	✓		✓		43.68	84.09
✓	✓	✓			40.64	84.04
✓	✓	✓	✓	✓	49.87	82.11
✓	✓	✓	✓		44.41	83.66

Table 5 : Quantitative ablation on various input conditions. The highlighted row represents baseline.

Training Phase					Inference Phase			Metrics
$\mathcal{L}_{\text{stand}}$	$\mathcal{L}_{\text{WTA}}$	$\mathcal{L}_{\text{SDF}}$	$\mathcal{L}_{\text{thumb}},\mathcal{L}_{\text{opp}}$	$\mathcal{L}_{\text{closure}}$	Naive Samp.	Rank.	TTP	$\text{SR}_{\text{scene}}$	CFR
✓						✓		37.79	81.37
	✓					✓		38.07	81.31
	✓	✓				✓		41.93	84.06
	✓	✓	✓			✓		43.19	84.40
	✓	✓	✓	✓	✓			36.08	79.71

Table 6 : Component-wise ablation study evaluated on both training and inference phases. TTP denotes Test-Time Optimization.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Parallel Hypotheses ( $M$ )							TTP
Metric	1	2	4	8	16	32	64	TTP
$\text{SR}_{\text{scene}}$	36.08	38.02	41.06	42.47	43.41	44.37	44.41	36.16
CFR	79.71	80.81	81.58	82.26	82.62	83.09	83.66	2.56
Inference Time (ms)	219.14	233.35	266.39	325.20	543.93	948.33	1598.41	6044.60

Appendix

Table D1 : Effect of parallel hypothesis generation vs. Test-Time Optimization.

$\text{SR}_{\text{scene}}$ failures		CFR failures
Failure type	Rate	Failure type	Rate
Insufficient contact	4.29%	Supporting-base collision	61.26%
Stability failure	95.71%	Non-target-object collision	38.74%

Appendix

Table D2 : Failure breakdown on the

\mathcal{D}_{\text{std}}

mini-split.

Rear offset (cm)	Height offset (cm)	IK feasibility (%)
0.0	+2.5	87.98
0.0	+5.0	87.94
2.5	+5.0	86.68
5.0	+2.5	85.63
7.5	+5.0	83.75

Appendix

Table D3 : IK feasibility under five representative Franka arm placements on the

\mathcal{D}_{\text{std}}

mini-split.

Figure E1 : Qualitative visualization of the $\mathcal{D}_{\text{std}}$ subset.

Figure E2 : Qualitative visualization of the $\mathcal{D}_{\text{mix}}$ subset.

Figure E3 : Qualitative visualization of the $\mathcal{D}_{\text{box}}$ subset.

Figure E4 : Qualitative visualization of the $\mathcal{D}_{\text{grid}}$ subset.

Figure E5 : Qualitative visualization of the $\mathcal{D}_{\text{shelf}}$ subset.

Figure E6 : Qualitative results of OmniDex model across diverse benchmark subsets.

Figure E7 : Representative real-robot grasping trials in cluttered scenes.

Explore similar work

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping
Jun 3, 2026Haofei Lu, Hongjia Liu, Yifei Dong +3Robotic Grasping Robotic RL
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden. · Robotics and Autonomous Systems at University of Turku, Finland.
SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments
Jul 22, 2026Yang Xu, Gurpreet Singh Mukker, Raymond Wang +3Robotic Grasping Language-Conditioned Robot Manipulation
University of Toronto · Vector Institute · University of British Columbia +1
AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection
Jun 22, 2026Mingi Choi, Gunhee Kim, Jisoo Kim +4Robotic Grasping Robotic Data Collection
Seoul National University · RLWRLD

Jun 3, 2026, cs.ROJ/K move · Enter open · S save

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

Haofei Lu, Hongjia Liu, Yifei Dong, Florian T. Pokorny+2

Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden. · Robotics and Autonomous Systems at University of Turku, Finland.

Robotic Grasping Robotic RL Diffusion Policy

Open paper AskPDF arXiv

This work addresses sequentially grasping multiple objects with a single dexterous hand without releasing those already held. Most dexterous grasping methods commit all of the hand's degrees of freedom to a single object, underutilizing its dexterity and leaving no redundancy for subsequent grasps. The proposed solution, MoDex, is a diffusion policy that predicts the next gripper pose directly from observations, conditioned on an opposition space and point cloud. The opposition space condition specifies which fingers participate in the current grasp, enabling the gripper to use only a subset of its available degrees of freedom while reserving the remaining degrees of freedom for subsequent grasps. To facilitate sim-to-real transfer, MoDex is trained in two stages: first through imitation learning on expert demonstrations, and subsequently through reinforcement learning fine-tuning, which consistently improves success rates over the pre-trained policy. We evaluate MoDex in simulation on a MuJoCo-based Franka Emika Panda robot equipped with an Allegro Hand and on the corresponding real-world hardware platform. Across both simulation and real-world experiments, MoDex achieves higher success rates than the evaluated learning-based baselines, improving performance by 2.92-17.92% and 6.67-17.78%, respectively. Project page: https://modex2026.github.io/.

Abstract

Explore similar work

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection