DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Jin Wu; Lianjie Yuan; Zeyan Sun; Yuanyuan Lei; Disi A; Bicheng Han; Fangzhou Xia

cs.ROSep 17, 2026

DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Authors: Jin Wu, Lianjie Yuan, Zeyan Sun, Yuanyuan Lei, Disi A, Bicheng Han, Fangzhou Xia

Organizations: The University of Texas at Austin · University of Florida · BrainCo

Abstract

Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/

Figures & tables

Fig. 1: Single-image asset factorization. (a) Semantic parsing separates interactive objects from the static scene. (b) Object localization, segmentation, completion, and mesh generation produce foreground assets. (c) Background completion and multi-view reconstruction produce the Gaussian scene.

Fig. 2: Metric scene and object alignment. (d) Photometric and sparse geometric constraints recover scene scale and camera pose. (e) Robot geometry refines marker-based calibration. (f) Object rescaling and canonical-axis alignment ground each asset metrically.

Modality	Components
Policy	Previous robot action.
Proprioception	Robot joint positions.
State	Object poses and orientation, articulated-joint displacement.
Contact	3D contact force at each fingertip.
Goal	Target pose and orientation, signed articulated-joint goal position.
Privileged	Robot joint and object velocities, hand-base and fingertip poses and spatial velocities, sampled object surface points.

TABLE I: Observation modalities exposed by the task interface.

Fig. 3: Simulator task construction. (g) Gaussian appearance, mesh collisions, and dynamic assets form the simulation scene. (h) Task primitives define objectives, randomization, and success criteria, with shared multimodal observations for learning and deployment.

	Segmentation and Localization		RGB Appearance		Depth	3D Reconstruction
Method	Mask IoU $\uparrow$	Boundary F1 $\uparrow$	RGB MAE $\downarrow$	$\Delta E_{76}$ $\downarrow$	Depth MAE (m) $\downarrow$	Centroid (m) $\downarrow$	Chamfer-L1 (m) $\downarrow$	F-score@1 cm $\uparrow$
Ours	$\boldsymbol{.731\!\pm\!.136}$	$\boldsymbol{.260\!\pm\!.120}$	$\boldsymbol{.182\!\pm\!.062}$	$\boldsymbol{27.90\!\pm\!7.64}$	$\boldsymbol{.0093\!\pm\!.0099}$	$\boldsymbol{.0115\!\pm\!.0160}$	$\boldsymbol{.0069\!\pm\!.0076}$	$\boldsymbol{.906\!\pm\!.132}$
Hunyuan3D-2.1	$.661\!\pm\!.180$	$.239\!\pm\!.137$	$.223\!\pm\!.063$	$34.74\!\pm\!10.12$	$.0130\!\pm\!.0115$	$.0143\!\pm\!.0173$	$.0103\!\pm\!.0122$	$.750\!\pm\!.250$
TRELLIS.2	$.559\!\pm\!.196$	$.163\!\pm\!.087$	$.250\!\pm\!.063$	$39.11\!\pm\!11.23$	$.0296\!\pm\!.0179$	$.0258\!\pm\!.0144$	$.0199\!\pm\!.0120$	$.346\!\pm\!.202$
SAM3D+FP	$.604\!\pm\!.143$	$.236\!\pm\!.118$	$.407\!\pm\!.118$	$58.68\!\pm\!13.02$	$.0166\!\pm\!.0140$	$.0183\!\pm\!.0173$	$.0118\!\pm\!.0081$	$.622\!\pm\!.235$
SAM3D-only	$.488\!\pm\!.285$	$.165\!\pm\!.114$	$.391\!\pm\!.111$	$55.95\!\pm\!12.95$	$.0235\!\pm\!.0235$	$.0299\!\pm\!.0358$	$.0231\!\pm\!.0252$	$.538\!\pm\!.339$

TABLE II: Object-instance reconstruction and alignment over 86 valid instances.

RGB Appearance				Depth				3D Reconstruction
RGB MAE $\downarrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$	$\Delta E_{76}$ $\downarrow$	Depth MAE (m) $\downarrow$	Depth RMSE (m) $\downarrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	Centroid (m) $\downarrow$	Chamfer-L1 (m) $\downarrow$	F-score@1 cm $\uparrow$
0.0583	20.74	0.761	8.22	0.00516	0.00655	0.010	1.000	0.00100	0.00633	0.900

TABLE III: Static-background fidelity of the HY-World reconstruction from the calibrated top-camera view.

Task	Real succ.	Rate (%)
OB: LiftToyBus	25/30	83.33
OB: LiftBlueMug	30/30	100.00
OB: PlaceToyBusInBasket	28/30	93.33
OB: HandoverGrayBowl	24/30	80.00
OB: BimanualLiftBasket	25/30	83.33
OB: ReorientThermometer	22/30	73.33

TABLE IV: Real replay of 30 simulation-successful rollouts per task across two robot embodiments.

	Imitation learning					Reinforcement learning
Task pair	ACT	DP	BC-RNN	OpenVLA	$\pi_{0.5}$	T–S
OB: LiftToyBus	67/55	71/50	5/0	14/5	73/65	53/20
OB: LiftBlueMug	82/100	89/90	19/5	35/10	85/100	21/10
OB: PlaceToyBusInBasket	24/15	32/0	6/0	42/10	36/10	16/0
OB: HandoverGrayBowl	75/65	15/0	0/0	17/0	63/40	53/10
OB: BimanualLiftBasket	70/80	69/45	27/5	8/0	84/90	62/30

TABLE V: Policy success rates (%) across 13 task–embodiment pairs. Each entry is simulation/real, evaluated over 100/20 trials. Imitation policies use simulation and real data; T–S is PPO teacher–student distillation.

Fig. 4: Experimental platforms. Left: KUKA LBR iiwa 7 with a LEAP Hand. Right: dual-arm OpenArm with BrainCo Revo1 hands. The cameras and task workspaces are shown for each platform.

Fig. 5: Representative top-camera comparisons for two workspaces: (a, c) DEXTERA simulations and (b, d) corresponding real observations. Rows show RGB appearance, instance masks, and depth, from top to bottom.

Fig. 6: Qualitative object-reconstruction comparison on a coconut-water carton, thermometer, and toy bus. Columns show the ground-truth object, SAM3D, TRELLIS.2, Hunyuan3D-2.1, and DEXTERA (ours). DEXTERA preserves object shape and appearance across geometrically and texturally distinct assets.

Fig. 7: Visual and interaction consistency during basket lifting and bowl handover. Rows (a, b) show simulated and real basket lifting, respectively; (c, d) show simulated and real bowl handover. Columns progress through successive interaction stages from left to right.

Fig. 8: Simulation-only training versus sim-real co-training for three policies on six task-embodiment pairs. Each panel separates simulation evaluation (left) from physical evaluation (right, shaded), with 100 and 20 trials per bar, respectively. Bar labels report success percentages. OB: OpenArm + BrainCo Revo1; KL: KUKA + LEAP Hand. Co-training improves physical success in 17 of 18 comparisons.

Explore similar work

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
May 3, 2026Zijian Zeng, Fei Ding, Huiming Yang +2Domain Randomization Dexterous Manipulation
Tsinghua University · Alibaba Group · Bengbu University +1
HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation
May 26, 2026Junyi Dong, Haotian Luo, Ziwei Xu +11Synthetic-to-Real Domain Adaptation Sim-to-Real Transfer
CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong
SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation
Jun 13, 2026Wenkang Hu, Haoran Wang, Yitong Li +10Zero-Shot Robotic Manipulation Robot Skill Learning
1Shanghai Jiao Tong University · 2Horizon Robotics · 3Style3D Research

May 3, 2026, cs.ROJ/K move · Enter open · S save

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

Zijian Zeng, Fei Ding, Huiming Yang, Xianwei Li+1

Tsinghua University · Alibaba Group · Bengbu University+1

Domain Randomization Dexterous Manipulation Sim-to-Real Transfer Generalization in Robotic Manipulation Robotic RL VLMs for Robotics

Open paper AskPDF arXiv

Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.

Abstract

Explore similar work

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation