DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Jin Wu; Lianjie Yuan; Zeyan Sun; Yuanyuan Lei; Disi A; Bicheng Han; Fangzhou Xia

cs.ROSep 17, 2026

DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Authors: Jin Wu, Lianjie Yuan, Zeyan Sun, Yuanyuan Lei, Disi A, Bicheng Han, Fangzhou Xia

Organizations: The University of Texas at Austin · University of Florida · BrainCo

Abstract

Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/

Figures & tables

Fig. 1: Single-image asset factorization. (a) Semantic parsing separates interactive objects from the static scene. (b) Object localization, segmentation, completion, and mesh generation produce foreground assets. (c) Background completion and multi-view reconstruction produce the Gaussian scene.

Fig. 2: Metric scene and object alignment. (d) Photometric and sparse geometric constraints recover scene scale and camera pose. (e) Robot geometry refines marker-based calibration. (f) Object rescaling and canonical-axis alignment ground each asset metrically.

Modality	Components
Policy	Previous robot action.
Proprioception	Robot joint positions.
State	Object poses and orientation, articulated-joint displacement.
Contact	3D contact force at each fingertip.
Goal	Target pose and orientation, signed articulated-joint goal position.
Privileged	Robot joint and object velocities, hand-base and fingertip poses and spatial velocities, sampled object surface points.

TABLE I: Observation modalities exposed by the task interface.

Fig. 3: Simulator task construction. (g) Gaussian appearance, mesh collisions, and dynamic assets form the simulation scene. (h) Task primitives define objectives, randomization, and success criteria, with shared multimodal observations for learning and deployment.

	Segmentation and Localization		RGB Appearance		Depth	3D Reconstruction
Method	Mask IoU $\uparrow$	Boundary F1 $\uparrow$	RGB MAE $\downarrow$	$\Delta E_{76}$ $\downarrow$	Depth MAE (m) $\downarrow$	Centroid (m) $\downarrow$	Chamfer-L1 (m) $\downarrow$	F-score@1 cm $\uparrow$
Ours	$\boldsymbol{.731\!\pm\!.136}$	$\boldsymbol{.260\!\pm\!.120}$	$\boldsymbol{.182\!\pm\!.062}$	$\boldsymbol{27.90\!\pm\!7.64}$	$\boldsymbol{.0093\!\pm\!.0099}$	$\boldsymbol{.0115\!\pm\!.0160}$	$\boldsymbol{.0069\!\pm\!.0076}$	$\boldsymbol{.906\!\pm\!.132}$
Hunyuan3D-2.1	$.661\!\pm\!.180$	$.239\!\pm\!.137$	$.223\!\pm\!.063$	$34.74\!\pm\!10.12$	$.0130\!\pm\!.0115$	$.0143\!\pm\!.0173$	$.0103\!\pm\!.0122$	$.750\!\pm\!.250$
TRELLIS.2	$.559\!\pm\!.196$	$.163\!\pm\!.087$	$.250\!\pm\!.063$	$39.11\!\pm\!11.23$	$.0296\!\pm\!.0179$	$.0258\!\pm\!.0144$	$.0199\!\pm\!.0120$	$.346\!\pm\!.202$
SAM3D+FP	$.604\!\pm\!.143$	$.236\!\pm\!.118$	$.407\!\pm\!.118$	$58.68\!\pm\!13.02$	$.0166\!\pm\!.0140$	$.0183\!\pm\!.0173$	$.0118\!\pm\!.0081$	$.622\!\pm\!.235$
SAM3D-only	$.488\!\pm\!.285$	$.165\!\pm\!.114$	$.391\!\pm\!.111$	$55.95\!\pm\!12.95$	$.0235\!\pm\!.0235$	$.0299\!\pm\!.0358$	$.0231\!\pm\!.0252$	$.538\!\pm\!.339$

TABLE II: Object-instance reconstruction and alignment over 86 valid instances.

RGB Appearance				Depth				3D Reconstruction
RGB MAE $\downarrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$	$\Delta E_{76}$ $\downarrow$	Depth MAE (m) $\downarrow$	Depth RMSE (m) $\downarrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	Centroid (m) $\downarrow$	Chamfer-L1 (m) $\downarrow$	F-score@1 cm $\uparrow$
0.0583	20.74	0.761	8.22	0.00516	0.00655	0.010	1.000	0.00100	0.00633	0.900

TABLE III: Static-background fidelity of the HY-World reconstruction from the calibrated top-camera view.

Task	Real succ.	Rate (%)
OB: LiftToyBus	25/30	83.33
OB: LiftBlueMug	30/30	100.00
OB: PlaceToyBusInBasket	28/30	93.33
OB: HandoverGrayBowl	24/30	80.00
OB: BimanualLiftBasket	25/30	83.33
OB: ReorientThermometer	22/30	73.33

TABLE IV: Real replay of 30 simulation-successful rollouts per task across two robot embodiments.

	Imitation learning					Reinforcement learning
Task pair	ACT	DP	BC-RNN	OpenVLA	$\pi_{0.5}$	T–S
OB: LiftToyBus	67/55	71/50	5/0	14/5	73/65	53/20
OB: LiftBlueMug	82/100	89/90	19/5	35/10	85/100	21/10
OB: PlaceToyBusInBasket	24/15	32/0	6/0	42/10	36/10	16/0
OB: HandoverGrayBowl	75/65	15/0	0/0	17/0	63/40	53/10
OB: BimanualLiftBasket	70/80	69/45	27/5	8/0	84/90	62/30

TABLE V: Policy success rates (%) across 13 task–embodiment pairs. Each entry is simulation/real, evaluated over 100/20 trials. Imitation policies use simulation and real data; T–S is PPO teacher–student distillation.

Fig. 4: Experimental platforms. Left: KUKA LBR iiwa 7 with a LEAP Hand. Right: dual-arm OpenArm with BrainCo Revo1 hands. The cameras and task workspaces are shown for each platform.

Fig. 5: Representative top-camera comparisons for two workspaces: (a, c) DEXTERA simulations and (b, d) corresponding real observations. Rows show RGB appearance, instance masks, and depth, from top to bottom.

Fig. 6: Qualitative object-reconstruction comparison on a coconut-water carton, thermometer, and toy bus. Columns show the ground-truth object, SAM3D, TRELLIS.2, Hunyuan3D-2.1, and DEXTERA (ours). DEXTERA preserves object shape and appearance across geometrically and texturally distinct assets.

Fig. 7: Visual and interaction consistency during basket lifting and bowl handover. Rows (a, b) show simulated and real basket lifting, respectively; (c, d) show simulated and real bowl handover. Columns progress through successive interaction stages from left to right.

Fig. 8: Simulation-only training versus sim-real co-training for three policies on six task-embodiment pairs. Each panel separates simulation evaluation (left) from physical evaluation (right, shaded), with 100 and 20 trials per bar, respectively. Bar labels report success percentages. OB: OpenArm + BrainCo Revo1; KL: KUKA + LEAP Hand. Co-training improves physical success in 17 of 18 comparisons.

Explore similar work

May 3, 2026cs.RO

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.

Zijian Zeng, Fei Ding, Huiming Yang +2

Tsinghua University · Alibaba Group · Bengbu University +1

Sim-To-Real Reinforcement Learning Contact-Rich Dexterous Manipulation Sim-To-Real Gap Dexterous Manipulation Robotic Systems

Ask PDF

May 26, 2026cs.RO

HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

Scaling data volume and diversity is critical for generalizing embodied intelligence. While synthetic data generation offers a scalable alternative to expensive physical data acquisition, transferring robotic manipulation policies from simulation to the real world (sim-to-real) remains a formidable challenge due to the domain gap. This paper presents HyperSim, a holistic framework spanning from synthetic data generation to policy training and seamless real-world deployment. To systematically bridge the sim-to-real gap, HyperSim is realized through three core pillars: high-fidelity environment synthesis, adversarial trajectory generation, and sim-and-real co-training. Collectively, these modules address domain discrepancies by enhancing visual fidelity, expanding data coverage, and enforcing domain-invariant representations. We rigorously validate HyperSim through a large-scale empirical study involving 400 real-world task executions across two representative manipulation models. Assessed across three fine-grained metrics, our complete pipeline achieves remarkable sim-to-real success rates of 80% and 95% with ACT and π_{0}, respectively. Furthermore, policies trained on our adversarial trajectories exhibit significantly enhanced robustness against dynamic uncertainties, achieving a 35% higher completion rate under physical perturbations.

Junyi Dong, Haotian Luo, Ziwei Xu +11

CloudRobo Lab, Huawei Cloud Computing Technologies Co.,Ltd. · Shanghai Jiao Tong University · The University of Hong Kong

Sim-To-Real Reinforcement Learning Robotic Manipulation Policies Sim-To-Real Gap Robotic Manipulation Nvidia Isaac Sim Synthetic Data+1

Ask PDF

Jun 13, 2026cs.RO

SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation

RGB sim-to-real for deformable manipulation has remained largely unsolved without real-world fine-tuning. We present SimWeaver, which trains zero-shot RGB VLA policies on 200 simulated demonstrations per task, reaching above 80% per-task and 91% average real-world success across 5 diverse deformable tasks including plastic-bag manipulation, without teleoperation or per-task calibration. SimWeaver combines a reliable measurement-backed simulator (SimWeaver-Sim) with an extensible asset framework supporting single-image generation(SimWeaver-Asset), a deterministic topology-aware trajectory synthesizer (SimWeaver-Syn), and a sim-to-real protocol with ISP-aware photometric augmentation (SimWeaver-Real). On silk grasping, the sim-trained policy reaches 100% under visual distribution shifts where real-data baselines drop to 9-70%, at two orders of magnitude lower per-trajectory cost. We will release SimWeaver and a representative asset subset. Project page: https://simweaver.github.io/

Wenkang Hu, Haoran Wang, Yitong Li +10

1Shanghai Jiao Tong University · 2Horizon Robotics · 3Style3D Research

Sim-To-Real Reinforcement Learning Sim-To-Real Gap Deformable Linear Objects Generalizable Vision-Language-Action Policies Weave Image Generation

Ask PDF