ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation

Arjun Bhardwaj; Maximum Wilder-Smith; Mayank Mittal; Vaishakh Patil; Marco Hutter

cs.ROApr 13, 2026

ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation

Authors: Arjun Bhardwaj, Maximum Wilder-Smith, Mayank Mittal, Vaishakh Patil, Marco Hutter

Organizations: ETH Zurich · NVIDIA

Abstract

In-hand object reorientation requires precise estimation of the object pose to handle complex task dynamics. While RGB sensing offers rich semantic cues for pose tracking, existing solutions rely on multi-camera setups or costly ray tracing. We present a sim-to-real framework for monocular RGB in-hand reorientation that integrates 3D Gaussian Splatting (3DGS) to bridge the visual sim-to-real gap. Our key insight is performing domain randomization in the Gaussian representation space: by applying physically consistent, pre-rendering augmentations to 3D Gaussians, we generate photorealistic, randomized visual data for object pose estimation. The manipulation policy is trained using curriculum-based reinforcement learning with teacher-student distillation, enabling efficient learning of complex behaviors. Importantly, both perception and control models can be trained independently on consumer-grade hardware, eliminating the need for large compute clusters. Experiments show that the pose estimator trained with 3DGS data outperforms those trained using conventional rendering data in challenging visual environments. We validate the system on a physical multi-fingered hand equipped with an RGB camera, demonstrating robust reorientation of five diverse objects even under challenging lighting conditions. Our results highlight Gaussian splatting as a practical path for RGB-only dexterous manipulation. For videos of the hardware deployments and additional supplementary materials, please refer to the project website: https://rffr.leggedrobotics.com/works/viserdex/

Figures & tables

Fig. 2 : Overview of our sim-to-real in-hand reorientation pipeline. We first train a teacher policy in simulation with full state access, then distill it into a recurrent student policy that operates from noisy observations. A monocular RGB pose estimator trained with 3D Gaussian Splatting data provides object poses to the student policy, enabling goal-conditioned dexterous manipulation on a real multi-fingered hand.

Fig. 3 : Pre-rasterization augmentation examples. Visualizations of the proposed SH-based perturbations applied to clustered Gaussians, producing structured variations in color, reflectance, and spatial appearance without ray-tracing.

Augmentation	Targets	Probability	Fraction	Range
Random Noise
Additive	SH0, SHN	0.2	1.0	$[-0.1,0.1]$
Scaling	SH0, SHN	0.2	1.0	$[0.8,1.2]$
Spatial Cluster
Additive	SH0, SHN	0.8	0.10	$[-0.1,0.1]$
Scaling	SH0, SHN	0.8	0.20	$[0.9,1.1]$

TABLE I : Pre-Rasterization Augmentation Parameters

Objects

Cube

3D Printed Toy

Rubber Duck

Tablet Bottle

Globe

Mean

Method

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

Nominal Conditions

Standard Tiled

11.9_{\pm 0.86}

57.2_{\pm 8.13}

14.1_{\pm 0.92}

37.7_{\pm 2.84}

9.5_{\pm 0.23}

67.0_{\pm 1.47}

10.0_{\pm 0.65}

45.7_{\pm 4.78}

15.0_{\pm 2.41}

59.1_{\pm 3.68}

12.1_{\pm 1.01}

53.3_{\pm 4.18}

DR Tiled

10.5_{\pm 0.75}

69.7_{\pm 9.05}

15.9_{\pm 0.64}

32.5_{\pm 2.78}

9.1_{\pm 0.23}

73.7_{\pm 2.22}

\mathbf{9.0_{\pm 0.52}}

50.7_{\pm 3.56}

16.7_{\pm 1.20}

51.3_{\pm 4.21}

12.2_{\pm 0.67}

55.6_{\pm 4.36}

Naïve GS

10.4_{\pm 1.35}

59.0_{\pm 4.66}

18.7_{\pm 0.95}

23.3_{\pm 3.18}

11.4_{\pm 0.26}

44.3_{\pm 4.88}

11.2_{\pm 0.63}

48.3_{\pm 4.59}

20.2_{\pm 1.45}

17.2_{\pm 4.61}

14.4_{\pm 0.93}

38.4_{\pm 4.38}

Ours

\mathbf{9.1_{\pm 0.51}}

\mathbf{73.1_{\pm 4.73}}

\mathbf{11.3_{\pm 0.82}}

\mathbf{56.5_{\pm 5.77}}

\mathbf{7.9_{\pm 0.24}}

\mathbf{78.0_{\pm 3.97}}

10.2_{\pm 0.84}

\mathbf{51.7_{\pm 5.06}}

\mathbf{12.3_{\pm 0.89}}

\mathbf{67.7_{\pm 3.06}}

\mathbf{10.2_{\pm 0.66}}

\mathbf{65.4_{\pm 4.52}}

TABLE II : Evaluation of learned pose estimator on real-world data under nominal and adversarial lighting. Our method is compared against three baselines: Standard Tiled , Randomized Tiled , and Naive GS rendering. We report Average Distance of Model Points (ADD) in mm and a strict accuracy metric (

<10\text{mm}

and

<10^{\circ}

), averaged over 5 random seeds.

Objects

Cube

3D Printed Toy

Rubber Duck

Tablet Bottle

Globe

Mean

Method

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

ADD

Accuracy

Nominal Conditions

w/o Random Noise

11.9_{\pm 3.24}

61.5_{\pm 11.21}

13.5_{\pm 1.14}

37.2_{\pm 9.13}

8.2_{\pm 0.39}

\mathbf{80.9_{\pm 2.88}}

10.7_{\pm 0.84}

46.7_{\pm 3.33}

14.5_{\pm 1.17}

66.5_{\pm 5.32}

11.8_{\pm 1.36}

58.6_{\pm 6.37}

w/o Spatial Clustering

9.2_{\pm 0.91}

\mathbf{74.9_{\pm 8.49}}

12.9_{\pm 0.86}

44.7_{\pm 7.06}

8.0_{\pm 0.22}

73.5_{\pm 3.27}

16.3_{\pm 1.38}

39.3_{\pm 5.64}

13.8_{\pm 2.15}

51.3_{\pm 5.73}

12.0_{\pm 1.10}

56.7_{\pm 6.04}

w/o Color Clustering

9.2_{\pm 0.41}

71.0_{\pm 5.94}

13.6_{\pm 1.60}

49.4_{\pm 2.42}

8.2_{\pm 0.59}

77.4_{\pm 5.64}

20.4_{\pm 0.56}

34.3_{\pm 4.03}

14.5_{\pm 1.69}

64.8_{\pm 3.09}

13.2_{\pm 0.97}

59.4_{\pm 4.22}

w/o Global Shift

11.2_{\pm 1.26}

67.2_{\pm 3.85}

\mathbf{10.9_{\pm 1.13}}

55.7_{\pm 6.58}

9.1_{\pm 0.41}

70.9_{\pm 3.85}

39.5_{\pm 2.84}

10.3_{\pm 2.45}

13.9_{\pm 0.25}

52.1_{\pm 5.17}

16.9_{\pm 1.18}

51.2_{\pm 4.38}

TABLE III : Ablation study of pre-rasterization augmentations under nominal and adversarial conditions for pose estimation. We compare our method with models trained with specific augmentation groups removed. We report Average Distance of Model Points (ADD) in mm and a strict accuracy metric (

<10\text{mm}

and

<10^{\circ}

), averaged over 5 random seeds.

Fig. 5 : Impact of performance-based curricula on training efficiency. Curves show learning progress for full curriculum compared with ablations with individual components removed. Note that No Curriculum and w/o Penalty Curriculum coincide at zero.

Fig. 6 : Rollout sequence of the hand reorienting an object to the target pose. The images are from the robot camera feed.

	Consecutive Successes (Over Five Runs)
Objects	DeXtreme [ 7 ]	Ours (Nominal)	Ours (Adversarial)
Cube	$27.8_{\pm 19.0}$	${35.4_{\pm 13.8}}$	$25.6_{\pm 8.9}$
3D Printed Toy	-	${28.2_{\pm 12.6}}$	$12.0_{\pm 6.9}$
Rubber Duck	-	${24.2_{\pm 15.3}}$	$9.0_{\pm 5.0}$
Tablet Bottle	-	${12.6_{\pm 8.8}}$	$4.2_{\pm 0.7}$
Globe	-	${87.6_{\pm 41.4}}$	$76.2_{\pm 66.2}$

TABLE IV : Real-world deployment on different objects.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Consecutive Successes
Objects	Teacher Policy (w/o obs noise)	Student Policy (w/o obs noise)	Student Policy (with obs noise)
Cube	$111.4_{\pm 24.7}$	${92.1_{\pm 5.7}}$	$82.3_{\pm 8.4}$
3D Printed Toy	$106.0_{\pm 25.0}$	${74.9_{\pm 4.7}}$	$74.6_{\pm 6.2}$
Rubber Duck	$97.1_{\pm 14.1}$	${46.2_{\pm 1.8}}$	$41.6_{\pm 2.3}$
Tablet Bottle	$118.4_{\pm 2.8}$	${77.0_{\pm 4.2}}$	$69.4_{\pm 2.9}$
Globe	$163.6_{\pm 3.4}$	${138.1_{\pm 2.4}}$	$129.5_{\pm 5.9}$

Appendix

TABLE V : Simulation results on different objects. We evaluate the policy across 256 environments with randomized physical conditions (DR) and report the mean and standard deviation over five episodes.

Fig. 7 : Top: Temporal evolution of translation and rotation errors during a real-world rollout. Red regions indicate intervals where artificial noise is injected into the pose estimator input. The belief decoder (orange) effectively filters these high-frequency perturbations, maintaining significantly lower error compared to the corrupted input (green). Bottom: Visualization of a specific failure case (corresponding to the blue box in the plot). The belief decoder successfully maintains a stable pose estimate close to the ground truth, effectively rejecting a catastrophic 180 ∘ flip returned by the pose estimator.

Objects	Trans Error	Rot Error	Trans Error Correlation	Rot Error Correlation
Cube	$9.05$	$14.6$	$0.4$	$0.38$
3D Printed Toy	$11.14$	$33.59$	$0.42$	$0.12$
Rubber Duck	$8.85$	$18.84$	$0.02$	$0.17$
Tablet Bottle	$10.9$	$38.27$	$0.14$	$0.08$
Globe	$12.01$	$32.42$	$0.04$	$0.20$
Mean	$10.39$	$27.54$	$0.20$	$0.19$

Appendix

TABLE VI : Pose estimation during real-world hardware deployment. We report the translation error (in mm) and the rotation error (in degrees). We also report Pearson correlation coefficient between the occlusion ratio and errors.

Objects

Cube

3D Printed Toy

Rubber Duck

Tablet Bottle

Globe

Mean

Method

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Nominal Conditions

Standard Tiled

10.8_{\pm 0.78}

5.6_{\pm 1.57}

11.2_{\pm 0.32}

19.2_{\pm 3.79}

8.1_{\pm 0.22}

7.5_{\pm 0.75}

8.2_{\pm 0.39}

13.3_{\pm 1.41}

13.9_{\pm 2.51}

8.4_{\pm 0.59}

10.4_{\pm 1.20}

10.8_{\pm 1.99}

Domain Randomized

10.0_{\pm 0.80}

\mathbf{4.0_{\pm 0.15}}

15.4_{\pm 0.74}

\mathbf{8.8_{\pm 0.78}}

8.3_{\pm 0.26}

5.0_{\pm 0.20}

7.8_{\pm 0.13}

\mathbf{10.2_{\pm 1.29}}

14.5_{\pm 1.29}

14.5_{\pm 2.31}

11.2_{\pm 0.77}

8.5_{\pm 1.24}

Naive GS

8.7_{\pm 1.25}

6.9_{\pm 1.22}

15.3_{\pm 0.96}

23.2_{\pm 2.04}

10.6_{\pm 0.32}

6.0_{\pm 0.27}

8.6_{\pm 0.54}

16.0_{\pm 3.54}

18.8_{\pm 1.50}

11.7_{\pm 1.38}

12.4_{\pm 1.01}

12.8_{\pm 2.01}

Ours

\mathbf{8.2_{\pm 0.64}}

4.8_{\pm 0.18}

\mathbf{10.2_{\pm 1.01}}

\mathbf{8.8_{\pm 0.82}}

\mathbf{7.5_{\pm 0.26}}

\mathbf{4.2_{\pm 0.08}}

\mathbf{7.5_{\pm 0.74}}

10.3_{\pm 1.93}

\mathbf{11.7_{\pm 0.83}}

\mathbf{5.3_{\pm 0.55}}

\mathbf{9.0_{\pm 0.74}}

\mathbf{6.7_{\pm 0.97}}

Appendix

TABLE VII : Evaluation of learned pose estimator on real-world data under nominal and adversarial lighting. Our method is compared against three baselines: Standard Tiled , Randomized Tiled , and Naive GS rendering. We report the translation error (in mm) and the rotation error (in degrees), averaged over 5 random seeds.

Objects

Cube

3D Printed Toy

Rubber Duck

Tablet Bottle

Globe

Mean

Method

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Trans Error

Rot Error

Nominal Conditions

w/o Random Noise

10.8_{\pm 3.05}

6.8_{\pm 2.27}

12.5_{\pm 0.91}

10.0_{\pm 2.63}

7.7_{\pm 0.44}

4.7_{\pm 0.29}

8.3_{\pm 0.62}

12.6_{\pm 2.20}

14.0_{\pm 1.16}

5.5_{\pm 0.59}

10.7_{\pm 1.55}

7.9_{\pm 1.86}

w/o Spatial Clustering

\mathbf{8.1_{\pm 1.23}}

5.6_{\pm 3.46}

11.7_{\pm 0.93}

10.9_{\pm 1.48}

\mathbf{7.2_{\pm 0.28}}

5.8_{\pm 0.24}

10.7_{\pm 0.71}

26.7_{\pm 3.23}

13.1_{\pm 2.36}

6.6_{\pm 0.86}

10.2_{\pm 1.31}

11.1_{\pm 2.25}

w/o Color Clustering

\mathbf{8.1_{\pm 0.62}}

5.4_{\pm 0.54}

12.2_{\pm 1.78}

11.7_{\pm 1.61}

7.6_{\pm 0.65}

4.7_{\pm 0.16}

14.0_{\pm 0.71}

33.8_{\pm 2.11}

13.7_{\pm 1.81}

6.5_{\pm 0.98}

11.1_{\pm 1.25}

12.4_{\pm 1.29}

w/o Global Shift

8.3_{\pm 0.60}

11.0_{\pm 4.64}

\mathbf{9.9_{\pm 1.10}}

9.5_{\pm 1.69}

8.1_{\pm 0.34}

6.6_{\pm 0.53}

29.5_{\pm 3.50}

67.9_{\pm 1.83}

11.9_{\pm 0.43}

12.4_{\pm 1.10}

13.5_{\pm 1.68}

21.5_{\pm 2.42}

Appendix

TABLE VIII : Ablation study of pre-rasterization augmentations under nominal and adversarial conditions for pose estimation. We compare our method with models trained with specific augmentation groups removed. We report the translation error (in mm) and the rotation error (in degrees), averaged over 5 random seeds.

Term	Weight	Equation	Description
Task Rewards
Orientation Tracking	1.0	$(d(\theta)+\epsilon)^{-1}$	$d(\theta)$ : orientation error, $\epsilon=0.1$ .
Success Bonus	250.0	$\mathbb{I}(d(\theta)\leq\epsilon_{\text{success}})$	Bonus when $d(\theta)\leq\epsilon_{\text{success}}=0.1$ rad.
Termination Penalties
Object Dropped	$-10.0$	$\mathbb{I}(\text{drop})$	Penalty when the object falls.
Curriculum Penalties (Weights are gradually increased during training)

Appendix

TABLE IX : Reward Terms

Term	Dim.	Description
Proprioceptive Observations $\boldsymbol{\mathcal{O}_{\text{prop}}}$
Joint Positions	16	Measured robot joint angles.
Action History	64	Joint position commands from the last four time steps.
Goal Orientation	4	Target orientation (quaternion).
Palm Link Position	3	Position of the palm link (constant)
Remaining Time	1	Normalized time remaining in the episode.

Appendix

TABLE X : Observation Space

Parameter	Type	Distribution	Range / Details
Startup Randomization
Robot Link Mass	Scaling	Log-Uniform	$\times[0.75,1.5]$
Robot Link Friction	Absolute	Uniform	Static/Dynamic: $[0.0,0.3]$
Fingertip Friction	Absolute	Uniform	Static/Dynamic: $[0.3,0.8]$
Robot Restitution	Absolute	Uniform	$[0.0,0.4]$
Object Scale	Scaling	Uniform	$\times[0.8,1.2]$ (per axis)

Appendix

TABLE XI : Domain Randomization Parameters

Parameter	Value
Policy Architecture
Actor Hidden Layers	$[1024,1024,1024,512]$
Critic Hidden Layers	$[1024,1024,1024,512]$
Exte. Encoder Hidden Layers	$[64,64]$
Exte. Encoder Latent Dimension	$24$
Priv. Encoder Hidden Layers	$[256,256]$

Appendix

TABLE XII : Teacher Policy Architecture and PPO Training Configuration

Parameter	Value
Student Architecture
Actor MLP	$[1024,1024,512,512]$
Exteroceptive MLP	$[256,256]$
Exteroceptive Latent Dim	$64$
Privileged Latent Dim	$256$
Activation Function	ELU

Appendix

TABLE XIII : Student Policy and Distillation Hyperparameters

Noise Component	Distribution / Type	Parameters / Range
Object Position
Temporal Downsampling	Discrete Sampling ( $k$ )	Update period: $1$ – $3$ steps
Stochastic Jitter	Bernoulli Delay ( $p$ )	Delay probability: $0.0$ – $0.1$
Tracking Failure	Random Replacement	Failure probability: $0.0$ – $0.3$
Biased Noise	Additive Uniform	Noise: $\mathcal{U}[-12,12]$ mm Bias: $\mathcal{U}[-12,12]$ mm
Object Orientation

Appendix

TABLE XIV : Perception Noise Generator Parameters

Augmentation	Probability	Range
Photometric Augmentations
Color Jitter	$0.2$	$[0.8,1.2]$
Hue Shift	$0.2$	$[-0.2,0.2]$
Brightness Scaling	$0.5$	$[0.5,1.5]$
Contrast Scaling	$0.5$	$[0.5,1.5]$
Gamma	$0.5$	$[0.5,1.5]$

Appendix

TABLE XV : Image Augmentation Operators and Parameters

Explore similar work

Sep 17, 2026

AnyViewDex: View-Invariant Dexterous Manipulation from RGB Observations

Visuomotor policies for multi-fingered dexterous manipulation are highly sensitive to camera viewpoint shifts. To achieve view invariance, recent methods increasingly rely on explicit 3D modalities like RGB-D or point clouds, which can introduce hardware dependencies, calibration requirements, and vulnerability to sensor noise during real-world deployment. In this work, we show that view-invariant control can be achieved without explicit test-time 3D sensing by encoding geometric knowledge into the visual representation during simulation. We present AnyViewDex, an asymmetric training pipeline that combines multi-view contrastive alignment with privileged 3D geometric supervision. By regressing absolute 3D object coordinates during simulated training, this auxiliary objective provides a geometric grounding signal that mitigates the spatial collapse of the globally pooled contrastive embedding. At deployment, the policy operates zero-shot using only uncalibrated monocular RGB and proprioception. We validate this approach across both reinforcement learning and student-teacher distillation. In hardware evaluation on an xArm7 with a 16-DoF LEAP Hand, AnyViewDex reaches 76.7% grasping success across eight unseen objects and six uncalibrated viewpoints (480 trials; 2,400 across all ablation conditions), indicating that geometrically grounded monocular policies transfer zero-shot without test-time depth. Project Page: https://anyviewdex.github.io/

Soham Patil, Om Sanjay Gunjal, Sourabh Bhosale +3

Robotics Research Center, IIIT-H, India · Department of Electronics Engineering, Veermata Jijabai Technological Institute Mumbai, India · Department of Electrical Engineering and Computer Sciences, Indian Institute of Science Education and Research Bhopal, India

Robotic Grasping Zero-Shot Robotic Manipulation Visuomotor Policy Learning Robotic Manipulation

Ask PDF

Oct 10, 2026

DexCast: Learning Real-World Dexterous Control from Human Videos

Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.

Zihan Gao, Yuhong Zhang, Mengzhe Wang +7

Tsinghua University · TsingTec · Fudan University +3

Robotic Manipulation Robot Imitation Learning Motion Representation Learning

Ask PDF

Mar 22, 2026

Geometrically Plausible Object Pose Refinement using Differentiable Simulation

State-of-the-art object pose estimation methods are prone to generating geometrically infeasible pose hypotheses. This problem is prevalent in dexterous manipulation, where estimated poses often intersect with the robotic hand or are not lying on a support surface. We propose a multi-modal pose refinement approach that combines differentiable physics simulation, differentiable rendering and visuo-tactile sensing to optimize object poses for both spatial accuracy and physical consistency. Simulated experiments show that our approach reduces the intersection volume error between the object and robotic hand by 73% when the initial estimate is accurate and by over 87% under high initial uncertainty, significantly outperforming standard ICP-based baselines. Furthermore, the improvement in geometric plausibility is accompanied by a concurrent reduction in translation and orientation errors. Achieving pose estimation that is grounded in physical reality while remaining faithful to multi-modal sensor inputs is a critical step toward robust in-hand manipulation.

Anil Zeybek, Rhys Newbury, Snehal Dikhale +3

Technical University of Munich, Germany · Physical AI, Australia · Monash University, Australia +1

Object Pose Estimation Robotic Manipulation

Ask PDF