GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation

Arjun Bhardwaj; Mayank Mittal; Maximum Wilder-Smith; Vaishakh Patil; Marco Hutter

cs.RONov 1, 2026

GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation

Authors: Arjun Bhardwaj, Mayank Mittal, Maximum Wilder-Smith, Vaishakh Patil, Marco Hutter

Organizations: ETH Zurich · ETH Zurich & NVIDIA

Abstract

Dexterous in-hand manipulation is commonly formulated around explicit object pose estimation, yet reliable pose tracking is challenging under severe hand-object occlusions and often requires object-specific supervision. This dependence creates a bottleneck between perception and control that can limit robustness and scalability. We present an end-to-end visuomotor framework for dexterous in-hand reorientation that operates on RGB observations and proprioception without intermediate pose estimation. Our approach jointly learns visual representations and control policies through behavior distillation from privileged state teachers and auxiliary reconstruction objectives. Furthermore, we introduce image-goal-conditioned reorientation, allowing desired object configurations to be specified directly through images rather than canonical pose representations. We evaluate our approach across a diverse set of object geometries and instances in simulation, where it consistently outperforms pose-based methods and policies built on pretrained visual encoders. Finally, we demonstrate direct sim-to-real transfer of a single image-goal-conditioned policy across five real-world objects, showing that accurate dexterous reorientation can be achieved without explicit pose estimation. For videos and code release, please refer to the project website: https://rffr.leggedrobotics.com/works/goaldex/

Figures & tables

Figure 2: Training architecture for image-goal-conditioned visuomotor distillation. Current RGB observations and goal images are encoded into visual representations and fused with proprioceptive inputs. A recurrent policy integrates the multimodal features over time to predict joint position targets, while auxiliary reconstruction and continuous contrastive objectives shape the latent representation using privileged simulation supervision.

Task Performance

Privileged-State Reconstruction Error

Consecutive Successes

\uparrow

Time to Success (s)

\downarrow

Success Prediction F1 Score

\uparrow

Prediction Failure Rate (%)

\downarrow

Object Position (mm)

\downarrow

Object Orientation (rad)

\downarrow

Relative Orientation (rad)

\downarrow

Object Scale (%)

\downarrow

Object Mass (%)

\downarrow

Proprioception Only Baseline

✓

\times

\times

\times

✓

\times

1.05_{\pm 0.00}

8.31_{\pm 0.00}

0.016_{\pm 0.000}

99.18_{\pm 0.00}

18.83_{\pm 0.12}

2.220_{\pm 0.017}

1.177_{\pm 0.007}

10.6_{\pm 0.1}

33.2_{\pm 0.7}

Pose-Based Baselines (extends [ 2 ] to multiple objects)

✓

\times

\times

✓

\times

8.64_{\pm 10.17}

7.96_{\pm 0.53}

0.373_{\pm 0.127}

44.77_{\pm 2.77}

5.40_{\pm 0.36}

1.169_{\pm 0.041}

0.517_{\pm 0.054}

8.6_{\pm 0.4}

24.8_{\pm 1.3}

✓

\times

✓

\times

11.23_{\pm 9.65}

7.73_{\pm 0.74}

0.439_{\pm 0.111}

43.76_{\pm 3.32}

5.43_{\pm 0.06}

1.654_{\pm 0.255}

0.614_{\pm 0.036}

8.9_{\pm 0.4}

\underline{23.3_{\pm 1.1}}

Table 1: Effect of sensory inputs and goal representations. JP: joint proprioception; P: predicted object pose; C: object class encoding; I: current RGB image; GP: pose-space goal; GI: image-space goal.

	Task Performance				Privileged-State Reconstruction Error
Architecture	Consecutive Successes $\uparrow$	Time to Success (s) $\downarrow$	Success Prediction F1 Score $\uparrow$	Prediction Failure Rate (%) $\downarrow$	Object Position (mm) $\downarrow$	Object Orientation (rad) $\downarrow$	Relative Orientation (rad) $\downarrow$	Object Scale (%) $\downarrow$	Object Mass (%) $\downarrow$
Frozen ImageNet	$0.02_{\pm 0.00}$	$8.18_{\pm 0.49}$	$0.029_{\pm 0.003}$	$98.65_{\pm 0.18}$	$11.70_{\pm 0.10}$	$1.238_{\pm 0.022}$	$1.179_{\pm 0.012}$	$9.8_{\pm 0.1}$	$30.6_{\pm 3.3}$
Frozen DinoV2	$0.00_{\pm 0.00}$	-	$0.000_{\pm 0.000}$	$100.00_{\pm 0.00}$	$23.20_{\pm 0.17}$	$2.229_{\pm 0.015}$	$1.805_{\pm 0.009}$	$10.3_{\pm 0.2}$	$39.0_{\pm 3.0}$
Frozen DepthAnythingV2	$0.12_{\pm 0.00}$	$8.46_{\pm 0.00}$	$0.036_{\pm 0.000}$	$96.72_{\pm 0.00}$	$12.60_{\pm 0.10}$	$1.309_{\pm 0.022}$	$1.335_{\pm 0.017}$	$9.4_{\pm 0.6}$	$32.0_{\pm 1.5}$
Pose-only Reconstruction	$\underline{66.03_{\pm 9.34}}$	$5.55_{\pm 0.21}$	$0.912_{\pm 0.002}$	$\underline{9.70_{\pm 0.70}}$	$4.57_{\pm 0.12}$	$0.195_{\pm 0.007}$	$0.149_{\pm 0.003}$	-	-
No Memory	$38.64_{\pm 1.89}$	$\underline{5.51_{\pm 0.28}}$	$0.872_{\pm 0.006}$	$14.50_{\pm 0.22}$	$5.27_{\pm 0.06}$	$0.224_{\pm 0.004}$	$0.171_{\pm 0.005}$	$8.2_{\pm 0.1}$	$24.9_{\pm 1.3}$

Table 2: Comparison of visual encoding, reconstruction objectives, and temporal memory variants.

	Task Performance				Privileged-State Reconstruction Error
Alignment Strategy	Consecutive Successes $\uparrow$	Time to Success (s) $\downarrow$	Success Prediction F1 Score $\uparrow$	Prediction Failure Rate (%) $\downarrow$	Object Position (mm) $\downarrow$	Object Orientation (rad) $\downarrow$	Relative Orientation (rad) $\downarrow$	Object Scale (%) $\downarrow$	Object Mass (%) $\downarrow$
Separate	$56.99_{\pm 1.89}$	$\underline{4.50_{\pm 1.12}}$	$0.913_{\pm 0.004}$	$10.50_{\pm 0.61}$	$4.57_{\pm 0.06}$	$0.171_{\pm 0.004}$	$0.154_{\pm 0.004}$	$6.0_{\pm 0.2}$	$25.1_{\pm 1.6}$
EMA Updates	$\underline{67.95_{\pm 6.80}}$	$5.65_{\pm 0.40}$	$0.857_{\pm 0.004}$	$10.30_{\pm 0.58}$	$4.67_{\pm 0.12}$	$0.196_{\pm 0.001}$	$0.208_{\pm 0.003}$	$7.4_{\pm 0.1}$	$\underline{24.7_{\pm 1.8}}$
Shared	$67.18_{\pm 4.43}$	$4.69_{\pm 0.42}$	$\mathbf{0.936_{\pm 0.002}}$	$9.32_{\pm 0.12}$	$\underline{4.20_{\pm 0.00}}$	$\mathbf{0.150_{\pm 0.003}}$	$\mathbf{0.133_{\pm 0.002}}$	$\mathbf{2.7_{\pm 0.2}}$	$25.5_{\pm 2.0}$
InfoNCE	$63.01_{\pm 15.51}$	$4.65_{\pm 0.37}$	$\underline{0.934_{\pm 0.001}}$	$\mathbf{8.11_{\pm 0.54}}$	$\mathbf{3.97_{\pm 0.06}}$	$0.162_{\pm 0.003}$	$\mathbf{0.133_{\pm 0.002}}$	$5.3_{\pm 0.1}$	$\mathbf{22.6_{\pm 0.6}}$
Ours	$\mathbf{77.30_{\pm 3.25}}$	$\mathbf{4.35_{\pm 0.24}}$	$0.921_{\pm 0.000}$	$\underline{8.80_{\pm 0.15}}$	$4.50_{\pm 0.00}$	$\underline{0.151_{\pm 0.004}}$	$0.136_{\pm 0.003}$	$\underline{3.5_{\pm 0.1}}$	$24.7_{\pm 2.6}$

Table 3: Effect of object–goal alignment on task performance and state reconstruction.

	Task Performance				Privileged-State Reconstruction Error
Method	Consecutive Successes $\uparrow$	Time to Success (s) $\downarrow$	Success Prediction F1 Score $\uparrow$	Prediction Failure Rate (%) $\downarrow$	Object Position (mm) $\downarrow$	Object Orientation (rad) $\downarrow$	Relative Orientation (rad) $\downarrow$	Object Scale (%) $\downarrow$	Object Mass (%) $\downarrow$
Privileged Teacher	$\mathbf{110.84_{\pm 3.98}}$	$\mathbf{2.42_{\pm 0.37}}$	-	-	-	-	-	-	-
Image-Goal Conditioned	$10.11_{\pm 2.06}$	$6.74_{\pm 0.05}$	$0.636_{\pm 0.003}$	$29.31_{\pm 2.17}$	$7.13_{\pm 0.06}$	$0.399_{\pm 0.002}$	$0.393_{\pm 0.003}$	$9.8_{\pm 0.4}$	$31.8_{\pm 2.2}$
Image-Goal Conditioned 2x	$41.42_{\pm 0.00}$	$5.30_{\pm 0.00}$	$\mathbf{0.882_{\pm 0.000}}$	$\mathbf{9.61_{\pm 0.00}}$	$\mathbf{4.60_{\pm 0.10}}$	$\mathbf{0.154_{\pm 0.004}}$	$\mathbf{0.190_{\pm 0.008}}$	$\mathbf{5.1_{\pm 0.2}}$	$\mathbf{29.0_{\pm 2.8}}$

Table 4: Scaling to 25 objects in simulation.

2\times

denotes twice as many parallel training environments.

Figure 4: Language-guided reorientation via visual goal selection. Given a task prompt, a vision-language model selects task-relevant goal images from candidate object viewpoints, which are then executed by the low-level controller. Colored boxes show the task prompts, the middle row shows selected goal images, and the bottom row shows camera observations after successful reorientation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method

Consecutive Successes

\uparrow

Time to Success (s)

\downarrow

Success Prediction F1 Score

\uparrow

Prediction Failure Rate (%)

\downarrow

Oracle Pose Policies

Oracle Pose

✓

\times

\times

\times

\times

✓

\times

22.49_{\pm 6.77}

5.38_{\pm 0.83}

0.782_{\pm 0.000}

11.13_{\pm 0.00}

Oracle Pose + Class

✓

\times

\times

\times

✓

\times

43.77_{\pm 7.06}

4.43_{\pm 0.41}

\underline{0.869_{\pm 0.002}}

13.04_{\pm 1.19}

Oracle Pose + Scale

✓

\times

✓

\times

\times

✓

\times

37.05_{\pm 0.00}

4.43_{\pm 0.00}

0.868_{\pm 0.000}

11.63_{\pm 0.00}

Oracle Pose + Class + Scale

✓

\times

\times

✓

\times

98.64_{\pm 0.00}

4.08_{\pm 0.00}

0.952_{\pm 0.000}

\underline{3.13_{\pm 0.00}}

Privileged Policies

Appendix

Table 6: Comparison with oracle-pose and privileged policies. JP: joint proprioception; OP: oracle object pose; C: object class encoding; S: object size; F: teacher’s privileged observations; I: current RGB image; GP: pose-space goal; GI: image-space goal.

Panel of Figure 5 — Figure 5: Per-object performance of the privileged teacher and our image-goal conditioned policy. Bars are per-object means; dashed lines mark each policy’s mean over all objects. Objects are ordered by our policy’s performance (consecutive successes: highest first; time to success: fastest first).

Model	Rollouts	Pos. (mm) $\downarrow$	Ori. (rad) $\downarrow$	Goal Angle (rad) $\downarrow$	Success F1 $\uparrow$
Pose estimator	Teacher	$6.76_{\pm 0.09}$	$0.245_{\pm 0.015}$	$0.153_{\pm 0.016}$	$0.846_{\pm 0.027}$
Pose estimator	Student	$6.82_{\pm 0.18}$	$0.250_{\pm 0.022}$	$0.147_{\pm 0.022}$	–
GoalDex recon. head	Teacher	$4.50_{\pm 0.00}$	$0.151_{\pm 0.004}$	$0.136_{\pm 0.003}$	$0.921_{\pm 0.000}$

Appendix

Table 7: Standalone evaluation of the pose estimator used to train pose-based baselines.

Observation Term	Dim.	Description
Proprioceptive Observations ( $\boldsymbol{\mathcal{O}_{\text{prop}}}$ )
Joint Positions	16	Measured robot joint angles.
Action History	64	Joint position commands from the last four time steps.
Remaining Time	1	Time remaining in the episode before timeout (seconds).
Group Total	81
Goal Observations ( $\boldsymbol{\mathcal{O}_{\text{goal}}}$ )

Appendix

Table 8: Observation Space of the Privileged Teacher Policy

Network Architecture
Parameter	Value
Actor MLP Dims	[1024, 1024, 1024, 1024]
Critic MLP Dims	[1024, 1024, 1024, 1024]
Goal Observation Encoder Dims	[256, 256]
Goal Observation Latent Dim	64
State Observation MLP Dims	[256, 256]

Appendix

Table 9: Training Parameters for the Privileged Teacher Policy

Observation Term	Dim.	Description
Proprioceptive Observations ( $\boldsymbol{\mathcal{O}_{\text{prop}}}$ )
Joint Positions	16	Measured robot joint angles.
Action History	64	Joint position commands from the last four time steps.
Remaining Time	1	Time remaining in the episode before timeout (seconds).
Group Total	81
Image Observations ( $\boldsymbol{\mathcal{O}_{\text{image}}}$ )

Appendix

Table 10: Observation Space of the Student Policy

Network Architecture
Parameter	Value
Actor MLP Dims	[1024, 1024, 512, 512]
Reconstruction MLP Dims	[1024, 1024, 512, 512]
Proprioceptive Encoder Dims	[1024, 1024]
Proprioceptive Latent Dim	512
Visual Encoder Backbone	ResNet-34

Appendix

Table 11: Architecture and Training Parameters for the Student Policy

Network Architecture
Parameter	Value
Input	Object-masked RGB, $120{\times}120$
Visual Encoder Backbone	ResNet-34
Backbone Input Resolution	$224{\times}224$
Head MLP Dims	[1024, 512]
Activation Function	ReLU

Appendix

Table 12: Architecture and Training Parameters for the Pose Estimator

Dexterous Manipulation

Dexterous manipulation studies how robots use multi-fingered hands to grasp, reorient, and interact with objects through contact.

See all 144 papers

Oct 10, 2026

DexCast: Learning Real-World Dexterous Control from Human Videos

Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.

Zihan Gao, Yuhong Zhang, Mengzhe Wang +7

Tsinghua University · TsingTec · Shenzhen University +2

Robotic Manipulation Robot Imitation Learning Motion Representation Learning

Ask PDF

Apr 13, 2026

ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation

In-hand object reorientation requires precise estimation of the object pose to handle complex task dynamics. While RGB sensing offers rich semantic cues for pose tracking, existing solutions rely on multi-camera setups or costly ray tracing. We present a sim-to-real framework for monocular RGB in-hand reorientation that integrates 3D Gaussian Splatting (3DGS) to bridge the visual sim-to-real gap. Our key insight is performing domain randomization in the Gaussian representation space: by applying physically consistent, pre-rendering augmentations to 3D Gaussians, we generate photorealistic, randomized visual data for object pose estimation. The manipulation policy is trained using curriculum-based reinforcement learning with teacher-student distillation, enabling efficient learning of complex behaviors. Importantly, both perception and control models can be trained independently on consumer-grade hardware, eliminating the need for large compute clusters. Experiments show that the pose estimator trained with 3DGS data outperforms those trained using conventional rendering data in challenging visual environments. We validate the system on a physical multi-fingered hand equipped with an RGB camera, demonstrating robust reorientation of five diverse objects even under challenging lighting conditions. Our results highlight Gaussian splatting as a practical path for RGB-only dexterous manipulation. For videos of the hardware deployments and additional supplementary materials, please refer to the project website: https://rffr.leggedrobotics.com/works/viserdex/

Arjun Bhardwaj, Maximum Wilder-Smith, Mayank Mittal +2

ETH Zurich · NVIDIA

Robotic Manipulation Dexterous Manipulation Sim-to-Real Transfer Object Pose Estimation Domain Randomization 3D Gaussian Splatting

Ask PDF

Aug 1, 2026

DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation

Dexterous object rotation is a sequential contact problem: each support, release, and re-contact decision must both produce the desired object motion, and prepare the hand configuration for continued rotation. Existing reinforcement learning methods discover such movement patterns through trial and error on specific robotic hand embodiments, without explicitly accounting for how each contact transition affects the hand's ability to sustain object rotation in subsequent steps. We introduce DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution. This prior captures how successful human contact transitions reshape the object-rotation directions available to the hand. DexMani then learns this manipulability evolution and uses it to guide downstream reinforcement learning, enabling rotation skills to be acquired across robot embodiments with distinct kinematics and active-contact configurations. Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates in every evaluated setting for both seen and unseen objects. DexMani reaches an average success rate of 57.5% on LEAP Hand, outperforming other baselines and producing smoother rotatory motions. Project site: https://dexmani.github.io

Xiaoyang Chen, Shengcheng Luo, Haoran Guo +4

Shanghai Jiao Tong University · ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) +1

Dexterous Manipulation Robot Skill Learning In-Hand Manipulation Cross-Embodiment Robot Learning Robotic RL Robot Imitation Learning

Ask PDF

Abstract

Dexterous Manipulation

Explore similar work

DexCast: Learning Real-World Dexterous Control from Human Videos

ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation

DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation