GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation

Arjun Bhardwaj; Mayank Mittal; Maximum Wilder-Smith; Vaishakh Patil; Marco Hutter

cs.RONov 1, 2026

GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation

Authors: Arjun Bhardwaj, Mayank Mittal, Maximum Wilder-Smith, Vaishakh Patil, Marco Hutter

Organizations: ETH Zurich · ETH Zurich & NVIDIA

Abstract

Dexterous in-hand manipulation is commonly formulated around explicit object pose estimation, yet reliable pose tracking is challenging under severe hand-object occlusions and often requires object-specific supervision. This dependence creates a bottleneck between perception and control that can limit robustness and scalability. We present an end-to-end visuomotor framework for dexterous in-hand reorientation that operates on RGB observations and proprioception without intermediate pose estimation. Our approach jointly learns visual representations and control policies through behavior distillation from privileged state teachers and auxiliary reconstruction objectives. Furthermore, we introduce image-goal-conditioned reorientation, allowing desired object configurations to be specified directly through images rather than canonical pose representations. We evaluate our approach across a diverse set of object geometries and instances in simulation, where it consistently outperforms pose-based methods and policies built on pretrained visual encoders. Finally, we demonstrate direct sim-to-real transfer of a single image-goal-conditioned policy across five real-world objects, showing that accurate dexterous reorientation can be achieved without explicit pose estimation. For videos and code release, please refer to the project website: https://rffr.leggedrobotics.com/works/goaldex/

Figures & tables

Figure 2: Training architecture for image-goal-conditioned visuomotor distillation. Current RGB observations and goal images are encoded into visual representations and fused with proprioceptive inputs. A recurrent policy integrates the multimodal features over time to predict joint position targets, while auxiliary reconstruction and continuous contrastive objectives shape the latent representation using privileged simulation supervision.

Task Performance

Privileged-State Reconstruction Error

Consecutive Successes

\uparrow

Time to Success (s)

\downarrow

Success Prediction F1 Score

\uparrow

Prediction Failure Rate (%)

\downarrow

Object Position (mm)

\downarrow

Object Orientation (rad)

\downarrow

Relative Orientation (rad)

\downarrow

Object Scale (%)

\downarrow

Object Mass (%)

\downarrow

Proprioception Only Baseline

✓

\times

\times

\times

✓

\times

1.05_{\pm 0.00}

8.31_{\pm 0.00}

0.016_{\pm 0.000}

99.18_{\pm 0.00}

18.83_{\pm 0.12}

2.220_{\pm 0.017}

1.177_{\pm 0.007}

10.6_{\pm 0.1}

33.2_{\pm 0.7}

Pose-Based Baselines (extends [ 2 ] to multiple objects)

✓

\times

\times

✓

\times

8.64_{\pm 10.17}

7.96_{\pm 0.53}

0.373_{\pm 0.127}

44.77_{\pm 2.77}

5.40_{\pm 0.36}

1.169_{\pm 0.041}

0.517_{\pm 0.054}

8.6_{\pm 0.4}

24.8_{\pm 1.3}

✓

\times

✓

\times

11.23_{\pm 9.65}

7.73_{\pm 0.74}

0.439_{\pm 0.111}

43.76_{\pm 3.32}

5.43_{\pm 0.06}

1.654_{\pm 0.255}

0.614_{\pm 0.036}

8.9_{\pm 0.4}

\underline{23.3_{\pm 1.1}}

Table 1: Effect of sensory inputs and goal representations. JP: joint proprioception; P: predicted object pose; C: object class encoding; I: current RGB image; GP: pose-space goal; GI: image-space goal.

	Task Performance				Privileged-State Reconstruction Error
Architecture	Consecutive Successes $\uparrow$	Time to Success (s) $\downarrow$	Success Prediction F1 Score $\uparrow$	Prediction Failure Rate (%) $\downarrow$	Object Position (mm) $\downarrow$	Object Orientation (rad) $\downarrow$	Relative Orientation (rad) $\downarrow$	Object Scale (%) $\downarrow$	Object Mass (%) $\downarrow$
Frozen ImageNet	$0.02_{\pm 0.00}$	$8.18_{\pm 0.49}$	$0.029_{\pm 0.003}$	$98.65_{\pm 0.18}$	$11.70_{\pm 0.10}$	$1.238_{\pm 0.022}$	$1.179_{\pm 0.012}$	$9.8_{\pm 0.1}$	$30.6_{\pm 3.3}$
Frozen DinoV2	$0.00_{\pm 0.00}$	-	$0.000_{\pm 0.000}$	$100.00_{\pm 0.00}$	$23.20_{\pm 0.17}$	$2.229_{\pm 0.015}$	$1.805_{\pm 0.009}$	$10.3_{\pm 0.2}$	$39.0_{\pm 3.0}$
Frozen DepthAnythingV2	$0.12_{\pm 0.00}$	$8.46_{\pm 0.00}$	$0.036_{\pm 0.000}$	$96.72_{\pm 0.00}$	$12.60_{\pm 0.10}$	$1.309_{\pm 0.022}$	$1.335_{\pm 0.017}$	$9.4_{\pm 0.6}$	$32.0_{\pm 1.5}$
Pose-only Reconstruction	$\underline{66.03_{\pm 9.34}}$	$5.55_{\pm 0.21}$	$0.912_{\pm 0.002}$	$\underline{9.70_{\pm 0.70}}$	$4.57_{\pm 0.12}$	$0.195_{\pm 0.007}$	$0.149_{\pm 0.003}$	-	-
No Memory	$38.64_{\pm 1.89}$	$\underline{5.51_{\pm 0.28}}$	$0.872_{\pm 0.006}$	$14.50_{\pm 0.22}$	$5.27_{\pm 0.06}$	$0.224_{\pm 0.004}$	$0.171_{\pm 0.005}$	$8.2_{\pm 0.1}$	$24.9_{\pm 1.3}$

Table 2: Comparison of visual encoding, reconstruction objectives, and temporal memory variants.

	Task Performance				Privileged-State Reconstruction Error
Alignment Strategy	Consecutive Successes $\uparrow$	Time to Success (s) $\downarrow$	Success Prediction F1 Score $\uparrow$	Prediction Failure Rate (%) $\downarrow$	Object Position (mm) $\downarrow$	Object Orientation (rad) $\downarrow$	Relative Orientation (rad) $\downarrow$	Object Scale (%) $\downarrow$	Object Mass (%) $\downarrow$
Separate	$56.99_{\pm 1.89}$	$\underline{4.50_{\pm 1.12}}$	$0.913_{\pm 0.004}$	$10.50_{\pm 0.61}$	$4.57_{\pm 0.06}$	$0.171_{\pm 0.004}$	$0.154_{\pm 0.004}$	$6.0_{\pm 0.2}$	$25.1_{\pm 1.6}$
EMA Updates	$\underline{67.95_{\pm 6.80}}$	$5.65_{\pm 0.40}$	$0.857_{\pm 0.004}$	$10.30_{\pm 0.58}$	$4.67_{\pm 0.12}$	$0.196_{\pm 0.001}$	$0.208_{\pm 0.003}$	$7.4_{\pm 0.1}$	$\underline{24.7_{\pm 1.8}}$
Shared	$67.18_{\pm 4.43}$	$4.69_{\pm 0.42}$	$\mathbf{0.936_{\pm 0.002}}$	$9.32_{\pm 0.12}$	$\underline{4.20_{\pm 0.00}}$	$\mathbf{0.150_{\pm 0.003}}$	$\mathbf{0.133_{\pm 0.002}}$	$\mathbf{2.7_{\pm 0.2}}$	$25.5_{\pm 2.0}$
InfoNCE	$63.01_{\pm 15.51}$	$4.65_{\pm 0.37}$	$\underline{0.934_{\pm 0.001}}$	$\mathbf{8.11_{\pm 0.54}}$	$\mathbf{3.97_{\pm 0.06}}$	$0.162_{\pm 0.003}$	$\mathbf{0.133_{\pm 0.002}}$	$5.3_{\pm 0.1}$	$\mathbf{22.6_{\pm 0.6}}$
Ours	$\mathbf{77.30_{\pm 3.25}}$	$\mathbf{4.35_{\pm 0.24}}$	$0.921_{\pm 0.000}$	$\underline{8.80_{\pm 0.15}}$	$4.50_{\pm 0.00}$	$\underline{0.151_{\pm 0.004}}$	$0.136_{\pm 0.003}$	$\underline{3.5_{\pm 0.1}}$	$24.7_{\pm 2.6}$

Table 3: Effect of object–goal alignment on task performance and state reconstruction.

	Task Performance				Privileged-State Reconstruction Error
Method	Consecutive Successes $\uparrow$	Time to Success (s) $\downarrow$	Success Prediction F1 Score $\uparrow$	Prediction Failure Rate (%) $\downarrow$	Object Position (mm) $\downarrow$	Object Orientation (rad) $\downarrow$	Relative Orientation (rad) $\downarrow$	Object Scale (%) $\downarrow$	Object Mass (%) $\downarrow$
Privileged Teacher	$\mathbf{110.84_{\pm 3.98}}$	$\mathbf{2.42_{\pm 0.37}}$	-	-	-	-	-	-	-
Image-Goal Conditioned	$10.11_{\pm 2.06}$	$6.74_{\pm 0.05}$	$0.636_{\pm 0.003}$	$29.31_{\pm 2.17}$	$7.13_{\pm 0.06}$	$0.399_{\pm 0.002}$	$0.393_{\pm 0.003}$	$9.8_{\pm 0.4}$	$31.8_{\pm 2.2}$
Image-Goal Conditioned 2x	$41.42_{\pm 0.00}$	$5.30_{\pm 0.00}$	$\mathbf{0.882_{\pm 0.000}}$	$\mathbf{9.61_{\pm 0.00}}$	$\mathbf{4.60_{\pm 0.10}}$	$\mathbf{0.154_{\pm 0.004}}$	$\mathbf{0.190_{\pm 0.008}}$	$\mathbf{5.1_{\pm 0.2}}$	$\mathbf{29.0_{\pm 2.8}}$

Table 4: Scaling to 25 objects in simulation.

2\times

denotes twice as many parallel training environments.

Figure 4: Language-guided reorientation via visual goal selection. Given a task prompt, a vision-language model selects task-relevant goal images from candidate object viewpoints, which are then executed by the low-level controller. Colored boxes show the task prompts, the middle row shows selected goal images, and the bottom row shows camera observations after successful reorientation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method

Consecutive Successes

\uparrow

Time to Success (s)

\downarrow

Success Prediction F1 Score

\uparrow

Prediction Failure Rate (%)

\downarrow

Oracle Pose Policies

Oracle Pose

✓

\times

\times

\times

\times

✓

\times

22.49_{\pm 6.77}

5.38_{\pm 0.83}

0.782_{\pm 0.000}

11.13_{\pm 0.00}

Oracle Pose + Class

✓

\times

\times

\times

✓

\times

43.77_{\pm 7.06}

4.43_{\pm 0.41}

\underline{0.869_{\pm 0.002}}

13.04_{\pm 1.19}

Oracle Pose + Scale

✓

\times

✓

\times

\times

✓

\times

37.05_{\pm 0.00}

4.43_{\pm 0.00}

0.868_{\pm 0.000}

11.63_{\pm 0.00}

Oracle Pose + Class + Scale

✓

\times

\times

✓

\times

98.64_{\pm 0.00}

4.08_{\pm 0.00}

0.952_{\pm 0.000}

\underline{3.13_{\pm 0.00}}

Privileged Policies

Appendix

Table 6: Comparison with oracle-pose and privileged policies. JP: joint proprioception; OP: oracle object pose; C: object class encoding; S: object size; F: teacher’s privileged observations; I: current RGB image; GP: pose-space goal; GI: image-space goal.

Panel of Figure 5 — Figure 5: Per-object performance of the privileged teacher and our image-goal conditioned policy. Bars are per-object means; dashed lines mark each policy’s mean over all objects. Objects are ordered by our policy’s performance (consecutive successes: highest first; time to success: fastest first).

Model	Rollouts	Pos. (mm) $\downarrow$	Ori. (rad) $\downarrow$	Goal Angle (rad) $\downarrow$	Success F1 $\uparrow$
Pose estimator	Teacher	$6.76_{\pm 0.09}$	$0.245_{\pm 0.015}$	$0.153_{\pm 0.016}$	$0.846_{\pm 0.027}$
Pose estimator	Student	$6.82_{\pm 0.18}$	$0.250_{\pm 0.022}$	$0.147_{\pm 0.022}$	–
GoalDex recon. head	Teacher	$4.50_{\pm 0.00}$	$0.151_{\pm 0.004}$	$0.136_{\pm 0.003}$	$0.921_{\pm 0.000}$

Appendix

Table 7: Standalone evaluation of the pose estimator used to train pose-based baselines.

Observation Term	Dim.	Description
Proprioceptive Observations ( $\boldsymbol{\mathcal{O}_{\text{prop}}}$ )
Joint Positions	16	Measured robot joint angles.
Action History	64	Joint position commands from the last four time steps.
Remaining Time	1	Time remaining in the episode before timeout (seconds).
Group Total	81
Goal Observations ( $\boldsymbol{\mathcal{O}_{\text{goal}}}$ )

Appendix

Table 8: Observation Space of the Privileged Teacher Policy

Network Architecture
Parameter	Value
Actor MLP Dims	[1024, 1024, 1024, 1024]
Critic MLP Dims	[1024, 1024, 1024, 1024]
Goal Observation Encoder Dims	[256, 256]
Goal Observation Latent Dim	64
State Observation MLP Dims	[256, 256]

Appendix

Table 9: Training Parameters for the Privileged Teacher Policy

Observation Term	Dim.	Description
Proprioceptive Observations ( $\boldsymbol{\mathcal{O}_{\text{prop}}}$ )
Joint Positions	16	Measured robot joint angles.
Action History	64	Joint position commands from the last four time steps.
Remaining Time	1	Time remaining in the episode before timeout (seconds).
Group Total	81
Image Observations ( $\boldsymbol{\mathcal{O}_{\text{image}}}$ )

Appendix

Table 10: Observation Space of the Student Policy

Network Architecture
Parameter	Value
Actor MLP Dims	[1024, 1024, 512, 512]
Reconstruction MLP Dims	[1024, 1024, 512, 512]
Proprioceptive Encoder Dims	[1024, 1024]
Proprioceptive Latent Dim	512
Visual Encoder Backbone	ResNet-34

Appendix

Table 11: Architecture and Training Parameters for the Student Policy

Network Architecture
Parameter	Value
Input	Object-masked RGB, $120{\times}120$
Visual Encoder Backbone	ResNet-34
Backbone Input Resolution	$224{\times}224$
Head MLP Dims	[1024, 512]
Activation Function	ReLU

Appendix

Table 12: Architecture and Training Parameters for the Pose Estimator

Dexterous Manipulation

Dexterous manipulation studies how robots use multi-fingered hands to grasp, reorient, and interact with objects through contact.

See all 144 papers

DexCast: Learning Real-World Dexterous Control from Human Videos
Oct 10, 2026Zihan Gao, Yuhong Zhang, Mengzhe Wang +7Robotic Manipulation Robot Imitation Learning
Tsinghua University · TsingTec · Shenzhen University +2
ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation
Apr 13, 2026Arjun Bhardwaj, Maximum Wilder-Smith, Mayank Mittal +2Robotic Manipulation Dexterous Manipulation
ETH Zurich · NVIDIA
DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation
Aug 1, 2026Xiaoyang Chen, Shengcheng Luo, Haoran Guo +4Dexterous Manipulation Robot Skill Learning
Shanghai Jiao Tong University · ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) +1

Oct 10, 2026J/K move · Enter open · S save

DexCast: Learning Real-World Dexterous Control from Human Videos

Zihan Gao, Yuhong Zhang, Mengzhe Wang, Kaisheng Liu+6

Tsinghua University · TsingTec · Shenzhen University+2

Robotic Manipulation Robot Imitation Learning Motion Representation Learning

Open paper AskPDF arXiv

Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.

Abstract

Dexterous Manipulation

Explore similar work

DexCast: Learning Real-World Dexterous Control from Human Videos

ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation

DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation