State-of-the-art object pose estimation methods are prone to generating geometrically infeasible pose hypotheses. This problem is prevalent in dexterous manipulation, where estimated poses often intersect with the robotic hand or are not lying on a support surface. We propose a multi-modal pose refinement approach that combines differentiable physics simulation, differentiable rendering and visuo-tactile sensing to optimize object poses for both spatial accuracy and physical consistency. Simulated experiments show that our approach reduces the intersection volume error between the object and robotic hand by 73% when the initial estimate is accurate and by over 87% under high initial uncertainty, significantly outperforming standard ICP-based baselines. Furthermore, the improvement in geometric plausibility is accompanied by a concurrent reduction in translation and orientation errors. Achieving pose estimation that is grounded in physical reality while remaining faithful to multi-modal sensor inputs is a critical step toward robust in-hand manipulation.
Figures & tables
Figure 1: Given an initial estimate, our approach iteratively updates the object pose using differentiable simulation, the relative positioning of the estimated pose and the robotic hand as well as data from vision and tactile sensors.
Figure 2: System Diagram. RGB-D and Tactile sensor inputs, and robot configuration is used to define 4 gradients. At each iteration, one of the gradients is selected according to a heuristic.
Figure 3: We simulate a serial robotic arm, with an allegro robotic hand fitted with tactile sensors.
Figure 4: The Contact Area (CA) metric is shown in red between the fingers and the Mustard Bottle object. 100k points are sampled on the surface of the object and contact is determined if the closest distance between the point and the robot is less than a threshold ϵ .
Method
PE ↓
OE ↓
CA
IV
∣ΔCA∣↓
∣ΔIV∣↓
Scale/Unit
cm
deg
cm 2
cm 3
cm 2
cm 3
Ground Truth
-
-
36.28
0.17
0
0
Initial Pose
0.65
6.98
39.05
2.36
2.77
2.19
ICP
3.46
43.01
65.26
28.73
28.98
28.56
ICP w/ Checkpointing
1.30
13.77
48.31
8.71
12.03
8.54
Ours
0.62
6.69
41.73
0.76
5.45
0.59
Table I : Pose refinement results on 9 YCB objects. Best results are indicated in bold .
Method
PE ↓
OE ↓
CA
IV
∣ΔCA∣↓
∣ΔIV∣↓
Scale/Unit
cm
deg
cm 2
cm 3
cm 2
cm 3
Ground Truth
-
-
36.28
0.17
0
0
Initial Pose
0.65
6.98
39.05
2.36
2.77
2.19
Mesh-to-Depth
2.54
8.72
54.52
14.31
18.24
14.14
Mesh-to-Tactile
3.24
9.51
90.63
32.68
54.35
32.51
Diff. Rendering
0.85
6.65
43.99
3.60
7.71
3.43
Table II : Comparison to individual gradients. Pose refinement results on 9 YCB objects. Best results are indicated in bold .
Method
PE ↓
OE ↓
CA
IV
∣ΔCA∣↓
∣ΔIV∣↓
Scale/Unit
cm
deg
cm 2
cm 3
cm 2
cm 3
Ground Truth
-
-
36.28
0.17
0
0
Initial Pose
2.21
15.25
61.34
21.57
25.06
21.40
ICP
3.65
42.55
61.08
26.10
24.80
25.93
ICP w/ Checkpointing
2.53
23.29
55.87
19.57
19.59
19.40
Ours
1.45
13.14
42.65
2.8
6.37
2.63
Table III : Refinement results when noise is added to the initial pose estimate (9 YCB objects). Best results are indicated in bold .
No Noise
Artificial Noise
# Iterations
PE ↓
OE ↓
CA
IV ↓
PE ↓
OE ↓
CA
IV ↓
Scale/Unit
cm
deg
cm 2
cm 3
cm
deg
cm 2
cm 3
Initial Pose
0.65
6.44
47.10
2.00
2.31
15.15
73.06
27.55
Ours (100 steps)
0.80
5.88
40.65
0.12
1.75
13.74
47.85
1.80
Ours (200 steps)
0.92
5.97
40.21
0.08
1.73
13.17
44.23
0.95
Ours (400 steps)
1.06
6.50
40.30
0.06
1.73
12.42
42.13
0.54
Table IV : Analysis on the number of refinement steps. Only the Mustard object was used. Regularization and checkpointing was not active. Best results are indicated in bold .