Robust estimation of object poses in robotic manipulation is often addressed using foundational general estimators, that aim to handle diverse error sources naively within a single model. Still, they struggle due to environmental uncertainties, while requiring long inference times and heavy computation. In contrast, we propose a modular, uncertainty-aware framework that attributes pose estimation errors to specific error sources and applies targeted mitigation strategies only when necessary. Instantiated with Iterative Closest Point (ICP) as a simple and lightweight pose estimator, we leverage our framework for real-world robotic grasping tasks. By decomposing pose estimation into failure detection, error attribution, and targeted recovery, we significantly improve the robustness of ICP and achieve competitive performance compared to foundation models, while relying on a substantially simpler and faster pose estimator.
Figures & tables
Fig. 1: Overview of our framework. After the scene is recorded, ICP provides an initial pose estimate. A grasp success predictor detects possible grasp failure, which queries an error attribution system. Based on the detected error, one of the mitigation strategies will be selected. In this example, the failure is attributed to structural noise in the point cloud. A point cloud reconstruction module recovers the clean point cloud, and the pose estimate is corrected, leading to a successful grasp. Since ICP is robust to surface noise, reconstructing the rough shape is sufficient in this example.
Fig. 2: Overview of the deployed framework. (A) A point cloud sampled from the object mesh is aligned to the recorded scene point cloud via ICP. (B) An MLP predicts grasp success based on the calculated transformation estimate and alignment metrics. (C) In the case of predicted grasp failure, a PointBERT [ 34 ] classifier attributes the failure to a specific source, here shown for noise. (D) Based on the classification, targeted mitigation strategies - BO-ICP [ 27 ] , the TGV-Planner [ 31 ] , and a custom point cloud reconstruction model - are applied to recover the pose estimate. In the illustrated case, the clean point cloud is reconstructed from the noisy point cloud and used for ICP alignment, which corrects the pose estimate.
Fig. 3: Point cloud reconstruction module. A transformer based encoder creates tokens from point patches of the noisy point cloud, while a DGCNN [ 20 ] encodes point-wise features of the mesh. From the point-wise mesh features, a subset is selected via FPS. The tokens from the noisy point cloud are merged with the selected point-wise mesh features by calculating a weighted sum with learned weights. An encoder predicts displacement vectors for each patch center point of the noisy point cloud, which is propagated to the surrounding points.
Fig. 4: Synthetic scenes showing the error cases created using SAPIEN [ 35 ] . For bad initialization, the initial pose hypothesis is shown in red.
Fig. 5: Overview of the real-world scenes used for evaluation and training. For each object, we show one example of each error case.
Fig. 6: Confusion matrix of the failure detection MLP evaluated on 108 grasping tasks.
Model
Mean Acc.
True Bad Init. Pred. Occlusion
True Occlusion Pred. Bad Init.
PointBERT
99.47%
1.02%
0.61%
DGCNN [ 17 ]
85.20%
20.60%
23.00%
TABLE I: Accuracy of uncertainty attribution and confusion rates of bad initialization (Bad Init.) and occlusion compared to the results in [ 17 ] .
Model
Mean Acc.
Std. Dev.
PointBERT
83.83%
2.27%
DGCNN
71.49%
2.55%
PointNet
50.64%
3.40%
TABLE II: Accuracy of error attribution on real-world scenes.
Error Case
ICP success
ICP&Mitigation success
FP success
Bad Init.
0%
55%
85%
Noise
15%
80%
30%
Occlusion
10%
70 %
70%
TABLE III: Grasp success comparison of our framework against FP [ 11 ] for each error case.
State-of-the-art object pose estimation methods are prone to generating geometrically infeasible pose hypotheses. This problem is prevalent in dexterous manipulation, where estimated poses often intersect with the robotic hand or are not lying on a support surface. We propose a multi-modal pose refinement approach that combines differentiable physics simulation, differentiable rendering and visuo-tactile sensing to optimize object poses for both spatial accuracy and physical consistency. Simulated experiments show that our approach reduces the intersection volume error between the object and robotic hand by 73% when the initial estimate is accurate and by over 87% under high initial uncertainty, significantly outperforming standard ICP-based baselines. Furthermore, the improvement in geometric plausibility is accompanied by a concurrent reduction in translation and orientation errors. Achieving pose estimation that is grounded in physical reality while remaining faithful to multi-modal sensor inputs is a critical step toward robust in-hand manipulation.
Anil Zeybek, Rhys Newbury, Snehal Dikhale +3
Technical University of Munich, Germany · Physical AI, Australia · Monash University, Australia +1
3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based benchmark that evaluates 6D pose estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using robots.
Varun Burde, Pavel Burget, Torsten Sattler
Faculty of Electrical Engineering, Czech Technical University in Prague, Czechia · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, Czechia
One common approach to pose estimation involves predicting object keypoints in an image, followed by using Perspective-n-Point algorithms to compute the object's rotation and translation relative to the camera. While rotations preserve object shapes, this property is often neglected in keypoint-based pose estimation methods, where keypoints are typically predicted independently from each other. As imprecise keypoint predictions negatively affects pose estimation accuracy, it also limits its reliability in downstream tasks. In this work, we explore whether such inaccurate pose estimates can be identified by simply examining spatial locations between 2D keypoints. We propose a set of hand-crafted geometric features that capture the self-consistency of keypoint predictions, including pairwise distances, reprojection consistency, as well as render and mask consistency. Despite its simplicity, a logistic regression classifier trained on these features reliably detects pose estimation failures, outperforming confidence-based approaches like conformal keypoint predictions that rely solely on keypoint uncertainty.
Robin Chan
Mathematical Modeling of Industrial Life Cycles, Institute of Mathematics, Technische Universität Berlin