Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success
Authors: Varun Burde, Pavel Burget, Torsten Sattler
Organizations: Faculty of Electrical Engineering, Czech Technical University in Prague, Czechia · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, Czechia
3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based benchmark that evaluates 6D pose estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using robots.
Figures & tables
Fig. 1: Overview of our evaluation pipeline. First, a canonical grasp library is pre-computed for each object. Then, for a given scene, a pose estimator provides Tc2oest . This pose is used to derive a target gripper pose, Tw2gest , which is executed on the ground-truth object. The outcome is recorded to calculate the Estimated Success Rate ( Sest ) (Sec. III-D2 ) and correlated with the initial pose error.
Fig. 2: Baseline gripper performance analysis, visualizing the Grasp Generation Success Rate ( Sgen ) (Sec. III-D1 ) across various grippers and objects under ideal conditions. (a) Per-object Sgen for each gripper. (b) Distribution of the best-performing gripper for each object. (c) Average Sgen per gripper. (d) A Physics-Based Outcome Breakdown (Sec. III-D3 ) of grasp failures for each gripper with gripper jaw widths annotated in millimeters.
Fig. 3: Analysis of Grasping Performance vs. Pose Estimation Error. Left Panel: Scatter plots showing the relation between various pose error metrics and the Estimated Success Rate ( Sest ), averaged over both FoundationPose and MegaPose across 8,250 trials and 18,882,842 simulations (Sec. III-D2 ). Right Panel: A detailed Physics-Based Outcome Breakdown (Sec. III-D3 ) of grasp attempts per object. The green portion of each bar represents the final Sest , while other colors show the proportions of different failure modes.
Fig. 4: Impact of 3D Model Fidelity on Grasp Candidates. Left panel: The Grasp Generation Success Rate ( Sgen ) (Sec. III-D1 ) for various reconstruction methods. Right panel: A Physics-Based Outcome Breakdown (Sec. III-D3 ) for grasps planned on these meshes. Note the significant increase in ’Collision’ failures for lower-quality models.
Fig. 5: Comparative Analysis of Grasping Success under Compounded Errors. This figure compares the final grasping success, measured by the Estimated Success Rate ( Sest ) (Sec. III-D2 ), when combining different sources of geometric and pose uncertainty. Left: Performance under the ’GT → Reconstructed mesh’ condition. Right: Performance under the ’Reconstructed mesh → GT’ condition.
Robotic grasping is a fundamental capability in robotic manipulation. Yet grasping remains challenging under partial observations. Reliable grasping depends on both local contact cues and object-level 3D structure. Existing geometry-aware grasping methods recognize the value of reconstruction, but they typically treat geometry as an intermediate prediction rather than a reusable object prior for grasping. In this paper, we present GraspFoM, a unified framework that leverages 3D foundation priors (SAM3D) to build a shared 3D object latent for both reconstruction and grasp pose prediction. Built on this shared object latent, we introduce an anchor-initialized truncated pose-reasoning diffuser that predicts continuous and multimodal grasp poses without directly relying on discrete grasp candidates. We further investigate the interaction between reconstruction and grasping through a reconstruction-aware scorer and a residual latent updater. Reconstruction provides grounded geometric cues, while grasp supervision refines the shared object latent toward grasp-relevant affordances. GraspFoM jointly predicts grasp poses and reconstructs high-fidelity 3D assets in mesh and 3DGS forms. Comprehensive experiments demonstrate that GraspFoM achieves state-of-the-art results on both reconstruction and grasping. Notably, these improvements require only a small number of additional trainable parameters. Component-wise ablation studies also demonstrate the contribution of each component.
Dongli Wu, Xiaobao Wei, Hao Wang +5
1The Hong Kong University of Science and Technology (Guangzhou) · 2Peking University · 3The Hong Kong University of Science and Technology
Robust estimation of object poses in robotic manipulation is often addressed using foundational general estimators, that aim to handle diverse error sources naively within a single model. Still, they struggle due to environmental uncertainties, while requiring long inference times and heavy computation. In contrast, we propose a modular, uncertainty-aware framework that attributes pose estimation errors to specific error sources and applies targeted mitigation strategies only when necessary. Instantiated with Iterative Closest Point (ICP) as a simple and lightweight pose estimator, we leverage our framework for real-world robotic grasping tasks. By decomposing pose estimation into failure detection, error attribution, and targeted recovery, we significantly improve the robustness of ICP and achieve competitive performance compared to foundation models, while relying on a substantially simpler and faster pose estimator.
Loris Schneider, Yitian Shi, Rosa Wolf +3
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany. · German Aerospace Center (DLR), Weßling, Germany.
Humans grasp unfamiliar objects by combining an initial visual estimate with tactile and proprioceptive feedback during interaction. We present ShapeGrasp, a robotic implementation of this approach. The proposed method is an iterative grasp-and-complete pipeline that couples implicit surface visuo-haptic shape completion (creation of full 3D shape from partial information) with physics-based grasp planning. From a single RGB-D view, ShapeGrasp infers a complete shape (point cloud or triangular mesh), generates candidate grasps via rigid-body simulation, and executes the best feasible grasp. Each grasp attempt yields additional geometric constraints -- tactile surface contacts and space occupied by the gripper body -- which are fused to update the object shape. Failures trigger pose re-estimation and regrasping using the refined shape. We evaluate ShapeGrasp in the real world using two different robots and grippers. To the best of our knowledge, this is the first approach that updates shape representations following a real-world grasp. We achieved superior results over baselines for both grippers (grasp success rate of 84% with a three-finger gripper and 91% with a two-finger gripper), while improving the 3D shape reconstruction quality in all evaluation metrics used.
Lukas Rustler, Matej Hoffmann
Department of Cybernetics, Faculty of Electrical Engineering, Czech Technical University in Prague, Prague, Czech Republic.