Object Pose Estimation

Follow this topic Ask about this topic

Object pose estimation determines where an object is and how it is oriented, often as a 6D pose and sometimes including its size or articulation. Methods use keypoints or dense correspondences with geometric solvers such as PnP and ICP, alongside learned RGB-D and point-cloud models, foundation features, and generative pose prediction. Symmetry-aware representations, temporal tracking, and tactile or multi-view cues help address occlusion and ambiguous geometry, with benchmarks such as BOP, LINEMOD, and NOCS testing different settings. A central challenge is making systems generalize beyond familiar objects and controlled benchmarks to cluttered, deformable, reflective, or otherwise changing real-world scenes while remaining reliable for tasks such as robotic grasping.

Latest papers

187 papers25 papers in the last four weeks, up 47% on the four weeks before. 0.1% of all new papers.

Mar 22, 2026

Bayesian Active Object Recognition and 6D Pose Estimation from Multimodal Contact Sensing

We present an active tactile exploration framework for joint object recognition and 6D pose estimation. The proposed method integrates wrist force/torque sensing, GelSight tactile sensing, and free-space constraints within a Bayesian inference framework that maintains a belief over object class and pose during active tactile exploration. By combining contact and non-contact evidence, the framework reduces ambiguity and improves robustness in the joint class-pose estimation problem. To enable efficient inference in the large hypothesis space, we employ a customized particle filter that progressively samples particles based on new observations. The inferred belief is further used to guide active exploration by selecting informative next touches under reachability constraints. For effective data collection, a motion planning and control framework is developed to plan and execute feasible paths for tactile exploration, handle unexpected contacts and GelSight-surface alignment with tactile servoing. We evaluate the framework in simulation and on a Franka Panda robot using 11 YCB objects. Results show that incorporating tactile and free-space information substantially improves recognition and pose estimation accuracy and stability, while reducing the number of action cycles compared with force/torque-only baselines. Code, dataset, and supplementary material will be made available online.

Haodong Zheng, Gabriele M. Caddeo, Andrei C. Jalba +3

Object Pose Estimation Robotic Tactile Sensing

Mar 22, 2026

Geometrically Plausible Object Pose Refinement using Differentiable Simulation

State-of-the-art object pose estimation methods are prone to generating geometrically infeasible pose hypotheses. This problem is prevalent in dexterous manipulation, where estimated poses often intersect with the robotic hand or are not lying on a support surface. We propose a multi-modal pose refinement approach that combines differentiable physics simulation, differentiable rendering and visuo-tactile sensing to optimize object poses for both spatial accuracy and physical consistency. Simulated experiments show that our approach reduces the intersection volume error between the object and robotic hand by 73% when the initial estimate is accurate and by over 87% under high initial uncertainty, significantly outperforming standard ICP-based baselines. Furthermore, the improvement in geometric plausibility is accompanied by a concurrent reduction in translation and orientation errors. Achieving pose estimation that is grounded in physical reality while remaining faithful to multi-modal sensor inputs is a critical step toward robust in-hand manipulation.

Anil Zeybek, Rhys Newbury, Snehal Dikhale +3

Technical University of Munich, Germany · Physical AI, Australia · Monash University, Australia +1

Object Pose Estimation Robotic Manipulation

Mar 20, 2026

Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation

In this paper, we address the problem of 6-DoF object pose estimation from a single RGB image. Indirect methods that typically predict intermediate 2D keypoints, followed by a Perspective-n-Point solver, have shown great performance. Direct approaches, which regress the pose in an end-to-end manner, are usually computationally more efficient but less accurate. However, direct pose regression heads rely on globally pooled features, ignoring spatial second-order statistics despite their informativeness in pose prediction. They also predict, in most cases, discontinuous pose representations that lack robustness. Herein, we therefore propose a covariance-pooled representation that encodes convolutional feature distributions as a symmetric positive definite (SPD) matrix. Moreover, we propose a novel pose encoding in the form of an SPD matrix via its Cholesky decomposition. Pose is then regressed in an end-to-end manner with a manifold-aware network head, taking into account the Riemannian geometry of SPD matrices. Experiments and ablations consistently demonstrate the relevance of second-order pooling and continuous representations for direct pose regression, including under partial occlusion.

Nassim Ali Ousalah, Peyman Rostami, Vincent Gaudillière +4

CVI2, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg⋆ · Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France† · Infinite Orbits⋊ +1

Riemannian Neural Networks Object Pose Estimation Geometric Representation Learning

Mar 20, 2026

UniPR: Unified Object-level Real-to-Sim Perception and Reconstruction from a Single Stereo Pair

Perceiving and reconstructing objects from images are critical for real-to-sim transfer tasks, which are widely used in the robotics community. Existing methods rely on multiple submodules such as detection, segmentation, shape reconstruction, and pose estimation to complete the pipeline. However, such modular pipelines suffer from inefficiency and cumulative error, as each stage operates on only partial or locally refined information while discarding global context. To address these limitations, we propose UniPR, the first end-to-end object-level real-to-sim perception and reconstruction framework. Operating directly on a single stereo image pair, UniPR leverages geometric constraints to resolve the scale ambiguity. We introduce Pose-Aware Shape Representation to eliminate the need for per-category canonical definitions and to bridge the gap between reconstruction and pose estimation tasks. Furthermore, we construct a large-vocabulary stereo dataset, LVS6D, comprising over 6,300 objects, to facilitate large-scale research in this area. Extensive experiments demonstrate that UniPR reconstructs all objects in a scene in parallel within a single forward pass, achieving significant efficiency gains and preserves true physical proportions across diverse object types, highlighting its potential for practical robotic applications.

Chuanrui Zhang, Yingshuang Zou, ZhengXian Wu +3

NTU · Tencent Robotics X · HKUST +2

Real-to-Sim Transfer 3D Reconstruction Object Pose Estimation

Mar 19, 2026

GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awareness

Generalized 3D hand-object pose estimation from a single RGB image remains challenging due to the large variations in object appearances and interaction patterns, especially under heavy occlusion. We propose GenHOI, a framework for generalized hand-object pose estimation with occlusion awareness. GenHOI integrates hierarchical semantic knowledge with hand priors to enhance model generalization under challenging occlusion conditions. Specifically, we introduce a hierarchical semantic prompt that encodes object states, hand configurations, and interaction patterns via textual descriptions. This enables the model to learn abstract high-level representations of hand-object interactions for generalization to unseen objects and novel interactions while compensating for missing or ambiguous visual cues. To enable robust occlusion reasoning, we adopt a multi-modal masked modeling strategy over RGB images, predicted point clouds, and textual descriptions. Moreover, we leverage hand priors as stable spatial references to extract implicit interaction constraints. This allows reliable pose inference even under significant variations in object shapes and interaction patterns. Extensive experiments on the challenging DexYCB and HO3Dv2 benchmarks demonstrate that our method achieves state-of-the-art performance in hand-object pose estimation.

Hui Yang, Wei Sun, Jian Liu +6

NERC-RVC, Hunan University · National University of Singapore · Lancaster University +2

3D Hand Pose Estimation OOD Generalization Object Pose Estimation

Mar 17, 2026

FSMC-Pose: Frequency and Spatial Fusion with Multiscale Self-calibration for Cattle Mounting Pose Estimation

Mounting posture is an important visual indicator of estrus in dairy cattle. However, achieving reliable mounting pose estimation in real-world environments remains challenging due to cluttered backgrounds and frequent inter-animal occlusion. We present FSMC-Pose, a top-down framework that integrates a lightweight frequency-spatial fusion backbone, CattleMountNet, and a multiscale self-calibration head, SC2Head. Specifically, we design two algorithmic components for CattleMountNet: the Spatial Frequency Enhancement Block (SFEBlock) and the Receptive Aggregation Block (RABlock). SFEBlock separates cattle from cluttered backgrounds, while RABlock captures multiscale contextual information. The Spatial-Channel Self-Calibration Head (SC2Head) attends to spatial and channel dependencies and introduces a self-calibration branch to mitigate structural misalignment under inter-animal overlap. We construct a mounting dataset, MOUNT-Cattle, covering 1176 mounting instances, which follows the COCO format and supports drop-in training across pose estimation models. Using a comprehensive dataset that combines MOUNT-Cattle with the public NWAFU-Cattle dataset, FSMC-Pose achieves higher accuracy than strong baselines, with markedly lower computational and parameter costs, while maintaining real-time inference on commodity GPUs. Extensive experiments and qualitative analyses show that FSMC-Pose effectively captures and estimates cattle mounting pose in complex and cluttered environments. Dataset and code are available at https://github.com/elianafang/FSMC-Pose.

Fangjing Li, Zhihai Wang, Xinxin Ding +5

Beijing Jiaotong University · NERCITA · Tsinghua University +1

Object Pose Estimation Agricultural Image Analysis Multi-Scale Feature Fusion

Mar 17, 2026

Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery

We introduce a speech-guided embodied agent framework for video-guided skull base surgery that dynamically executes perception and image-guidance tasks in response to surgeon queries. The proposed system integrates natural language interaction with real-time visual perception directly on live intraoperative video streams, thereby enabling surgeons to request computational assistance without disengaging from operative tasks. Unlike conventional image-guided navigation systems that rely on external optical trackers and additional hardware setup, the framework operates purely on intraoperative video. The system begins with interactive segmentation and labeling of the surgical instrument. The segmented instrument is then used as a spatial anchor that is autonomously tracked in the video stream to support downstream workflows, including anatomical segmentation, interactive registration of preoperative 3D models, monocular video-based estimation of the surgical tool pose, and image guidance through real-time anatomical overlays. We evaluate the proposed system in video-guided skull base surgery scenarios and benchmark its tracking performance against a commercially available optical tracking system. Across three experimental trials, the hybrid vision-based method achieved a mean absolute tool-tip position error of 2.32 Plus/Minus 1.10 mm in the camera frame, with inter-frame yaw and pitch propagation discrepancies of 0.18 Plus/Minus 0.25° and 0.21 Plus/Minus 0.30°, respectively. The system completes tool segmentation and anatomy registration within approximately two minutes, substantially reducing setup complexity relative to conventional tracking workflows. These results demonstrate that speech-guided embodied agents can provide accurate spatial guidance while improving workflow integration and enabling rapid deployment of video-guided surgical systems.

Jecia Z. Y. Mao, Francis X. Creighton, Russell H. Taylor +1

Surgical Instrument Tracking Surgical Video Understanding Surgical Robotics Object Pose Estimation

Mar 12, 2026

Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization

Accurate and efficient tracking of surgical instruments is fundamental for Robot-Assisted Minimally Invasive Surgery. Although vision-based robot pose estimation has enabled markerless calibration without tedious physical setups, reliable tool tracking for surgical robots still remains challenging due to partial visibility and specialized articulation design of surgical instruments. Previous works in the field are usually prone to unreliable feature detections under degraded visual quality and data scarcity, whereas rendering-based methods often struggle with computational costs and suboptimal convergence. In this work, we incorporate CMA-ES, an evolutionary optimization strategy, into a versatile tracking pipeline that jointly estimates surgical instrument pose and joint configurations. Using batch rendering to efficiently evaluate multiple pose candidates in parallel, the method significantly reduces inference time and improves convergence robustness. The proposed framework further generalizes to joint angle-free and bi-manual tracking settings, making it suitable for both vision feedback control and online surgery video calibration. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method significantly outperforms prior approaches in both accuracy and runtime. Source code and data are available at https://github.com/hanyang-hu/online_dvrk_tracking.

Hanyang Hu, Zekai Liang, Florian Richter +1

Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA.

Visual Object Tracking Covariance Matrix Adaptation Evolution Strategy Object Pose Estimation Surgical Instrument Tracking Surgical Video Understanding Surgical Robotics

Mar 12, 2026

High-Precision 6DOF Pose Estimation via Global Phase Retrieval in Fringe Projection Profilometry for 3D Mapping

Digital fringe projection (DFP) enables micrometer-level 3D reconstruction, yet extending it to large-scale mapping remains challenging because six-degree-of-freedom pose estimation often cannot match the reconstruction's precision. Conventional iterative closest point (ICP) registration becomes inefficient on multi-million-point clouds and typically relies on downsampling or feature-based selection, which can reduce local detail and degrade pose precision. Drift-correction methods improve long-term consistency but do not resolve sampling sensitivity in dense DFP point clouds.We propose a high-precision pose estimation method that augments a moving DFP system with a fixed, intrinsically calibrated global projector. Using the global projector's phase-derived pixel constraints and a PnP-style reprojection objective, the method estimates the DFP system pose in a fixed reference frame without relying on deterministic feature extraction, and we experimentally demonstrate sampling invariance under coordinate-preserving subsampling. Experiments demonstrate sub-millimeter pose accuracy against a reference with quantified uncertainty bounds, high repeatability under aggressive subsampling, robust operation on homogeneous surfaces and low-overlap views, and reduced error accumulation when used to correct ICP-based trajectories. The method extends DFP toward accurate 3D mapping in quasi-static scenarios such as inspection and metrology, with the trade-off of time-multiplexed acquisition for the additional projector measurements.

Sehoon Tak, Keunhee Cho, Sangpil Kim +1

Department of Mechanical Engineering, Yonsei University, Seoul 03722, South Korea. · Department of Artificial Intelligence, Korea University, Seoul

3D Mapping Object Pose Estimation Camera Pose Estimation Fringe Projection Profilometry Point Cloud Registration

Mar 11, 2026

TacLoc: Global Tactile Localization on Objects from a Registration Perspective

Pose estimation is essential for robotic manipulation, particularly when visual perception is occluded during gripper-object interactions. Existing tactile-based methods generally rely on tactile simulation or pre-trained models, which limits their generalizability and efficiency. In this study, we propose TacLoc, a novel tactile localization framework that formulates the problem as a one-shot point cloud registration task. TacLoc introduces a graph-theoretic partial-to-full registration method, leveraging dense point clouds and surface normals from tactile sensing for efficient and accurate pose estimation. Without requiring rendered data or pre-trained models, TacLoc achieves improved performance through normal-guided graph pruning and a hypothesis-and-verification pipeline. TacLoc is evaluated extensively on the YCB dataset. We further demonstrate TacLoc on real-world objects across two different visual-tactile sensors.

Zirui Zhang, Boyang Zhang, Fumin Zhang +1

Object Pose Estimation Point Cloud Registration Tactile Sensing

Mar 9, 2026

Event-based Motion & Appearance Fusion for 6D Object Pose Tracking

Object pose tracking is a fundamental and essential task for robotics to perform tasks in the home and industrial settings. The most commonly used sensors to do so are RGB-D cameras, which can hit limitations in highly dynamic environments due to motion blur and frame-rate constraints. Event cameras have remarkable features such as high temporal resolution and low latency, which make them a potentially ideal vision sensors for object pose tracking at high speed. Even so, there are still only few works on 6D pose tracking with event cameras. In this work, we take advantage of the high temporal resolution and propose a method that uses both a propagation step fused with a pose correction strategy. Specifically, we use 6D object velocity obtained from event-based optical flow for pose propagation, after which, a template-based local pose correction module is utilized for pose correction. Our learning-free method has comparable performance to the state-of-the-art algorithms, and in some cases out performs them for fast-moving objects. The results indicate the potential for using event cameras in highly-dynamic scenarios where the use of deep network approaches are limited by low update rates.

Zhichao Li, Chiara Bartolozzi, Lorenzo Natale +1

Event-driven Perception for Robotics, Istituto Italiano di Tecnologia, Italy · Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Italy · University of Genoa, Genoa, Italy

Object Pose Estimation Event-Based Vision

Mar 7, 2026

CanoVerse: 3D Object Scalable Canonicalization and Dataset for Generation and Pose

3D learning systems implicitly assume that objects occupy a coherent reference frame. Nonetheless, in practice, every asset arrives with an arbitrary global rotation, and models are left to resolve directional ambiguity on their own. This persistent misalignment suppresses pose-consistent generation, and blocks the emergence of stable directional semantics. To address this issue, we construct \methodName{}, a massive canonical 3D dataset of 320K objects over 1,156 categories -- an order-of-magnitude increase over prior work. At this scale, directional semantics become statistically learnable: Canoverse improves 3D generation stability, enables precise cross-modal 3D shape retrieval, and unlocks zero-shot point-cloud orientation estimation even for out-of-distribution data. This is achieved by a new canonicalization framework that reduces alignment from minutes to seconds per object via compact hypothesis generation and lightweight human discrimination, transforming canonicalization from manual curation into a high-throughput data generation pipeline. The Canoverse dataset will be publicly released upon acceptance. Project page: https://github.com/123321456-gif/Canoverse

Li Jin, Yuchen Yang, Weikai Chen +11

SDU · LIGHTSPEED · UNC Chapel Hill +2

3D Shape Retrieval 3D Representation Learning 3D Asset Generation Object Pose Estimation

Mar 4, 2026

Yolo-Key-6D: Single Stage Monocular 6D Pose Estimation with Keypoint Enhancements

Estimating the 6D pose of objects from a single RGB image is a critical task for robotics and extended reality applications. However, state-of-the-art multi stage methods often suffer from high latency, making them unsuitable for real time use. In this paper, we present Yolo-Key-6D, a novel single stage, end-to-end framework for monocular 6D pose estimation designed for both speed and accuracy. Our approach enhances a YOLO based architecture by integrating an auxiliary head that regresses the 2D projections of an object's 3D bounding box corners. This keypoint detection task significantly improves the network's understanding of 3D geometry. For stable end-to-end training, we directly regress rotation using a continuous 9D representation projected to SO(3) via singular value decomposition. On the LINEMOD and LINEMOD-Occluded benchmarks, YOLO-Key-6D achieves competitive accuracy scores of 96.24% and 69.41%, respectively, with the ADD(-S) 0.1d metric, while proving itself to operate in real time. Our results demonstrate that a carefully designed single stage method can provide a practical and effective balance of performance and efficiency for real world deployment.

Kemal Alperen Çetiner, Hazım Kemal Ekenel

ASELSAN, Aerospace Technologies, Türkiye · Istanbul Technical University, Department of Computer Engineering, Türkiye · New York University Abu Dhabi, Division of Engineering, UAE

Object Pose Estimation

Mar 4, 2026

Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole

Reliable insertion of industrial connectors remains a central challenge in robotics, requiring sub-millimeter precision under uncertainty and often without full visual access. Vision-based approaches struggle with occlusion and limited generalization, while learning-based policies frequently fail to transfer to unseen geometries. To address these limitations, we leverage tactile sensing, which captures local surface geometry at the point of contact and thus provides reliable information even under occlusion and across novel connector shapes. Building on this capability, we present Touch2Insert, a tactile-based framework for arbitrary peg insertion. Our method reconstructs cross-sectional geometry from high-resolution tactile images and estimates the relative pose of the hole with respect to the peg in a zero-shot manner. By aligning reconstructed shapes through registration, the framework enables insertion from a single contact without task-specific training. To evaluate its performance, we conducted experiments with three diverse connectors in both simulation and real-robot settings. The results indicate that Touch2Insert achieved sub-millimeter pose estimation accuracy for all connectors in simulation, and attained an average success rate of 86.7% on the real robot, thereby confirming the robustness and generalizability of tactile sensing for real-world robotic connector insertion.

Masaru Yajima, Yuma Shin, Rei Kawakami +2

Institute of Science Tokyo, Tokyo, Japan. · Mitsubishi Electric, Kanagawa, Japan.

Robotic Insertion Tactile Sensing for Robotic Manipulation Zero-Shot Robotic Manipulation Object Pose Estimation

Mar 4, 2026

Tracking Feral Horses in Aerial Video Using Oriented Bounding Boxes

The social structures of group-living animals such as feral horses are diverse and remain insufficiently understood, even within a single species. To investigate group dynamics, aerial videos are often utilized to track individuals and analyze their movement trajectories, which are essential for evaluating inter-individual interactions and comparing social behaviors. Accurate individual tracking is therefore crucial. In multi-animal tracking, axis-aligned bounding boxes (bboxes) are widely used; however, for aerial top-view footage of entire groups, their performance degrades due to complex backgrounds, small target sizes, high animal density, and varying body orientations. To address this issue, we employ oriented bounding boxes (OBBs), which include rotation angles and reduce unnecessary background. Nevertheless, current OBB detectors such as YOLO-OBB restrict angles within a 180

^{\circ}

range, making it impossible to distinguish head from tail and often causing sudden 180

^{\circ}

flips across frames, which severely disrupts continuous tracking. To overcome this limitation, we propose a head-orientation estimation method that crops OBB-centered patches, applies three detectors (head, tail, and head-tail), and determines the final label through IoU-based majority voting. Experiments using 299 test images show that our method achieves 99.3% accuracy, outperforming individual models, demonstrating its effectiveness for robust OBB-based tracking.

Saeko Takizawa, Tamao Maeda, Shinya Yamamoto +1

Graduate School of Information Science, University of Hyogo, Hyogo, Japan · Research Center for Integrative Evolutionary Science, The Graduate University for Advanced Studies, Kanagawa, Japan · Institute for the Future of Human Society, Kyoto University, Kyoto, Japan

Multi-Object Tracking Object Detection Aerial Object Detection Object Pose Estimation

Mar 3, 2026

Self-supervised Domain Adaptation for Visual 3D Pose Estimation of Nano-drone Racing Gates by Enforcing Geometric Consistency

We consider the task of visually estimating the relative pose of a drone racing gate in front of a nano-quadrotor, using a convolutional neural network pre-trained on simulated data to regress the gate's pose. Due to the sim-to-real gap, the pre-trained model underperforms in the real world and must be adapted to the target domain. We propose an unsupervised domain adaptation (UDA) approach using only real image sequences collected by the drone flying an arbitrary trajectory in front of a gate; sequences are annotated in a self-supervised fashion with the drone's odometry as measured by its onboard sensors. On this dataset, a state consistency loss enforces that two images acquired at different times yield pose predictions that are consistent with the drone's odometry. Results indicate that our approach outperforms other SoA UDA approaches, has a low mean absolute error in position (x=26, y=28, z=10 cm) and orientation (

ψ

=13

{^{\circ}}

), an improvement of 40% in position and 37% in orientation over a baseline. The approach's effectiveness is appreciable with as few as 10 minutes of real-world flight data and yields models with an inference time of 30.4ms (33 fps) when deployed aboard the Crazyflie 2.1 Brushless nano-drone.

Nicholas Carlotti, Michele Antonazzi, Elia Cereda +4

Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland · University of Milan, Italy · Integrated Systems Laboratory (IIS), ETHZ, Zürich, Switzerland

Relative Pose Estimation Object Pose Estimation Synthetic-to-Real Domain Adaptation Unsupervised Domain Adaptation Self-Supervised Learning

Mar 3, 2026

Tracing Back Error Sources to Explain and Mitigate Pose Estimation Failures

Robust estimation of object poses in robotic manipulation is often addressed using foundational general estimators, that aim to handle diverse error sources naively within a single model. Still, they struggle due to environmental uncertainties, while requiring long inference times and heavy computation. In contrast, we propose a modular, uncertainty-aware framework that attributes pose estimation errors to specific error sources and applies targeted mitigation strategies only when necessary. Instantiated with Iterative Closest Point (ICP) as a simple and lightweight pose estimator, we leverage our framework for real-world robotic grasping tasks. By decomposing pose estimation into failure detection, error attribution, and targeted recovery, we significantly improve the robustness of ICP and achieve competitive performance compared to foundation models, while relying on a substantially simpler and faster pose estimator.

Loris Schneider, Yitian Shi, Rosa Wolf +3

Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany. · German Aerospace Center (DLR), Weßling, Germany.

Object Pose Estimation Robust Estimation Point Cloud Registration

Mar 1, 2026

MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation

Freehand 3D ultrasound (US) reconstruction promises volumetric imaging with the flexibility of standard 2D probes, yet existing tracking paradigms face a restrictive trilemma: marker-based systems demand prohibitive costs, inside-out methods require intrusive sensor attachment, and sensorless approaches suffer from severe cumulative drift. To overcome these limitations, we present MLRecon, a robust markerless 3D US reconstruction framework delivering drift-resilient 6D probe pose tracking using a single commodity RGB-D camera. Leveraging the generalization power of vision foundation models, our pipeline enables continuous markerless tracking of the probe, augmented by a vision-guided divergence detector that autonomously monitors tracking integrity and triggers failure recovery to ensure uninterrupted scanning. Crucially, we further propose a dual-stage pose refinement network that explicitly disentangles high-frequency jitter from low-frequency bias, effectively denoising the trajectory while maintaining the kinematic fidelity of operator maneuvers. Experiments demonstrate that MLRecon significantly outperforms competing sensorless and sensor-aided methods, achieving average position errors as low as 0.88 mm on complex trajectories and yielding high-quality 3D reconstructions with sub-millimeter mean surface accuracy. This establishes a new benchmark for low-cost, accessible volumetric US imaging in resource-limited clinical settings.

Yi Zhang, Puxun Tu, Kun Wang +3

Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Ultrasound in Medicine, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China · Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

Ultrasound Imaging Object Pose Estimation

Feb 28, 2026

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation

Existing end-to-end approaches of robotic manipulation often lack generalization to unseen objects or tasks due to limited data and poor interpretability. While recent Multimodal Large Language Models (MLLMs) demonstrate strong commonsense reasoning, they struggle with geometric and spatial understanding required for pose prediction. In this paper, we propose RobMRAG, a 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation (MRAG) framework for zero-shot robotic manipulation. Specifically, we construct a multi-source manipulation knowledge base containing object contact frames, task completion frames, and pose parameters. During inference, a Hierarchical Multimodal Retrieval module first employs a three-priority hybrid retrieval strategy to find task-relevant object prototypes, then selects the geometrically closest reference example based on pixel-level similarity and Instance Matching Distance (IMD). We further introduce a 3D-Aware Pose Refinement module based on 3D Gaussian Splatting into the MRAG framework, which aligns the pose of the reference object to the target object in 3D space. The aligned results are reprojected onto the image plane and used as input to the MLLM to enhance the generation of the final pose parameters. Extensive experiments show that on a test set containing 30 categories of household objects, our method improves the success rate by 7.76% compared to the best-performing zero-shot baseline under the same setting, and by 6.54% compared to the state-of-the-art supervised baseline. Our results validate that RobMRAG effectively bridges the gap between high-level semantic reasoning and low-level geometric execution, enabling robotic systems that generalize to unseen objects while remaining inherently interpretable.

Zilong Xie, Jingyu Gong, Xin Tan +2

School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China

Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation VLMs for Robotics Multimodal RAG Object Pose Estimation

Feb 28, 2026

COG: Confidence-aware Optimal Geometric Correspondence for Unsupervised Single-reference Novel Object Pose Estimation

Estimating the 6DoF pose of a novel object with a single reference view is challenging due to occlusions, view-point changes, and outliers. A core difficulty lies in finding robust cross-view correspondences, as existing methods often rely on discrete one-to-one matching that is non-differentiable and tends to collapse onto sparse key-points. We propose Confidence-aware Optimal Geometric Correspondence (COG), an unsupervised framework that formulates correspondence estimation as a confidence-aware optimal transport problem. COG produces balanced soft correspondences by predicting point-wise confidences and injecting them as optimal transport marginals, suppressing non-overlapping regions. Semantic priors from vision foundation models further regularize the correspondences, leading to stable pose estimation. This design integrates confidence into the correspondence finding and pose estimation pipeline, enabling unsupervised learning. Experiments show unsupervised COG achieves comparable performance to supervised methods, and supervised COG outperforms them.

Yuchen Che, Jingtu Wu, Hao Zheng +1

Institute of Science Tokyo · RIKEN · Tohoku University

Object Pose Estimation Unsupervised Learning Optimal Transport

Feb 25, 2026

WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos

Egocentric manipulation videos are highly challenging due to severe occlusions during interactions and frequent object entries and exits from the camera view as the person moves. Current methods typically focus on recovering either hand or object pose in isolation, but both struggle during interactions and fail to handle out-of-sight cases. Moreover, their independent predictions often lead to inconsistent hand-object relations. We introduce WHOLE, a method that holistically reconstructs hand and object motion in world space from egocentric videos given object templates. Our key insight is to learn a generative prior over hand-object motion to jointly reason about their interactions. At test time, the pretrained prior is guided to generate trajectories that conform to the video observations. This joint generative reconstruction substantially outperforms approaches that process hands and objects separately followed by post-processing. WHOLE achieves state-of-the-art performance on hand motion estimation, 6D object pose estimation, and their relative interaction reconstruction. Project website: https://judyye.github.io/whole-www

Yufei Ye, Jiaman Li, Ryan Rong +1

Stanford University · Amazon FAR (Frontier AI & Robotics)

Hand-Object Interaction Reconstruction Hand-Object Interaction 3D Reconstruction Egocentric Video Understanding 3D Hand Reconstruction Object Pose Estimation+2

Feb 23, 2026

Simulation-Ready Cluttered Scene Estimation via Physics-aware Joint Shape and Pose Optimization

Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end Simulation-ready Physics-Aware Reconstruction for Cluttered Scenes (SPARCS) pipeline, which integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses. Project webpage: https://rory-weicheng.github.io/SPARCS/.

Wei-Cheng Huang, Jiaheng Han, Xiaohan Ye +2

Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign · Department of Computer Science, The University of Hong Kong · Meta Reality Labs

Physics-Aware 3D Scene Reconstruction 3D Reconstruction Object Pose Estimation Differentiable Physics

Feb 23, 2026

Learning Positive-Incentive Point Sampling in Neural Implicit Fields for Object Pose Estimation

Learning neural implicit fields of 3D shapes is a rapidly emerging field that enables shape representation at arbitrary resolutions. Due to the flexibility, neural implicit fields have succeeded in many research areas, including shape reconstruction, novel view image synthesis, and more recently, object pose estimation. Neural implicit fields enable learning dense correspondences between the camera space and the object's canonical space-including unobserved regions in camera space-significantly boosting object pose estimation performance in challenging scenarios like highly occluded objects and novel shapes. Despite progress, predicting canonical coordinates for unobserved camera-space regions remains challenging due to the lack of direct observational signals. This necessitates heavy reliance on the model's generalization ability, resulting in high uncertainty. Consequently, densely sampling points across the entire camera space may yield inaccurate estimations that hinder the learning process and compromise performance. To alleviate this problem, we propose a method combining an SO(3)-equivariant convolutional implicit network and a positive-incentive point sampling (PIPS) strategy. The SO(3)-equivariant convolutional implicit network estimates point-level attributes with SO(3)-equivariance at arbitrary query locations, demonstrating superior performance compared to most existing baselines. The PIPS strategy dynamically determines sampling locations based on the input, thereby boosting the network's accuracy and training efficiency. Our method outperforms the state-of-the-art on three pose estimation datasets. Notably, it demonstrates significant improvements in challenging scenarios, such as objects captured with unseen pose, high occlusion, novel geometry, and severe noise.

Yifei Shi, Boyan Wan, Xin Xu +1

College of Intelligence Science and Technology, National University of Defense Technology, China · College of Computer Science, National University of Defense Technology, China

Object Pose Estimation Implicit Neural Representations Adaptive Sampling

Feb 23, 2026

Generative 6D Pose Estimation via Conditional Flow Matching

Existing methods for instance-level 6D pose estimation typically rely on neural networks that either directly regress the pose in

\mathrm{SE}(3)

or estimate it indirectly via local feature matching. The former struggle with object symmetries, while the latter fail in the absence of distinctive local features. To overcome these limitations, we propose a novel formulation of 6D pose estimation as a conditional flow matching problem in

\mathbb{R}^3

. We introduce Flose, a generative method that infers object poses via a denoising process conditioned on local features. While prior approaches based on conditional flow matching perform denoising solely based on geometric guidance, Flose integrates appearance-based semantic features to mitigate ambiguities caused by object symmetries. We further incorporate RANSAC-based registration to handle outliers. We validate Flose on five datasets from the established BOP benchmark. Flose outperforms prior methods with an average improvement of +4.5 Average Recall. Project Website : https://tev-fbk.github.io/Flose/

Amir Hamza, Davide Boscaini, Weihang Li +2

Fondazione Bruno Kessler · University of Trento · Technical University of Munich +1

Object Pose Estimation Conditional Flow Matching

Feb 23, 2026

DICArt: Advancing Category-level Articulated Object Pose Estimation in Discrete State-Spaces

Articulated object pose estimation is a core task in embodied AI. Existing methods typically regress poses in a continuous space, but often struggle with 1) navigating a large, complex search space and 2) failing to incorporate intrinsic kinematic constraints. In this work, we introduce DICArt (DIsCrete Diffusion for Articulation Pose Estimation), a novel framework that formulates pose estimation as a conditional discrete diffusion process. Instead of operating in a continuous domain, DICArt progressively denoises a noisy pose representation through a learned reverse diffusion procedure to recover the GT pose. To improve modeling fidelity, we propose a flexible flow decider that dynamically determines whether each token should be denoised or reset, effectively balancing the real and noise distributions during diffusion. Additionally, we incorporate a hierarchical kinematic coupling strategy, estimating the pose of each rigid part hierarchically to respect the object's kinematic structure. We validate DICArt on both synthetic and real-world datasets. Experimental results demonstrate its superior performance and robustness. By integrating discrete generative modeling with structural priors, DICArt offers a new paradigm for reliable category-level 6D pose estimation in complex environments.

Li Zhang, Mingyu Mei, Ailing Wang +7

University of Science and Technology of China · Zhejiang University · East China Normal University +6

Object Pose Estimation Diffusion Models Articulated Object Reconstruction

Feb 19, 2026

Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success

3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based benchmark that evaluates 6D pose estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using robots.

Varun Burde, Pavel Burget, Torsten Sattler

Faculty of Electrical Engineering, Czech Technical University in Prague, Czechia · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, Czechia

Robot Manipulation Benchmarks Robotic Grasping Object Pose Estimation 3D Reconstruction 3D Mesh Reconstruction

Feb 18, 2026

MarUco: A Markerless 6D Pose Estimation Framework for Closed-Loop Control of Surgical Continuum Manipulators

Flexible endoscopic continuum manipulators offer high dexterity and access to complex anatomy, but nonlinear hysteresis limits feedforward control accuracy. Closed-loop control can compensate for these errors but requires accurate six-degree-of-freedom (6D) end-effector pose feedback. We present MarUco, a markerless 6D pose estimation framework for closed-loop control using only stereo vision during operation. A photorealistic pseudo-rigid-body simulation pipeline generates large-scale annotated training data without manual labeling. A multifeature fusion network integrates masks, keypoints, heatmaps, and bounding boxes from both stereo views to estimate an initial pose, followed by a learned single-pass render-and-compare module that refines the pose without iterative optimization. Kinematics-free hand-eye calibration estimates camera-to-robot-base extrinsics, and self-supervised adaptation uses unlabeled real stereo pairs to mitigate sim-to-real pose error. Across 1,000 real samples, MarUco achieves translation and rotation errors of 0.78

\pm

0.50 mm and 3.07

\pm

1.25°, respectively, with a total processing time of 52.4 ms per stereo pair. In closed-loop experiments over eight reference paths, MarUco achieves a mean terminal translation error of 1.8 mm, an 88% reduction relative to uncompensated open-loop control. To the best of our knowledge, this is the first markerless position-based visual servoing framework for continuum manipulators.

Junhyun Park, Chunggil An, Myeongbo Park +3

Department of Robotics and Mechatronics Engineering, DGIST, Daegu 42988, Republic of Korea · AI Research Lab, DEEPNOID, Seoul, Republic of Korea

Object Pose Estimation Continuum Robot Control Vision-Based Robot Control Surgical Robotics

Feb 18, 2026

Image Measurement Method for Automatic Insertion of Forks into Inclined Pallet

In order to insert a fork into a hole of a pallet by a forklift located in front of a pallet, it is necessary to control the height position, reach position, and tilt angle of the fork to match the position and orientation of the hole of the pallet. In order to make AGF (Autonomous Guided Forklift) do this automatically, we propose an image measurement method to measure the pitch inclination of the pallet in the camera coordinate system from an image obtained by using a wide-angle camera. In addition, we propose an image measurement method to easily acquire the calibration information between the camera coordinate system and the fork coordinate system necessary to apply the measurements in the camera coordinate system to the fork control. In the experiment space, a wide-angle camera was fixed at the backrest of a reach type forklift. The wide-angle images taken by placing a pallet in front of the camera were processed. As a result of evaluating the error by comparing the image measurement value with the hand measurement value when changing the pitch inclination angle of the pallet, the relative height of the pallet and the fork, and whether the pallet is loaded or not, it was confirmed that the error was within the allowable range for safely inserting the fork.

Nobuyuki Kita, Takuro Kato

TICO-AIST Cooperative Research Laboratory for Advanced Logistics, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan

Robotic Insertion Industrial Robotics Object Pose Estimation

Feb 16, 2026

Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract cancers but is limited by surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise tissue resection, potentially improving outcomes. This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery. Our goal is to utilize monocular endoscopic image-based perception algorithms to identify position and orientation of flexible instruments and measure their distances from tissues. We introduce 2D and 3D learning-based perception algorithms and develop a physically-realistic simulator that models flexible instruments dynamics. This simulator generates realistic endoluminal scenes, enabling control of flexible robots and substantial data collection. Using a continuum robot prototype, we conducted module and system-level evaluations. Results show that our algorithms improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.

Ruofeng Wei, Kai Chen, Yui Lun Ng +7

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Department of Mechanical Engineering, The University of Hong Kong, China. · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, and also with the Department of Mechanical Engineering, The University of Hong Kong.

Robotic Endoscopy Robotic Perception Surgical Robotics Continuum Robot Control Object Pose Estimation Monocular 3D Object Detection

Dec 4, 2025

Equivariant symmetry-aware head pose estimation for fetal MRI

We present E(3)-Pose, a novel fast pose estimation method that jointly and explicitly models rotation equivariance and object symmetry. Our work is motivated by the challenging problem of accounting for fetal head motion during a diagnostic MRI scan. We aim to enable automatic adaptive prescription of diagnostic 2D MRI slices with 6-DoF head pose estimation, supported by rapid low-resolution 3D MRI volumes acquired before each 2D slice. Existing pose estimation methods struggle to generalize to clinical volumes due to pose ambiguities induced by inherent anatomical symmetries, as well as low resolution, noise, and artifacts. In contrast, E(3)-Pose captures anatomical symmetries and rigid pose equivariance by construction, and yields robust estimates of the fetal head pose. Our experiments on publicly available and representative clinical fetal MRI datasets demonstrate the superior robustness and generalization of our method across domains. Crucially, E(3)-Pose achieves state-of-the-art accuracy on clinical MRI volumes, supporting future clinical translation. Our implementation is publicly available at github.com/MedicalVisionGroup/E3-Pose.

Ramya Muthukrishnan, Borjan Gagoski, Aryn Lee +4

MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, USA · Boston Children’s Hospital and Harvard Medical School, Boston, USA · Inria, Université Côte d’Azur, Sophia Antipolis, France

Symmetry-Aware Object Pose Estimation Object Pose Estimation Equivariant Neural Networks Magnetic Resonance Imaging Fetal MRI