Event-based Motion & Appearance Fusion for 6D Object Pose Tracking

Zhichao Li; Chiara Bartolozzi; Lorenzo Natale; Arren Glover

cs.CVMar 9, 2026

Event-based Motion & Appearance Fusion for 6D Object Pose Tracking

Authors: Zhichao Li, Chiara Bartolozzi, Lorenzo Natale, Arren Glover

Organizations: Event-driven Perception for Robotics, Istituto Italiano di Tecnologia, Italy · Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Italy · University of Genoa, Genoa, Italy

Abstract

Object pose tracking is a fundamental and essential task for robotics to perform tasks in the home and industrial settings. The most commonly used sensors to do so are RGB-D cameras, which can hit limitations in highly dynamic environments due to motion blur and frame-rate constraints. Event cameras have remarkable features such as high temporal resolution and low latency, which make them a potentially ideal vision sensors for object pose tracking at high speed. Even so, there are still only few works on 6D pose tracking with event cameras. In this work, we take advantage of the high temporal resolution and propose a method that uses both a propagation step fused with a pose correction strategy. Specifically, we use 6D object velocity obtained from event-based optical flow for pose propagation, after which, a template-based local pose correction module is utilized for pose correction. Our learning-free method has comparable performance to the state-of-the-art algorithms, and in some cases out performs them for fast-moving objects. The results indicate the potential for using event cameras in highly-dynamic scenarios where the use of deep network approaches are limited by low update rates.

Figures & tables

Fig. 1 : Overview of the proposed pipeline for 6D object pose tracking with an event camera. Events captured by an event camera are utilized to update a velocity independent representation (EROS) [ 19 ] , simultaneously with event-based optical flow extraction. The 6D object pose is propagated based on the 6D object velocity estimated from the optical flow measurement. Several appearance templates are generated based on the propagated pose but with additional small pose perturbations, from which edges are extracted. The propagated 6D pose is corrected based on the best matched template compared to the EROS representation. Finally, an Unscented Kalman filter is adopted to further smooth the corrected pose over time.

Panel of Figure 2 — Fig. 2 : Visualization of (a) raw events, (b) EROS representation, and (c) template generated using the model rendered at a perpetuated pose with edge gradient extraction.

Panel of Figure 3 — Fig. 3 : Snapshots of objects in synthetic data sequences: (a) regular motion sample, and (b) fast motion sample.

Sequence		mustard_regular		potted_regular		cracker_regular		tomato_regular
Methods	Modality	$e_{p}(\sigma)(cm)$	$e_{r}(\sigma)(deg)$	$e_{p}(\sigma)(cm)$	$e_{r}(\sigma)(deg)$	$e_{p}(\sigma)(cm)$	$e_{r}(\sigma)(deg)$	$e_{p}(\sigma)(cm)$	$e_{r}(\sigma)(deg)$
ROFT [ 24 ]	RGBD	1.16(0.59)	4.88(1.62)	3.41(1.44)	19.81(8.78)	1.54(0.40)	2.61(1.04)	1.77(0.63)	9.29(3.87)
se(3)-TrackNet [ 12 ]	RGBD	0.77(0.09)	2.78(1.11)	0.84(0.34)	11.37(5.81)	0.86(0.52)	5.20(3.32)	28.27(8.81)	34.90(9.41)
FoundationPose [ 13 ]	RGBD	0.86(0.40)	6.29(3.36)	0.56(0.18)	1.72(0.75)	0.40(0.15)	0.94(0.35)	0.48(0.21)	1.53(0.70)
Hybrid method [ 16 ]	RGBDE	1.61(0.68)	5.78(1.75)	4.42(3.31)	12.96(7.99)	1.95(1.09)	7.30(3.66)	2.13(0.84)	9.25(4.08)
EDOPT [ 19 ]	Events	1.93(1.04)	14.17(8.40)	15.55(8.90)	21.89(10.57)	1.10(0.59)	4.11(2.31)	13.15(7.33)	71.96(36.60)

TABLE I : 6D object pose tracking performance of multiple algorithms using different information modalities on synthetic object motion sequences. The results of translations and rotations are measured by root mean square error (RMSE) and standard deviation (

\sigma

). EDOPT_Vel means that the EDOPT algorithm integrates the velocity calculated from two recent continuously tracked object poses, and Ours (only Vel) refers to propagating the pose using only 6D velocity.

Fig. 4 : Dual-camera setup. The left device is an event camera with resolution 1280 $\times$ 720. The right RGB-D camera is a RealSense D415, only used for evaluating base-line algorithms and doesn’t form an input to our algorithm.

Fig. 5 : Propagation and local correction strategy. The (left) figure shows error accumulation if velocity integration is used alone. The (middle) figure demonstrates a failure case of perturbation correction, when the noise or discretisation error allow the true pose to leave the perturbation region without correction. The (right) figure shows the proposed method combining both approaches, enabling the pose variation between the propagated pose and the ground truth pose to more likely remain within the bounds of the local correction space, and conversely correct integration error.

Fig. 6 : Qualitative results of mustard_bottle and cracker_box from real camera data.

TABLE II : Ablation study.

Object Pose Estimation

Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.

See all 187 papers

Event6D: Event-based Novel Object 6D Pose Tracking
Mar 30, 2026Jae-Young Kang, Hoonhee Cho, Taeyeop Lee +4Object Pose Estimation Event-Based Vision
KAIST · NVIDIA
Keypoint-based Dynamic Object 6-DoF Pose Tracking via Event Camera
Apr 25, 2026Zhe Wang, Qijin Song, Zihao Li +2Event-Based Vision Object Pose Estimation
ShanghaiTech Automation and Robotics (STAR) Center, School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, China · WLSA Shanghai Academy · School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, China
RoPT: Robust 6DoF Object Pose Tracking in Video
Oct 20, 2026Yongqing Liang, Yuzhong Chen, Sihang Li +7Visual Object Tracking Object Pose Estimation
New York University · General Motors

Mar 30, 2026J/K move · Enter open · S save

Event6D: Event-based Novel Object 6D Pose Tracking

Jae-Young Kang, Hoonhee Cho, Taeyeop Lee, Minjun Kang+3

KAIST · NVIDIA

Object Pose Estimation Event-Based Vision

Open paper AskPDF arXiv

Event cameras provide microsecond latency, making them suitable for 6D object pose tracking in fast, dynamic scenes where conventional RGB and depth pipelines suffer from motion blur and large pixel displacements. We introduce EventTrack6D, an event-depth tracking framework that generalizes to novel objects without object-specific training by reconstructing both intensity and depth at arbitrary timestamps between depth frames. Conditioned on the most recent depth measurement, our dual reconstruction recovers dense photometric and geometric cues from sparse event streams. Our EventTrack6D operates at over 120 FPS and maintains temporal consistency under rapid motion. To support training and evaluation, we introduce a comprehensive benchmark suite: a large-scale synthetic dataset for training and two complementary evaluation sets, including real and simulated event datasets. Trained exclusively on synthetic data, EventTrack6D generalizes effectively to real-world scenarios without fine-tuning, maintaining accurate tracking across diverse objects and motion patterns. Our method and datasets validate the effectiveness of event cameras for event-based 6D pose tracking of novel objects. Code and datasets are publicly available at https://chohoonhee.github.io/Event6D.

Abstract

Object Pose Estimation

Explore similar work

Event6D: Event-based Novel Object 6D Pose Tracking

Keypoint-based Dynamic Object 6-DoF Pose Tracking via Event Camera

RoPT: Robust 6DoF Object Pose Tracking in Video