Reflection-Robust 6DoF Object Tracking with Light Fields
Authors: Nikolai Goncharov, Donald G. Dansereau
Organizations: Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, University of Sydney, Sydney, NSW, Australia
Tracking the 6DoF pose of a moving rigid object is fundamental to robotics and autonomous driving, but existing trackers assume that object appearance is stable across a sequence, an assumption that breaks down on reflective surfaces whose appearance changes as they mirror the environment. We introduce a light field based reflection-robust 6DoF tracker that turns this apparent nuisance into a pose cue. Per frame, our method recovers depth robustly against reflections, back-projects it into a point cloud, and estimates surface normals. It then decomposes the object's view-dependent appearance into a diffuse albedo and the environment map it reflects, resulting in a relightable surface light field. Starting from a coarse initialization, we relight it by the recovered environment map and optimize the pose on the photometric loss. Because a moving object mirrors new parts of the scene, the environment map fills in as the sequence proceeds, sharpening this signal over time. To evaluate this approach, we introduce a light field tracking dataset re-rendered from a robotic manipulation benchmark at four controlled reflectivity levels, each paired with simulated depth that reproduces how consumer RGB-D sensors fail on shiny surfaces. Additionally, we evaluate on two captured light field sequences. Our method trails the strongest baselines on diffuse objects and is the only one that holds its accuracy on fully reflective objects, where every baseline degrades.
Figures & tables
Figure 1: Reflection as a pose signal. (a) A reflective surface patch P in flatland, with normal n and center xt , translates by Δx under environment illumination. (b) The EPIs Lt(s,u) and Lt+1(s,u) are correlated ( ⋆ ), the loss minimum revealing xt+1 . (c) Loss versus recovered translation x . Monocular color (orange), the classical cue, gives only a shallow dip, the EPIs (blue) capture view-dependence but ignore re-illumination, and only the relit EPIs (red) drive the loss to zero at the true coordinate.
Figure 2: Method overview. Top: each light field frame is processed, robustly against reflections, into depth and normals, and the view-dependent colors of the surface points are separated into per-point albedos D(xi) and a shared environment map E^(ω) ( Secs. 3.4 and 3.5 ). Bottom left: consecutive surface light fields are related by the unknown transform \prescriptt−1Tt ; the true illumination is never observed, and we substitute it with the map E^t−1 accumulated over previous frames. Bottom right: LoFTR [ 38 ] matches on the diffuse colors initialize \prescriptt−1Tt ; the source is then transformed, relit under E^t−1 , and rendered into I^pred , whose residual against the target updates the pose ( Sec. 3.3 ).
Figure 3: Our multi-reflectivity tracking dataset. (a) The mustard0 sequence; each frame is available at four reflectivity levels. (b) The four YCB objects: bleach cleanser, cracker box, mustard bottle, and sugar box. (c) Simulated sensor depth from SGBM [ 15 ] on a rendered stereo pair with a projected dot pattern; as reflectivity grows the depth collapses into holes, as on an RGB-D camera.
Figure 4: Tracking accuracy versus object reflectivity r , for the objects (top) and cube (bottom), averaged over the eight sequences, reported as ADD (AUC), ATE, and ARE; Tab. 1 gives ADD-S. Ours (red) stays flat on the objects while the baselines fall off with reflectivity; on the severely underconstrained planar cube the advantage does not hold.
ADD-S ↑
ATE [m] ↓
ARE [ ∘ ] ↓
r
0.0
0.5
0.7
1.0
0.0
0.5
0.7
1.0
0.0
0.5
0.7
1.0
PnP
0.704
0.341
0.165
0.139
0.267
0.254
1.698
1.692
24.8
45.7
95.4
65.4
ICP
0.794
0.566
0.442
0.370
0.046
0.116
0.183
0.205
27.5
44.2
56.8
68.6
LoFTR
0.896
0.835
0.802
0.659
0.025
0.035
0.044
0.074
9.8
15.2
19.7
40.0
FP
0.953
0.937
0.733
0.691
0.010
0.011
0.133
0.182
2.2
11.5
27.4
20.2
BSDF
0.954
0.936
0.912
0.857
0.011
0.014
0.017
0.029
3.7
5.2
8.6
16.4
Table 1: Average tracking performance over the 8 object sequences per reflectivity level r (FP: FoundationPose, BSDF: BundleSDF). Ours is the only method stable across the range and best on every metric at full reflectivity. Best in bold , second in italic . ADD is given in the supplementary material.
mustard bottle
same bottle, foil-wrapped
Method
ADD-S ↑
ATE [m] ↓
ARE [ ∘ ] ↓
ADD-S ↑
ATE [m] ↓
ARE [ ∘ ] ↓
PnP
0.322
0.136
31.2
0.205
0.269
128.4
ICP
0.135
0.193
103.2
0.207
0.172
80.7
LoFTR [ 38 ]
0.449
0.103
108.3
0.353
0.113
99.9
FoundationPose [ 45 ]
0.868
0.038
41.1
0.462
0.108
129.3
BundleSDF [ 44 ]
0.921
0.013
19.9
0.941
0.011
11.2
Table 2: Tracking on the captured sequences , two sequences of one mustard bottle, plain and foil-wrapped. ADD-S as AUC, ATE in m, ARE in degrees. Best in bold , second in italic . FoundationPose registers on every frame, its tracker being built for smaller steps.
Figure 7
Figure 6: Relightable surface light field representation , recovered feed-forward from one light field capture on estimated depth at r=0.7 . Rows are objects; columns show the recovered geometry, the diffuse–environment decomposition, and the relit render.
Table 9
Figure S1 : Dataset comparison. Original YCBInEOAT [ 43 ] captures (top) and our re-rendered central subviews at the same frames (bottom). The object follows the same ground truth trajectory in both, while the lab scene is replaced by a controlled background so reflectivity can be varied without changing the motion.
Figure S2 : Dataset composition compared to YCBInEOAT [ 43 ] . (a) Per-sequence frame counts before (gray) and after (blue) motion filtering: filtering removes redundant near-stationary frames, reducing the total by roughly an order of magnitude while preserving every sequence. (b) Distribution of per-frame motion in the filtered dataset across the six degrees of freedom: the retained motion stays broad and well spread on every axis.
Figure S3 : Simulated sensor depth versus reflectivity. Per-frame depth RMSE distributions on the objects (left) and cube (center), and the mean fraction of missing pixels (right), as functions of reflectivity r . From r=0 to r=1 the median RMSE grows roughly tenfold and the cube loses up to 87% of its pixels.
Figure S4 : FoundationPose reference models across reflectivity. One neural reference model per object (columns) and reflectivity level (rows). As the surface turns specular, holes open where the geometry can no longer be recovered, the reflected environment is baked into the surface, and the print detail the tracker matches against disappears.
Object
PnP
ICP
LoFTR
FP
BSDF
Ours
Bleach
0.080 ✗
0.147 ✗
0.554
0.525
0.810
0.795
Cracker box
0.056 ✗
0.086 ✗
0.525
0.881
0.526
0.736
Mustard
0.129 ✗
0.289 ✗
0.771
0.914
0.909
0.861
Sugar box
0.048 ✗
0.090 ✗
0.090 ✗
0.311 ✗
0.643
0.741
Objects lost ↓
4
4
1
1
0
0
Table S4 : Per-object ADD ↑ at full reflectivity ( r=1.0 ), averaged over each object’s sequences. An object is lost (✗) when mean ADD falls below 0.5 ; the last row counts the objects lost. Best per row in bold .
Figure S5 : Tracking under ground truth versus deployment depth. Mean ADD-S over the eight sequences on the objects (left) and cube (right), under ground truth depth (dashed) and each method’s deployment depth (solid; simulated sensor depth for the baselines, our estimated light field depth for ours). With clean depth the baselines barely degrade on the objects, so their collapse under sensor depth is caused by depth corruption rather than reflective appearance. The cube is the exception: BundleSDF collapses at full reflectivity under either depth.
Figure S6 : Depth input quality. Our light field depth ( LF depth , dashed) against the simulated sensor depth (solid) versus reflectivity r , for the objects (top) and cube (bottom): depth RMSE (left, log scale) and fraction of missing pixels (right), both lower-is-better, averaged over the eight sequences. The sensor depth collapses as the surface turns specular while the light field depth holds, except on the cube at r=1.0 , where the planar mirror makes it ambiguous.
Figure S7 : Environment map reconstruction over time on the reflective cube ( r=0.7 , estimated depth). Top: the central subview at four frames. Bottom: the incrementally recovered environment map and its observed fraction, with the ground truth panorama on the right.
Figure S8 : Reflection separation at r=0.7 , frame 20. Each object (rows) is decomposed into a diffuse component bound to the surface and a reflected component bound to the environment. Left block: the light field input, the ground truth diffuse albedo, and our recovered diffuse under ground truth and estimated depth. Right block: our recovered environment map under both depths, with the ground truth map below. Ground truth depth yields the cleaner separation, while the surface normal noise of estimated depth blurs the environment map and leaks reflected content into the diffuse component.
Figure S9 : Qualitative ablation on the most reflective sugar_box sequence. Each row tracks the object over time (columns left to right), with the yellow outline and axes showing the estimated pose and the row’s mean ADD over this sequence. The top row is the ground truth, below it the full model and then variants that each remove one component (photometric refinement, reflection separation, and finally light field depth replaced by sensor depth). Drift grows as each component is removed, most severely once sensor depth replaces the light field depth.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
Real-time 6-DoF object pose tracking is essential for many robotics applications, and several approaches exist. Yet even today's approaches remain unreliable under temporary full occlusions and rapid object motions. Once tracking is lost, most methods struggle to detect the failure and recover automatically, often requiring manual re-initialization. In this paper, we address the problem of robust model-based 6-DoF tracking of unseen objects from RGB-D data, especially in scenarios with occlusion and fast motion. We propose a novel method that combines efficient learning-based keypoint matching with optimization-based alignment and introduces a novel failure detection and recovery module. Our system monitors pose reliability, detects tracking divergence or occlusions, and performs a global re-detection and pose estimation step that robustly verifies recovery candidates before resuming tracking. Our evaluation on standard tracking benchmarks and on a new dataset of occluded and fast-moving scenes shows that our method matches state-of-the-art accuracy on easy tracking sequences, maintains high tracking speed at 57.6 frames per second, and provides the most robust tracking performance under challenging conditions. Thus, we believe that our approach is a relevant step forward in robust 6-DoF object tracking from RGB-D data.
Balázs Opra, Léo Ghafari, Thomas Stewart +1
Woven by Toyota, Inc. · University of Bonn, Germany. · University of Bonn, Center for Robotics, and the Lamarr Institute for Machine Learning and Artificial Intelligence.
Robust object 6D pose tracking is critical for robotic systems operating in dynamic and occluded scenes. Per-frame estimators are accurate but computationally expensive, while current trackers struggle with fast motion and complete occlusion due to their reliance on continuous visibility. To address these challenges, we present RRTrack, an efficient, recoverable object 6D pose tracker that enables robust tracking through fast motion and target disappearance--reappearance. RRTrack introduces a 2D--6D closed-loop tracking strategy that integrates memory-based video object segmentation (VOS) with 6D pose refinement. The 2D branch maintains target localization, and the 6D branch verifies geometric consistency before memory updates. In addition, a DINOv2-based dual-bank template matching module is developed to recover lost targets by jointly exploiting offline synthetic templates and online observation anchors while maintaining real-time efficiency. We also introduce a synthetic RGB-D benchmark comprising three robotic scenarios with fast motion and full occlusion. Experimental results on the synthetic benchmark demonstrate that RRTrack improves equal-subset mean ADD-S AR by 66.3% and ADD-S AUC by 65.7% over FoundationPose while achieving 55.2 FPS. Real-world experiments further validate the robustness of RRTrack under noisy sensing conditions. Project page: https://github.com/7kevin24/RRTrack
Junyue Li, Ye Zheng, Yifan Chen +2
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Institute of Artificial Intelligence (TeleAI), China Telecom, China · College of Future Information Technology, Fudan University, Shanghai 200433, China
Accurate 6-DoF object pose tracking is essential for robotic interaction, yet conventional monocular pipelines frequently suffer from catastrophic divergence under rapid object motion. We present RoPT, a robust tracking framework engineered to overcome this limitation by unifying visual perception, temporal kinematics, and global re-localization into a tightly coupled closed loop. Unlike existing open-loop trackers, RoPT delivers three critical capabilities that fundamentally alter system resilience. First, the visual perception component tracks object masks to robustly handle partial and severe occlusions. Second, a kinematic factor graph regularizes frame-to-frame updates, forecasts forward pose predictions, and drives a parallel multi-hypothesis verification scheme to bypass local minima. Finally, an active re-localization module transforms the pipeline into a closed-loop system, making it failure-aware and capable of deterministic self-recovery. We demonstrate the effectiveness of our framework across challenging benchmarks alongside real-world robotic demonstrations, achieving significantly higher tracking survivability under rapid motion.