Monocular 3D object tracking aims to estimate temporally consistent 3D object poses across video frames, enabling autonomous agents to reason about scene dynamics. However, existing state-of-the-art approaches are fully supervised and rely on dense 3D annotations over long video sequences, which are expensive to obtain and difficult to scale. In this work, we address this fundamental limitation by proposing the first sparsely supervised framework for monocular 3D object tracking. Our approach decomposes the task into two sequential sub-problems: 2D query matching and 3D geometry estimation. Both components leverage the spatio-temporal consistency of image sequences to augment a sparse set of labeled samples and learn rich 2D and 3D representations of the scene. Leveraging these learned cues, our model automatically generates high-quality 3D pseudolabels across entire videos, effectively transforming sparse supervision into dense 3D track annotations. This enables existing fully-supervised trackers to effectively operate under extreme label sparsity. Extensive experiments on the KITTI and nuScenes datasets demonstrate that our method significantly improves tracking performance, achieving an improvement of up to 15.50 p.p. while using at most four ground truth annotations per track.
Figures & tables
Figure 1: Sparse3DTrack: The first sparsely supervised approach for monocular 3D object tracking. Our framework couples spatio-temporal consistency of image sequences with sparse annotations (green) to generate high-quality 3D tracking pseudolabels (blue).
Figure 2: Overview of our Sparse3DTrack framework for sparsely supervised monocular 3D object tracking. Sparse3DTrack decomposes the overall problem into two sequential sub-tasks, namely, 2D query matching which finds object correspondences across frames and 3D geometry estimation which estimates the 3D pose and dimensions of the localized object. Our framework is extremely label-efficient and enables object tracking using at most four annotations per track.
Figure 3
KITTI Tracking
nuScenes
Base
2D
3D
MOTA ↑
MOTP ↓
IDF1 ↑
AMOTA ↑
AMOTP ↓
MOTA ↑
MOTP ↓
CenterTrack [ 44 ]
Sparse
14.21
53.15
19.97
14.06
160.24
12.20
68.82
SPOT [ 43 ]
FCOS3D [ 32 ]
19.05
46.78
31.00
10.26
161.31
7.82
89.51
SPOT [ 43 ]
CenterNet [ 45 ]
21.83
42.74
31.33
18.33
142.83
12.84
62.81
Ours
Ours
29.71
47.21
35.66
23.97
151.63
19.48
74.14
SAM2 [ 26 ]
FCOS3D [ 32 ]
26.31
48.77
37.80
0.71
173.24
1.31
107.13
Table 1: Evaluation of our approach on the KITTI Tracking and nuScenes datasets. The results of our model are highlighted in grey, and the best values are indicated in bold. Here, “Sparse” and “Oracle” refer to the cases when the base network is trained using only the sparse labels (lower bound) and 100% ground truth labels (upper bound), respectively. All values are represented in [ % ].
Table 5Table 6
Figure 5: Qualitative results of the 3D pseudolabels generated by Sparse3DTrack and the corresponding monocular 3D tracking performance when CenterTrack is trained using these pseudolabels. Each object track is visualized in a unique color, and its previous trajectory is illustrated using a sequence of dots of the same color. Note from (a, b) that Sparse3DTrack generates accurate and temporally consistent 3D psuedolabels even when objects undergo total occlusion, allowing CenterTrack to effectively track objects on unseen image sequences (c, d).
Figure S.1 : Visualization of the 3D pseudolabels generated by Sparse3DTrack. Each object track is visualized in a unique color, and its previous trajectory is illustrated using a sequence of dots of the same color. Note the accurate and temporally consistent predictions across both the KITTI and nuScenes datasets.
Figure S.2 : Visualization of the failure modes experienced by Sparse3DTrack. Each object track is visualized in a unique color, and its previous trajectory is illustrated using a sequence of dots of the same color. Observe that failures typically occur when the target object is far away, near image edges, or subject to significant occlusion and object clutter.
Figure S.3 : Visualization of the tracking results generated by the base network, CenterTrack, when trained using our 3D pseudolabels. Each object track is visualized in a unique color, and its previous trajectory is illustrated using a sequence of dots of the same color. Observe that CenterTrack generalizes to unseen image sequences and maintains accurate track predictions under occlusion, across varying geographic regions, and under different lighting conditions.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
Monocular 3D object detection has achieved impressive performance on densely annotated datasets. However, it struggles when only a fraction of objects are labeled due to the high cost of 3D annotation. This sparsely annotated setting is common in real-world scenarios where annotating every object is impractical. To address this, we propose a novel framework for sparsely annotated monocular 3D object detection with two key modules. First, we propose Road-Aware Patch Augmentation (RAPA), which leverages sparse annotations by augmenting segmented object patches onto road regions while preserving 3D geometric consistency. Second, we propose Prototype-Based Filtering (PBF), which generates high-quality pseudo-labels by filtering predictions through prototype similarity and depth uncertainty. It maintains global 2D RoI feature prototypes and selects pseudo-labels that are both feature-consistent with learned prototypes and have reliable depth estimates. Our training strategy combines geometry-preserving augmentation with prototype-guided pseudo-labeling to achieve robust detection under sparse supervision. Extensive experiments demonstrate the effectiveness of the proposed method. The source code is available at https://github.com/VisualAIKHU/MonoSAOD .
Monocular 3D object detection is crucial for scalable perception across fields like autonomous driving, robotics, and surveillance. However, progress is hindered by limited 3D annotations and the inherent ambiguity of single-image geometry. Existing methods often rely on strong geometric assumptions or carefully curated datasets, which limit their applicability to real-world scenarios. In this paper, we present PLOT (Pseudo-Labeling via Object Tracking), a framework that generates 3D annotations from monocular videos without auxiliary sensors or model retraining. PLOT tracks object and background trajectories to estimate camera motion and perform object association in pose-unknown settings. These trajectories provide point correspondences that align frame-wise pseudo-LiDARs, which are then fused via simple optimization into a unified object shape robust to occlusion and viewpoint shifts. Recognizing temporal coherence as a fundamental requirement for reliable shape fusion and video perception, we design a global object memory that preserves consistent object identities across frames. PLOT achieves robust annotation quality and strong generalization on both M3OD video benchmarks and in-the-wild videos, proving its effectiveness across diverse and unconstrained domains. Project page: https://plot-eccv.github.io.
Seokyeong Lee, Sithu Aung, Junyong Choi +3
KAIST AI · VRG, FEE, Czech Technical University in Prague · Samsung Research +3
Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.