TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Ayush Jain; Sreeharsha Paruchuri; Ishita Gupta; Fan Zhang; Tanner Schmidt; Jakob Engel; Katerina Fragkiadaki; Adam W. Harley

cs.CVSep 24, 2026

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Authors: Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley

Organizations: Carnegie Mellon University · Meta

Abstract

Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.

Figures & tables

Figure 1: TrackEverything tracks all points across all frames of long videos (1000+ frames) in 3D world-coordinate space. Left: a persistent, voxel-de-duplicated 3D scene representation grows with unique scene content rather than video length, remaining practical while prior all-frame dense 3D trackers run out of memory. Right: for each input video (top) we show the output dense 3D tracks (bottom). Trails are shown for dynamic points; static content is rendered as a 3D point cloud.

Figure 2: TrackEverything architecture. TrackEverything processes videos in sliding windows of length $L$ . Encoding: Video frames are encoded and unprojected into a world-coordinate 3D feature cloud using depths/pointmaps from sensors or geometry predictors. Endpoint Refinement: Conditioned on source/target position embeddings and 3D WAFT feature sampling, a recurrent transformer iteratively predicts each point’s 3D offset $\Delta\mathbf{x}$ at the target timestep, visibility, and a static/dynamic classification. Trajectory Refinement: Trajectories for dynamic points are initialized via constant-velocity interpolation and refined via within-track temporal self-attention and cross-attention from track summary tokens ( [cls] ) to the scene cloud. Output and De-duplication: At window boundaries, points arriving at identical 3D voxels are merged by mean-pooling before initializing the subsequent window, bounding representation size across long videos.

Figure 3: Complexity analysis on PointOdyssey. Accuracy (APD-M), end-to-end latency, and peak GPU memory vs. video length at 384 query points (top row, a–c), and query point count at 120 frames (bottom row, d–f). ‘ x ’ marks out-of-memory. Dense baselines exceed memory in 100-200 frames; sparse baselines track only a few thousand points, and OOM at 600 frames. TrackEverything uniquely tracks all points across all frames with modest latency and under 30GB GPU memory.

Table 2: Additional experiments and ablations. † reported by D4RT. TE: TrackEverything.

Panel of Figure 4 — Figure 4: Qualitative comparison against baselines. TrackEverything tracks every visible point across time, including newly emerging content. Notice that only one cow is visible in the first frame, while the second cow emerges later. Sparse trackers need human-specified queries and are usually demoed with a first-frame grid; densifying queries over later frames quickly becomes prohibitive ( Figure 3 ). First-frame dense trackers have the same blind spot by design. In all visualizations, we show trails for dynamic points only; the method tracks all points and accounts for camera motion, but we omit static-point trails in the 2D renderings for clarity.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Table 4: Early merge on full-length TAPVid-3D. APD-P uses VGGT-

\Omega

, as in Table 1 . FPS averages are the mean of ADT, DriveTrack, and PStudio.

Figure 5: Sliding-window tracking and cross-window de-duplication pipeline. Top row (Window $i$ ): A temporal window of frames is processed by TrackEverything to predict dense 3D tracks across all points. At the final timestep of the window, the predicted 3D point locations (here 46k points) are voxelized by quantizing coordinates into spatial voxels and mean-pooling co-located tokens. This cross-frame de-duplication prunes redundant surface observations (reducing point count from 46k to 16k). Bottom row (Window $i+1$ ): When the subsequent frame window arrives, its frames are lifted into a 3D point cloud using a feedforward geometry predictor (or sensor depths). These newly observed points are concatenated with the de-duplicated points carried over from previous windows (serving as persistent 3D scene memory) to construct the active set of points to track. TrackEverything then produces dense 3D trajectories for all points across all timesteps of Window $i+1$ . By chaining sliding-window processing with boundary de-duplication, TrackEverything tracks extended video sequences while bounding active token counts to unique physical scene geometry rather than video duration.

Figure 6: 3D WAFT projection and iterative refinement dynamics. Top: Projections across time under zero-velocity initialization. Source points initialized at frame $T=0$ (red dots) are projected to future timesteps ( $T=7$ and $T=15$ ) using camera extrinsics and intrinsics. Red dots denote projected 2D locations under the zero-velocity prior ( $\mathbf{x}_{\mathrm{tgt}}^{(0)}=\mathbf{x}_{\mathrm{src}}$ in world space); green dots mark true ground-truth (GT) locations, with green arrows at $T=15$ indicating displacement error. For static objects, camera projection accounts for ego-motion and aligns projected points near GT; for dynamic objects, world motion leads to significant spatial offset. Bottom: Zoomed-in patches across refinement iterations ( $\text{Iter}\in\{0,1,2\}$ ) at $T=7$ and $T=15$ . Static point (top): The zero-velocity initialization at $\text{Iter}=0$ already places the sampled point close to the true location. The bilinearly sampled target feature $\mathbf{g}_{\mathrm{tgt}}$ closely matches the source feature $\mathbf{g}_{\mathrm{src}}$ , signaling to the model that no large displacement is required and maintaining stability across iterations 1 and 2. Dynamic point (bottom): At $\text{Iter}=0$ , the zero-velocity projection misses the moving object by multiple pixels (landing on the background), creating an appearance mismatch between $\mathbf{g}_{\mathrm{src}}$ and $\mathbf{g}_{\mathrm{tgt}}$ in the concatenated WAFT feature. Driven by this mismatch, the endpoint refiner iteratively updates 3D positions: by $\text{Iter}=1$ the point moves substantially closer, and by $\text{Iter}=2$ the sampled point locks onto the correct surface feature.

Panel of Figure 7 — Figure 7: Qualitative comparison against baselines on the Breakdance sequence (Dance-jump and hand manipulation continue on the following pages). Rows show TrackEverything, SpatialTracker-v2, CoTracker3, and DeltaV2; time advances left to right. TrackEverything tracks every visible point across time, including newly emerging content. Sparse trackers need human-specified queries and are usually demoed with a first-frame grid; densifying queries over later frames quickly becomes prohibitive ( Figure 3 ). First-frame dense trackers have the same blind spot by design. In all visualizations, we show trails for dynamic points only; the method tracks all points and accounts for camera motion, but we omit static-point trails in the 2D renderings for clarity.

Panel of Figure 8 — Figure 8: Qualitative comparison against baselines (continued). Dance-jump sequence.

Panel of Figure 9 — Figure 9: Qualitative comparison against baselines (continued). Hand-manipulation sequence.

Figure 10: Qualitative ablations of iterative refinement and WAFT. (a) Without iterative refinement, tracks are noisier and less coherent; with refinement, trajectories follow motion more cleanly. (b) Without WAFT, tracks drift with long stray trails on fast motion; with WAFT, trajectories remain tightly localized on the moving subject.

Panel of Figure 11 — Figure 11: Static/Dynamic Classification. Points classified as dynamic are shown in red, static in blue. Time advances left to right. Notice how in the last example, the model correctly classifies some tigers as static while others as dynamic, and the static/dynamic assignment shifts as objects start or stop moving within a window.

		ADT					DriveTrack					PStudio					Avg
Method	Geo	P	M	AJ	OA	EPE $\downarrow$	P	M	AJ	OA	EPE $\downarrow$	P	M	AJ	OA	EPE $\downarrow$	P	M
Sparse tracking
CoTracker3	Pi3	38.7	88.0	27.2	88.8	0.16	29.2	41.9	20.3	88.5	2.13	23.7	83.1	14.5	85.0	0.18	30.5	71.0
CoTracker3	$\Omega$	43.1	87.8	31.3	88.8	0.16	29.7	42.8	21.4	88.5	2.33	29.6	87.9	19.0	85.0	0.13	34.1	72.8
SpatialTracker-v2	–	37.7	87.0	26.9	90.9	0.15	28.6	41.3	20.0	89.1	1.62	25.1	84.1	17.2	82.1	0.15	30.5	70.8
SpatialTracker-v2	Pi3	39.9	88.1	28.9	91.0	0.16	29.8	42.5	20.6	89.0	1.63	20.6	80.7	12.5	83.7	0.20	30.1	70.4

Appendix

Table 5: TAPVid-3D detailed metrics, 48-frame. Bold / underline: best / second-best within the block (D4RT excluded). P/M: APD-P/APD-M; AJ: Average Jaccard; OA: occlusion accuracy; EPE: end-point error.

\Omega

: VGGT-

\Omega

; Pi3: Pi3; –: native geometry.

		ADT					DriveTrack					PStudio					Avg
Method	Geo	P	M	AJ	OA	EPE $\downarrow$	P	M	AJ	OA	EPE $\downarrow$	P	M	AJ	OA	EPE $\downarrow$	P	M
Sparse tracking
CoTracker3	Pi3	31.4	82.3	18.9	84.8	0.20	24.3	40.5	17.0	89.7	2.30	18.7	80.2	10.6	81.7	0.20	24.8	67.7
CoTracker3	$\Omega$	34.5	84.4	21.5	84.8	0.18	27.4	43.9	19.8	89.7	2.21	27.1	87.9	16.5	81.7	0.13	29.7	72.1
SpatialTracker-v2	–	oom	oom	oom	oom	oom	oom	oom	oom	oom	oom	23.9	84.0	15.5	80.5	0.15	–	–
SpatialTracker-v2	Pi3	32.1	82.4	–	–	–	26.2	43.5	18.7	91.0	1.46	17.2	78.5	10.3	82.3	0.22	25.2	68.1

Appendix

Table 6: TAPVid-3D detailed metrics, full-length. Bold / underline: best / second-best within the block (D4RT excluded). N/A: method cannot run on full-length video. Column definitions match Table 5 .

Figure 12: Complexity analysis on PointOdyssey (APD-P). Accuracy, latency, and peak GPU memory as functions of (top row, a–c) video length at 384 query points, and (bottom row, d–f) number of query points at 120 frames. ‘x’ marks out-of-memory.

Explore similar work

May 12, 2026cs.CV

TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking

Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.

Jisu Nam, Jahyeok Koo, Soowon Son +4

KAIST AI · Google DeepMind

3D Tracking Video Diffusion Models Video Latents Discriminative Congruence Transform Prior Knowledge Integration

Ask PDF

Sep 8, 2026cs.CV

Point4D: Long-range 4D Motion Reconstruction

We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io

Minsik Jeon, Jay Karhade, Deva Ramanan +1

Carnegie Mellon University

4D Reconstruction 3D Tracking Four-Dimensional Long-Video Benchmarks Motion

Ask PDF

Sep 1, 2026cs.CV

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

Multi-camera systems are increasingly practical for robotics, AR/VR, and autonomous driving because complementary views reduce depth ambiguity and preserve visibility under occlusion. Existing point-tracking benchmarks, however, focus on a single video or static multi-camera rigs. None test long-term 3D point tracking across several synchronized views under camera motion. We introduce TAPVid-MV (Tracking Any Point in Video across Multiple Views), the first benchmark for this setting. It contains a curated set of 284 sequences, 1,142 calibrated camera streams, and 109,769 point tracks across seven subsets spanning indoor and outdoor domains, from robotics and human activity to driving and synthetic procedural scenes. We obtain these trajectories using dataset-specific auxiliary modalities: sensor depth, LiDAR, SLAM and SfM points, human meshes, posed object meshes, and simulation. Every sequence and trajectory is visually verified by human annotators. Across more than 30 baselines, no method comes close to solving the task. Surprisingly, existing multi-view point trackers do not consistently outperform monocular point trackers. By evaluating reconstruction and point tracking on the same datasets, TAPVid-MV helps distinguish errors in recovered geometry from errors in point correspondence. Through this joint analysis, we identify geometry recovery as a major bottleneck for accurate 3D point tracking. Beyond multi-view 3D point tracking, our released annotations support monocular 2D and 3D point tracking, future-trajectory prediction, and 4D reconstruction.

Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman +9

1Google DeepMind · University College London · 3ETH Zürich +1

3D Tracking Multi-View Camera 3D Generation

Ask PDF