ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild

Ilya Fradlin; Christian Schmidt; Jens Piekenbrinck; Karim Knaebel; Gonzalo Martin Garcia; Bastian Leibe

cs.CVOct 1, 2026

ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild

Authors: Ilya Fradlin, Christian Schmidt, Jens Piekenbrinck, Karim Knaebel, Gonzalo Martin Garcia, Bastian Leibe

Organizations: RWTH Aachen University

Abstract

Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary aspects of scene geometry and motion, yet bringing them together requires establishing correspondence across viewpoints, capture times, and visibility changes. We introduce ARROW, a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. At its core is a novel order-invariant querying approach, which allows the association of queries with observations across arbitrary inputs. We show that exposing the model to more diverse sets of inputs during training results in improved task performance. Moreover, the resulting model is capable of generalization to a wider range of tasks including multi-view tracking. Trained with this strategy, ARROW establishes a new state of the art in 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, while remaining competitive across 3D reconstruction tasks. Code and weights are publicly available.

Figures & tables

Figure 1: ARROW reconstructs 4D scenes over arbitrary sets of input images. It sets a new state of the art in 3D point tracking & multi-view tracking and performs on par in 3D reconstruction.

Panel of Figure 2 — Figure 2: From indexed to input-dependent correspondence. (a) Order-aware global context enables position-based associations. (b) Order-invariant global context exposes no fixed order to exploit. (c) Content-derived identity embeddings enable order-invariant associations.

Figure 3: ARROW architecture. The multi-view encoder ${\mathcal{E}}$ tokenizes each frame independently and employs global attention to produce a shared global context ${\bm{F}}$ together with an ID token ${\bm{h}}_{i}$ for each frame. The query encoder combines a source-centered RGB crop, the Fourier-encoded $(u,v)$ coordinate, and the ID tokens of the queried source, target, and reference observations into a latent query token ${\bm{z}}_{{\bm{q}}}$ . Each latent query token independently cross-attends to the global context, so a single encoding serves arbitrarily many queries and yields a consistent 4D representation of the scene.

Panel of Figure 4 — Figure 4: 4D reconstruction and tracking. V-DPM, 4RC, OmniX, and ARROW on four DAVIS ( Perazzi et al., 2016 ) sequences. Input frames are on the left; each panel shows its 3D reconstruction with trajectory overlays. ARROW shows clean reconstructions and consistent long-term tracks.

	Aria Digital Twin				Dynamic Replica				PointOdyssey				Panoptic Studio
Method	APD $\uparrow$		EPE $\downarrow$		APD $\uparrow$		EPE $\downarrow$		APD $\uparrow$		EPE $\downarrow$		APD $\uparrow$		EPE $\downarrow$
	All	Dyn.	All	Dyn.	All	Dyn.	All	Dyn.	All	Dyn.	All	Dyn.	All	Dyn.	All	Dyn.
SpatialTrackerV2	91.75	$83.18$	0.0964	$0.0832$	$69.90$	$67.18$	$0.2986$	$0.3348$	$67.36$	$68.03$	$0.4299$	$0.4789$	$73.42$	$73.42$	$0.2397$	$0.2397$
St4RTrack	$72.89$	$77.98$	$0.2958$	$0.1087$	$74.29$	$76.85$	$0.2600$	$0.1973$	$67.45$	$69.89$	$0.3144$	$0.2722$	$70.79$	$70.79$	$0.2480$	$0.2480$
Trace Anything	$75.71$	$76.07$	$0.2517$	$0.1875$	$60.64$	$60.58$	$0.5751$	$0.4730$	$39.51$	$43.70$	$1.0546$	$0.8906$	$71.17$	$71.17$	$0.2788$	$0.2788$
V-DPM	$85.35$	$73.27$	$0.1763$	$0.1455$	$77.28$	$80.13$	$0.2306$	$0.1904$	$80.47$	$82.59$	0.1886	0.1672	$77.11$	$77.11$	$0.1881$	$0.1881$

Table 1: WorldTrack 3D point tracking. We report the average percentage of points within distance thresholds (APD, %) and mean 3D end-point error (EPE, meters). All and Dyn. denote overall and dynamic-only scores. Results use global median scale alignment. ARROW sets a new state of the art.

	Camera-coordinate 3D tracking												World-coordinate 3D tracking
Method	DriveTrack			Aria Digital Twin			Panoptic Studio			Avg.			DriveTrack		ADT		Avg.
	AJ $\uparrow$	APD $\uparrow$	OA $\uparrow$	AJ $\uparrow$	APD $\uparrow$	OA $\uparrow$	AJ $\uparrow$	APD $\uparrow$	OA $\uparrow$	AJ $\uparrow$	APD $\uparrow$	OA $\uparrow$	APD $\uparrow$	L1 $\downarrow$	APD $\uparrow$	L1 $\downarrow$	APD $\uparrow$	L1 $\downarrow$
St4RTrack	—	—	—	—	—	—	—	—	—	—	—	—	$0.020$	$0.499$	$0.062$	$0.839$	$0.041$	$0.669$
CoTracker3 + UD2	$0.038$	$0.062$	0.856	$0.102$	$0.146$	0.937	0.119	0.176	0.862	$0.086$	$0.128$	0.885	—	—	—	—	—	—
CoTracker3 + VGGT	0.129	0.189	0.856	$0.132$	$0.190$	0.937	$0.045$	$0.070$	0.862	$0.102$	$0.150$	0.885	0.205	$0.170$	$0.192$	$0.736$	$0.199$	$0.453$
SpatialTrackerV2	$0.064$	$0.100$	0.865	0.260	0.342	0.936	$0.097$	$0.152$	$0.805$	0.140	0.198	0.869	$0.101$	0.139	0.336	0.238	0.219	0.189

Table 2: TAPVid-3D evaluation in camera and world coordinates. Without ground-truth intrinsics. L1 is measured after normalizing predictions and ground truth by their median distances to the reference camera. Baseline results are taken from D4RT ( Zhang et al., 2026 ) . UD2 denotes UniDepthV2 ( Piccinelli et al., 2026 ) . ARROW leads in average AJ, APD, and normalized L1.

Method	Panoptic Studio				DexYCB				Multi-View Kubric				Avg.
	APD $\uparrow$	EPE $\downarrow$	OA $\uparrow$	AJ $\uparrow$	APD $\uparrow$	EPE $\downarrow$	OA $\uparrow$	AJ $\uparrow$	APD $\uparrow$	EPE $\downarrow$	OA $\uparrow$	AJ $\uparrow$	APD $\uparrow$	EPE $\downarrow$	OA $\uparrow$	AJ $\uparrow$
MVTracker + VGGT	$54.6$	$0.2369$	$69.5$	$37.0$	$39.7$	0.1063	$82.6$	$30.0$	$31.7$	$0.3070$	$72.4$	$21.2$	$42.0$	$0.2167$	$74.8$	$29.4$
MVTracker + DA3	59.9	$0.2054$	$69.1$	39.1	$32.4$	$0.1732$	$80.8$	$23.0$	43.6	0.1595	76.4	30.4	45.3	0.1793	75.4	30.8
MVTracker + VGGT- $\Omega$ *	$49.1$	$0.2923$	$63.6$	$28.5$	49.3	0.1057	$78.3$	39.9	$28.4$	$0.2279$	$72.5$	$18.0$	$42.3$	$0.2087$	$71.5$	$28.8$
LAPA + VGGT- $\Omega$ *	$41.4$	$0.3668$	85.8	$31.0$	$31.3$	$0.1543$	85.2	$24.4$	$13.6$	$0.3954$	$51.6$	$7.0$	$28.8$	$0.3055$	$74.2$	$20.8$
TAPIP3D + VGGT- $\Omega$ *	$50.1$	$0.2835$	87.6	38.8	50.2	0.0991	88.6	38.6	32.4	0.2002	83.5	21.8	$44.3$	$0.1943$	86.6	33.1

Table 3: RGB-only multi-view 3D tracking. We report APD and EPE on ground-truth-visible positions, as well as OA and AJ.

+

denotes methods requiring an external reconstruction model.

	Camera pose									3D reconstruction
Method	Bonn			Sintel			Panoptic Studio $\dagger$			NRGBD		7-Scenes
	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	Chamfer $\downarrow$	NC $\uparrow$	Chamfer $\downarrow$	NC $\uparrow$
VGGT	$0.0151$	$0.0066$	$0.223$	$0.1657$	$0.0512$	$0.484$	$0.4310$	$0.9849$	$16.914$	$0.0429$	$0.910$	$0.0538$	0.754
$\pi^{3}$	$0.0114$	$0.0063$	$0.225$	0.0735	$0.0417$	0.263	$0.1506$	$0.3755$	$5.663$	$0.0300$	$0.910$	$0.0721$	$0.736$
VGGT- $\Omega$ *	0.0081	$0.0041$	0.213	$0.0777$	0.0305	0.195	$0.3195$	$0.6475$	$8.947$	$0.0354$	$0.895$	0.0411	0.754
DA3	0.0088	$0.0047$	0.212	$0.1287$	$0.0531$	$0.357$	$0.2084$	$0.5030$	$6.037$	0.0287	0.927	$0.0598$	$0.752$

Table 4: Pose estimation and 3D reconstruction. We compare against dedicated 3D reconstruction models (top) and 4D reconstruction and tracking approaches (bottom). Although ARROW is not solely trained for pose prediction or 3D reconstruction, it still achieves competitive results.

Panel of Table 6 — Table 6: Cumulative ablations. S5 is the $100\mathrm{k}$ -step DA3-Large ablation model and reports absolute scores; other rows report signed differences from S5. APD, AUC, and F1 differences are in percentage points. : order-dependent querying (D4RT); : order-invariant querying (ARROW). “video” comprises the datasets in Table 7 excluding Hypersim, ScanNet++, BlendedMVS, and Structured3D.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Scene type	Tracks	Motion mask	Multi- camera	# Scenes	# Frames	Weight
Syn4D ( Jiang et al., 2026b )	dynamic	✓	✓	✓	$504$	1.2M	$13.50$
Dynamic Replica ( Karaev et al., 2023 )	dynamic	✓	✓	✓	$503$	151K	$12.50$
Stereo4D ( Jin et al., 2025 )	dynamic	✓	✗	✗	$31,420$	6M	$11.50$
PointOdyssey ( Zheng et al., 2023 )	dynamic	✓	✓	✗	$123$	238K	$10.50$
Kubric-4D ( Van Hoorick et al., 2024 )	dynamic	✓	✓	✓	$2,992$	2.9M	$8.00$
CoTracker3-Kubric ( Karaev et al., 2025 )	dynamic	✓	✓	✗	$5,837$	700K	$6.00$

Appendix

Table 7: ARROW training mixture. “Multi-camera” denotes datasets with multiple video streams per scene. “Weight” denotes the probability to sample a scene from this dataset during training.

Dataset	Image sampling		Query sampling
	Policy	Local window	Track (%)	B (%)	D (%)	G (%)	Geometry
Syn4D	Mixed	$\times 10$	$80$	$25$	$40$	$35$	Edges
Dynamic Replica	Local	$\times 10$	$80$	$25$	$40$	$35$	Edges
Stereo4D	Local	$\times 4$	$100$	—	—	—	—
PointOdyssey, standard	Local	$\times 10$	$70$	$25$	$40$	$35$	Edges
flat background	Local	$\times 10$	$80$	$35$	$55$	$10$	Uniform

Appendix

Table 8: Dataset-specific sampling. We sample frames based on temporal locality, covisibility, or randomly from all available observations. “Mixed” indicates sampling from multiple camera streams. Where 3D tracks are available, we use them to construct a large percentage of all queries. The remaining query budget is divided into queries constructed from motion boundaries (B), dynamic regions (D), or based on geometric clues (G). Geometry queries are based on depth and normal discontinuities or uniformly sampled over the image plane.

Panel of Figure 6 — Figure 6: Geometric query sampling. On static scenes, we oversample queries near depth and normal discontinuities. From left to right: the RGB of a sample frame; the detected edges; the sampled source $(u,v)$ coordinates. The fraction of queries constructed this way varies by dataset (see Table 8 ).

Panel of Figure 7 — Figure 7: Motion boundary oversampling. (a) Oversampling near motion boundaries encourages the model to distinguish static and moving points. Blue marks indicate queries sampled from 3D tracks, and orange points indicate queries oversampled near motion boundaries. (b) Qualitative comparison of the impact of oversampled motion boundaries. Left: RGB input; middle: tracks without motion boundary oversampling; right: tracks with motion boundary oversampling. The fraction of queries constructed this way varies by dataset (see Table 8 ).

Configuration	Aria Digital Twin		Dynamic Replica		PointOdyssey		Panoptic Studio		Avg.
	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$
Frozen encoder	-6.85	+0.0612	-8.20	+0.0706	-4.39	+0.0851	-15.41	+0.1487	-8.71	+0.0914
Multi-level context	+0.33	-0.0013	-0.33	-0.0018	-0.21	+0.0042	+1.32	-0.0088	+0.28	-0.0019
Patch-feature query initialization	-0.59	+0.0072	-2.59	+0.0311	-0.33	+0.0270	-2.06	+0.0166	-1.39	+0.0205
Stage S5 ( Table 6 )	$87.52$	$0.1495$	81.01	0.2026	74.36	0.2519	77.13	0.1894	$80.01$	$0.1983$
S5 + EMA	+1.37	-0.0110	-0.23	+0.0046	+0.26	-0.0049	-0.09	+0.0060	+0.33	-0.0013

Appendix

Table 9: Design-choice ablations on WorldTrack. All rows use the

100\mathrm{k}

-step DA3-Large ablation setup of Table 6 . Entries report APD (%) and EPE (m) over all points. S5 reports absolute scores; other rows report signed differences from S5. APD differences are in percentage points. The first three variants are trained separately with the indicated change. The EMA row evaluates exponential-moving-average weights from the S5 training run.

Table 10: Short-context ablations on WorldTrack. We use

24

-frame full-context inference for all stages. Stages use the

100\mathrm{k}

-step DA3-Large setup of Table 6 . : order-dependent querying (D4RT); : order-invariant querying (ARROW). Entries report APD (%) and EPE (m) over all points. S5 reports absolute scores; other rows report signed differences from S5. APD differences are in percentage points.

Figure 8: Long tracking performance. All-point APD over the indicated sequence prefix with full-context inference (solid) or

48

-frame windows with

8

-frame overlap (dashed). Full-context inference performs better than windowed inference for all methods except OpenD4RT. ARROW outperforms other methods under matched scene size and inference method. Evaluations run on NVIDIA GH200 GPUs. V-DPM evaluation on 200 frames did not finish within 80 GPU-hours. Other missing points ran out of memory.

Panel of Figure 9 — Figure 9: Geometry and annotation issues in PointOdyssey. From left to right: objects floating above the ground instead of resting on it, a background image mapped onto a flat plane rather than modeled scene geometry, and depth annotation artifacts that produce fragmented geometry. Each example shows a 3D visualization and the corresponding RGB image (bottom left) and depth map (bottom right). These corrupted cases motivate our manual scene review, scene-specific sampling rules, and exclusion of unreliable supervision.

Panel of Figure 10 — Figure 10: Video depth comparison. Top: RGB frames in temporal order, with the selected frame outlined. Middle: the selected RGB frame and depth predictions. Bottom: matching crops of the tree trunk (A) and the bench (B), marked by dashed boxes in the full views. ARROW preserves fine structural details in both highlighted regions.

Panel of Figure 11 — Figure 11: Qualitative UV tracking. ARROW predicts image coordinates $(\hat{u},\hat{v})$ and visibility for the swing (top) and breakdance (bottom) sequences. A circled dot ( $\odot$ ) marks each source query in its source frame; all other markers are predictions. Filled dots ( $\bullet$ ) indicate visible points and hollow circles ( $\circ$ ) indicate occluded points. Colors identify corresponding points, and curves connect their locations across the displayed frames. These examples illustrate occlusion handling, tracking through challenging articulated motion, and consistent tracking of static scene points under camera motion.

Panel of Figure 12 — Figure 12: 3D reconstruction. Multi-view (top two rows) and monocular (bottom two rows) 3D reconstruction results compared with state-of-the-art methods.

Panel of Figure 13 — Figure 13: Multi-view 3D tracking. Examples from Panoptic Studio (top) and Multi-View Kubric (bottom). The three input columns on the left show successive frames from three camera views.

Explore similar work

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views
Sep 1, 2026Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman +93D Tracking Multi-View
1Google DeepMind · University College London · 3ETH Zürich +1
Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes
Jun 29, 2026Xi Li, Linyuan Li, Yan Wu +43D Reconstruction Local Camera Pose
Realsee, China · Quanzhou University of Information Engineering, China
UniQueR: Unified Query-based Feedforward 3D Reconstruction
Mar 24, 2026Chensheng Peng, Quentin Herau, Jiezhi Yang +6Feed-Forward 3D Reconstruction Differentiable Rendering
Applied Intuition · UC Berkeley · Stanford University

Sep 1, 2026, cs.CVJ/K move · Enter open · S save

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman, Yi Yang+8

1Google DeepMind · University College London · 3ETH Zürich+1

3D Tracking Multi-View Camera 3D Generation

Open paper AskPDF arXiv

Multi-camera systems are increasingly practical for robotics, AR/VR, and autonomous driving because complementary views reduce depth ambiguity and preserve visibility under occlusion. Existing point-tracking benchmarks, however, focus on a single video or static multi-camera rigs. None test long-term 3D point tracking across several synchronized views under camera motion. We introduce TAPVid-MV (Tracking Any Point in Video across Multiple Views), the first benchmark for this setting. It contains a curated set of 284 sequences, 1,142 calibrated camera streams, and 109,769 point tracks across seven subsets spanning indoor and outdoor domains, from robotics and human activity to driving and synthetic procedural scenes. We obtain these trajectories using dataset-specific auxiliary modalities: sensor depth, LiDAR, SLAM and SfM points, human meshes, posed object meshes, and simulation. Every sequence and trajectory is visually verified by human annotators. Across more than 30 baselines, no method comes close to solving the task. Surprisingly, existing multi-view point trackers do not consistently outperform monocular point trackers. By evaluating reconstruction and point tracking on the same datasets, TAPVid-MV helps distinguish errors in recovered geometry from errors in point correspondence. Through this joint analysis, we identify geometry recovery as a major bottleneck for accurate 3D point tracking. Beyond multi-view 3D point tracking, our released annotations support monocular 2D and 3D point tracking, future-trajectory prediction, and 4D reconstruction.

Abstract

Explore similar work

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes

UniQueR: Unified Query-based Feedforward 3D Reconstruction