UniQuery4R: Unified 4D Scene Reconstruction from a Single Query

Tiancheng Chen; Sheng Tang; Wenhua Jin; Weiqi Zhang; Juntong Fang; Junsheng Zhou; Zesong Li

cs.CVAug 18, 2026

UniQuery4R: Unified 4D Scene Reconstruction from a Single Query

Authors: Tiancheng Chen, Sheng Tang, Wenhua Jin, Weiqi Zhang, Juntong Fang, Junsheng Zhou, Zesong Li

Organizations: Kosmo Research · Automotive Engineering Department, Jilin University · School of Software, Tsinghua University

Abstract

Reconstructing dynamic 4D scenes requires jointly estimating correspondence, geometry, object motion, and camera motion. Existing feed-forward methods typically predict dense task-specific maps or independently process source-target pairs, leading to unnecessary computation for sparse queries and limited feature reuse across different frame pairs. We present UniQuery4R, a query-conditioned framework that encodes a multi-frame clip once and selects the source view, target view, and continuous source-image coordinate only at decoding time via source-to-target cross-attention. Each query jointly predicts target correspondence, target-time 3D position, and scene flow, along with source depth, while camera parameters are estimated per view. This design allows the encoded clip to be reused across arbitrary source-target selections and supports both sparse inference and dense reconstruction through batched queries, without learned temporal embeddings tied to a fixed clip length. We further introduce a direction-magnitude parameterization of scene flow with separate supervision for moving and static points. Among the evaluated methods, UniQuery4R achieves the best macro-average results on WorldTrack for both scene-flow estimation and dynamic-point reconstruction.

Figures & tables

Figure 1: Conceptual overview. A continuous source-pixel query over a jointly encoded clip predicts target correspondence, target-time 3D position, scene flow, and source depth, while camera parameters are estimated per view.

Method	Input Clip Length	Query Source	Key / Value	Supported Output
V-DPM	Variable	N/A	N/A	Dense
D4RT	Fixed (48 frames)	Input embedding	All-frame patches	Sparse / Dense
UniQuery4R (ours)	Variable	Sampled from $F_{s}$	$F_{t}$ patches	Sparse / Dense

Table 1: Paradigm comparison with D4RT and V-DPM. UniQuery4R differs in temporal horizon, query formation, and target-side attention. “Variable” means no learned fixed clip length (memory-bounded in practice).

Figure 2: (a) Overall architecture. A clip is patch-embedded with camera tokens and jointly encoded by frame/global attention. At decoding time, $\mathbf{q}=(u,v,s,t)$ selects source/target; the multi-scale query decoder yields $\mathbf{q}_{s}$ and target features $\mathbf{M}_{t}$ , then $\mathbf{q}_{s}$ cross-attends to $\mathbf{M}_{t}$ ( $\times 4$ ) to form $\mathbf{q}_{c}$ . Src $\to$ Tgt MLPs predict $\mathbf{P}$ , $\Delta\mathbf{P}$ , and $\mathbf{f}$ from $\mathbf{q}_{c}$ ; a Src-only MLP predicts local depth from $\mathbf{q}_{s}$ ; a camera head reads per-view tokens. (b) Multi-scale query decoder with gated fusion that builds $\mathbf{q}_{s}$ from the source pyramid.

Output	Sym.	Q?	Description
Local, per-query — from $\mathbf{q}_{c}$
Dynamic point	$\mathbf{P}\!\in\!\mathbb{R}^{3}$	Y	target-time pos. (ref.-cam.)
Scene flow	$\Delta\mathbf{P}\!\in\!\mathbb{R}^{3}$	Y	$s\!\to\!t$ displ. (ref.-cam.)
2D corresp.	$\mathbf{f}\!\in\!\mathbb{R}^{2}$	Y	target-image coordinate
Conf./prec.	$(c_{P},\mathbf{c}_{f})$	Y	point and correspondence
Local, per-query — from $\mathbf{q}_{s}$

Table 2: UniQuery4R outputs for query

\mathbf{q}=(u,v,s,t)

. “Q?” denotes query dependence; camera parameters are predicted per view.

Method	ADT					NRGBD					Sintel
	A 5 $\uparrow$	A 30 $\uparrow$	ATE $\downarrow$	R t $\downarrow$	R r $\downarrow$	A 5 $\uparrow$	A 30 $\uparrow$	ATE $\downarrow$	R t $\downarrow$	R r $\downarrow$	A 5 $\uparrow$	A 30 $\uparrow$	ATE $\downarrow$	R t $\downarrow$	R r $\downarrow$
VGGT ( Wang et al. 2025 )	0.134	0.724	0.219	0.269	3.457	0.676	0.926	0.047	0.071	0.994	0.212	0.475	0.100	0.050	0.588
DA3-Giant ( Lin et al. 2026b )	0.145	0.741	0.160	0.223	2.221	0.772	0.944	0.034	0.045	0.848	0.214	0.495	0.065	0.044	0.373
VGGT- $\Omega$ ( Wang et al. 2026a )	0.446	0.851	0.087	0.091	2.107	0.704	0.933	0.040	0.059	1.008	0.286	0.583	0.028	0.022	0.230
Any4D ( Karhade et al. 2026 )	0.070	0.597	0.346	0.432	4.566	0.348	0.833	0.105	0.163	2.247	0.034	0.290	0.144	0.104	2.226
V-DPM ( Sucar et al. 2026 )	0.400	0.861	0.076	0.103	2.374	0.664	0.925	0.045	0.066	1.113	0.175	0.505	0.082	0.054	0.520

Table 5: Camera pose (A 5 /A 30 : AUC@5/@30) and trajectory (ATE; R t /R r : RPE t /RPE r ) on ADT, NRGBD, and Sintel. Best bolded; second-best underlined.

Output	From	Ch.	Act.	Notes
$\mathbf{P}$	$\mathbf{q}_{c}$	3	inv-log	target-time 3D
$c_{P}$	$\mathbf{q}_{c}$	1	—	logit
$\tilde{\mathbf{v}},\tilde{m}$	$\mathbf{q}_{c}$	$3{+}1$	dir.–mag.	$\to\Delta\mathbf{P}$
$\mathbf{f}$	$\mathbf{q}_{c}$	2	identity	abs. UV
$\mathbf{c}_{f}$	$\mathbf{q}_{c}$	4	—	logits
$d,c_{d}$	$\mathbf{q}_{s}$	$1{+}1$	$\exp$ /—	source depth

Table A.1: Query prediction heads.

Term	Default	Dataset overrides
Source depth $\mathcal{L}_{d}$	$1$	DL3DV/Waymo: $0$ ; VKITTI2/Stereo4D: $0$
Target point $\mathcal{L}_{P}$	$10$	DL3DV: $.1$ ; Waymo/VKITTI2: $0$ ; Stereo4D: $.1$
Displacement $\mathcal{L}_{\Delta}$	$50$	Waymo/VKITTI2/Stereo4D: $30$
Direction/static magnitude	$1$	none
2D correspondence $\mathcal{L}_{f}$	$5$	PointOdyssey: $0$
Camera $\mathcal{L}_{\pi}$	$1$	outer multiplier $10$

Table A.2: Loss weights used to train the final model. Camera translation, quaternion, and FoV components have internal weights

1

10

, and

0.5

, respectively. The four refinement predictions are weighted by

\{0.6^{3},0.6^{2},0.6,1\}

and then averaged.

Source	#samples	Weight
PointOdyssey	111	0.4
Kubric-4D	12,928	1.0
CoTracker3Kubric	5,827	1.0
Internal dataset	2,596	1.0
Hypersim	758	0.1
DL3DV	54,903	0.1

Table A.3: Training mixture (indexed samples and relative weights).

Setting	Value
Hardware	$16\times$ NVIDIA H20
Iterations	$150$ k
Numerical precision	bfloat16
Batch size (per GPU / global)	$1$ / $16$
Backbone learning rate	$10^{-5}$
Other-module learning rate	$10^{-4}$

Table A.4: Optimization hyperparameters for the final model.

Encode vs. clip length		Encode vs. resolution		Encode/decode vs. query count
$T$	Enc. (ms)		$H{\times}W$	Enc. (ms)	Mode	$Q$ (chunks)	Enc. (ms)	Query dec. (ms)
2	$118.90{\pm}14.48$		$252^{2}$	$107.62{\pm}13.36$	S	256 (1)	357.79	$5.36{\pm}0.10$
4	$200.06{\pm}10.73$		$364^{2}$	$197.37{\pm}5.29$	S	1,024 (1)	359.38	$8.33{\pm}0.05$
8	$359.76{\pm}8.92$		$504^{2}$	$359.76{\pm}8.92$	S	4,096 (1)	358.54	$21.80{\pm}1.15$
12	$544.36{\pm}4.79$		$518^{2}$	$356.40{\pm}6.90$	S	8,192 (1)	361.27	$36.85{\pm}0.32$
16	$765.93{\pm}6.41$				S	16,384 (1)	366.20	$69.66{\pm}0.66$

Table B.1: Measured runtime points underlying the scaling results in the main paper. The clip-length sweep fixes

H=W=504

and

Q=4096

; the resolution sweep fixes

T=8

and

Q=4096

. The query sweep uses

T=8

; S denotes sparse decoding at

504^{2}

, while D denotes dense decoding at the indicated resolution. Parentheses report the number of chunks with a maximum chunk size of 30k. Results use FP16 on one NVIDIA A800 with five warm-up runs, 20 measured repeats, and CUDA synchronization.

Figure B.1: First-layer source-to-target cross-attention over increasing temporal gaps. Each row fixes a source query and visualizes its normalized attention on progressively more distant target frames.

Encoder initialization	APD $\uparrow$	EPE $\downarrow$
VGGT- $\Omega$ ( Wang et al. 2026a )	80.88	0.1857
DA3-Giant ( Lin et al. 2026b )	81.29	0.1821
Internal foundation model	83.62	0.1597

Table B.2: WorldTrack dynamic-point macro-average under matched training (

150

k iterations,

8{\times}

A800) with different encoder initializations.

Method	EPE $\downarrow$
SpaTrackerV2 ( Xiao et al. 2025 )	5.5187
St4RTrack ( Feng et al. 2025 )	3.8485
TraceAnything ( Liu et al. 2026 )	7.3385
Any4D ( Karhade et al. 2026 )	5.2362
V-DPM ( Sucar et al. 2026 )	2.7367
4RC ( Luo et al. 2026 )	3.2957

Table B.3: Dynamic-point EPE (m) on SynthVerse (

50

sequences

\times

64

frames; WorldTrack protocol). Zero-shot evaluation excludes SynthVerse training; the last row uses the official training split with benchmark sequences held out.

Figure C.1: Additional 3D tracking on diverse dynamic scenes. Each color denotes one fixed source query over time.

Figure C.2: Qualitative 2D correspondence for selected source–target pairs.

Figure C.3: Motion masks from thresholded consecutive scene flow ( $\gamma{=}0.02$ m).

Figure C.4: Static-scene reconstruction with estimated cameras.

Explore similar work

Depth Anything V4: Dynamic 4D Scene Reconstruction via Riemannian Flow Matching on 4D Gaussian Splatting
Aug 18, 2026Jiaming Fan, Jian Lu, Jinling Jia +14D Reconstruction Dynamic Scene Reconstruction
College of Artificial Intelligence, Nanjing University of Posts and Telecommunications · College of Automation, Nanjing University of Posts and Telecommunications · Institute of Ad-vanced Technology for Carbon Neutrality, Nanjing University of Posts and Telecommunications
OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields
Jul 12, 2026Yanqin Jiang, Tengfei Wang, Zhengwei Wang +63D Point Tracking 4D Reconstruction
MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent Hunyuan +3
Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction
May 29, 2026Mungyeom Kim, Minkyeong Jeon, Honggyu An +10Point Tracking 3D Gaussian Splatting
KAIST AI · ETH Zürich, ETH AI Center · Sony AI +1

Aug 18, 2026J/K move · Enter open · S save

Depth Anything V4: Dynamic 4D Scene Reconstruction via Riemannian Flow Matching on 4D Gaussian Splatting

Jiaming Fan, Jian Lu, Jinling Jia, Chenbin Zhang

College of Artificial Intelligence, Nanjing University of Posts and Telecommunications · College of Automation, Nanjing University of Posts and Telecommunications · Institute of Ad-vanced Technology for Carbon Neutrality, Nanjing University of Posts and Telecommunications

4D Reconstruction Dynamic Scene Reconstruction Monocular 3D Reconstruction 3D Gaussian Splatting Novel View Synthesis Riemannian Flow Matching

Open paper AskPDF arXiv

We present Depth Anything V4 (DAV4), a framework for dynamic 4D scene reconstruction from monocular video. Our key contribution is the application of Riemannian Flow Matching (RFM) to 4D Gaussian Splatting parameters, defining probability paths directly on non-Euclidean manifolds (scale, rotation, opacity), ensuring all intermediate states are valid. Through controlled experiments, we isolate RFM's contribution from test-time optimization (TTO) and pre-training. A deterministic MLP baseline with the same data, architecture, and TTO achieves F-score 0.762; RFM achieves 0.806 - the +0.044 gain is RFM's isolated contribution. We provide corrected computational cost analysis: pre-training is 360 GPU-hours, amortizing for large-scale deployment (over 10,000 scenes). Uncertainty is quantified via Negative Gaussian Log-Likelihood and Expected Calibration Error. DAV4 outperforms prior Depth Anything models and per-scene 4D-GS on dynamic reconstruction and novel-view synthesis, while using no human-annotated depth labels as training losses.

Abstract

Explore similar work

Depth Anything V4: Dynamic 4D Scene Reconstruction via Riemannian Flow Matching on 4D Gaussian Splatting

OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields

Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction