Gen4P: Video Generators are 4D Perceivers

Yitong Xia; Zeren Jiang; Chuanxia Zheng

cs.CVOct 15, 2026

Gen4P: Video Generators are 4D Perceivers

Authors: Yitong Xia, Zeren Jiang, Chuanxia Zheng

Organizations: PVG, Nanyang Technological University · VGG, University of Oxford

Abstract

We introduce Gen4P, a unified framework for recovering 4D geometry and motion from monocular video. We show that a pretrained video generator can be repurposed into a feed-forward 4D perceiver without iterative denoising. Building on on-demand querying, Gen4P processes clean video latents in a single encoder pass and predicts the requested 4D attributes through a lightweight cross-attention decoder. The on-demand query formulation enables flexible spatial sampling, and our proposed continuous-time queries further provide temporal flexibility. To strengthen geometric knowledge, we propose a straightforward plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior, improving overall performance without adding extra trainable parameters. Gen4P achieves state-of-the-art or competitive performance on tasks of 3D point tracking, video depth and camera pose estimation. Qualitative results on in-the-wild videos illustrate its capability of tracking complex motion while maintaining the delicate structure. Together, these results demonstrate the potential of video generative pretraining for unified 4D perception. See https://gen4p.github.io for more visual results.

Figures & tables

Figure 1: Gen4P repurposes a multi-step Video Gen erator into a single-step 4 D P erceiver, enabling unified feed-forward prediction of dense 3D tracks, point maps, depth maps, and camera poses from a video input. Gen4P is able to track complex motion while maintaining the delicate structure of in-the-wild content, like the thin legs of the flamingo and long-range skiing motion.

Figure 2: Overview of Gen4P. The input video is encoded into scene tokens $\mathbf{F}$ in a single pass by the Video Generative Encoder , which consists of a frozen VAE encoder and DiT, an optional frozen Depth-LoRA, and a trainable 4D-LoRA. The cross-attention Query Decoder is queried independently for each query $\mathbf{q}(u,v,t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}})$ , where the query timestamps are parameterized as continuous time embeddings. The decoder outputs the corresponding 3D and 2D point attributes, enabling the full construction of 3D tracks, depth maps and camera poses.

Panel of Figure 3 — Figure 3: Noise Level – Tracking.

Figure 4: Qualitative Results of 3D Tracking across different methods, consisting of both static and dynamic content. We colorize the dynamic foreground tracklines.

Method	ADT		DR		PO		PStudio
	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$
	All / Dyn.	All / Dyn.	All / Dyn.	All / Dyn.	All / Dyn.	All / Dyn.	All	All
MonST3R ( Zhang et al., 2025a )	73.58 / 82.17	0.326 / 0.176	53.79 / 52.87	0.514 / 0.547	32.95 / 35.02	0.902 / 1.095	58.14	0.454
STv2 ( Xiao et al., 2025 )	91.89 / 84.44	0.095 / 0.143	70.71 / 68.52	0.287 / 0.333	69.48 / 65.35	0.388 / 0.500	73.52	0.237
St4RTrack ( Feng et al., 2025 )	73.02 / 78.01	0.294 / 0.209	74.34 / 77.35	0.260 / 0.207	67.34 / 72.03	0.317 / 0.247	70.67	0.249
Any4D ( Karhade et al., 2026 )	58.47 / 69.73	0.408 / 0.271	79.14 / 70.31	0.217 / 0.309	67.28 / 68.69	0.347 / 0.366	64.31	0.296

Table 1: Quantitative Results on World-Coordinate Tracking. We report the standard Average Percent of Points within Delta (APD

~\uparrow

) and End-Point Error (EPE

~\downarrow

) for All points and only Dynamic points in separate groups. Each group is aligned to the GT tracks using a global median alignment over 64 frames. We highlight the best , second and third best results.

Method	Bonn		Sintel		KITTI
Method	AbsRel $\downarrow$	$\delta_{<1.25}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{<1.25}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{<1.25}$ $\uparrow$
MonST3R ( Zhang et al., 2025a )	0.062	96.36	0.300	58.01	0.089	91.11
Spann3R ( Wang and Agapito, 2025 )	0.139	83.21	0.306	68.14	0.114	89.49
VGGT ( Wang et al., 2025a )	0.049	97.40	0.211	72.55	0.059	96.15
DepthAnythingv3 ( Lin et al., 2025 )	0.049	97.28	0.200	70.80	0.043	97.98
MapAnything ( Keetha et al., 2026 )	0.058	96.00	0.293	69.70	0.057	97.28

Table 2: Quantitative Results on Video Depth Estimation. Results are reported for three unseen datasets: Bonn, Sintel, and KITTI. We highlight the best , second and third best results.

Method	Sintel			TUM-Dynamics			ScanNetv2
Method	ATE $\downarrow$	$\mathrm{RPE_{T}}\downarrow$	$\mathrm{RPE_{R}}\downarrow$	ATE $\downarrow$	$\mathrm{RPE_{T}}\downarrow$	$\mathrm{RPE_{R}}\downarrow$	ATE $\downarrow$	$\mathrm{RPE_{T}}\downarrow$	$\mathrm{RPE_{R}}\downarrow$
VGGT ( Wang et al., 2025a )	0.172	0.061	0.471	0.012	0.010	0.309	0.034	0.015	0.376
DepthAnythingv3 ( Lin et al., 2025 )	0.122	0.043	0.348	0.012	0.008	0.310	0.032	0.012	0.355
MapAnything ( Keetha et al., 2026 )	0.218	0.075	0.655	0.028	0.023	0.372	0.051	0.024	0.719
Any4D ( Karhade et al., 2026 )	0.212	0.121	2.650	0.044	0.023	0.439	0.063	0.028	1.061
V-DPM ( Sucar et al., 2026 )	0.139	0.053	0.454	0.023	0.010	0.321	0.035	0.014	0.394

Table 3: Quantitative Results on Camera Pose Estimation. We report the standard Absolute Translation Error (

\mathrm{ATE}\downarrow

), Relative Pose Error-Translation (

\mathrm{RPE_{T}}\downarrow

), and Relative Pose Error-Rotation (

\mathrm{RPE_{R}}\downarrow

). We highlight the best and second best results.

Panel of Figure 5 — Figure 5: Spatial and Temporal Flexibility of our method, with 4RC as baseline. (a) Video depth estimation across different input resolutions on Sintel, with 4RC evaluated on the closest valid resolution. (b) Point tracking across different sequence lengths on DR.

Panel of Figure 6 — Figure 6: Displacement Map and Point Cloud. Points are mean-normalized to unit scale before computing accumulated motions.

Item	Adaptation	Params. (B)	Tracking				Video Depth
			DR		ADT		Bonn
			APD $\uparrow$ All / Dyn.	EPE $\downarrow$ All / Dyn.	APD $\uparrow$ All / Dyn.	EPE $\downarrow$ All / Dyn.	AbsRel $\downarrow$ All / Dyn.	$\delta_{<1.25}$ $\uparrow$ All / Dyn.
(a)	$\Delta W_{\mathrm{4D}}^{512\dagger}$	0.84	83.51 / 91.26	0.186 / 0.101	74.16 / 91.37	0.358 / 0.101	0.074	97.13
(b)	$\Delta W_{\mathrm{4D}}^{512}$	0.84	83.24 / 91.31	0.186 / 0.098	77.54 / 92.19	0.312 / 0.077	0.074	97.14
(c)	$\Delta W_{\mathrm{4D}}^{1024}$	1.54	82.80 / 91.54	0.195 / 0.096	76.83 / 92.83	0.325 / 0.077	0.078	96.96
(d)	$W_{\mathrm{DiT}}$	1.53	81.85 / 90.63	0.207 / 0.105	76.34 / 92.37	0.328 / 0.079	0.081	96.62

Table 4: Ablation Study on Different Modules. We report 3D tracking on 64 evaluated frames on DR and ADT as All / Dynamic groups, and video depth estimation on 110 evaluated frames with 49-frame inference chunks on Bonn.

\dagger

denotes discrete time embedding.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Additional Qualitative Results of motion tracking, point cloud reconstruction and camera pose estimation.

Panel of Figure 8 — Figure 8: Confidence Filtering for flying point suppression.

Dataset	Native Res.	Native AR.	Encoder Frame Size					Est. Subclip Count Per Size	Sampling Prob.
Dataset	Native Res.	Native AR.	$\mathbf{480\times 480}$	$\mathbf{480\times 640}$	$\mathbf{480\times 832}$	$\mathbf{352\times 832}$	$\mathbf{256\times 832}$	Est. Subclip Count Per Size	Sampling Prob.
Syn4D	$720\times 1280$	$9{:}16$	$720\times 720$	$720\times 960$	$720\times 1248$	$528\times 1248$	$384\times 1248$	25k	16.064%
DynamicReplica	$720\times 1280$	$9{:}16$	$720\times 720$	$720\times 960$	$720\times 1248$	$528\times 1248$	$384\times 1248$	13k	16.064%
PointOdyssey	$540\times 960$	$9{:}16$	$540\times 540$	$540\times 720$	$540\times 936$	$396\times 936$	$288\times 936$	20k	16.064%
Kubric Cotracker3	$512\times 512$	$1{:}1$	$512\times 512$	-	$285\times 494$	-	-	45k	12.851%
DL3DV	$528\text{--}539\times 927\text{--}972$	$\sim 4{:}7$	$525\times 525$	$525\times 700$	$525\times 910$	$385\times 910$	$280\times 910$	18K	3.213%

Appendix

Table A1: Dataset statistics. The middle five columns are indexed by ‘Encoder Frame Size’, with each entry recording the corresponding query frame size with the same aspect ratio.

Scene Group	Excluded IDs	#Raw Sequences	#WT Clips	#Valid Clips
Apartment_release_clean	131, 133–136, 138, 140–150	17	19	19
Apartment_release_decoration	131–140	10	11	11
Apartment_release_decoration_skeleton	131–140	10	11	10
Apartment_release_golden_skeleton	100	1	1	1
Apartment_release_meal	131–133, 136, 138–140	7	8	8
Total	WorldTrack-ADT Test Set	45	50	49

Appendix

Table A2: Excluded ADT test sequences during data preprocessing. One of the 50 evaluation clips contains no visible frame-0 query points, leaving 49 valid evaluation results.

Setting	ADT		DR		PO		PStudio
	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$	APD $\uparrow$	EPE $\downarrow$
	All / Dyn.	All / Dyn.	All / Dyn.	All / Dyn.	All / Dyn.	All / Dyn.	All	All
Ours- II	91.93 / 86.82	0.112 / 0.160	88.18 / 92.52	0.130 / 0.090	78.11 / 84.92	0.196 / 0.138	82.11	0.149
- w/o ADT	74.69 / 84.79	0.261 / 0.148	87.50 / 92.57	0.136 / 0.090	76.07 / 83.60	0.214 / 0.149	82.38	0.147

Appendix

Table A3: Additional quantitative results on world-coordinate tracking.

Track / %	$5\times 10^{-3}$ m	$1\times 10^{-2}$ m	$5\times 10^{-2}$ m
ADT	67.23	68.45	74.30
DR	71.81	72.33	73.62
PO	47.07	47.25	48.49
PStudio	0.00	0.00	2.00
Average	56.82	57.29	59.34

Appendix

Table A4: Percentage of frame-0 visible tracks classified as static under different motion thresholds.

4D Scene Understanding

4D scene understanding reconstructs and reasons about how objects, geometry, and actions change across space and time.

See all 53 papers

Beyond Pixels: From Video Priors to 4D Worlds
Aug 11, 2026Zihao Liu, Xiaolong Shen, Zhenglin Zhou +2Video Diffusion Models 4D Scene Generation
ReLER, CCAI, Zhejiang University
Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation
Dec 4, 2025Yanran Zhang, Ziyi Wang, Wenzhao Zheng +3Video Diffusion Models 3D Scene Generation
Department of Automation, Tsinghua University, China · GigaAI
Point4D: Long-range 4D Motion Reconstruction
Sep 8, 2026Minsik Jeon, Jay Karhade, Deva Ramanan +1Point Tracking 3D Point Tracking
Carnegie Mellon University

Aug 11, 2026J/K move · Enter open · S save

Beyond Pixels: From Video Priors to 4D Worlds

Zihao Liu, Xiaolong Shen, Zhenglin Zhou, Ruijie Quan+1

ReLER, CCAI, Zhejiang University

Video Diffusion Models 4D Scene Generation

Open paper AskPDF arXiv

4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.

Abstract

4D Scene Understanding

Explore similar work

Beyond Pixels: From Video Priors to 4D Worlds

Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

Point4D: Long-range 4D Motion Reconstruction