Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Yushi Lan; Zeren Jiang; Koichi Namekata; Xingang Pan; Chuanxia Zheng; Andrea Vedaldi

cs.CVOct 18, 2026

Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Authors: Yushi Lan, Zeren Jiang, Koichi Namekata, Xingang Pan, Chuanxia Zheng, Andrea Vedaldi

Organizations: University of Oxford · Nanyang Technological University

Abstract

Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.

Figures & tables

Figure 1: 4D reconstruction as a scaffold for video reshooting. Top: input video with its 3D trajectories (projected onto the image plane for visualization). Middle: reconstruction with the source camera (red) and target cameras (blue) for camera paths from Vista4D’s benchmark. Bottom: Holo4D outputs for each path; the middle frame overlays the projected trajectories (subset shown).

Figure 2: Holo4D overview. Top (4D scaffold): a 4D reconstructor $\mathcal{R}$ recovers persistent 3D trajectories, paired with sampled source colors, which are projected into the target cameras and encoded into track features $\mathbf{T}$ . Bottom (video diffusion): source and noisy target latents are concatenated along width; each DiT block concatenates $\mathbf{T}$ with its hidden features along channels (null on the source half) and adds camera features $\mathbf{C}$ encoded from Plücker rays. Flames and snowflakes mark trained and frozen modules ( Appendix B ). Trajectory panels show a subset of the dense tracks; the top-right overlay is for visualization only. The output is obtained by iterative denoising; Figure 11 details the DiT block.

Figure 3: Video reshooting on Vista4D’s benchmark trajectories. Top: dense source geometry, source camera (red), and requested camera path (blue; circle: $j=0$ , frusta: $j=24,48$ ). The track condition projects source trajectories into the target view; dark regions have no support. The Holo4D frame at $j=24$ overlays a subset of the tracks. Red boxes (with magnified insets) mark Vista4D’s errors: the diner is turned toward the camera, and the car loses its rear window. Holo4D uses $50$ steps with the two-stage backend (VGGT- $\Omega$ + CoWTracker; left) or 4RC (right); baseline inputs and released settings are described in Appendix E .

Figure 4: Novel-view synthesis on iPhone. Rows show spin, block, and teddy at $j=24$ , with ground-truth target views (GT) on the right. Holo4D uses the two-stage backend and $50$ steps.

Method	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ReCamMaster	10.34	0.478	0.628	10.56	0.326	0.689
TrajectoryCrafter	11.56	0.438	0.569	11.14	0.288	0.631
EX-4D	13.31	0.520	0.464	12.91	0.366	0.532
Gen3C	12.42	0.469	0.489	12.20	0.312	0.550
Vista4D	12.84	0.487	0.484	12.46	0.328	0.535
[0pt][0pt] Holo4D -14B α	13.22	0.484	0.435	12.68	0.326	0.509

Table 1: Novel-view synthesis on iPhone [ 81 ] .

\alpha

: 4RC;

\beta

: two-stage backend; same generator, no GMOS. “m”: masked metrics. Bold/underline: best/second best.

		Camera control			VBench [ 84 ]
Method	Control	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	Aest. $\uparrow$	Imag. $\uparrow$	Subj. $\uparrow$	Backg. $\uparrow$	Temp. $\uparrow$
ReCamMaster	Extrinsics	9.547	15.37	0.099	0.493	0.503	0.922	0.942	0.256
TrajectoryCrafter	Point cloud	3.533	4.817	0.207	0.500	0.548	0.894	0.923	0.253
EX-4D	Point cloud	2.903	3.560	0.587	0.442	0.377	0.827	0.899	0.238
Gen3C	Point cloud	5.168	7.581	0.386	0.454	0.450	0.856	0.911	0.248
Vista4D	Point cloud	3.576	3.608	0.132	0.478	0.534	0.904	0.926	0.256

Table 2: Camera control and video quality on DAVIS. Holo4D outperforms rendered-geometry baselines in camera control, imaging quality, and subject and background consistency. ATE:

\times 10^{2}

; RPE-T:

\times 10^{3}

; RPE-R: degrees. Bold and underline mark best and second-best results.

Figure 5: Zero-shot robot reshooting. Two DROID [ 85 ] episodes (top: source) reshot along a crane-above path and a near orbit (bottom), at $j=0,24,48$ . Holo4D -14B uses no robot training data; inference uses 4RC and $50$ steps. More examples and track conditions are in Figure 17 .

Method	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
ReCamMaster	12.68	62.20	0.749	13.66	0.118	0.236	1849
TrajectoryCrafter	5.018	20.61	0.225	13.64	0.256	0.178	1878
EX-4D	8.163	57.89	0.563	12.69	0.273	0.271	1916
Gen3C	5.724	40.03	0.469	13.10	0.103	0.211	1633
Vista4D	5.953	25.19	0.255	14.28	0.285	0.151	1214
[0pt][0pt] Holo4D -14B	1.065	6.154	0.073	16.52	0.441	0.119	973

Table 3: Novel-view synthesis on Syn4D. Ground-truth source geometry at inference removes upstream reconstruction error. ATE:

\times 10^{2}

; RPE-T:

\times 10^{3}

; RPE-R: degrees. Bold and underline mark best and second-best results.

Figure 6: Long-video rollout. A $196$ -frame DROID [ 85 ] episode reshot to a static target view in five $49$ -frame windows (4RC, $50$ steps): Holo4D -14B for window 1, the I2V continuation model for windows 2–5 ( Appendix B ). Brackets mark the window boundaries shown.

Condition	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$
w/o source color	2.327	2.125	0.094
w/o target depth	1.882	2.135	0.068
+ rendered RGB/masks	1.83	2.48	0.098
[0pt][0pt] Full ( Holo4D )	1.621	2.077	0.065

Table 4: Stage-1 conditioning design on DAVIS. All variants condition on trajectories. ATE:

\times 10^{2}

; RPE-T:

\times 10^{3}

; RPE-R: degrees.

Source geometry	ATE $\downarrow$	RPE-R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Open-D4RT ∗	1.858	0.151	13.10	0.158	0.169
V-DPM	1.660	0.079	14.96	0.268	0.151
4RC	1.353	0.087	14.98	0.278	0.131
CoWTracker ∗	1.205	0.107	15.09	0.294	0.137
GT geometry	1.065	0.073	16.52	0.441	0.119
$\alpha=0.1$	3.033	0.221	13.44	0.171	0.172

Table 5: Reconstruction backends and track noise on Syn4D. Top: backends ( ∗ VGGT-

\Omega

depth and cameras); bottom: GT tracks with relative noise

\alpha

. ATE

\times 10^{2}

; RPE-R in degrees. No GMOS. Bold/underline: best/second best per block.

Method	VGGT- $\Omega$			Pi3
Method	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	TransErr 2 $\downarrow$	RotErr $\downarrow$
Vista4D	46.04	36.81	0.255	4.573	4.169
Holo4D -14B α	46.31	21.33	1.142	2.164	14.98
Holo4D -14B β	13.76	9.064	0.301	2.367	6.098

Table 6: Camera control on Vista4D’s

110

trajectories.

\alpha

: 4RC;

\beta

: two-stage. ATE

\times 10^{2}

; RPE-T

\times 10^{3}

; rotations in degrees.

	Overall		LIBERO-plus perturbation category
Initialization	LIBERO	LIB.-plus	Background	Camera	Language	Light	Layout	Robot	Noise
Official release	96.80	50.91	50.09	16.82	69.94	81.79	61.97	46.00	39.41
Control (Wan2.2)	95.60	50.81	56.32	9.01	70.79	85.20	66.62	49.10	31.73
[0pt][0pt] Holo4D init.	96.45	52.10	63.38	8.01	71.57	89.14	66.16	49.16	32.92

Table 7: Policy learning with Fast-WAM. Success (%) after

20

k LIBERO steps. Control and Holo4D differ only in video self-attention initialization; the official release is a separate reference. Bold/underline: best/second best.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

	Submitted version	This version
Source video	VAE latents, width-concatenated with target latents	Unchanged
RGB-D branch	Target-view RGB, normalized depth, and validity rendered from a static/dynamic point cache; Conv3d geometry tokens	Removed
Track branch	Tracked positions; z-buffer aggregation in the described design; fused with RGB-D tokens	Positions, source color, target-camera depth, and validity; channel-wise max with occupancy; injected along channels in every DiT block
Target cameras	Plücker rays	Plücker rays, encoded into camera features
Backbone	Wan2.1-I2V-14B; Wan2.1-T2V-1.3B	Wan2.1-T2V-14B
Resolution	$144\times 256$ , then $288\times 512$	$144\times 256$ , then $384\times 672$

Appendix

Table 8: Submitted version versus this version. The submitted column summarizes the described design; the rendering used for the submitted results is distinguished below.

Panel of Figure 7 — Figure 7: Submitted pipeline (reproduced from the submitted version). Static and dynamic source pixels were rendered into a hybrid target-view RGB-D cache (projected RGB, depth, and visibility), which conditioned the video diffusion model together with Plücker rays and projected 3D tracks. This version removes the rendered RGB-D branch and keeps the track branch ( Figure 2 ).

Panel of Figure 9 — Figure 9: Submitted qualitative comparison on DAVIS (frames from the submitted version). “Holo4D (submitted)” is the submitted 14B checkpoint with rendered RGB-D and track conditioning; the second row shows source pixels reprojected into the target view (labelled GT in the submission). See Figures 3 and 16 for the final model.

Figure 11: Trajectory conditioning. (a) Each trajectory is projected into target frame $j$ ; its observation $\mathbf{z}_{n,j}$ (frame time, normalized position, validity, target-camera depth, and source color) is encoded, scattered to the target grid with a channel-wise maximum and an occupancy channel, and brought to the DiT token grid by a causal 3D adapter, giving $\mathbf{T}$ . (b) Source and noisy target latents are concatenated along width once, at the input; in every block, $\mathbf{T}$ occupies the target half (null on the source half) and the camera features $\mathbf{C}$ from one shared encoder are repeated on both halves. (c) Each of the $40$ DiT blocks concatenates $\mathbf{T}$ with the normalized hidden features, projects back to width $D$ with $W^{(\ell)}$ initialized as $[I\,|\,0]$ , and adds $\mathbf{C}$ before self-attention; the camera projector is identity-initialized and gated. Flames and snowflakes mark trained and frozen modules (network icons: dark = trained, light = frozen); token shading runs from the first to the last frame. Standard cross-attention/FFN residuals and the auxiliary query-frame input are omitted.

Index	Dataset	Real / Syn.	Dyn. / Static	Depth, cameras	3D tracks	Stage 1	Stage 2
1	Syn4D [ 73 ]	Synthetic	Dynamic	Ground truth	Ground truth	31.65	40.00
2	MultiCamVideo [ 1 ]	Synthetic	Dynamic	VGGT- $\Omega$	CoWTracker	0.00	30.00
3	DL3DV [ 75 ]	Real	Static	DA3 + dataset	Fixed points	32.91	20.00
4	MVS-Synth [ 77 ]	Synthetic	Static	Ground truth	Fixed points	6.33	10.00
5	Kubric [ 74 ]	Synthetic	Dynamic	Ground truth	Box motion	12.66	0.00
6	TartanAir [ 78 ]	Synthetic	Static	Ground truth	Fixed points	12.66	0.00

Appendix

Table 9: Training data. Sampling weights (%) per stage, not proportions of distinct videos. Static scenes need no tracker: backprojected source points stay fixed in 3D. Dynamic pairs come from synchronized multiview videos; static pairs from different chunks of the same video. DA3: DepthAnything 3 [ 4 ] .

Method	Backbone	Res.
ReCamMaster	Wan2.1-T2V-1.3B	480 $\times$ 832
TrajectoryCrafter	CogVideoX-5B	384 $\times$ 672
EX-4D	Wan2.1-I2V-14B	512 $\times$ 512
Gen3C	Cosmos-7B	704 $\times$ 1280
Vista4D	Wan2.1-T2V-14B	384 $\times$ 672
[0pt][0pt] Holo4D -14B	Wan2.1-T2V-14B	384 $\times$ 672

Appendix

Table 10: Backbones and native resolutions.

Method	Steps	Aest. $\uparrow$	Imag. $\uparrow$	Subj. $\uparrow$	Backg. $\uparrow$	Temp. $\uparrow$
Vista4D α	50	0.478	0.534	0.904	0.926	0.256
Holo4D -14B α	50	0.493	0.601	0.910	0.931	0.253
Holo4D -14B β	50	0.478	0.589	0.891	0.919	0.252
Holo4D -14B β	25	0.471	0.580	0.883	0.910	0.249
Holo4D -5B α	50	0.501	0.559	0.894	0.928	0.253
Holo4D -5B β	50	0.489	0.554	0.881	0.920	0.252

Appendix

Table 11: VBench across reconstruction backends on DAVIS.

\alpha

: 4RC;

\beta

: two-stage backend with GMOS. 5B rows use Wan2.2-TI2V-5B at

704\times 1280

. Columns follow Table 2 .

Method	Steps	VGGT- $\Omega$			Pi3
Method	Steps	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	TransErr 2 $\downarrow$	RotErr $\downarrow$
Vista4D α	50	3.576	3.608	0.132	6.201	2.070
Holo4D -14B α	50	1.657	2.415	0.102	3.206	2.332
Holo4D -14B β	50	0.719	1.299	0.101	3.359	2.284
Holo4D -14B β	25	0.729	1.095	0.077	3.625	1.785
Holo4D -5B α	50	1.464	2.952	0.116	3.195	2.275

Appendix

Table 12: Source geometry and sampling steps on DAVIS.

\alpha

: 4RC;

\beta

: two-stage backend with GMOS. 5B rows use Wan2.2-TI2V-5B at

704\times 1280

(

384\times 672

for

\dagger

). † Trained without Syn4D, MVS-Synth, and DL3DV, with a dense track condition without persistent identities. ATE is scaled by

10^{2}

; RPE-T and TransErr 2 by

10^{3}

. Rotations are in degrees.

Mask	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	TransErr 2 $\downarrow$	RotErr $\downarrow$
Motion-derived	0.984	1.527	0.114	3.351	2.364
[0pt][0pt] GMOS	0.719	1.299	0.101	3.359	2.284

Appendix

Table 13: Segmentation for static/dynamic track fusion on DAVIS. Two-stage backend with the same first-frame tracks,

384\times 672

50

steps; only the foreground mask differs. ATE:

\times 10^{2}

; RPE-T and TransErr 2 :

\times 10^{3}

; rotations: degrees.

Figure 12: GMOS in the two-stage backend. VGGT- $\Omega$ supplies depth and cameras and CoWTracker supplies tracks. GMOS segments the moving regions in the query frame; trajectories outside the mask are held at their query-frame 3D positions, and trajectories inside keep their tracked motion. Both sets form one track condition, shown projected into the target view at $j=24$ . Background regions occluded by the foreground in the query frame have no trajectories. DAVIS blackswan , final checkpoint, $50$ denoising steps.

Method	Geometry	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	TransErr 2 $\downarrow$	RotErr $\downarrow$
Vista4D	GT	2.233	12.12	0.079	13.36	0.372
[0pt][0pt] Holo4D -14B	GT	1.299	8.121	0.051	1.881	0.242
[0pt][0pt] Holo4D -5B	GT	1.957	11.19	0.080	5.023	0.515
Vista4D	Pi3	2.348	10.15	0.067	19.26	0.651
[0pt][0pt] Holo4D -14B	Pi3 + GT 2D	1.330	8.158	0.052	5.715	0.534
[0pt][0pt] Holo4D -5B	Pi3 + GT 2D	1.996	10.40	0.084	13.50	0.740

Appendix

Table 14: Camera control with GT and estimated source geometry on Syn4D. Nine held-out clips; Pi3 camera recovery. ATE:

\times 10^{2}

; RPE-T and TransErr 2 :

\times 10^{3}

; rotations: degrees. Bold/underline: best/second best per geometry group. Protocols: Appendix E .

Condition	VGGT- $\Omega$			Pi3, source-aligned		Rot. gain $\uparrow$
Condition	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	TransErr 2 $\downarrow$	RotErr $\downarrow$	Rot. gain $\uparrow$
Plücker only	15.03	60.92	1.423	0.9866	31.26	0.0218
+ rendered RGB-D	16.21	41.69	1.565	0.7625	33.59	0.0027
+ rendered RGB-D + tracks	16.14	45.22	1.642	0.7117	33.25	0.0096
+ multi-frame RGB-D + tracks	16.39	34.69	1.635	0.6854	33.29	0.0149

Appendix

Table 15: Conditioning in early 1.3B checkpoints. Rotation gain is the fitted ratio of generated to requested rotation. ATE is scaled by

10^{2}

; RPE-T by

10^{3}

. Rotations are in degrees.

Conditioning variants (training not matched)
Condition	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$
Plücker rays only	10.65	7.32	0.373
PCD(RGB-D) + valid mask, no tracks	10.36	9.16	0.236
PCD(RGB-D) + 3D tracks	2.35	2.38	0.086
Tracks without per-track color	2.33	2.13	0.094
[0pt][0pt] 3D tracks only ( Holo4D )	1.62	2.08	0.065

Appendix

Table 16: Track and rendered conditioning on DAVIS. PCD denotes rendered point clouds. The unmatched upper block is not ranked; lower-block bold and underline mark best and second-best results. ATE and RPE-T are scaled by

10^{2}

and

10^{3}

; rotations are in degrees.

Configuration	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Plücker rays only	11.23	0.442	0.630	11.36	0.281	0.717
$+$ RGB render	11.91	0.466	0.611	11.95	0.314	0.725
$+$ depth + valid mask	11.93	0.453	0.640	11.93	0.296	0.758
$+$ 3D tracks (single-frame)	13.58	0.477	0.490	13.20	0.319	0.568
$+$ multi-frame static cache	13.65	0.479	0.497	13.28	0.322	0.581

Appendix

Table 17: Earlier conditioning variants on DyCheck. Source geometry: (a) 4RC; (b) two-stage backend. “m” denotes masked metrics; bold and underline mark best and second-best results per panel.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$
ViDAR [ 26 ]	12.90	0.385	0.734	13.01	0.525	0.653
[0pt][0pt] Holo4D -14B α	12.68	0.326	0.509	13.22	0.484	0.435
[0pt][0pt] Holo4D -14B β	13.20	0.348	0.465	14.03	0.507	0.379

Appendix

Table 18: Novel-view synthesis compared with ViDAR. Under Vista4D’s iPhone protocol, ViDAR is optimized per scene on the first

49

source frames, center-cropped and resized to

288\times 512

. We evaluate

49

target frames, unlike ViDAR’s full-sequence benchmark.

\alpha

: 4RC;

\beta

: two-stage backend. “m” denotes masked metrics. Bold and underline mark best and second-best results.

Figure 13: Syn4D with ground-truth source geometry. Two held-out pairs with large viewpoint changes; the right column is the rendered target view. Rendered-geometry baselines reproject ground-truth source depth, Holo4D receives ground-truth 3D trajectories, and ReCamMaster uses only the cameras ( Table 3 ). EX-4D generates square frames, shown as a central crop. Holo4D places the moving objects as in the ground truth; all methods differ from it in unobserved regions.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	FVD $\downarrow$
ReCamMaster	12.40	0.277	0.761	11.69	63.79	0.776	1908
TrajectoryCrafter	9.418	0.176	0.725	9.636	39.98	0.522	3073
EX-4D	9.865	0.163	0.863	18.76	89.83	2.438	2625
Gen3C	11.72	0.216	0.686	9.336	52.28	0.395	1693
Vista4D	10.98	0.158	0.668	9.972	41.56	0.419	1525
[0pt][0pt] Holo4D -14B	11.70	0.180	0.667	3.267	32.09	0.272	1711

Appendix

Table 19: Novel-view synthesis on Syn4D with 4RC source geometry. Geometry-conditioned methods receive 4RC source geometry in their native form; ReCamMaster uses no geometry. FVD uses the paired target videos. ATE and RPE-T are scaled by

10^{2}

and

10^{3}

; rotations are in degrees. Bold/underline: best/second best.

	Syn4D (exact depth)				iPhone (pseudo-reference)
	VGGT- $\Omega$		4RC		VGGT- $\Omega$		4RC
Method	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
ReCamMaster	0.762	0.311	0.738	0.298	0.152	0.765	0.121	0.835
TrajectoryCrafter	0.673	0.367	0.713	0.337	0.109	0.848	0.098	0.881
EX-4D	0.673	0.298	0.766	0.232	0.109	0.823	0.112	0.877
Gen3C	0.740	0.337	0.736	0.326	0.098	0.862	0.112	0.895

Appendix

Table 20: Depth reconstructed from generated videos. Syn4D uses exact target depth; iPhone uses reconstructed target-video depth as a pseudo-reference.

\alpha

: 4RC;

\beta

: two-stage source backend, without GMOS. Evaluation and aggregation are specified in Appendices F and I .

Source geometry	ATE $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
VGGT- $\Omega$ + Open-D4RT	1.858	11.22	0.151	13.10	0.158	0.169	1337
V-DPM	1.660	8.904	0.079	14.96	0.268	0.151	1252
4RC	1.353	7.148	0.087	14.98	0.278	0.131	1168
VGGT- $\Omega$ + CoWTracker	1.205	7.688	0.107	15.09	0.294	0.137	1216
Ground-truth tracks	1.065	6.154	0.073	16.52	0.441	0.119	1098
GT tracks, $\alpha=0.1$	3.033	22.53	0.221	13.44	0.171	0.172	1375

Appendix

Table 21: Source geometry and track noise on Syn4D. The same generator receives estimated source geometry from four reconstruction backends (top; no GMOS), ground-truth tracks (middle), or ground-truth tracks with Gaussian noise whose standard deviation is

\alpha

times that of the source tracks (bottom).

32

held-out pairs,

25

denoising steps. ATE and RPE-T are scaled by

10^{2}

and

10^{3}

; rotations are in degrees. FVD uses an earlier

256\times 256

scoring and is comparable only within this table. Bold/underline rank estimated backends.

Figure 14: Changing the source reconstructor without retraining. The same checkpoint receives ground-truth trajectories or trajectories from four reconstruction backends, with target cameras, sampler, and seed fixed ( Table 21 ). Geometric errors change the generated layout and appearance, as seen with V-DPM and Open-D4RT.

Stage	Time (s)	Share
4RC feed-forward	$1.46\pm 0.02$	$1.5\%$
Static cache + rendering	$7.54\pm 0.30$	$7.8\%$
Dynamic render + fusion	$1.43\pm 0.31$	$1.5\%$
3D-track encoding	$0.06\pm 0.00$	$0.1\%$
Denoising + decoding	$86.50\pm 0.26$	$89.2\%$
Warm-pipeline total	$\mathbf{96.99\pm 0.67}$	$\mathbf{100\%}$

Appendix

Table 22: Warm inference runtime on one H200. Times are mean

\pm

standard deviation over four videos, including reconstruction, proxy rendering, and generation but excluding model loading and video writing.

Table 35

Figure 15: Additional video reshooting examples. Source and generated frames for six trajectories from Vista4D’s benchmark (top) and six from our DAVIS evaluation set (bottom), named above each example. The middle generated frame ( $j=24$ ) overlays a subset of the projected trajectories, with trails covering the previous $12$ frames. All examples use the two-stage backend and $50$ denoising steps; none appears in the other figures.

Figure 16: Comparison with all baselines on two further Vista4D trajectories. Left: a dolly-out from a golfer that reveals a golf cart; right: a rising, zooming-out crane shot of a man on a hillside. Layout as in Figure 3 . TrajectoryCrafter, EX-4D, and Gen3C render the cart incomplete or distort the people, EX-4D leaves large holes, and ReCamMaster stays close to the source framing; Vista4D and Holo4D both follow the requested paths. Holo4D -14B uses 4RC source reconstruction and $50$ denoising steps; baselines use the same settings as in Figure 3 .

Figure 17: Zero-shot novel-view synthesis of robot videos. Each example shows the source video (top), projected 3D trajectories with source colors (middle; dark regions have no support, and trails show a subset over the preceding $12$ frames), and the generated view (bottom). Top: DROID [ 85 ] episodes with 4RC reconstruction; bottom: humanoid clips with the two-stage backend. Holo4D -14B uses no robot training data and no fine-tuning; $50$ denoising steps.

Figure 18: More long-video rollouts. From top: three DROID [ 85 ] episodes of $196$ , $214$ , and $246$ frames ( $5$ , $5$ , and $6$ windows) reshot to static target views, and a $94$ -frame humanoid clip ( $2$ windows) reshot along a close crane-above path. Six evenly spaced frames per video. Window 1 uses Holo4D -14B with 4RC (DROID) or the two-stage backend (humanoid); later windows use the I2V continuation model, trained with OmniWorld-DROID, and 4RC; $50$ steps ( Appendix B ).

Figure 19: Content outside the initial scaffold. The marked person enters the source view late and has no first-frame query points, so generated views must complete them without trajectory support (no target-view ground truth is available).

	LIBERO			LIBERO-plus
Suite	Official	Ctrl.	Ours	Tasks	Official	Ctrl.	Ours
Spatial	97.40	97.00	97.60	2,402	54.58	59.87	55.08
Object	99.40	99.20	99.20	2,518	69.46	62.43	65.45
Goal	96.40	95.20	96.00	2,591	37.32	38.67	44.89
LIBERO-10	94.00	91.00	93.00	2,519	42.83	43.03	43.35
All	96.80	95.60	96.45	10,030	50.91	50.81	52.10

Appendix

Table 25: Fast-WAM details. Closed-loop success rates (%) per suite on LIBERO (

500

trials per suite) and LIBERO-plus (one trial per task); All : suite mean (LIBERO), task-weighted mean (LIBERO-plus). Official: released Fast-WAM checkpoint; Ctrl.: Wan2.2-initialized control; Ours: Holo4D self-attention initialization.

Asset	License and scope
Training and evaluation data
Syn4D	CC BY 4.0; underlying assets retain their individual terms.
MultiCamVideo , DyCheck iPhone	Apache 2.0 (dataset releases).
DL3DV-10K	CC BY-NC 4.0 and DL3DV-10K Terms of Use.
MVS-Synth	Custom terms: research and educational use only.
Kubric , RealEstate10K	Apache 2.0 (code); rendered assets/source videos have separate terms.

Appendix

Table 26: External asset licenses. Asset names link to official license notices or release cards. Code licenses do not automatically cover separate weights or source footage.

Camera-Controlled Video Generation

Camera-controlled video generation creates videos that follow specified viewpoints and camera paths while preserving coherent scenes and motion.

See all 135 papers

Vista4D: Video Reshooting with 4D Point Clouds
Apr 23, 2026Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca +9Camera-Controlled Video Generation 4D Reconstruction
1Eyeline Labs · 3Columbia University · 4UCLA +3
Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting
Apr 23, 2026Avinash Paliwal, Adithya Iyer, Shivin Yadav +2Camera-Controlled Video Generation Video Diffusion Models
Morphic Inc.
Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering
Oct 10, 2026Hanxin Zhu, Jianqi Ma, Cong Wang +10Dynamic Novel View Synthesis 4D Scene Generation
USTC · Tencent · CASIA +3

Apr 23, 2026J/K move · Enter open · S save

Vista4D: Video Reshooting with 4D Point Clouds

Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca, Yash Kant+8

1Eyeline Labs · 3Columbia University · 4UCLA+3

Camera-Controlled Video Generation 4D Reconstruction Dynamic Scene Reconstruction 4D Scene Understanding

Open paper AskPDF arXiv

We present Vista4D, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud. Specifically, given an input video, our method re-synthesizes the scene with the same dynamics from a different camera trajectory and viewpoint. Existing video reshooting methods often struggle with depth estimation artifacts of real-world dynamic videos, while also failing to preserve content appearance and failing to maintain precise camera control for challenging new trajectories. We build a 4D-grounded point cloud representation with static pixel segmentation and 4D reconstruction to explicitly preserve seen content and provide rich camera signals, and we train with reconstructed multiview dynamic data for robustness against point cloud artifacts during real-world inference. Our results demonstrate improved 4D consistency, camera control, and visual quality compared to state-of-the-art baselines under a variety of videos and camera paths. Moreover, our method generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition. See our project page for results, code, and models: https://eyeline-labs.github.io/Vista4D

Abstract

Camera-Controlled Video Generation

Explore similar work

Vista4D: Video Reshooting with 4D Point Clouds

Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting

Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering