Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

Yongqi Mao; Zijia Dai; Zhishuo Liu; Wei Xu; Kaiwei Wang; Guotao Meng

cs.CVAug 28, 2026

Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

Authors: Yongqi Mao, Zijia Dai, Zhishuo Liu, Wei Xu, Kaiwei Wang, Guotao Meng

Organizations: Zhejiang University · Manifold Tech · ShanghaiTech University · University of Cambridge

Abstract

Video re-shooting is the task of re-rendering a monocular video of a dynamic scene along a user-specified camera trajectory. A dominant paradigm explicitly provides the target geometry by lifting the source video into a 4D point cloud using per-frame depth, and rasterizing this representation along the target trajectory to obtain point-cloud renders as geometric conditions for trajectory-controlled generation. The point cloud render and source video may provide conflicting visual cues throughout denoising, as both are directly used as conditions. We refer to this conflict as a trust dilemma, which can compromise trajectory control, 3D consistency of dynamic content, and visual quality, particularly under distribution shifts. We argue that because a point-cloud render is pixel-aligned with the target view, it does not need to be continuously provided as a conditioning signal throughout denoising. We propose Manifold4D, which injects the render directly into the initial noise of flow matching, so that generation no longer departs from standard Gaussian noise but from a geometry-bearing noise distribution that we term the point cloud rendered manifold, leaving the source video as the only visual condition. The render is thus used exactly once, and the network is never asked to learn how to read it; in subsequent denoising steps the model can focus on the source video. In our experiments, Manifold4D attains the best camera-control accuracy on every metric, lowering rotation error by 25--27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and attaining the best overall photometric quality and 3D consistency of dynamic content.

Figures & tables

Figure 1: Video re-shooting. Given a monocular video and a target camera trajectory, Manifold4D re-shoots the video along the new trajectory, achieving the best trajectory control among existing methods while preserving visual quality. The bottom panel shows a re-shooting example. GEN3C and TrajectoryCrafter inherit the artifacts of the imperfect render, while Vista4D leaves the subject misaligned with the render geometry. In the top-right plot, the vertical axis averages the VBench aesthetic and imaging quality scores; the horizontal axis denotes the camera-control error relative to the best method (see Appendix A for details).

Figure 2: How each method consumes the render. Prior methods concatenate the render at every denoising step, so the network must learn to interpret and utilize it. Ours directly adds it into the starting noisy latent, requiring no learning. This explicit, one-time consumption is the key difference.

Figure 3: Pipeline overview. Given a monocular video and a target camera trajectory, Manifold4D takes a 4D point cloud obtained from an existing reconstruction and motion decomposition pipeline and renders target-trajectory views from it. The rendered views are encoded into VAE latents, which are injected into the noise and patchified into tokens. The source video is similarly encoded into tokens, concatenated with the noise tokens, and fed to the DiT for denoising, together with the encoded camera trajectory.

Method	DAVIS-Traj (72 clips)			Vista4D-Eval (103 clips)			Vista4D-Eval raw (110 clips)
Method	RotErr $\downarrow$	TransErr $\downarrow$	IntrErr $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	IntrErr $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	IntrErr $\downarrow$
ReCamMaster	14.04	0.581	2.77	11.81	0.495	10.63	13.73	0.549	11.23
TrajectoryCrafter	1.56	0.274	1.14	5.40	0.462	8.40	8.24	0.518	9.14
GEN3C	1.45	0.236	1.13	3.77	0.441	8.74	4.31	0.460	9.33
Vista4D	2.56	0.240	1.47	3.89	0.403	7.66	4.84	0.432	8.10
Ours	1.09	0.161	1.04	2.76	0.394	7.65	4.26	0.424	8.09

Table 1: Camera control accuracy. Manifold4D attains the highest accuracy on every metric, approaching or surpassing the point cloud render itself on rotation error. Bold denotes the best and underline the second-best result.

Figure 4: Control error and visual quality under growing motion amplitude. Manifold4D maintains the lowest pose error at almost every amplitude, staying close to the point cloud render, while other methods degrade markedly. Visual quality (Aesthetic and Imaging) likewise remains the highest, especially at large amplitudes. Red tick labels mark the amplitudes of DAVIS-Traj .

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	EPE $\downarrow$
ReCamMaster	10.89	0.3363	0.7161	11.00	0.3326	0.5376	6.66
TrajectoryCrafter	15.52	0.4338	0.5162	16.41	0.4476	0.3494	1.58
GEN3C	14.75	0.3937	0.4609	15.87	0.4119	0.3103	1.47
Vista4D	15.78	0.3913	0.3884	16.30	0.3997	0.2600	1.28
Ours	16.21	0.3847	0.3582	16.65	0.3887	0.2393	1.25

Table 2: Novel view photometric quality on iPhone Dataset. Manifold4D achieves state-of-the-art performance across key photometric and optical-flow metrics.

Method	DAVIS-Traj				Vista4D-Eval
Method	FID $\downarrow$	FVD $\downarrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	FID $\downarrow$	FVD $\downarrow$	Aesthetic $\uparrow$	Imaging $\uparrow$
ReCamMaster	65.55	1122.9	48.099	57.204	92.79	1181.1	52.338	58.694
TrajectoryCrafter	85.27	1571.6	46.899	57.038	116.31	1581.4	47.579	63.300
GEN3C	89.76	1463.9	47.245	53.218	113.43	1534.3	48.852	62.001
Vista4D	55.64	1208.2	51.096	64.914	98.12	1404.6	55.012	70.933
Ours	57.55	1222.7	51.138	65.384	97.37	1360.2	53.757	70.509

Table 3: Visual quality. FID/FVD are computed against the corresponding source clips; Aesthetic and Imaging are VBench scores (

\times 100

Figure 5: Qualitative results on monocular input. Dashed guides are placed on the point cloud render and copied to every row: green marks the expected position of the dynamic subject, red a static background structure. Our method shows stable control and pleasing visual quality while Vista4D’s dynamic subject drifts away from the green guide. GEN3C and TrajectoryCrafter inherit the sparsity of the render, leaving subjects fragmented; ReCamMaster stays clean but ignores the requested view.

Criterion	ReCamMaster	TrajectoryCrafter	GEN3C	Vista4D	Ours
Trajectory following	1.4%	17.0%	20.5%	28.3%	63.6%
3D consistency	1.8%	15.6%	9.5%	8.7%	62.4%
Overall quality	12.9%	3.2%	5.8%	54.7%	49.6%

Table 4: User study. Selection rate in %. Manifold4D is preferred by large margins on trajectory following and 3D consistency of the dynamic subject, and remains competitive on overall quality.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Trajectory patterns of DAVIS-Traj . Each pattern orbits the dynamic subject while keeping it on the optical axis. f1 (blue) sweeps $\pm 60$ ∘ of yaw; f2 (green) sweeps $\pm 30$ ∘ of yaw with $-10$ ∘ of pitch at $1.3\times$ distance; f3 (yellow) sweeps $\pm 30$ ∘ of yaw with $+10$ ∘ of pitch while dollying in from $1.0$ to $0.7$ . The red curve shows the source camera trajectory.

Method	Base model	Output	Time (min)
ReCamMaster	Wan2.1-T2V-1.3B	81 $\times$ 480 $\times$ 832	9.0
TrajectoryCrafter	CogVideoX-Fun 5B	49 $\times$ 384 $\times$ 672	2.7
GEN3C	Cosmos-Predict1 7B	121 $\times$ 704 $\times$ 1280	18.5
Vista4D	Wan2.1-T2V-14B	49 $\times$ 384 $\times$ 672	19.3
Ours	Wan2.1-T2V-14B	49 $\times$ 384 $\times$ 672	13.0

Appendix

Table 6: Base models and inference time. All methods run with 50 diffusion steps on a single NVIDIA A100 80 GB GPU and generate at their native resolution.

Method	DAVIS-Traj (72 clips)			Vista4D-Eval (106 clips)			Vista4D-Eval raw (110 clips)
Method	RotErr $\downarrow$	TransErr $\downarrow$	IntrErr $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	IntrErr $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	IntrErr $\downarrow$
ReCamMaster	14.00	0.586	3.19	13.23	0.549	10.67	13.56	0.563	10.67
TrajectoryCrafter	1.79	0.256	1.52	6.62	0.500	7.55	7.71	0.504	7.57
GEN3C	1.88	0.267	1.41	4.05	0.438	7.52	4.25	0.433	7.52
Vista4D	2.90	0.304	2.06	4.07	0.382	7.23	4.67	0.388	7.26
Ours	1.56	0.247	1.59	3.82	0.419	6.96	4.39	0.427	6.97

Appendix

Table 7: Camera control accuracy with Pi3 reconstruction. Left : filtered (RotErr

\leq 60^{\circ}

in any method, maximum common clip set). Right : raw Vista4D-Eval (all 110 clips, no filtering). DAVIS-Traj has no filtered clips so only one column group is shown. 3-seed mean. Bold denotes the best and underline denotes the second best among generated methods (Point cloud render excluded from ranking and typeset in italics).

	RotErr $\downarrow$ (deg)									TransErr $\downarrow$ (normalized)
Yaw	$10$	$20$	$30$	$40$	$50$	$60$	$70$	$80$	$90$	$10$	$20$	$30$	$40$	$50$	$60$	$70$	$80$	$90$
ReCamMaster	7.32	9.21	11.44	14.43	17.67	19.55	22.28	38.94	43.19	0.290	0.348	0.510	0.622	0.560	0.689	0.721	0.918	0.853
TrajectoryCrafter	0.41	0.50	0.82	1.39	1.61	2.79	2.93	3.85	7.15	0.028	0.035	0.060	0.600	0.101	0.896	0.539	0.367	0.415
GEN3C	0.98	1.08	1.23	2.01	1.91	2.44	2.92	12.58	6.49	0.038	0.066	0.080	0.557	0.344	0.401	0.496	1.011	0.651
Vista4D	0.58	0.96	1.78	2.19	2.45	2.45	4.53	6.30	6.17	0.029	0.050	0.091	0.103	0.163	0.146	0.206	0.303	0.302
Ours	0.40	0.55	0.79	1.03	1.25	1.35	1.95	2.46	2.77	0.034	0.044	0.066	0.079	0.085	0.110	0.139	0.167	0.200

Appendix

Table 8: Camera control accuracy under growing motion amplitude. RotErr and TransErr at each amplitude, averaged over ten DAVIS-Traj scenes. The amplitude is the per-side yaw angle, from

10^{\circ}

90^{\circ}

(total sweep

20^{\circ}

–

180^{\circ}

). The point cloud render is included as a geometric reference and excluded from ranking.

DAVIS-Traj
Method	FID $\downarrow$	FVD $\downarrow$	CLIP-T $\uparrow$	SC $\uparrow$	BC $\uparrow$	MS $\uparrow$	AQ $\uparrow$	IQ $\uparrow$
ReCamMaster	65.55	1122.9	0.2392	88.333	89.807	97.932	48.099	57.204
TrajectoryCrafter	85.27	1571.6	0.2361	84.866	88.519	95.911	46.899	57.038
GEN3C	89.76	1463.9	0.2298	84.543	90.624	95.627	47.245	53.218
Vista4D	55.64	1208.2	0.2508	86.980	90.420	95.677	51.096	64.914
Ours	57.55	1222.7	0.2444	86.393	89.367	95.122	51.138	65.384

Appendix

Table 10: Full visual quality and temporal stability metrics. FID/FVD are computed against the corresponding source clips, CLIP-T is the frame-wise text-image similarity, and the remaining five columns are VBench scores (

\times 100

): subject consistency (SC), background consistency (BC), motion smoothness (MS), aesthetic quality (AQ) and imaging quality (IQ). Mean over 3 seeds.

Figure 7: Novel-view comparison on real iPhone multi-view captures. Vista4D shows mild 3D inconsistency (red lines) and does not follow the point cloud colors; TrajectoryCrafter and GEN3C appear flat and miss fine human details; ReCamMaster lacks camera control.

Figure 8: Robustness to incorrect dynamic geometry. We omit the motion mask to simulate an erroneous point cloud: dynamic points from all frames are stacked together, smearing the rendered subject. Manifold4D still recovers the correct motion from the source video.

Figure 9: Qualitative ablation comparison. The two architecture variants against the full model on Vista4D-Eval clips. The Vista4D-style three-stream layout exhibits 3D inconsistency on dynamic objects, while the GEN3C-style variant shows noticeable degradation in visual quality in some scenes. The full model preserves both geometric consistency and visual quality.

Figure 10: User study evaluation interface. Left: the point cloud render is overlaid on each generated video (toggleable with the Q key) for the trajectory-following and 3D-consistency questions. Right: the full generated video is played without overlay for the visual-quality question, presented separately to mitigate bias from the preceding overlay-based questions.

Camera-Controlled Video Generation

Camera-controlled video generation creates videos that follow specified viewpoints and camera paths while preserving coherent scenes and motion.

See all 143 papers

Vista4D: Video Reshooting with 4D Point Clouds
Apr 23, 2026Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca +9Camera-Controlled Video Generation 4D Reconstruction
1Eyeline Labs · 3Columbia University · 4UCLA +3
Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion
Oct 18, 2026Yushi Lan, Zeren Jiang, Koichi Namekata +3Camera-Controlled Video Generation Dynamic Novel View Synthesis
University of Oxford · Nanyang Technological University
Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks
Jun 14, 2026Feng Qiao, Zhaochong An, Zhexiao Xiong +2Multi-View Consistency Camera-Controlled Video Generation
Washington University in St. Louis · University of Copenhagen

Apr 23, 2026J/K move · Enter open · S save

Vista4D: Video Reshooting with 4D Point Clouds

Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca, Yash Kant+8

1Eyeline Labs · 3Columbia University · 4UCLA+3

Camera-Controlled Video Generation 4D Reconstruction Dynamic Scene Reconstruction 4D Scene Understanding

Open paper AskPDF arXiv

We present Vista4D, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud. Specifically, given an input video, our method re-synthesizes the scene with the same dynamics from a different camera trajectory and viewpoint. Existing video reshooting methods often struggle with depth estimation artifacts of real-world dynamic videos, while also failing to preserve content appearance and failing to maintain precise camera control for challenging new trajectories. We build a 4D-grounded point cloud representation with static pixel segmentation and 4D reconstruction to explicitly preserve seen content and provide rich camera signals, and we train with reconstructed multiview dynamic data for robustness against point cloud artifacts during real-world inference. Our results demonstrate improved 4D consistency, camera control, and visual quality compared to state-of-the-art baselines under a variety of videos and camera paths. Moreover, our method generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition. See our project page for results, code, and models: https://eyeline-labs.github.io/Vista4D

Abstract

Camera-Controlled Video Generation

Explore similar work

Vista4D: Video Reshooting with 4D Point Clouds

Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks