Video re-shooting is the task of re-rendering a monocular video of a dynamic scene along a user-specified camera trajectory. A dominant paradigm explicitly provides the target geometry by lifting the source video into a 4D point cloud using per-frame depth, and rasterizing this representation along the target trajectory to obtain point-cloud renders as geometric conditions for trajectory-controlled generation. The point cloud render and source video may provide conflicting visual cues throughout denoising, as both are directly used as conditions. We refer to this conflict as a trust dilemma, which can compromise trajectory control, 3D consistency of dynamic content, and visual quality, particularly under distribution shifts. We argue that because a point-cloud render is pixel-aligned with the target view, it does not need to be continuously provided as a conditioning signal throughout denoising. We propose Manifold4D, which injects the render directly into the initial noise of flow matching, so that generation no longer departs from standard Gaussian noise but from a geometry-bearing noise distribution that we term the point cloud rendered manifold, leaving the source video as the only visual condition. The render is thus used exactly once, and the network is never asked to learn how to read it; in subsequent denoising steps the model can focus on the source video. In our experiments, Manifold4D attains the best camera-control accuracy on every metric, lowering rotation error by 25--27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and attaining the best overall photometric quality and 3D consistency of dynamic content.
Figures & tables
Figure 1: Video re-shooting. Given a monocular video and a target camera trajectory, Manifold4D re-shoots the video along the new trajectory, achieving the best trajectory control among existing methods while preserving visual quality. The bottom panel shows a re-shooting example. GEN3C and TrajectoryCrafter inherit the artifacts of the imperfect render, while Vista4D leaves the subject misaligned with the render geometry. In the top-right plot, the vertical axis averages the VBench aesthetic and imaging quality scores; the horizontal axis denotes the camera-control error relative to the best method (see Appendix A for details).
Figure 2: How each method consumes the render. Prior methods concatenate the render at every denoising step, so the network must learn to interpret and utilize it. Ours directly adds it into the starting noisy latent, requiring no learning. This explicit, one-time consumption is the key difference.
Figure 3: Pipeline overview. Given a monocular video and a target camera trajectory, Manifold4D takes a 4D point cloud obtained from an existing reconstruction and motion decomposition pipeline and renders target-trajectory views from it. The rendered views are encoded into VAE latents, which are injected into the noise and patchified into tokens. The source video is similarly encoded into tokens, concatenated with the noise tokens, and fed to the DiT for denoising, together with the encoded camera trajectory.
Method
DAVIS-Traj (72 clips)
Vista4D-Eval (103 clips)
Vista4D-Eval raw (110 clips)
RotErr ↓
TransErr ↓
IntrErr ↓
RotErr ↓
TransErr ↓
IntrErr ↓
RotErr ↓
TransErr ↓
IntrErr ↓
ReCamMaster
14.04
0.581
2.77
11.81
0.495
10.63
13.73
0.549
11.23
TrajectoryCrafter
1.56
0.274
1.14
5.40
0.462
8.40
8.24
0.518
9.14
GEN3C
1.45
0.236
1.13
3.77
0.441
8.74
4.31
0.460
9.33
Vista4D
2.56
0.240
1.47
3.89
0.403
7.66
4.84
0.432
8.10
Ours
1.09
0.161
1.04
2.76
0.394
7.65
4.26
0.424
8.09
Table 1: Camera control accuracy. Manifold4D attains the highest accuracy on every metric, approaching or surpassing the point cloud render itself on rotation error. Bold denotes the best and underline the second-best result.
Figure 4: Control error and visual quality under growing motion amplitude. Manifold4D maintains the lowest pose error at almost every amplitude, staying close to the point cloud render, while other methods degrade markedly. Visual quality (Aesthetic and Imaging) likewise remains the highest, especially at large amplitudes. Red tick labels mark the amplitudes of DAVIS-Traj .
Method
PSNR ↑
SSIM ↑
LPIPS ↓
mPSNR ↑
mSSIM ↑
mLPIPS ↓
EPE ↓
ReCamMaster
10.89
0.3363
0.7161
11.00
0.3326
0.5376
6.66
TrajectoryCrafter
15.52
0.4338
0.5162
16.41
0.4476
0.3494
1.58
GEN3C
14.75
0.3937
0.4609
15.87
0.4119
0.3103
1.47
Vista4D
15.78
0.3913
0.3884
16.30
0.3997
0.2600
1.28
Ours
16.21
0.3847
0.3582
16.65
0.3887
0.2393
1.25
Table 2: Novel view photometric quality on iPhone Dataset. Manifold4D achieves state-of-the-art performance across key photometric and optical-flow metrics.
Method
DAVIS-Traj
Vista4D-Eval
FID ↓
FVD ↓
Aesthetic ↑
Imaging ↑
FID ↓
FVD ↓
Aesthetic ↑
Imaging ↑
ReCamMaster
65.55
1122.9
48.099
57.204
92.79
1181.1
52.338
58.694
TrajectoryCrafter
85.27
1571.6
46.899
57.038
116.31
1581.4
47.579
63.300
GEN3C
89.76
1463.9
47.245
53.218
113.43
1534.3
48.852
62.001
Vista4D
55.64
1208.2
51.096
64.914
98.12
1404.6
55.012
70.933
Ours
57.55
1222.7
51.138
65.384
97.37
1360.2
53.757
70.509
Table 3: Visual quality. FID/FVD are computed against the corresponding source clips; Aesthetic and Imaging are VBench scores ( ×100 ).
Figure 5: Qualitative results on monocular input. Dashed guides are placed on the point cloud render and copied to every row: green marks the expected position of the dynamic subject, red a static background structure. Our method shows stable control and pleasing visual quality while Vista4D’s dynamic subject drifts away from the green guide. GEN3C and TrajectoryCrafter inherit the sparsity of the render, leaving subjects fragmented; ReCamMaster stays clean but ignores the requested view.
Criterion
ReCamMaster
TrajectoryCrafter
GEN3C
Vista4D
Ours
Trajectory following
1.4%
17.0%
20.5%
28.3%
63.6%
3D consistency
1.8%
15.6%
9.5%
8.7%
62.4%
Overall quality
12.9%
3.2%
5.8%
54.7%
49.6%
Table 4: User study. Selection rate in %. Manifold4D is preferred by large margins on trajectory following and 3D consistency of the dynamic subject, and remains competitive on overall quality.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Trajectory patterns of DAVIS-Traj . Each pattern orbits the dynamic subject while keeping it on the optical axis. f1 (blue) sweeps ±60 ∘ of yaw; f2 (green) sweeps ±30 ∘ of yaw with −10 ∘ of pitch at 1.3× distance; f3 (yellow) sweeps ±30 ∘ of yaw with +10 ∘ of pitch while dollying in from 1.0 to 0.7 . The red curve shows the source camera trajectory.
Method
Base model
Output
Time (min)
ReCamMaster
Wan2.1-T2V-1.3B
81 × 480 × 832
9.0
TrajectoryCrafter
CogVideoX-Fun 5B
49 × 384 × 672
2.7
GEN3C
Cosmos-Predict1 7B
121 × 704 × 1280
18.5
Vista4D
Wan2.1-T2V-14B
49 × 384 × 672
19.3
Ours
Wan2.1-T2V-14B
49 × 384 × 672
13.0
Appendix
Table 6: Base models and inference time. All methods run with 50 diffusion steps on a single NVIDIA A100 80 GB GPU and generate at their native resolution.
Method
DAVIS-Traj (72 clips)
Vista4D-Eval (106 clips)
Vista4D-Eval raw (110 clips)
RotErr ↓
TransErr ↓
IntrErr ↓
RotErr ↓
TransErr ↓
IntrErr ↓
RotErr ↓
TransErr ↓
IntrErr ↓
ReCamMaster
14.00
0.586
3.19
13.23
0.549
10.67
13.56
0.563
10.67
TrajectoryCrafter
1.79
0.256
1.52
6.62
0.500
7.55
7.71
0.504
7.57
GEN3C
1.88
0.267
1.41
4.05
0.438
7.52
4.25
0.433
7.52
Vista4D
2.90
0.304
2.06
4.07
0.382
7.23
4.67
0.388
7.26
Ours
1.56
0.247
1.59
3.82
0.419
6.96
4.39
0.427
6.97
Appendix
Table 7: Camera control accuracy with Pi3 reconstruction. Left : filtered (RotErr ≤60∘ in any method, maximum common clip set). Right : raw Vista4D-Eval (all 110 clips, no filtering). DAVIS-Traj has no filtered clips so only one column group is shown. 3-seed mean. Bold denotes the best and underline denotes the second best among generated methods (Point cloud render excluded from ranking and typeset in italics).
RotErr ↓ (deg)
TransErr ↓ (normalized)
Yaw
10
20
30
40
50
60
70
80
90
10
20
30
40
50
60
70
80
90
ReCamMaster
7.32
9.21
11.44
14.43
17.67
19.55
22.28
38.94
43.19
0.290
0.348
0.510
0.622
0.560
0.689
0.721
0.918
0.853
TrajectoryCrafter
0.41
0.50
0.82
1.39
1.61
2.79
2.93
3.85
7.15
0.028
0.035
0.060
0.600
0.101
0.896
0.539
0.367
0.415
GEN3C
0.98
1.08
1.23
2.01
1.91
2.44
2.92
12.58
6.49
0.038
0.066
0.080
0.557
0.344
0.401
0.496
1.011
0.651
Vista4D
0.58
0.96
1.78
2.19
2.45
2.45
4.53
6.30
6.17
0.029
0.050
0.091
0.103
0.163
0.146
0.206
0.303
0.302
Ours
0.40
0.55
0.79
1.03
1.25
1.35
1.95
2.46
2.77
0.034
0.044
0.066
0.079
0.085
0.110
0.139
0.167
0.200
Appendix
Table 8: Camera control accuracy under growing motion amplitude. RotErr and TransErr at each amplitude, averaged over ten DAVIS-Traj scenes. The amplitude is the per-side yaw angle, from 10∘ to 90∘ (total sweep 20∘ – 180∘ ). The point cloud render is included as a geometric reference and excluded from ranking.
Method
FID ↓
FVD ↓
CLIP-T ↑
SC ↑
BC ↑
MS ↑
AQ ↑
IQ ↑
DAVIS-Traj
ReCamMaster
65.55
1122.9
0.2392
88.333
89.807
97.932
48.099
57.204
TrajectoryCrafter
85.27
1571.6
0.2361
84.866
88.519
95.911
46.899
57.038
GEN3C
89.76
1463.9
0.2298
84.543
90.624
95.627
47.245
53.218
Vista4D
55.64
1208.2
0.2508
86.980
90.420
95.677
51.096
64.914
Ours
57.55
1222.7
0.2444
86.393
89.367
95.122
51.138
65.384
Appendix
Table 10: Full visual quality and temporal stability metrics. FID/FVD are computed against the corresponding source clips, CLIP-T is the frame-wise text-image similarity, and the remaining five columns are VBench scores ( ×100 ): subject consistency (SC), background consistency (BC), motion smoothness (MS), aesthetic quality (AQ) and imaging quality (IQ). Mean over 3 seeds.
Figure 7: Novel-view comparison on real iPhone multi-view captures. Vista4D shows mild 3D inconsistency (red lines) and does not follow the point cloud colors; TrajectoryCrafter and GEN3C appear flat and miss fine human details; ReCamMaster lacks camera control.
Figure 8: Robustness to incorrect dynamic geometry. We omit the motion mask to simulate an erroneous point cloud: dynamic points from all frames are stacked together, smearing the rendered subject. Manifold4D still recovers the correct motion from the source video.
Figure 9: Qualitative ablation comparison. The two architecture variants against the full model on Vista4D-Eval clips. The Vista4D-style three-stream layout exhibits 3D inconsistency on dynamic objects, while the GEN3C-style variant shows noticeable degradation in visual quality in some scenes. The full model preserves both geometric consistency and visual quality.
Figure 10: User study evaluation interface. Left: the point cloud render is overlaid on each generated video (toggleable with the Q key) for the trajectory-following and 3D-consistency questions. Right: the full generated video is played without overlay for the visual-quality question, presented separately to mitigate bias from the preceding overlay-based questions.
We present Vista4D, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud. Specifically, given an input video, our method re-synthesizes the scene with the same dynamics from a different camera trajectory and viewpoint. Existing video reshooting methods often struggle with depth estimation artifacts of real-world dynamic videos, while also failing to preserve content appearance and failing to maintain precise camera control for challenging new trajectories. We build a 4D-grounded point cloud representation with static pixel segmentation and 4D reconstruction to explicitly preserve seen content and provide rich camera signals, and we train with reconstructed multiview dynamic data for robustness against point cloud artifacts during real-world inference. Our results demonstrate improved 4D consistency, camera control, and visual quality compared to state-of-the-art baselines under a variety of videos and camera paths. Moreover, our method generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition. See our project page for results, code, and models: https://eyeline-labs.github.io/Vista4D
Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.
Yushi Lan, Zeren Jiang, Koichi Namekata +3
University of Oxford · Nanyang Technological University
Re-rendering an existing video from a novel camera viewpoint requires the output to follow the prescribed camera trajectory while preserving the appearance and dynamics of the original scene across every frame. Existing methods rely on per-frame pose embeddings, noisy point-cloud renderings, or implicit learned correspondences, none of which provides an explicit, temporally continuous link between source and target pixels. We propose Track2View, which conditions a video diffusion transformer on paired 3D point tracks: sparse trajectories of scene points projected into both the source and target camera views. These tracks provide explicit spatiotemporal correspondences that are temporally continuous by construction, encoding what content should appear where and when. At the core of Track2View is a dual-view track conditioner that transfers visual context from source to target view through parameter-free geometric operations and learned temporal aggregation, ensuring generalization to arbitrary camera trajectories without memorizing specific motions. We further introduce a data curation pipeline that extracts one-to-one track correspondences by running a 3D point tracker on temporally concatenated multi-camera view pairs. On a 400-video benchmark spanning static and dynamic scenes, Track2View achieves state-of-the-art results across visual quality, view synchronization, and camera accuracy, reducing rotation error by 30-65% and translation error by 61-72% relative to leading baselines. Project page is available at this https URL: https://qjizhi.github.io/track2view
Feng Qiao, Zhaochong An, Zhexiao Xiong +2
Washington University in St. Louis · University of Copenhagen