Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations (≈2.67s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of 0.07s per-frame at 1024×768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m
Figures & tables
Figure 1: Illustration of the proposed LiveStre4m method, a feed-forward model for live-streaming novel viewpoint video from two or more low-resolution input streams.
Figure 2: Overviews of LiveStre4m model architecture. The model receives multi-view video keyframes, the first such keyframe is used by the Camera Pose Predictor to regress camera poses. Then, the images and predicted camera information are processed by the Spatial Module to generate a low-resolution novel viewpoint of the scene. After two keyframes are processed, NIN interpolates and increases image resolution to generate the high-resolution video snippet. Snippets are accumulated in a temporally consistent video stream. To accumulate two input keyframe, the generated video has a small delay of less than a second when live-streaming novel viewpoint video.
Figure 3: Representation of the Spatial Module architecture. The module leverages ViTs to model scene 3D geometry and appearance as well as a VGG model to extract visual features. By combining appearance features extracted and the predicted geometry, a DPT head predicts 3D Gaussian parameters centered on the predicted pointmap to achieve photorealistic NVS.
Model
Neural3DVideo [ 14 ]
MeetRoom [ 13 ]
Runtime (s) ↓
PSNR ↑
Camera Free
#Views ↓
Runtime (s) ↓
PSNR ↑
Camera Free
#Views ↓
Video Optimization
K-planes [ 7 ]
48.00
32.17
✗
≥ 19
–
–
–
–
4DGS [ 31 ]
7.80
32.70
✗
≥ 19
–
–
–
–
Spacetime-GS [ 15 ]
48.00
33.71
✗
≥ 19
–
–
–
–
Frame Optimization
Table 1: Quantitative comparison of dynamic scene reconstruction methods on Neural3DVideo [ 14 ] and MeetRoom [ 13 ] datasets at 1352×1014 and 1280×720 resolution, respectively. Reported metrics include average runtime, PSNR, number of input views, and whether methods are camera-free. For a fair comparison, LiveStre4m is evaluated on the same A100 GPU as reported by IGS.
Model
Neural3DVideo [ 14 ]
MeetRoom [ 13 ]
Runtime (s) ↓
PSNR ↑
Resolution ↑
Pose Free
#Views ↓
Runtime (s) ↓
PSNR ↑
Resolution ↑
Pose Free
#Views ↓
FLARE [ 41 ]
0.249
21.45
512×384
✓
2
0.243
16.65
512×384
✓
2
LiveStre4m (Ours)
0.074
21.11
1024×768
✓
2
0.074
18.65
1024×768
✓
2
LiveStre4m (Ours)
0.062
22.44
512×384
✓
2
0.062
19.32
512×384
✓
2
Table 2: Quantitative comparison of feed-forward scene reconstruction methods on Neural3DVideo [ 14 ] and MeetRoom [ 13 ] datasets on a single H100 GPU
Figure 4: Qualitative results produced by with LiveStre4m, which synthesizes the target viewpoint using only two neighboring input views, without requiring optimization or ground-truth camera parameters. All images are shown at the same resolution as in Tab. 1 From top to bottom: sear_steak [ 14 ] , discussion [ 13 ] and trimming [ 13 ] scene results are provided.
Output Res. (p)
Runtime (s) ↓
PSNR ↑
512×384
0.062
20.57
640×512
0.063
20.93
1024×768
0.074
19.63
1152×768
0.078
19.54
1536×1152
0.124
17.34
Table 3: LiveStre4m performance comparison using different image resolutions, with the same 2 input views and target view, computed on a single H100 GPU.
Distance
# Inputs
Runtime (s) ↓
PSNR ↑
Closest
2
0.074
21.11
4
0.085
20.82
Intermediate
2
0.074
20.24
4
0.085
20.05
Distant
2
0.074
18.83
4
0.084
18.02
Table 4: Performance of LiveStre4m under varying input view configurations, considering both distance to the target view and the total number of input views, evaluated at 1024×768 resolution. Due to the camera distribution in the MeetRoom dataset [ 13 ] , consistent categorization into closest, intermediate, and distant views is not feasible. Therefore, results are reported only on the Neural3DVideo dataset [ 14 ] .
# Outputs
Runtime (s) ↓
PSNR ↑
1
0.075
19.63
2
0.079
19.24
4
0.088
18.90
Table 5: Ablation study on the number of synthetic views generated by LiveStre4m at 1024×768 resolution using a single H100 GPU.
Component
Runtime (ms)
Camera Pose Predictor
1.5
Spatial Module
52.1
Gaussian Rendering
9.6
Interpolation Net
19.3
Super Res Net
0.6
Table 6: Average runtime of each model component over a 300-frame video at 1024×768 resolution, measured on a single NVIDIA H100 GPU.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Qualitative comparison of reference and predicted viewpoints at frame 150 across all five scenes, rendered at 1024×768 resolution.
Output Res. ( p )
Scene
PSNR ↑
Runtime (s) ↓
1024×768
cut
20.43
0.074
sear
21.79
0.074
discussion
18.75
0.074
trimming
17.94
0.074
vrheadset
19.24
0.074
512×384
cut
21.16
0.062
Appendix
Table 7: Quantitative results on the scenes described in Sec. 4.1 using LiveStre4m, evaluated at two image resolutions in a single NVIDIA H100 GPU.
Model
VRU-Basketball
Runtime (s) ↓
PSNR ↑
Resolution ↑
FLARE [ 41 ]
0.248
17.42
512×384
LiveStre4m (Ours)
0.076
17.88
1024×768
LiveStre4m (Ours)
0.061
18.68
512×384
Appendix
Table 8: Quantitative results on the VRU-Basketball dataset [ 33 , 32 ] obtained with two feed-forward methods, the proposed LiveStre4m (ours) and FLARE [ 41 ] . Results obtained in a single H100 GPU.
Figure 6: Qualitative results comparing different time steps of the generated video of the GZ scene of the VRU-Basketball dataset [ 33 , 32 ] with the expected outputs rendered at 1024×768 resolution.
Model
Resolution
RRA@ 5∘↑
RTA@ 5∘↑
AUC@ 30∘↑
FLARE [ 41 ]
512×384
100.00
60.00
91.66
LiveStre4m
1024×768
100.00
60.00
92.09
LiveStre4m
512×384
100.00
60.00
83.00
Appendix
Table 9: Quantitative comparison of camera pose prediction accuracy. Metrics reported are RRA@ 5∘ , RTA@ 5∘ , and the combined AUC@ 30∘ (average of rotation and translation AUC).
We tackle the problem of sparse novel view synthesis (NVS) using video diffusion models; given K (≈5) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be invariant to permutations of that input set. To this end, we present FrameCrafter, which adapts video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional embeddings. Our results suggest that video models can be easily trained to "forget" about time with minimal supervision, producing competitive performance on sparse-view NVS benchmarks. Project page: https://frame-crafter.github.io/
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
Feed-forward 3D Gaussian Splatting (3DGS) allows efficient and high-fidelity novel view synthesis (NVS) from an offline recorded image sequence. However, achieving online NVS from streaming and unposed image inputs remains challenging. Although online feed-forward geometric estimation methods have been proposed for streaming depth and point cloud recovery, they cannot be adapted to NVS due to severe rendering artifacts. This is because NVS demands stricter multi-view consistency in Gaussian scales and pose-geometry alignment; even minor deviations would accumulate over time and visibly degrade rendering quality. To this end, we propose FreeStreamGS, a robust online feed-forward framework for efficient and high-quality NVS. We introduce two key mechanisms: a Decoupled Intrinsic Recovery Head that removes cumulative camera intrinsic bias and prevents scene scale jitter during long-term streaming, and a Dynamic Point Refinement Offset strategy that relaxes rigid unprojection to correct coupled pose-depth drift. Extensive experiments show that FreeStreamGS achieves rendering quality competitive with state-of-the-art offline feed-forward 3DGS methods, despite operating without access to future frames.
Ruiyang Chen, Feiran Li, Chu Zhou +3
Beijing University of Posts and Telecommunications, China · Independent Researcher · National Institute of Informatics, Japan