LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video

Pedro Quesado; Erkut Akdag; Yasaman Kashefbahrami; Willem Menu; Egor Bondarev

cs.CVApr 8, 2026

LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video

Authors: Pedro Quesado, Erkut Akdag, Yasaman Kashefbahrami, Willem Menu, Egor Bondarev

Organizations: AIMSGroup, Department of Electrical Engineering, Eindhoven University of Technology

Abstract

Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations ( $\approx 2.67$ s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of $0.07$ s per-frame at $1024 \times 768$ resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m

Figures & tables

Figure 1: Illustration of the proposed LiveStre4m method, a feed-forward model for live-streaming novel viewpoint video from two or more low-resolution input streams.

Figure 2: Overviews of LiveStre4m model architecture. The model receives multi-view video keyframes, the first such keyframe is used by the Camera Pose Predictor to regress camera poses. Then, the images and predicted camera information are processed by the Spatial Module to generate a low-resolution novel viewpoint of the scene. After two keyframes are processed, NIN interpolates and increases image resolution to generate the high-resolution video snippet. Snippets are accumulated in a temporally consistent video stream. To accumulate two input keyframe, the generated video has a small delay of less than a second when live-streaming novel viewpoint video.

Figure 3: Representation of the Spatial Module architecture. The module leverages ViTs to model scene 3D geometry and appearance as well as a VGG model to extract visual features. By combining appearance features extracted and the predicted geometry, a DPT head predicts 3D Gaussian parameters centered on the predicted pointmap to achieve photorealistic NVS.

Video Optimization
Model	Neural3DVideo [ 14 ]				MeetRoom [ 13 ]
Model	Runtime (s) $\downarrow$	PSNR $\uparrow$	Camera Free	#Views $\downarrow$	Runtime (s) $\downarrow$	PSNR $\uparrow$	Camera Free	#Views $\downarrow$
K-planes [ 7 ]	48.00	32.17	✗	$\geq$ 19	–	–	–	–
4DGS [ 31 ]	7.80	32.70	✗	$\geq$ 19	–	–	–	–
Spacetime-GS [ 15 ]	48.00	33.71	✗	$\geq$ 19	–	–	–	–
Frame Optimization

Table 1: Quantitative comparison of dynamic scene reconstruction methods on Neural3DVideo [ 14 ] and MeetRoom [ 13 ] datasets at

1352\times 1014

and

1280\times 720

resolution, respectively. Reported metrics include average runtime, PSNR, number of input views, and whether methods are camera-free. For a fair comparison, LiveStre4m is evaluated on the same A100 GPU as reported by IGS.

Model	Neural3DVideo [ 14 ]					MeetRoom [ 13 ]
Model	Runtime (s) $\downarrow$	PSNR $\uparrow$	Resolution $\uparrow$	Pose Free	#Views $\downarrow$	Runtime (s) $\downarrow$	PSNR $\uparrow$	Resolution $\uparrow$	Pose Free	#Views $\downarrow$
FLARE [ 41 ]	0.249	21.45	$512\times 384$	✓	2	0.243	16.65	$512\times 384$	✓	2
LiveStre4m (Ours)	0.074	21.11	$1024\times 768$	✓	2	0.074	18.65	$1024\times 768$	✓	2
LiveStre4m (Ours)	0.062	22.44	$512\times 384$	✓	2	0.062	19.32	$512\times 384$	✓	2

Table 2: Quantitative comparison of feed-forward scene reconstruction methods on Neural3DVideo [ 14 ] and MeetRoom [ 13 ] datasets on a single H100 GPU

Figure 4: Qualitative results produced by with LiveStre4m, which synthesizes the target viewpoint using only two neighboring input views, without requiring optimization or ground-truth camera parameters. All images are shown at the same resolution as in Tab. 1 From top to bottom: sear_steak [ 14 ] , discussion [ 13 ] and trimming [ 13 ] scene results are provided.

Output Res. (p)	Runtime (s) $\downarrow$	PSNR $\uparrow$
$512\times 384$	0.062	20.57
$640\times 512$	0.063	20.93
$1024\times 768$	0.074	19.63
$1152\times 768$	0.078	19.54
$1536\times 1152$	0.124	17.34

Table 3: LiveStre4m performance comparison using different image resolutions, with the same 2 input views and target view, computed on a single H100 GPU.

Distance	# Inputs	Runtime (s) $\downarrow$	PSNR $\uparrow$
Closest	2	0.074	21.11
Closest	4	0.085	20.82
Intermediate	2	0.074	20.24
Intermediate	4	0.085	20.05
Distant	2	0.074	18.83
Distant	4	0.084	18.02

Table 4: Performance of LiveStre4m under varying input view configurations, considering both distance to the target view and the total number of input views, evaluated at

1024\times 768

resolution. Due to the camera distribution in the MeetRoom dataset [ 13 ] , consistent categorization into closest, intermediate, and distant views is not feasible. Therefore, results are reported only on the Neural3DVideo dataset [ 14 ] .

# Outputs	Runtime (s) $\downarrow$	PSNR $\uparrow$
1	0.075	19.63
2	0.079	19.24
4	0.088	18.90

Table 5: Ablation study on the number of synthetic views generated by LiveStre4m at

1024\times 768

resolution using a single H100 GPU.

Component	Runtime (ms)
Camera Pose Predictor	$1.5$
Spatial Module	$52.1$
Gaussian Rendering	$9.6$
Interpolation Net	$19.3$
Super Res Net	$0.6$

Table 6: Average runtime of each model component over a 300-frame video at

1024\times 768

resolution, measured on a single NVIDIA H100 GPU.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative comparison of reference and predicted viewpoints at frame 150 across all five scenes, rendered at $1024\times 768$ resolution.

Output Res. ( $p$ )	Scene	PSNR $\uparrow$	Runtime (s) $\downarrow$
$1024\times 768$	cut	20.43	0.074
	sear	21.79	0.074
	discussion	18.75	0.074
	trimming	17.94	0.074
	vrheadset	19.24	0.074
$512\times 384$	cut	21.16	0.062

Appendix

Table 7: Quantitative results on the scenes described in Sec. 4.1 using LiveStre4m, evaluated at two image resolutions in a single NVIDIA H100 GPU.

Model	VRU-Basketball
Model	Runtime (s) $\downarrow$	PSNR $\uparrow$	Resolution $\uparrow$
FLARE [ 41 ]	0.248	17.42	$512\times 384$
LiveStre4m (Ours)	0.076	17.88	$1024\times 768$
LiveStre4m (Ours)	0.061	18.68	$512\times 384$

Appendix

Table 8: Quantitative results on the VRU-Basketball dataset [ 33 , 32 ] obtained with two feed-forward methods, the proposed LiveStre4m (ours) and FLARE [ 41 ] . Results obtained in a single H100 GPU.

Figure 6: Qualitative results comparing different time steps of the generated video of the GZ scene of the VRU-Basketball dataset [ 33 , 32 ] with the expected outputs rendered at $1024\times 768$ resolution.

Model	Resolution	RRA@ $5^{\circ}$ $\uparrow$	RTA@ $5^{\circ}$ $\uparrow$	AUC@ $30^{\circ}$ $\uparrow$
FLARE [ 41 ]	$512\times 384$	100.00	60.00	91.66
LiveStre4m	$1024\times 768$	100.00	60.00	92.09
LiveStre4m	$512\times 384$	100.00	60.00	83.00

Appendix

Table 9: Quantitative comparison of camera pose prediction accuracy. Metrics reported are RRA@

5^{\circ}

, RTA@

5^{\circ}

, and the combined AUC@

30^{\circ}

(average of rotation and translation AUC).

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 289 papers

FrameCrafter: Novel View Synthesis as Video Completion
Apr 9, 2026Qi Wu, Khiem Vuong, Minsik Jeon +2Video Diffusion Models Novel View Synthesis
Carnegie Mellon University
BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model
Feb 26, 2026Yuci Han, Charles Toth, John E. Anderson +2Novel View Synthesis Multi-View Image Synthesis
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
FreeStreamGS: Online Feed-forward 3D Gaussian Splatting from Unposed Streaming Inputs
Jun 2, 2026Ruiyang Chen, Feiran Li, Chu Zhou +33D Gaussian Splatting Streaming 3D Reconstruction
Beijing University of Posts and Telecommunications, China · Independent Researcher · National Institute of Informatics, Japan

Apr 9, 2026J/K move · Enter open · S save

FrameCrafter: Novel View Synthesis as Video Completion

Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan+1

Carnegie Mellon University

Video Diffusion Models Novel View Synthesis Permutation-Invariant Learning Multi-View Image Synthesis

Open paper AskPDF arXiv

We tackle the problem of sparse novel view synthesis (NVS) using video diffusion models; given

K

(

\approx 5

) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be

\textit{invariant}

to permutations of that input set. To this end, we present FrameCrafter, which adapts video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional embeddings. Our results suggest that video models can be easily trained to "forget" about time with minimal supervision, producing competitive performance on sparse-view NVS benchmarks. Project page: https://frame-crafter.github.io/

Abstract

Novel View Synthesis

Explore similar work

FrameCrafter: Novel View Synthesis as Video Completion

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model

FreeStreamGS: Online Feed-forward 3D Gaussian Splatting from Unposed Streaming Inputs