3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

Stefan Schulz; Fernando Edelstein; Hannah Dröge; Matthias B. Hullin; Markus Plack

cs.CVApr 13, 2026

3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

Authors: Stefan Schulz, Fernando Edelstein, Hannah Dröge, Matthias B. Hullin, Markus Plack

Organizations: University of Bonn

Abstract

Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/

Figures & tables

					Single GPU Reconstruction
Method	Venue	Real-time	Multi-Object	# Cameras	online	offline
Nerfacto-big Tancik et al. (2023)	SIG ’23	✘	✔	$>$ 30	✘	✘
Splatfacto-big Tancik et al. (2023)	’24	✘	✔	$>$ 30	✘	✘
RIFTCast Zingsheim et al. (2025)	ACM MM ’25	✔	✔	$>$ 10	✘	✔
ENeRF Lin et al. (2022)	SA ’22	(✔)	✔	$>$ 2	✔	✔
GPS-Gaussian Zheng et al. (2024)	CVPR ’24	(✔)	(✔)	2	✔	✔

Table 1: Method capabilities and limitations overview.

Figure 2: Overview of our real-time view interpolation framework. A lightweight Ghost-based backbone extracts multi-scale feature pyramids from the input images (left). A coarse-to-fine module then estimates depth and alpha for the target view using grouped correlations and residual refinement (center). Finally, a hierarchical fusion decoder aggregates projected features and synthesizes the novel RGB image (right).

Figure 3: Details on the Correlation Block (left) and Proj Block (right) from Fig. 2 . The correlation block computes pairwise group correlations from depth-projected features and masks, processes them with depth proposals in a shared encoder–decoder, and refines the fused latents with the upsampled depth. The projection block warps features using the current depth, predicts per-view weights, and aggregates them with a ConvNet into a latent representation.

		RIFTCast Zingsheim et al. (2025)			DNA Rendering Cheng et al. (2023)			LLFF Mildenhall et al. (2020)
		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Dense	Nerfacto-big* Tancik et al. (2023) (x)	26.2	0.891	0.078	29.5	0.920	0.061	(11.7)	(0.308)	(0.706)
	Splatfacto-big* Tancik et al. (2023) (x)	30.7	0.920	0.058	34.5	0.976	0.028	(13.7)	(0.438)	(0.669)
	RIFTCast Zingsheim et al. (2025) (x)	29.1	0.953	0.033	29.9	0.932	0.030	n/a	n/a	n/a
Sparse	FrugalNeRF* Lin et al. (2025) (3)	23.0	0.894	0.114	24.2	0.928	0.098	19.1	0.609	0.273
	ENeRF Lin et al. (2022) (3)	24.9	0.939	0.072	26.0	0.949	0.073	21.3	0.712	0.209

Table 2: Quantitative comparison across multiple datasets for human capture and scenes without humans. Values in parentheses indicate the number of source views used for novel view synthesis, where (x) denotes all available views. Colored boxes show the best and second-best results per metric of sparse-view methods (2-3 images). LLFF represents a challenging out-of-distribution scenario with complex scene geometry, which lies outside the training distribution of our model.

Panel of Figure 4 — Figure 4: Qualitative comparison on multiple datasets for human capture and non-human containing scenes with real-time capable methods. While the first method, RIFTCast Zingsheim et al. (2025) , requires all views available and isolated scene content, the other methods can achieve novel view synthesis with 2 to 3 views.

	1024 x 1024			2048 x 2048
	Memory $\downarrow$	Training time $\downarrow$	Inference time $\downarrow$	Memory $\downarrow$	Training time $\downarrow$	Inference time $\downarrow$
Nerfacto-big* Tancik et al. (2023) (x)	12.1 GB	100.3 min	7.8 ms	12.8 GB	108.3 min	9.6 ms
Splatfacto-big* Tancik et al. (2023) (x)	0 1.2 GB	11.2 min	1.3 ms	0 2.6 GB	13.1 min	1.8 ms
FrugalNeRF* Lin et al. (2025) (3)	20.2 GB	12.3 min	13.2 ms	20.4 GB	16.2 min	16.5 ms
ENeRF Lin et al. (2022) (3)	0 8.1 GB	0 ms	97.3 ms	13.1 GB	0 ms	233.7 ms
GPS-Gaussian Zheng et al. (2024) (2)	0 3.2 GB	0 ms	73.7 ms	0 8.1 GB	0 ms	119.2 ms

Table 3: Memory and Speed comparisons of the tested methods on a NVIDIA RTX 4090 GPU using the official implementations. We also provide a version of our method optimized with TensorRT (Ours RT ) to allow for real-time usage.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Memory $\downarrow$	FPS $\uparrow$
full model	25.9	0.952	0.074	2.2 GB	40.8
2 views	23.6	0.913	0.089	1.7 GB	49.4
# channels halved	24.9	0.927	0.090	2.2 GB	41.3
# channels doubled	26.1	0.952	0.072	2.6 GB	32.6
3-level pyramid	22.8	0.893	0.112	2.2 GB	42.4
6-level pyramid	25.8	0.953	0.077	2.2 GB	41.0

Table 4: Quantitative comparison of different model variants with TensorRT optimization. Showcase of the improvements and sweet spot our different modules and choices bring with them at 1024

\times

1024 resolution on DNA Rendering dataset.

Panel of Figure 5 — Figure 5: Generalization behavior of 3DTV across geometry, viewpoint, and resolution. Despite synthetic-only training, the model predicts coherent depth and stable synthesis under varying camera depths and scales to higher output resolutions without retraining.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Exemplary Delaunay triangulation for the RIFTCast [ 9 ] capture stage setup. The scene has been scaled for visualization purposes.

Figure 7: Evaluation of hyperparameter choices and their influence on the resulting triangulation.

Figure 8: (Cont.) Evaluation of hyperparameter choices and their influence on the resulting triangulation.

Loss Term	Phase 1.1 ( $\leq 25$ )	Phase 1.2 ( $26-100$ )	Phase 2 ( $>100$ )
$\lambda_{1}$	2.0	2.0	2.0
$\lambda_{2}$	2.0	2.0	2.0
$\lambda_{3}$	1.0	1.0	0.5
$\lambda_{4}$	1.0	1.0	0.5
$\lambda_{5}$	1.0	1.0	1.0
$\lambda_{6}$	0.0	0.5	0.5

Appendix

Table 5: Loss Weighting Schedule. Weights for each loss component across the three training phases.

Scene	Train Views	Test View
Fern	IMG_4035, IMG_4041, IMG_4029	IMG_4043
Flower	IMG_2962, IMG_2970, IMG_2983	IMG_2984
Horns	DJI_2020…842, DJI_2020…122, DJI_2020…633	DJI_2020…048
Leaves	IMG_2997, IMG_3004, IMG_3011	IMG_3010
Orchids	IMG_4468, IMG_4472, IMG_4476	IMG_4470
Room	DJI_2020…006, DJI_2020…455, DJI_2020…082	DJI_2020…910

Appendix

Table 6: LLFF View Selection. Training views for 2-view (*) and 3-view setups. Test views are held out for all methods; offline baselines utilize all remaining views.

Figure 9: Snap-Snap produces significant ghosting and duplication artifacts when rendering the frontal view from the RIFTCast [ 9 ] dataset.

Figure 10: Example of reconstruction failure in DNA Rendering [ 85 ] , where Snap-Snap fails to properly fuse the front and back views, resulting in duplicated geometry.

Panel of Figure 11 — Figure 11: Failure analysis of FWD: The model fails to maintain proper object proportions and exhibits significant blurring compared to the ground truth.

Panel of Figure 12 — Figure 12: Qualitative comparison of our method to online and offline reconstruction methods. (Part 1.1)

Panel of Figure 13 — Figure 13: Qualitative comparison of our method to online and offline reconstruction methods. (Part 1.2)

Panel of Figure 14 — Figure 14: Qualitative comparison of our method to online and offline reconstruction methods. (Part 2.1)

Panel of Figure 15 — Figure 15: Qualitative comparison of our method to online and offline reconstruction methods. (Part 2.2)

Panel of Figure 16 — Figure 16: Qualitative evaluation (Part 1) of our model and the different influences of individual parts to our overall network.

Panel of Figure 17 — Figure 17: Qualitative evaluation (Part 2) of our model and the different influences of individual parts to our overall network.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Full model (3 views)	25.7	0.941	0.073
2 views	23.0	0.909	0.093
# channels halved	24.8	0.915	0.094
# channels doubled	26.1	0.950	0.073
3-level pyramid	22.4	0.888	0.118
6-level pyramid	25.5	0.939	0.080

Appendix

Table 7: Quantitative Ablation on RIFTCast [ 9 ] . Evaluation of model variants with TensorRT optimization at

1024\times 1024

resolution. The results highlight the performance trade-offs regarding channel depth, pyramid levels, and our proposed depth modules.

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 292 papers

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras
Sep 14, 2026Hanzhang Tu, Zhanfeng Liao, Wei Min +23D Gaussian Splatting Camera Pose Estimation
Tsinghua University, Beijing, China · Shadow AI, Beijing, China
Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups
Mar 5, 2026Leif Van Holland, Domenic Zingsheim, Mana Takhsha +4Image Inpainting Streaming 3D Reconstruction
University of Bonn, Germany
Fully Distributed Multi-View 3D Tracking in Real-Time
Jun 11, 2026Byron Hernandez, Fangyu Li, Aotian Wu +33D MOT Multi-Object Tracking
University of Florida, Gainesville, FL, USA · NVIDIA Corporation, Santa Clara, CA, USA

Sep 14, 2026J/K move · Enter open · S save

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras

Hanzhang Tu, Zhanfeng Liao, Wei Min, Jiajun Zhang+1

Tsinghua University, Beijing, China · Shadow AI, Beijing, China

3D Gaussian Splatting Camera Pose Estimation Dynamic Scene Reconstruction Sparse-View 3D Reconstruction 3D Human Reconstruction Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Live free-viewpoint visualization of real humans is critical for immersive communication and interactive digital experiences. Existing methods either rely on computationally expensive optimization or require calibrated cameras and low-resolution inputs, making real-time high-resolution deployment impractical. In this work, we present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization from sparse, unposed RGB streams. Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance in a single forward pass. To achieve this, we start by designing a lightweight sparsity-aware multi-view transformer backbone that tokenizes foreground human regions while preserving global context through a shared scene token. We then employ a fully transformer-based Gaussian decoder to mitigate convolution-induced over-smoothing while keeping decoding sparse and efficient. In addition, we introduce a hybrid feature pyramid that injects multi-scale appearance cues into geometry prediction. We further introduce a lightweight differentiable Levenberg-Marquardt camera refinement layer to enhance multi-view consistency and geometric alignment. Moreover, to stabilize learning under sparse, unposed inputs, we transfer multi-view geometry priors from a large visual-geometry foundation model via teacher-student distillation. Finally, the predicted Gaussian maps are streamed with video codecs to remote devices for interactive free-viewpoint rendering. Extensive experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU. Additional captured sequences illustrate its performance across varied subjects, clothing, and motions under our multi-camera setup.

Abstract

Novel View Synthesis

Explore similar work

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras

Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups

Fully Distributed Multi-View 3D Tracking in Real-Time