State-of-the-art novel-view synthesis (NVS) methods commonly require per-scene optimization before rendering, limiting their applicability in live settings. Live or ad-hoc NVS, however, demands low-latency rendering during capture and strong generalization to new scenes. Some previous work tried to address this task by adapting training- and optimization-based methods, trading quality for low latency. We introduce a live NVS approach following traditional image-based rendering that operates on a posed set of depth sensor observations and images. Our method leverages explicit, sparse point-based geometry obtained from active sensors, resulting in a sparsely filled depth buffer that is completed in the (novel) target view's screen-space. Using this proxy, a posed set of images is projected into the target frame before fusing the resulting observations through a frequency-aware two-band blending strategy. The pipeline is entirely non-neural and optimization-free, requiring no training data or scene-specific optimization. As a result, our approach reduces the delay between capture and novel-view rendering and does not rely on favorable network generalization. We validate the method's applicability to heterogeneous sensor setups - including handheld RGB-D cameras, automotive LiDAR, and thermal cameras - as well as challenging scenarios, such as low light captures. Our experiments demonstrate that the approach outperforms prior methods in live reconstruction scenarios, while maintaining competitive performance in conventional NVS benchmarks.
Figures & tables
Figure 1
Figure 1: Pipeline architecture. Given posed images and depth observations, our method produces novel views in milliseconds. First, points are rendered to the target view’s screen space to construct a dense depth map through hierarchical depth completion. Auxiliary images are then warped onto the screen-space geometry, while resolving disocclusions. Finally, the images are blended using a two-band approach, weighted by an approximate distance transform as blending weight.
Figure 2: Depth test to resolve disocclusions. When fetching color from auxiliary views IA to target view T , depth map D is rendered as a mesh to IA ( light purple : D as mesh; blue : part of mesh visible in IA ): green : test passes; red : test fails → occlusion; yellow : false negative → test fails but point should be visible.
Figure 3: Naive blending of multiple images causes ghosting and loss of detail. Two-band blending [ BL*03 ] smooths low frequencies while retaining high-frequency detail, mitigating said artifacts. The high-band image has been adjusted for visibility.
LiveNVS
nuScenes
Waymo
Argoverse 2
( 1280×720 / 1920×1080 )
( 1600×900 )
( 1920×1280 )
( 1800×1550 )
Approach
PSNR ↑
LPIPS ↓
Time [ ms ]
PSNR ↑
LPIPS ↓
Time [ ms ]
PSNR ↑
LPIPS ↓
Time [ ms ]
PSNR ↑
LPIPS ↓
Time [ ms ]
Depth Anything 3
15.51
0.4909
6222
17.46
0.5638
6258
19.23
0.5343
4958
16.63
0.5135
3704
Long-LRM
19.93
0.2349
2686
23.55
0.3572
2638
25.23
0.3559
1939
23.54
0.3246
2791
DrivingForward
-
-
-
24.00
0.3221
590
24.36
0.4095
284
20.21
0.5106
285
Inovis
19.01
0.2736
157
22.20
0.4396
146
22.49
0.4157
243
19.80
0.3763
289
Table 1: Quantitative results for live NVS. A dash indicates that a method was not applicable to a dataset.
Figure 4: Qualitative comparison with On-the-fly NVS [ MSL25 ] , Long-LRM [ ZTZ25 ] and DrivingForward [ TTXM25 ] . We state measured capture-to-image times. Views taken from trajectories are highlighted by arrows. Extrapolating lane shifts are marked with .
Figure 5: Qualitative comparison in an offline setting. Total reconstruction time is reported for each method. On-the-fly NVS and Faster-GS processed the scenes completely before rendering, while our approach reconstructs high-quality novel views without training or optimization.
MipNerf 360
RGBT-Scenes
Tanks and Temples
Approach
PSNR ↑
LPIPS ↓
PSNR ↑
LPIPS ↓
PSNR ↑
LPIPS ↓
Faster-GS
27.31
0.1420
24.36
0.1422
22.38
0.2449
On-the-fly NVS
24.22
0.2659
21.23
0.2248
19.88
0.3928
Ours
25.20
0.1486
24.37
0.1371
22.10
0.2365
Table 2: Quantitative results for offline NVS on MipNerf 360, Tanks and Temples, and thermal RGBT-Scenes datasets.
Depth Source
Time per
TSDF
Ours
Image [ ms ]
LPIPS ↓ / Time [ ms ]
LPIPS ↓ / Time [ ms ]
Captured [ Ste23 ]
0
0.2438 / 74.39
0.1987 / 8.51
Monocular [ LCL*25 ]
30.47
0.2834 / 74.39
0.2725 / 8.51
Multi-view [ CF24 ]
40.72
0.2412 / 74.39
0.1976 / 8.51
Table 3: Quantitative comparison of input depth source (captured stereo [ Ste23 ] , monocular [ LCL25 ] and transformer MVS [ CF24 ] ) and target depth fusion (ours vs nvblox ’ TSDF [ MOW24 ] ) on the LiveNVS dataset.
Figure 6: Thermal images of the RGBT-Scenes dataset and automotive scenes captured during night.
Dataset
Proc. Res.
Sensor Setup
Scene Type
Waymo [ SKD*20 ]
1280p
LiDAR + 5 cameras
automotive
nuScenes [ CBL*20 ]
900p
LiDAR + 6 cameras
automotive
Argoverse 2 [ WQA*21 ]
1550p
LiDAR + 7 cameras
automotive
LiveNVS [ FRF*23 ]
720p / 1080p
RGB-D
handheld (+ SLAM)
MipNerf 360 [ BMV*22 ]
520p / 820p
RGB
handheld (+ COLMAP)
Tanks and Temples [ KPZK17 ]
1080p
RGB
handheld (+ COLMAP)
Table S1: Specification of the benchmark datasets, detailing processing resolution and sensor setup across various scene types.
Waymo Open Dataset [ SKD*20 ]
Argoverse 2 [ WQA*21 ]
2601205676330128831_4880_000_4900_000
00a6ffc1-6ce9-3bc3-a060-6006e9893a1a
4140965781175793864_460_000_480_000
0a132537-3aec-35bb-af13-7faa0811000d
7247823803417339098_2320_000_2340_000
0b324587-6097-3f92-a07a-a44f48c85d9e
8688567562597583972_940_000_960_000
01bb304d-7bd8-35f8-bbef-7086b688e35e
8993680275027614595_2520_000_2540_000
03b2cf2d-fb61-36fe-936f-36bbf197a8ac
10504764403039842352_460_000_480_000
03fba633-8085-30bc-b675-687a715536ac
Table S2: List of scenes chosen from Waymo Open Dataset and Argoverse 2 . Presented scenes show comparatively reduced dynamic content.
Figure S1: Live NVS results on automotive scenes captured during night.
Figure S2: Qualitative comparison in automotive live scenarios form the nuScenes , Waymo Open Dataset , and Argoverse 2 datasets.
Figure S3: Qualitative comparison in handheld live scenarios form the LiveNVS dataset.
Figure S4: Qualitative comparison in offline scenarios. Four MipNerf 360 and two Tanks and Temples scenes are displayed.
Figure S5: Thermal images of the RGBT-Scenes dataset.
Figure S6: Ablation of source of geometry. Comparison of input depth acquisition techniques (captured RGB-D from Zed 2i camera [ Ste23 ] , monocular depth estimation [ LCL25 ] , transformer-based multi-view depth estimation [ CF24 ] ) and target depth fusion (our depth completion vs nvblox TSDF [ MOW24 ] )
Figure S7: Failure case of multi-view depth estimation with MVSFormer++ . In this automotive example including a strictly forward trajectory without lateral baseline, the depth estimation produces lots of floaters, severely impacting image quality ( left ), while using LiDAR data as depth data reveals no such issues ( right ).
Figure S8: PSNR and render time against number of used auxiliary images k . The red dotted line illustrates the chosen trade-off k=4 .
Figure S9: Render quality with varying number of points on MipNerf 360 [ BMV*22 ] garden scene. Sparse depth maps, the respective dense depth maps, the rendering result, and ground truth are displayed under utilization of varying subsampling factors of points. Full point cloud contains 3.1 mio points.
Figure S10: Comparison of our pipeline with the depth completion replaced with the Point sample rendering [ GD98 ] approach.
Figure 23
Figure S11: Varying σ in presence of calibration errors. Input poses have been artificially jittered, to the point where the naive approach and also our proposed σ=0.95 is not able to prevent ghosting artifacts. Increasing σ in this scenario leads to high quality images despite calibration errors being present.
LiveNVS
Approach
PSNR ↑
LPIPS ↓
Time [ ms ]
Ours + Deep Blending [ HPP*18 ]
21.84
0.2128
147
Ours
22.03
0.1987
12.11
Table S3: Quantitative results of our softEDT -based blending compared to Deep Blending ’s [ HPP*18 ] neural blending network.
Figure S12: Ablation of blending. We compare our softEDT -based weight computation on the LiveNVS dataset with Deep Blending [ HPP*18 ] by replacing weight computation in our pipeline with their learned weight prediction network.
School of AI for Science, Peking University Shenzhen Graduate School, Shenzhen, 518055 China · School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School, Shenzhen, 518055 China