High-quality 3D streaming from multiple cameras is crucial for immersive experiences in many AR/VR applications. The limited number of views - often due to real-time constraints - leads to missing information and incomplete surfaces in the rendered images. Existing approaches typically rely on simple heuristics for the hole filling, which can result in inconsistencies or visual artifacts. We propose to complete the missing textures using a novel, application-targeted inpainting method independent of the underlying representation as an image-based post-processing step after the novel view rendering. The method is designed as a standalone module compatible with any calibrated multi-camera system. For this we introduce a multi-view aware, transformer-based network architecture using spatio-temporal embeddings to ensure consistency across frames while preserving fine details. Additionally, our resolution-independent design allows adaptation to different camera setups, while an adaptive patch selection strategy balances inference speed and quality, allowing real-time performance. We evaluate our approach against state-of-the-art inpainting techniques under the same real-time constraints and demonstrate that our model achieves the best trade-off between quality and speed, outperforming competitors in both image and video-based metrics.
Figures & tables
Figure 1: Streamed content from a multi-camera setup (left) is prone to incomplete textures (center) because of missing information in the sparse viewpoints. To fix this, we propose a transformer-based inpainting method that efficiently incorporates information from the original images and thus surpasses traditional inpainting on the reconstruction alone (right).
Figure 2: Overview of the proposed transformer-based inpainting pipeline. The framework consists of two main stages: (1) Feature Encoding , where input and context images are encoded into feature representations and split into patches equipped with their spatio-temporal coordinates, (2) Context Aggregation and Decoding , utilizing a series of transformer groups and the contextual information to update the inpaint patches, that are finally decoded and blended with the known image regions. The flame symbol indicates a module with trainable parameters.
Figure 3: Feature maps are split into overlapping patches: background‐only patches are pruned, and object patches are kept as context Rt (blue). In the inpaint feature map, patches with missing pixels (green) constitute Pt , while patches without missing pixels are added to the context. In this illustration, we show patches as non‐overlapping for clarity, even though they are overlapping in practice.
Whole Image
Inpainted Regions
Model
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
VFID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
RGVI [ 12 ] – offline –
33.478
0.9881
0.0140
1.6203
42.582
0.99834
0.00678
03.09
DSTT [ 46 ]
def
31.532
0.9827
0.0332
3.2139
35.091
0.99733
0.00761
12.82
pretrained
win
31.986
0.9835
0.0317
2.4210
36.252
0.99788
0.00477
05.96
mul
32.045
0.9835
0.0316
2.3864
36.490
0.99793
0.00470
01.71
DSTT [ 46 ]
def
31.341
0.9825
0.0341
3.4281
34.972
0.99731
0.00747
13.68
Table 1: Results of all baseline methods compared to our method on various metrics, measured either on the whole image or only for pixels belonging to the inpainted regions. DSTT, FuseFormer, and E2FGVI are reported in three variants: Default settings of the pretrained model (def), windowed approach (win) and multiple views as an input (mul). Colored boxes show best and second-best results per metric of online methods. RGVI is included for reference.
Whole Image
Inpainted Regions
Model
PSNR ↑
SSIM ↑
LPIPS ↓
VFID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
DSTT [ 46 ]
32.977
0.9822
0.0159
1.4968
36.051
0.9967
0.0109
00.71
Fuseformer [ 45 ]
32.678
0.9832
0.0206
1.4056
37.361
0.9975
0.0095
00.69
E2FGVI [ 36 ]
32.766
0.9833
0.0204
1.3748
37.535
0.9976
0.0092
00.82
Ours
33.059
0.9800
0.0287
0.9953
42.192
0.9989
0.0031
37.01
Table 2: Results of the online baseline methods (pretrained, mul) compared to our method (without fine-tuning) on various metrics on the RIFTCast dataset [ 91 ] . Colored boxes show best and second-best results per metric.
Figure 4: Visual comparison of our method against the pretrained multicam variants of the baseline methods. First column shows the input image from the reconstruction framework, last column shows the ground-truth view seen from the omitted camera.
Whole Image
Inpainted Regions
PSNR ↑
SSIM ↑
LPIPS ↓
VFID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
Single cam w/o masks
32.400
0.9846
0.0261
1.6009
34.426
0.9971
0.0092
87.46
Single cam
32.277
0.9843
0.0275
1.7331
40.004
0.9986
0.0034
79.43
w/o temp
32.237
0.9843
0.0275
1.7320
39.838
0.9986
0.0039
41.32
w/o RoPE
32.046
0.9839
0.0278
1.7963
38.920
0.9983
0.0050
41.05
Ours
32.616
0.9851
0.0262
1.6671
42.184
0.9991
0.0022
41.55
Table 3: Ablation study on the impact of key components of our method. Colored boxes show best and second results per metric.
Figure 5: Ablation study showing (from left to right): the input image, reconstruction using only a single camera view once without and once with masks, without leveraging past video frames, without Rotary Positional Encodings (RoPE), and the full proposed pipeline.
Figure 6: Results on the RIFTCast dataset [ 91 ] .
Figure 7: Results on a dynamic scene showing wrongly estimated colors, missing the pattern on the sleeve. Green color artifacts between the arms in the ground truth image are a result of inaccurate foreground masks in the dataset.
Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.
Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/
Stefan Schulz, Fernando Edelstein, Hannah Dröge +2
Live free-viewpoint visualization of real humans is critical for immersive communication and interactive digital experiences. Existing methods either rely on computationally expensive optimization or require calibrated cameras and low-resolution inputs, making real-time high-resolution deployment impractical. In this work, we present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization from sparse, unposed RGB streams. Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance in a single forward pass. To achieve this, we start by designing a lightweight sparsity-aware multi-view transformer backbone that tokenizes foreground human regions while preserving global context through a shared scene token. We then employ a fully transformer-based Gaussian decoder to mitigate convolution-induced over-smoothing while keeping decoding sparse and efficient. In addition, we introduce a hybrid feature pyramid that injects multi-scale appearance cues into geometry prediction. We further introduce a lightweight differentiable Levenberg-Marquardt camera refinement layer to enhance multi-view consistency and geometric alignment. Moreover, to stabilize learning under sparse, unposed inputs, we transfer multi-view geometry priors from a large visual-geometry foundation model via teacher-student distillation. Finally, the predicted Gaussian maps are streamed with video codecs to remote devices for interactive free-viewpoint rendering. Extensive experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU. Additional captured sequences illustrate its performance across varied subjects, clothing, and motions under our multi-camera setup.
Hanzhang Tu, Zhanfeng Liao, Wei Min +2
Tsinghua University, Beijing, China · Shadow AI, Beijing, China
We introduce a test-time framework for multiview Transformers (MVTs) that incorporates priors (e.g., camera poses, intrinsics, and depth) to improve 3D tasks without retraining or modifying pre-trained image-only networks. Rather than feeding priors into the architecture, we cast them as constraints on the predictions and optimize the network at inference time. The optimization loss consists of a self-supervised objective and prior penalty terms. The self-supervised objective captures the compatibility among multi-view predictions and is implemented using photometric or geometric loss between renderings from other views and each view itself. Any available priors are converted into penalty terms on the corresponding output modalities. Across a series of 3D vision benchmarks, including point map estimation and camera pose estimation, our method consistently improves performance over base MVTs by a large margin. On the ETH3D, 7-Scenes, and NRGBD datasets, our method reduces the point-map distance error by more than half compared with the base image-only models. Our method also outperforms retrained prior-aware feed-forward methods, demonstrating the effectiveness of our test-time constrained optimization (TCO) framework for incorporating priors into 3D vision tasks.