Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups

Leif Van Holland; Domenic Zingsheim; Mana Takhsha; Hannah Dröge; Patrick Stotko; Markus Plack; Reinhard Klein

cs.CVMar 5, 2026

Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups

Authors: Leif Van Holland, Domenic Zingsheim, Mana Takhsha, Hannah Dröge, Patrick Stotko, Markus Plack, Reinhard Klein

Organizations: University of Bonn, Germany

Abstract

High-quality 3D streaming from multiple cameras is crucial for immersive experiences in many AR/VR applications. The limited number of views - often due to real-time constraints - leads to missing information and incomplete surfaces in the rendered images. Existing approaches typically rely on simple heuristics for the hole filling, which can result in inconsistencies or visual artifacts. We propose to complete the missing textures using a novel, application-targeted inpainting method independent of the underlying representation as an image-based post-processing step after the novel view rendering. The method is designed as a standalone module compatible with any calibrated multi-camera system. For this we introduce a multi-view aware, transformer-based network architecture using spatio-temporal embeddings to ensure consistency across frames while preserving fine details. Additionally, our resolution-independent design allows adaptation to different camera setups, while an adaptive patch selection strategy balances inference speed and quality, allowing real-time performance. We evaluate our approach against state-of-the-art inpainting techniques under the same real-time constraints and demonstrate that our model achieves the best trade-off between quality and speed, outperforming competitors in both image and video-based metrics.

Figures & tables

Figure 1: Streamed content from a multi-camera setup (left) is prone to incomplete textures (center) because of missing information in the sparse viewpoints. To fix this, we propose a transformer-based inpainting method that efficiently incorporates information from the original images and thus surpasses traditional inpainting on the reconstruction alone (right).

Figure 2: Overview of the proposed transformer-based inpainting pipeline. The framework consists of two main stages: (1) Feature Encoding , where input and context images are encoded into feature representations and split into patches equipped with their spatio-temporal coordinates, (2) Context Aggregation and Decoding , utilizing a series of transformer groups and the contextual information to update the inpaint patches, that are finally decoded and blended with the known image regions. The flame symbol indicates a module with trainable parameters.

Panel of Figure 3 — Figure 3: Feature maps are split into overlapping patches: background‐only patches are pruned, and object patches are kept as context $\mathcal{R}_{t}$ (blue). In the inpaint feature map, patches with missing pixels (green) constitute $\mathcal{P}_{t}$ , while patches without missing pixels are added to the context. In this illustration, we show patches as non‐overlapping for clarity, even though they are overlapping in practice.

		Whole Image				Inpainted Regions
Model	Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$
RGVI [ 12 ] – offline –		$33.478$	$0.9881$	$0.0140$	$1.6203$	$42.582$	$0.99834$	$0.00678$	$\phantom{0}3.09$
DSTT [ 46 ]	def	$31.532$	$0.9827$	$0.0332$	$3.2139$	$35.091$	$0.99733$	$0.00761$	$12.82$
pretrained	win	$31.986$	$0.9835$	$0.0317$	$2.4210$	$36.252$	$0.99788$	$0.00477$	$\phantom{0}5.96$
	mul	$32.045$	$0.9835$	$0.0316$	$2.3864$	$36.490$	$0.99793$	$0.00470$	$\phantom{0}1.71$
DSTT [ 46 ]	def	$31.341$	$0.9825$	$0.0341$	$3.4281$	$34.972$	$0.99731$	$0.00747$	$13.68$

Table 1: Results of all baseline methods compared to our method on various metrics, measured either on the whole image or only for pixels belonging to the inpainted regions. DSTT, FuseFormer, and E2FGVI are reported in three variants: Default settings of the pretrained model (def), windowed approach (win) and multiple views as an input (mul). Colored boxes show best and second-best results per metric of online methods. RGVI is included for reference.

	Whole Image				Inpainted Regions
Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$
DSTT [ 46 ]	$32.977$	$0.9822$	$0.0159$	$1.4968$	$36.051$	$0.9967$	$0.0109$	$\phantom{0}0.71$
Fuseformer [ 45 ]	$32.678$	$0.9832$	$0.0206$	$1.4056$	$37.361$	$0.9975$	$0.0095$	$\phantom{0}0.69$
E2FGVI [ 36 ]	$32.766$	$0.9833$	$0.0204$	$1.3748$	$37.535$	$0.9976$	$0.0092$	$\phantom{0}0.82$
Ours	$33.059$	$0.9800$	$0.0287$	$0.9953$	$42.192$	$0.9989$	$0.0031$	$37.01$

Table 2: Results of the online baseline methods (pretrained, mul) compared to our method (without fine-tuning) on various metrics on the RIFTCast dataset [ 91 ] . Colored boxes show best and second-best results per metric.

Panel of Figure 4 — Figure 4: Visual comparison of our method against the pretrained multicam variants of the baseline methods. First column shows the input image from the reconstruction framework, last column shows the ground-truth view seen from the omitted camera.

	Whole Image				Inpainted Regions
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$
Single cam w/o masks	$32.400$	$0.9846$	$0.0261$	$1.6009$	$34.426$	$0.9971$	$0.0092$	$87.46$
Single cam	$32.277$	$0.9843$	$0.0275$	$1.7331$	$40.004$	$0.9986$	$0.0034$	$79.43$
w/o temp	$32.237$	$0.9843$	$0.0275$	$1.7320$	$39.838$	$0.9986$	$0.0039$	$41.32$
w/o RoPE	$32.046$	$0.9839$	$0.0278$	$1.7963$	$38.920$	$0.9983$	$0.0050$	$41.05$
Ours	$32.616$	$0.9851$	$0.0262$	$1.6671$	$42.184$	$0.9991$	$0.0022$	$41.55$

Table 3: Ablation study on the impact of key components of our method. Colored boxes show best and second results per metric.

Panel of Figure 5 — Figure 5: Ablation study showing (from left to right): the input image, reconstruction using only a single camera view once without and once with masks, without leveraging past video frames, without Rotary Positional Encodings (RoPE), and the full proposed pipeline.

Panel of Figure 6 — Figure 6: Results on the RIFTCast dataset [ 91 ] .

Panel of Figure 7 — Figure 7: Results on a dynamic scene showing wrongly estimated colors, missing the pattern on the sleeve. Green color artifacts between the arms in the ground truth image are a result of inaccurate foreground masks in the dataset.

Image Inpainting

Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.

See all 92 papers

3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis
Apr 13, 2026Stefan Schulz, Fernando Edelstein, Hannah Dröge +2Novel View Synthesis Multi-View Geometry
University of Bonn
Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras
Sep 14, 2026Hanzhang Tu, Zhanfeng Liao, Wei Min +23D Gaussian Splatting Camera Pose Estimation
Tsinghua University, Beijing, China · Shadow AI, Beijing, China
Learning 3D Reconstruction with Priors in Test Time
Apr 4, 2026Lei Zhou, Haoyu Wu, Akshat Dave +1Multi-View 3D Reconstruction Test-Time Optimization
Stony Brook University

Apr 13, 2026J/K move · Enter open · S save

3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

Stefan Schulz, Fernando Edelstein, Hannah Dröge, Matthias B. Hullin+1

University of Bonn

Novel View Synthesis Multi-View Geometry

Open paper AskPDF arXiv

Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/

Abstract

Image Inpainting

Explore similar work

3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras

Learning 3D Reconstruction with Priors in Test Time