Stereo video inpainting, which aims to fill the occluded regions of warped videos with visually coherent content while maintaining temporal consistency, remains a challenging open problem. The regions to be filled are scattered along object boundaries and occupy only a small fraction of each frame, leading to two key challenges. First, existing approaches perform poorly on such tasks due to the scarcity of high-quality stereo inpainting datasets, which limits their ability to learn effective inpainting priors. Second, these methods apply equal processing to all regions of the frame, even though most pixels require no modification, resulting in substantial redundant computation. To address these issues, we introduce three interconnected components. We first propose Gradient-Aware Parallax Warping (GAPW), which leverages backward warping and the gradient of the coordinate mapping function to obtain continuous edges and smooth occlusion regions. Then, a Parallax-Based Dual Projection (PBDP) strategy is introduced, which incorporates GAPW to produce geometrically consistent stereo inpainting pairs and accurate occlusion masks without requiring stereo videos. Finally, we present Sparsity-Aware Stereo Inpainting (SASI), which reduces over 70% of redundant tokens, achieving a 10.7x speedup during diffusion inference and delivering results comparable to its full-computation counterpart, enabling real-time processing of HD (768 x 1280) videos at 25 FPS on a single A100 GPU.
Figures & tables
Figure 1: Comparison among different methods at a resolution of 768×1280 : our method achieves a PSNR of 30.5 dB at 1 NFE, with an inference speed of 24.9 fps.
Figure 2: Comparison of forward warping and our GAPW in pixel mapping and occlusion mask generation.
Figure 3: ( a ) Illustration of the Parallax-Based Dual Projection, which utilizes Gradient-Aware Parallax Warping for reprojection to obtain the occlusion mask under input view. ( b ) Our proposed Sparsity-Aware Stereo Inpainting utilizes Mask-Based Token Selection to reduce the redundancy of visual tokens.
Figure 4: Qualitative comparison of data construction between TrajectoryCrafter [ 39 ] and ours.
Figure 5: Qualitative comparison of stereo video inpainting on the HD-100 test set ( 768×1280 ). Our inpainting results are much closer to the ground truth, showing sharper details and clearer structures.
Figure 6: Qualitative comparison of 2D-to-3D conversion. Each group shows the generated right view , zoomed-in patches , stereo-matched disparity , and anaglyph visualization from left to right. Our method produces sharp, high-quality views with accurate and detailed disparity.
Table 2: Quantitative comparison of 2D-to-3D methods on the Dynamic Replica valid set and the SVD AVP test set. † End-to-end approaches.
Dataset Strategy
PSNR ↑
SSIM ↑
LPIPS ↓
Random Mask
26.64
0.906
0.092
TrajectoryCrafter
31.14
0.923
0.047
Ours
32.48
0.933
0.049
Table 3: Ablation of inpainting performance on dataset strategy.
Dilate
Stride
r (%)
Latency ‡ ↓
PSNR ↑
SSIM ↑
LPIPS ↓
baseline
100.0
380.9
32.48
0.933
0.049
—
—
13.7
18.51
29.84
0.916
0.063
3
—
25.6
35.7
32.01
0.931
0.051
3
20
29.4
44.8
32.10
0.931
0.050
3
10
33.2
54.8
32.16
0.931
0.050
5
—
36.1
58.4
32.31
0.932
0.050
Table 4: HD-100 ( 576×1024 ) ablation with fixed NFE =4 , varying dilation kernel and temporal stride. Retention rate r is the fraction of tokens kept for DiT computation; ‡ Latency denotes DiT-only inference time in ms/frame. All settings achieve a substantial speedup with negligible quality loss. The light-gray row ( dilate=5 ) indicates the optimal setting adopted in all other experiments.
Figure 7: Ablation on max disparity. Our method consistently delivers high-quality, geometrically accurate results even under large disparity ranges.
Max Disparity
PSNR ↑
SSIM ↑
LPIPS ↓
0.02
32.18
0.915
0.043
0.05
32.16
0.905
0.049
0.07
30.48
0.900
0.053
0.08
29.91
0.895
0.058
Table 5: HD-100 ( 768×1280 ) ablation with fixed NFE =1 under wide-baseline settings. The light-gray row ( 0.07 ) corresponds to the setting adopted for all stereo inpainting experiments on the HD-100 test set. Larger disparities enlarge occluded areas, causing a slight expected drop in metrics, yet our method remains robust and effective.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Inference pipeline of 2D-to-3D conversion.
Figure 9: Module-wise latency breakdown and reduction on 768×1280 HD videos.
VAE
PSNR ↑
SSIM ↑
LPIPS ↓
WanVAE
30.59
0.906
0.040
Distilled (Ours)
30.48
0.900
0.053
Appendix
Table 6: Ablation on stereo inpainting quality on HD-100 ( 768×1280 ). Results are evaluated under the same final setting as in the main paper (see Tab. 1 ), using different VAEs. The distilled VAE achieves nearly identical quality to the original WanVAE.
VAE
Params
Encoder
Decoder
Total
WanVAE
126.8 M
47.7 ms
75.0 ms
122.7 ms
Distilled (Ours)
80.2 M
9.3 ms
18.5 ms
27.8 ms
Appendix
Table 7: Ablation on VAE parameter efficiency and latency at 1024×1024 resolution.
Train-time token filtering
PSNR ↑
SSIM ↑
LPIPS ↓
No (dense training)
32.48
0.933
0.049
Yes (sparse training)
32.31
0.932
0.050
Appendix
Table 8: Ablation on applying sparsity during training.
Figure 10: Qualitative comparison on HD-100 ( 768×1280 ). This figure complements Fig. 5 by including ZeroStereo [ 30 ] , a training-free stereo inpainting method that often produces random, spatially inconsistent fillings and distorted textures.
Metrics
Deep3D
Depthify.ai
Owl3D
StereoCrafter
ZeroStereo
Ours
AbsRel. ↓
0.029
0.035
0.042
0.039
0.018
0.019
δ<1.05↑
0.847
0.770
0.716
0.751
0.929
0.914
Appendix
Table 9: Geometry-consistency comparison on Dynamic Replica.
Figure 11: Comparison with SpatialDreamer [ 15 ] . Highlighted regions show truncation artifacts in SpatialDreamer, whereas our results preserve complete and consistent stereo geometry.
Figure 12: Comparison with M2SViD [ 26 ] . Our results show sharper details, larger disparities, and better color consistency, while M2SViD suffers from color drift and structural degradation.
Figure 13: Qualitative comparison on the SVD AVP test set ( 768×768 ). The second row shows per-pixel MSE maps, where blue denotes small errors and red large deviations from the ground truth. Our results exhibit the smallest errors and best overall consistency.
Figure 14: Qualitative comparison on the Dynamic Replica valid set( 720×1280 ). Our method yields the most accurate and artifact-free stereo completions, consistent with quantitative improvements in Tab. 2 .
Figure 15: Representative failure cases on transparent, reflective, and highly textured scenes from the SVD AVP test set.
Video outpainting aims to expand the visible content of a video beyond the original frame boundaries while preserving spatial fidelity and temporal coherence across frames. Existing methods primarily rely on large-scale generative models, such as diffusion models. However, generationbased approaches suffer from implicit temporal modeling and limited spatial context. These limitations lead to intraframe and inter-frame inconsistencies, which become particularly pronounced in dynamic scenes and large outpainting scenarios. To overcome these challenges, we propose Seen-to-Scene, a novel framework that unifies propagationbased and generation-based paradigms for video outpainting. Specifically, Seen-to-Scene leverages flow-based propagation with a flow completion network pre-trained for video inpainting, which is fine-tuned in an end-to-end manner to bridge the domain gap and reconstruct coherent motion fields. To further improve the efficiency and reliability of propagation, we introduce a reference-guided latent propagation that effectively propagates source content across frames. Extensive experiments demonstrate that our method achieves superior temporal coherence and visual realism with efficient inference, surpassing even prior state-of-the-art methods that require input-specific adaptation.
While video generation models excel at producing high-quality monocular videos, generating 3D stereoscopic and spatial videos for immersive applications remains an underexplored challenge. We present a pose-free and training-free method that leverages an off-the-shelf monocular video generation model to produce immersive 3D videos. Our approach first warps the generated monocular video into pre-defined camera viewpoints using estimated depth information, then applies a novel frame matrix inpainting framework. This framework utilizes the original video generation model to synthesize missing content across different viewpoints and timestamps, ensuring spatial and temporal consistency without requiring additional model fine-tuning. Moreover, we develop a \dualupdate~scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. The resulting multi-view videos are then adapted into stereoscopic pairs or optimized into 4D Gaussians for spatial video synthesis. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, such as Sora, Lumiere, WALT, and Zeroscope. The experiments demonstrate that our method has a significant improvement over previous methods. Project page at: https://daipengwa.github.io/S-2VG_ProjectPage/
Peng Dai, Feitong Tan, Qiangeng Xu +6
Department of Electrical and Electronic Engineering at The University of Hong Kong, Hong Kong.
Video inverse problems are fundamental to streaming, telepresence, and AR/VR, where high perceptual quality must coexist with tight latency constraints. Diffusion-based priors currently deliver state-of-the-art reconstructions, but existing approaches either adapt image diffusion models with ad hoc temporal regularizers - leading to temporal artifacts - or rely on native video diffusion models whose iterative posterior sampling is far too slow for real-time use. We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior. We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass, inheriting the teacher's strong temporal modeling while completely removing iterative test-time optimization. The distillation is prior-driven: it only requires the teacher diffusion model and known degradation operators, and does not rely on externally paired clean/noisy video data. To further boost throughput, we replace the video-diffusion backbone VAE with a high-efficiency LeanVAE via an innovative teacher-space regularized distillation scheme, enabling low-latency latent-space processing. Across streaming random inpainting, Gaussian deblurring and super-resolution, InstantViR matches or surpasses the reconstruction quality of diffusion-based baselines while running at over 35 FPS on NVIDIA A100 GPUs, achieving up to 100 times speedups over iterative video diffusion solvers. These results show that diffusion-based video reconstruction is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.
Weimin Bai, Suzhe Xu, Yiwei Ren +4
Peking University · Huaqiao University · Kuaishou Technology