ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models
Authors: Riccardo de Lutio, Tobias Fischer, Yen-Yu Chang, Yuxuan Zhang, Jay Zhangjie Wu, Xuanchi Ren, Tianchang Shen, Katarina Tothova, +2 more
Organizations: NVIDIA, USA · NVIDIA, Switzerland · ETHZ, Switzerland · Cornell University, USA · NVIDIA, Canada · University of Toronto, Canada · Vector Institute, Canada
Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas. Methods that leverage generative priors to correct artifacts in these areas hold promise but currently suffer from two shortcomings. The first is scalability, as existing methods use image diffusion models or bidirectional video models that are limited in the number of views they can generate in a single pass (and thus require a costly iterative distillation process for consistency). The second is quality itself, as generators used in prior work tend to produce outputs that are inconsistent with existing scene content and fail entirely in completely unobserved regions. To solve these, we propose a two-stage pipeline that leverages two key insights. First, we train a powerful bidirectional generative model with a novel opacity mixing strategy that encourages consistency with existing observations while retaining the model's ability to extrapolate novel content in unseen areas. Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass. This model can directly produce novel views or serve as pseudo-supervision to improve the underlying 3D representation in a simple and highly efficient manner. We evaluate our method extensively and demonstrate that it can generate plausible reconstructions in scenarios where existing approaches fail completely. When measured on commonly benchmarked datasets, we outperform all existing baselines by a wide margin, exceeding prior state-of-the-art methods by 1-3 dB PSNR.
Figures & tables
Figure 1. ArtiFixer enhances and extends existing 3D reconstructions in a highly efficient and scalable manner. Given an initial reconstruction and optional reference views and text prompt, it auto-regressively generates novel content that maintains a high degree of consistency with existing observations. ArtiFixer can directly produce hundreds of novel views in a single inference pass or serve as pseudo-supervision to improve the underlying 3D reconstruction. Project page: https://research.nvidia.com/labs/sil/projects/artifixer
Figure 2. Method overview. We first train a bidirectional flow matching model that transports degraded RGB renderings into clean outputs. We encode the input RGB into latent space and mix with Gaussian noise using the rendered opacity maps to avoid mode collapse in unseen regions. We inject fine-grained opacity information and camera control along with optional clean reference views and a text prompt. In the second phase of our pipeline, we distill the teacher into an auto-regressive causal model via Self Forcing-style DMD distillation ( Huang et al., 2025 ) , which can be directly used to render novel views or used as pseudo-supervision to distill back into the underlying 3D representation.
Figure 3. Transformer block. We start from a pretrained text-to-video model ( Wan et al., 2025b ) and inject camera and opacity information into each transformer block via linear layers after applying self-attention and layer normalization. We patchify reference views into visual tokens, apply relative camera conditioning via PRoPE ( Li et al., 2025a ) , and add Kn and Vn projections to the cross-attention operation. We zero-initialize fr , fo , and Vn to ensure compatibility with the pretrained initialization.
Figure 4. Opacity mixing. Given a degraded rendering and optional reference views and text prompt ( left ), we predict an artifact-free rendering at a target viewpoint. Starting from Gaussian noise and channel concatenating the degraded rendering as in prior work ( Wu et al., 2025c ; Yin et al., 2025b ) produces renderings that are semantically similar to the reference views, but with notable inconsistencies (such as the table in the top row ). Directly starting from the degraded rendering instead of Gaussian noise improves consistency, but degrades quality noticeably when extrapolating to areas outside those covered by the degraded renderings ( bottom row ). Instead, we mix Gaussian noise into the rendering based on its opacity map. The resulting input retains the consistency benefits of the original while enabling a strong generative capability in entirely novel regions.
Nerfbusters ( Warburg et al., 2023 )
DL3DV ( Ling et al., 2024 )
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
Nerfacto ( Tancik et al., 2023 )
17.29
0.621
0.402
134.65
17.16
0.581
0.430
112.30
3DGS ( Kerbl et al., 2023 )
17.66
0.678
0.327
113.84
17.18
0.588
0.384
107.23
Nerfbusters ( Warburg et al., 2023 )
17.72
0.647
0.352
116.83
17.45
0.606
0.370
96.61
GANeRF ( Roessle et al., 2023 )
17.42
0.611
0.354
115.60
17.54
0.610
0.342
81.44
NeRFLiX ( Zhou et al., 2023 )
17.91
0.656
0.346
113.59
17.56
0.610
0.359
80.65
Table 1. Artifact removal on Nerfbusters and DL3DV . All ArtiFixer variants outperform prior methods by a considerable margin, improving PSNR by 2 dB.
PSNR ↑
SSIM ↑
LPIPS ↓
Method
3-view
6-view
9-view
3-view
6-view
9-view
3-view
6-view
9-view
Zip-NeRF ( Barron et al., 2023 )
12.77
13.61
14.30
0.271
0.284
0.312
0.705
0.663
0.633
3DGS ( Kerbl et al., 2023 )
13.06
14.96
16.79
0.251
0.355
0.447
0.576
0.505
0.446
2DGS ( Huang et al., 2024 )
13.07
15.02
16.67
0.243
0.338
0.423
0.580
0.506
0.449
FSGS ( Zhu et al., 2024 )
14.17
16.12
17.94
0.318
0.415
0.492
0.578
0.517
0.468
FreeNeRF ( Yang et al., 2023 )
12.87
13.35
14.59
0.260
0.283
0.319
0.715
0.717
0.695
Table 2. Sparse view reconstruction methods on the Mip-NeRF 360 dataset. We exceed existing work by a wide margin across every metric.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
3DGUT ( Wu et al., 2025b )
16.12
0.537
0.445
92.94
Difix3D (Nerfacto) ( Wu et al., 2025a )
14.16
0.453
0.545
74.59
Difix3D (3DGS) ( Wu et al., 2025a )
16.60
0.599
0.405
52.70
Difix3D+ (Nerfacto) ( Wu et al., 2025a )
13.74
0.434
0.483
30.07
Difix3D+ (3DGS) ( Wu et al., 2025a )
16.34
0.564
0.382
21.77
Fixer (offline) ( NVIDIA, 2025 )
13.09
0.355
0.584
135.43
Table 3. Novel content generation. We reconstruct DL3DV scenes following a protocol that creates large areas unobserved by training views. We outperform the next-best method (GenFusion) by almost 3 dB in PSNR.
Figure 5. ArtiFixer variants. Most visible differences occur in highly corrupted regions. ArtiFixer3D ’s explicit 3D consistency improves fidelity with the source images and mitigates transient corruption ( middle ), at the cost of some sharpness, which ArtiFixer3D+ restores. Nonetheless, all variants outperform prior work by a substantial margin.
Method
Direct Input
Opacity Mixing
Diffusion Forcing
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
Channel Concatenation
✗
✗
✓
14.52
0.391
0.490
87.551
w/o Opacity Mixing
✓
✗
✓
17.34
0.440
0.429
87.058
w/o Initialization
✓
✓
✗
17.58
0.450
0.416
74.924
Full Method
✓
✓
✓
17.99
0.461
0.408
69.43
Table 4. Diagnostics. We evaluate reconstruction quality on Mip-NeRF 360. Denoising input renderings instead of conditioning via channel concatenation is crucial to producing outputs consistent with source images.
Figure 6. Reference views. Without the initial rendering condition, ArtiFixer can generate predictions from the reference views. Although fidelity drops somewhat, the high-level structure of the scene remains intact.
Figure 7. Text-to-video generation. To illustrate our model’s generative ability, we generate videos from text prompts alone. With opacity mixing, it retains similar quality to its base model ( Wan et al., 2025b ) .
Method
FPS ↑
Wan 2.1 T2V-14B ( Wan et al., 2025b )
0.12
Wan 2.1 T2V-1.3B ( Wan et al., 2025b )
0.49
ArtiFixer / ArtiFixer3D+ (14B)
8.36
ArtiFixer / ArtiFixer3D+ (1.3B)
34.38
ArtiFixer3D
268
Table 5. Inference speed. Causal distillation yields a 70× speedup over the bidirectional Wan 2.1 backbones. ArtiFixer3D renders directly from 3DGUT. Additional configurations are reported in Table 7 .
Figure 8. DL3DV results . We compare ArtiFixer3D+ to its initial 3DGUT ( Wu et al., 2025b ) input, two baselines that build upon bidirectional video diffusion models ( top rows ), and two that leverage image models ( bottom rows ). GenFusion ( Wu et al., 2025c ) ’s video model generates 16 frames at a time, requiring an iterative distillation process that leads to blurry results, especially in empty areas. Gen3C ( Ren et al., 2025 ) ’s renderings are sharper but often do not respect the source content (background in top row ), have incorrect geometry ( second row ), and exhibit color shift ( sixth row ). Methods that directly take renderings as input without opacity mixing ( Wu et al., 2025a ; NVIDIA, 2025 ) fail to reconstruct empty regions. Our method can reconstruct plausible and consistent geometry even when the initial rendering is highly degraded. Please refer to our project website for comparison videos.
Figure 9. Mip-NeRF 360 results . We present visualizations of Mip-NeRF’s most challenging split (3-view). Our results far exceed all prior work both quantitatively and qualitatively. Our method is able to recover the correct geometry from the reference views even in scenarios where the input rendering is completely inaccurate (table in third row ).
Figure 10. Nerfbusters results . As with the other datasets, our method is the only one to generate plausible visuals in unseen regions while preserving the fidelity of the original content.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Δ PSNR
Δ SSIM
Δ LPIPS
Mip-NeRF 360 (3 views)
+0.14
+0.003
− 0.002
Mip-NeRF 360 (6 views)
+0.07
+0.002
− 0.001
Mip-NeRF 360 (9 views)
+0.03
+0.003
− 0.001
DL3DV
+0.02
0.000
− 0.001
Nerfbusters
− 0.07
+0.001
0.000
Appendix
Table 6. Text conditioning. We measure the impact of VLM-generated prompts vs. no prompt for ArtiFixer3D+ . Text prompts provide a small benefit in sparse settings that diminishes with denser captures.
FPS ↑
Method
GPUs
1 step
2 steps
3 steps
4 steps
ArtiFixer (14B)
1
29.42
16.07
11.03
8.36
ArtiFixer (14B)
4
58.72
35.91
24.65
19.18
ArtiFixer (1.3B)
1
86.75
57.76
43.20
34.38
ArtiFixer (1.3B)
4
101.77
69.44
53.77
49.24
Appendix
Table 7. Inference configurations. Fewer denoising steps and context parallelism across multiple GPUs further improve throughput, with the 1.3B variant reaching up to 101.77 FPS.
Figure 11. Denoising steps . We vary the number of denoising steps when beginning from the initial degraded rendering. ArtiFixer can render plausible content in as few as 1 step, although sharpness and temporal consistency suffer somewhat in empty areas.
PSNR ↑
SSIM ↑
LPIPS ↓
Method
3-view
6-view
9-view
3-view
6-view
9-view
3-view
6-view
9-view
GenFusion ( Wu et al., 2025c )
15.29
17.16
18.36
0.369
0.447
0.496
0.585
0.500
0.465
GSFixer ( Yin et al., 2025b )
15.61
17.27
18.63
0.370
0.426
0.481
0.559
0.478
0.420
CAT3D ( Gao* et al., 2024 )
16.62
17.72
18.67
0.377
0.425
0.460
0.515
0.482
0.460
ArtiFixer3D+ (1.3B)
16.60
18.04
19.44
0.414
0.466
0.513
0.486
0.435
0.394
ArtiFixer3D+ (14B)
17.51
18.95
20.16
0.444
0.498
0.537
0.441
0.396
0.359
Appendix
Table 8. Impact of model scale on Mip-NeRF 360. Our 1.3B variant matches CAT3D within 0.02 dB on the 3-view split and exceeds other video model baselines despite using fewer parameters.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
GenFusion ( Wu et al., 2025c )
17.03
0.624
0.392
132.91
Gen3C ( Ren et al., 2025 )
15.50
0.491
0.476
68.36
ArtiFixer3D+ (1.3B)
19.04
0.635
0.352
22.3
ArtiFixer3D+ (14B)
20.15
0.662
0.307
13.91
Appendix
Table 9. Impact of model scale on novel content generation (DL3DV). Even with a 1.3B backbone, ArtiFixer3D+ outperforms the other video model baselines by a wide margin.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
3DGS ( Kerbl et al., 2023 )
9.57
0.108
0.779
SparseNeRF ( Wang et al., 2023 )
9.23
0.191
0.632
DNGaussian ( Li et al., 2024 )
10.23
0.156
0.643
ReconX ( Liu et al., 2026 )
14.28
0.394
0.564
ArtiFixer3D+
14.75
0.464
0.463
Appendix
Table 10. Tanks and Temples (2-view). ArtiFixer3D+ outperforms all baselines.
Method
MASt3R ↓
RAFT ↓
Fixer ( NVIDIA, 2025 )
0.1288
0.1236
Difix3D+ ( Wu et al., 2025a )
0.0974
0.0959
GenFusion ( Wu et al., 2025c )
0.0817
0.0786
Gen3C ( Ren et al., 2025 )
0.0766
0.0757
ArtiFixer
0.0749
0.0749
ArtiFixer3D+
0.0697
0.0697
Appendix
Table 11. Multi-view consistency. We measure multi-view consistency via MEt3R ( Asim et al., 2025 ) with MASt3R and RAFT backbones. All ArtiFixer variants outperform baselines, with ArtiFixer3D achieving the best results due to its explicit multi-view-consistent 3D representation.
3D Gaussian Splatting (3DGS) delivers high-fidelity real-time rendering but suffers from geometric and photometric degradations under sparse-view constraints. Current generative restoration approaches are often limited by insufficient temporal coherence, a lack of explicit spatial constraints, and a lack of large-scale training data, resulting in multi-view inconsistencies, erroneous geometric hallucinations, and limited generalization to diverse real-world artifact distributions. In this paper, we present ArtifactWorld, a framework that resolves 3DGS artifact repair through systematic data expansion and a homogeneous dual-model paradigm. To address the data bottleneck, we establish a fine-grained phenomenological taxonomy of 3DGS artifacts and construct a comprehensive training set of 107.5K diverse paired video clips to enhance model robustness. Architecturally, we unify the restoration process within a video diffusion backbone, utilizing an isomorphic predictor to localize structural defects via an artifact heatmap. This heatmap then guides the restoration through an Artifact-Aware Triplet Fusion mechanism, enabling precise, intensity-guided spatio-temporal repair within native self-attention. Extensive experiments demonstrate that ArtifactWorld achieves state-of-the-art performance in sparse novel view synthesis and robust 3D reconstruction. Code available at: https://github.com/fyting/ArtifactWorld.
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
Feed-forward 3D reconstruction has revolutionized 3D vision, providing a powerful baseline for downstream tasks such as novel-view synthesis with 3D Gaussian Splatting. Previous works explore fixing the corrupted rendering results with a diffusion model. However, they lack geometric concern and fail at filling the missing area on the extrapolated view. In this work, we introduce Leveling3D, a novel pipeline that integrates feed-forward 3D reconstruction with geometrical-consistent generation to enable holistic simultaneous reconstruction and generation. We propose a geometry-aware leveling adapter, a lightweight technique that aligns internal knowledge in the diffusion model with the geometry prior from the feed-forward model. The leveling adapter enables generation on the artifact area of the extrapolated novel views caused by underconstrained regions of the 3D representation. Specifically, to learn a more diverse distributed generation, we introduce the palette filtering strategy for training, and a test-time masking refinement to prevent messy boundaries along the fixing regions. More importantly, the enhanced extrapolated novel views from Leveling3D could be used as the inputs for feed-forward 3DGS, leveling up the 3D reconstruction. We achieve SOTA performance on public datasets, including tasks such as novel-view synthesis and depth estimation.