Novel View Synthesis (NVS) generates novel views of a scene from a sparse set of input images and is a key component in applications such as immersive rendering and 3D content generation. Existing NVS approaches broadly exhibit an efficiency--quality tradeoff: regression-based models generate novel views with relatively low latency but can struggle with perceptual quality, particularly in challenging regions or regions not visible in the input views (disoccluded regions), while diffusion-based models produce higher-quality and temporally consistent views at substantially greater computational cost. This motivates a hybrid approach that leverages the complementary strengths of regression and diffusion to bridge this efficiency--quality gap. We present Sphinx, a training-free hybrid inference and serving system for NVS that combines regression-level efficiency with diffusion-level quality. Sphinx first uses fast regression-based NVS to generate coarse novel views, which provide a strong initialization for diffusion and avoid expensive generation from pure noise. It then selectively applies diffusion refinement where additional computation is needed, reducing the denoising workload while preserving the quality benefits of diffusion-based NVS. To efficiently serve the resulting heterogeneous requests, Sphinx further introduces an SLO-aware serving framework that dynamically allocates GPU resources based on each request's remaining computation and urgency. Evaluated across diverse NVS models, datasets, and serving configurations, Sphinx achieves an average 2.3 times inference speedup (up to 2.7 times) with less than 10% perceptual degradation, while its SLO-aware scheduler achieves up to 10 times higher SLO attainment than FCFS scheduling across hardware platforms.
Figures & tables
Figure 1. Overview of Sphinx . Adaptive inference reduces denoising and spatial computation, while SLO-aware serving efficiently schedules the resulting heterogeneous requests.
Figure 2. Example NVS task: two input camera frames are used to generate four intermediate novel views between them.
Figure 3. Latency–quality gap between regression- and diffusion-based NVS, measured on a NVIDIA H200 GPU.
Figure 4. Hybrid inference combining regression-based fast initialization with diffusion-based selective denoising.
Figure 5. (a) Relationship between regression-quality ratio and resulting quality factor across different k ; (b) decision logic for adaptive k selection.
Figure 6. Distribution of selected denoising steps across target frames on the RE10K dataset; bar width indicates frequency, black lines denote the 25th–75th percentile range, and white dots mark the mean.
Figure 7. Temporal-cache design. We cache temporal-attention latents at a full denoising step Xt and reuse them during the next (T−1) partial steps for inactive frames.
Figure 8. Regression output (MVSplat) and ground truth. Red-highlighted disoccluded regions show artifacts or missing details, while the rest is largely reliable.
Figure 9. Per-step latency breakdown of FrameCrafter and SEVA on a single A40. Red boxes mark self-attention components targeted by sparse attention.
Figure 10. Principal Component Analysis (PCA) visualization of input-image embeddings from 3,000 scenes across RE10K, DL3DV, and ACID.
Figure 11. Overview of Sphinx ’s adaptive inference pipeline.
Figure 12. Request heterogeneity in starting denoising steps and selected-token ratios introduced by adaptive inference.
Figure 13. Latency of a five step FrameCrafter denoising chunk on four NVIDIA A40 GPUs across token selection ratios and SP degrees. Higher token ratios increase latency, while larger SP degrees reduce it.
Figure 16. Comparison of SEVA performance across RE10K, DL3DV, and ACID under different quality settings. Each bar represents a refinement mode (Fine, Coarse, Selective), and colors correspond to quality factors 0.98, 0.95, and 0.90.
Figure 17. Comparison of FrameCrafter performance across RE10K, DL3DV, and ACID under different quality settings. Each bar represents a refinement mode (Fine, Coarse, Selective), and colors correspond to quality factors 0.98, 0.95, and 0.90.
Figure 18. Distribution of target frames requiring different numbers of denoising steps for SEVA and FrameCrafter on the RE10K dataset at a quality factor of 0.95.
SEVA
Quality Metrics
Benchmark
Method
MUSIQ ↑
PSNR ↑
SSIM ↑
LPIPS ↓
RE10K
Diffusion-Only
63.55
21.28
0.736
0.250
STA
55.27
12.84
0.472
0.495
Sphinx ( α=0.98 )
63.40
21.14
0.737
0.250
Sphinx ( α=0.98 ) + selective refinement
62.53
21.00
0.737
0.257
Sphinx ( α=0.95 )
62.46
20.63
0.730
0.262
Table 1. SEVA quality results on RE10K, DL3DV, and ACID. Higher is better except for LPIPS.
FrameCrafter
Quality Metrics
Benchmark
Method
MUSIQ ↑
PSNR ↑
SSIM ↑
LPIPS ↓
RE10K
Diffusion-Only
58.99
19.39
0.640
0.193
STA
57.13
14.51
0.476
0.439
Sphinx ( α=0.98 )
57.51
19.28
0.644
0.210
Sphinx ( α=0.98 ) + selective refinement
55.03
19.09
0.650
0.235
Sphinx ( α=0.95 )
56.45
19.09
0.648
0.228
Table 2. FrameCrafter quality results on RE10K, DL3DV, and ACID. Higher is better except for LPIPS.
Figure 19. SLO failure rate under varying request rates and SLO scales on a server with 4 NVIDIA A40 GPUs.
Figure 20. SLO failure rate under varying request rates and SLO scales on a server with 8 NVIDIA H200 GPUs.
Figure 21. P95 tail latency of serving strategies across request rates on 4 NVIDIA A40 GPUs.
Figure 22. P95 tail latency of serving strategies across request rates on 8 NVIDIA H200 GPUs.
Figure 23. CDF of end-to-end latency for different serving strategies under request rates of 1.4 and 2.0 requests/min on a server with 4 NVIDIA A40 GPUs. The red dashed line indicates the SLO threshold under an SLO scale of 1.5× .
Figure 24. CDF of end-to-end latency for different serving strategies under request rates of 7.0 and 9.0 requests/min on a server with 8 NVIDIA H200 GPUs. The red dashed line indicates the SLO threshold under an SLO scale of 1.5× .
Figure 25. Performance–quality and energy–quality trade-offs across quality factors and inference methods.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 26. End-to-end latency breakdown of vanilla diffusion and Sphinx .
Figure 27. Qualitative comparison of novel view synthesis results across different methods. Each row shows target view generation between two input frames. While many frames are generated between two input frames, this figure only shows one generated frame for all baselines (regression, Sphinx , and diffusion) due to space limitation.
Single-image novel view synthesis (NVS) enables photorealistic rendering of un- observed viewpoints from a single input. Practical NVS systems require two key capabilities: robust reconstruction of occluded regions and high inference effi- ciency. While hybrid decoupled frameworks combining feedforward 3D Gaussian Splatting (3DGS) and diffusion models show promise for large-view-deviation NVS, they suffer from capacity competition: a fixed number of Gaussians forces resource shifts from visible to newly disoccluded areas, degrading original scene fidelity when the target view deviates significantly from the input. To address this, we propose Spackle, a lightweight residual learning framework that mit- igates capacity competition without sacrificing efficiency. Spackle operates in three stages: predicting base 3DGS attributes from given views, automatically identifying poorly reconstructed regions, and learning a residual 3DGS optimized exclusively for these areas. At inference, we combine the baseline and aug- mented Gaussians for NVS. We conduct comprehensive experiments and show that Spackle achieves state-of-the-art performance on large-view-deviation cases.
Xuanzhi Liu, Yuhe Zhou, Xinyi Wu +4
Shenzhen University of Advanced Technology · HONOR
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
We tackle the problem of sparse novel view synthesis (NVS) using video diffusion models; given K (≈5) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be invariant to permutations of that input set. To this end, we present FrameCrafter, which adapts video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional embeddings. Our results suggest that video models can be easily trained to "forget" about time with minimal supervision, producing competitive performance on sparse-view NVS benchmarks. Project page: https://frame-crafter.github.io/