Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.
Figures & tables
Figure 1: Novel view synthesis highlights. Given a single input view, its corresponding depth map, and Plücker ray conditioning, our method generates novel views that remain 3D-consistent with the target across challenging datasets (DL3DV, RealEstate10K, T&T) and selective in-the-wild scenes. For the datasets, we illustrate a single novel view generation alongside the ground-truth target, whereas for the in-the-wild setting, we demonstrate two distinct novel view generations.
Figure 2: PCC computation pipeline. The source view (a) is from the original scene, while the target view (f) has been modified: a red sphere is added and a yellow sphere is removed. Top row ( Δt , detecting missing geometry): the source depth (b) is unprojected to 3D and reprojected into the target camera frame (c); black regions indicate pixels outside the reprojection coverage. Where the yellow sphere was removed, target camera rays pass through and hit the cubes in the back instead of striking the sphere surface, producing large positive Δt values. These inconsistent pixels are highlighted in red (e), with nf=13703 pixels exceeding the error threshold δ . Bottom row ( Δs , detecting hallucinated geometry): The modified target depth is reprojected back into the source frame (h). The added red sphere reprojects into a region with no corresponding geometry in the source perspective, yielding large positive Δs values and nb=8581 inconsistent pixels (j). The final PCC score of 0.0232 quantifies the overall geometric disagreement between the two views.
Figure 3: ESC Framework. The framework extends Qwen MM-DiT with depth information and Plücker rays conditioning by leveraging LoRA adapters and a Ray ControlNet. At inference time, we leverage PCC-based sampling to enforce improved generated scene geometry.
Figure 4
Perceptual Quality
Geometry
Model
LPIPS ↓
DINO-Sim ↑
DreamSim ↓
PCC ↓
DL3DV Dataset
PhotoNVS
0.5743±0.1229
0.5647±0.0900
0.3876±0.1293
0.2628±0.1677
VIVID
0.6407±0.1047
0.6909±0.2080
0.2953±0.1288
0.2475±0.1624
OVIE
0.7099±0.1490
0.2755±0.3507
0.6480±0.2730
0.3122±0.1598
GLD †
0.5665±0.1013
0.7019±0.1499
0.2549±0.1175
0.2487±0.1642
Table 1: Quantitative comparison on novel view synthesis across three datasets. We evaluate perceptual quality using LPIPS, DINO similarity, and DreamSim, and geometric consistency using PCC, on the DL3DV, RE10K, and Tanks & Temples datasets. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) is better. We compare our method (ESC) and its variants against prior baselines (PhotoNVS, VIVID, OVIE, GLD, LagerNVS), with results reported as mean ± standard deviation. The best, second-best, and third-best results in each column are highlighted. ESC consistently outperforms all baselines across perceptual metrics, and the geometry-guided ESC-PCC-BoN variant further improves geometric consistency. Models marked with † were trained on the training split of the dataset. Models marked with ‡ were trained on a restricted subset comprising less than half of the full training split.
Figure 6: Metric correlation analyses. Per-view correlation between PCC and 2D metrics across both experiments. Each point represents a single (variant, view) pair. Red : scale jitter experiment—PCC correlates strongly with all 2D metrics ( r=−0.84 for PSNR, −0.90 for SSIM, +0.83 for LPIPS), since geometric distortion simultaneously changes appearance. Blue : occlusion experiment—correlations weaken substantially ( r=−0.24 for PSNR, −0.30 for SSIM, +0.22 for LPIPS), as views with large visual differences (high LPIPS, low PSNR) maintain low PCC because each scene is internally 3D-consistent. The strong-to-weak correlation shift between experiments demonstrates that PCC and 2D metrics measure fundamentally orthogonal properties.
Figure 7: Sensitivity of the DA3 PCC metric to controlled scale perturbations. Starting from a clean reference scene, we re-render each object with a random per-object scale jitter of 1–40% and measure (a) PCC, (b) pose AUC@3, 5, 15, 30, (c) relative rotation error (RPE-R), and (d) relative translation error (RPE-T), with depth and cameras predicted by DA3 and FG/BG-split affine alignment (no top-% pruning). With one exception, all evaluated metrics track geometric error monotonically alongside increasing jitter magnitude. Specifically, PCC rises from 0.07 at 0% jitter to 0.62 at 40%, AUC@30 falls from 0.84 to 0.51 (with AUC@3 collapsing from 0.67 to 0 by the 20% mark), and RPE-T increases from 0.77 to 12.35. RPE-R is the only metric that fails to reliably reflect the introduced noise, remaining largely stagnant and breaking monotonicity at 2%, 7%, and 30%.
Figure 8: Geometrically inconsistent point detections in a counterfactual scene. The real pair depicts a 3D-consistent scene in which all objects remain stationary as the camera moves between the two viewpoints. The edited pair shares the same source image (same left image), and its target image (right image) is rendered from the same camera viewpoint as in the consistent pair. However, one or more objects are displaced in the second’s target, producing a counterfactual scene that is incompatible with the 3D structure observed in source. By projecting the point cloud derived from each image into the other view, the PCC-based procedure identifies points whose projected geometry is inconsistent with the modified scene. These inconsistencies are highlighted as red and blue pixels, corresponding to inconsistencies detected by the forward and backward projections, respectively.
Figure 9: Effect of frame gap on model performance. Performance consistently degrades across all models as the frame gap increases. Nevertheless, our method outperforms all baselines at every frame-gap setting on all metrics except PSNR and SSIM, for which LagerNVS achieves the best results.
Perceptual Quality
Geometry
Model
LPIPS ↓
DINO-Sim ↑
DreamSim ↓
PCC ↓
DL3DV Dataset
ESC-Shadow
0.4952±0.1386
0.8035±0.1540
0.1676±0.1304
0.2533±0.1640
ESC-Base
0.4448±0.1569
0.8366±0.1389
0.1420±0.1252
0.2448±0.1646
ESC-PCC-BoN
0.4340±0.1556
0.8386±0.1388
0.1402±0.1232
0.2257±0.1556
RE10K Dataset
Table 2: Ablation study of the ESC framework. Quantitative evaluation comparing ESC-shadow, ESC-base, and the full ESC-PCC-BoN pipeline across the DL3DV, RE10K, and Tanks & Temples datasets.
Figure 10: Correlation between DA3-derived and predicted PCC metrics.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: One Blender + Kubric scene rendered from all 15 grid viewpoints, with ground-truth disparity overlaid. Each tile shows the rendered RGB view; the bottom-right is the corresponding ground-truth depth converted to inverse-depth (disparity). Rows / columns follow the shared yaw × pitch grid (yaws {−90∘,−45∘,0∘,+45∘,+90∘} , pitches {−20∘,0∘,+20∘} ).
Source
Num. of Scenes Loaded
Weight
Min gap
Max gap
DL3DV-10K
8,559
0.35
5
25
RealEstate10K
26,628
0.65
10
50
Appendix
Table 3: Mixed training mixture. Two real-world sources are sampled in proportion to the listed weight . Min/Max gap is the temporal stride (in frames) between the source and target frame of a training pair; the wider RE10K range reflects its faster intra-scene camera motion.
Dataset
Stride
Min Gap
Max Gap
Num. Pairs
Num. Scenes
DL3DV-10K
5
5
25
28,780
54
RealEstate10K
10
10
50
18,084
1,991
Tanks & Temples
1
1
25
5,642
7
Appendix
Table 4: Evaluation benchmark configurations, denoting the sequence frame stride, minimum/maximum frame separation gaps, total evaluated pairs, and scene counts.
Hyper-parameter
Value
Base model
Qwen/Qwen-Image-Edit-2509
Transformer backbone
linoyts/Qwen-Image-Edit-Rapid-AIO
Lightning weights
Qwen-Image-Edit-2511-Lightning-8steps-V1.0-bf16
LoRA rank / α
128 / 128 ( 64 / 64 for Blender training)
LoRA target modules
q, k, v, out, MLP, mod, in, proj_out
LoRA learning rate
5×10−5
Appendix
Table 5: Mixed training configuration. Values reproduced verbatim from the released config.json (mixed-DA3 run).
Figure 12: Scale jitter experiment. Baseline (top) with no jitter, and two variants where each object is randomly scaled by ± 5% and ± 20%. All metrics degrade together: PCC increases (worse 3D consistency), while PSNR / SSIM / LPIPS decrease (worse visual similarity). Unlike the occlusion experiment, scale jitter is detectable by all metrics since it changes both geometry and appearance uniformly.
Figure 13: Occlusion experiment. Scene A (top) and two variants B 1 , B 2 where objects are added behind the anchor camera (View 01). Metrics: PCC / PSNR / SSIM / LPIPS . For Scene A, 2D metrics are perfect since each view is compared to itself. For Scene B variants, PCC remains unchanged ( ≈ Scene A) because each scene is internally 3D-consistent, while PSNR / SSIM / LPIPS correctly detect the visual differences from added objects.
Figure 14: Ground-truth versus DA3-estimated geometry for a two-view pair. We visualize three distinct 3D reconstructions of a two-view scene. Top two rows: The source and target RGB inputs alongside their corresponding ground-truth (GT) depth maps (from Blender) and scale-aligned DA3 depth predictions, all color-coded by distance. Bottom row (left to right): (1) The reference GT joint point cloud, obtained by unprojecting both GT depths using GT camera poses (PCC = 0.008). (2) A mixed reconstruction utilizing the source GT depth and the target DA3 depth. The target depth is affine-aligned to the GT scale using a foreground/background split (determined via Otsu’s thresholding on the pooled GT depth) to specifically isolate the error introduced by DA3 (PCC = 0.099). (3) The fully estimated point cloud, unprojected using DA3 depths and GT cameras for both views(PCC = 0.203).
Figure 15: Projection Consistency Constraint (PCC DA3 ) on DL3DV. Each column shows a source–target view pair: the top two rows are the source and target RGB frames, and the bottom two rows show the fused two-view point cloud (top-down and front) built from the GT source depth and DA3-estimated target depth (FG/BG affine-aligned), reprojected under the ground-truth poses. Consistent points keep their RGB color; red/blue highlight forward/backward reprojection inconsistencies, and the per-column PCC DA3 score (lower is better) quantifies them.
Figure 16: Projection Consistency Constraint (PCC DA3 ) on re10k. Each column shows a source–target view pair: the top two rows are the source and target RGB frames, and the bottom two rows show the fused two-view point cloud (top-down and front) built from the GT source depth and DA3-estimated target depth (FG/BG affine-aligned), reprojected under the ground-truth poses. Consistent points keep their RGB color; red/blue highlight forward/backward reprojection inconsistencies, and the per-column PCC DA3 score (lower is better) quantifies them.
2D image metrics
3D / camera pose metrics
PSNR ↑
SSIM ↑
LPIPS ↓
PCC GT↓
PCC DA3↓
AUC@5 ↑
RPE-R ↓
RPE-T ↓
Baseline
∞
1.000
0.000
0.002
0.074
0.743
12.79
0.77
Jitter ± 1%
42.1
0.996
0.004
0.004
0.095
0.700
13.08
0.79
Jitter ± 2%
39.1
0.990
0.008
0.006
0.123
0.686
12.93
0.93
Jitter ± 5%
34.6
0.976
0.021
0.015
0.232
0.486
13.23
2.22
Jitter ± 10%
31.1
0.962
0.043
0.029
0.346
0.329
13.87
4.12
Appendix
Table 6: Mean metrics across views for the scale-jitter and occlusion experiments. Under scale jitter, 2D quality (PSNR/SSIM/LPIPS) and the DA3-based geometry/pose metrics (PCC DA3 , AUC@5, RPE-T) all degrade monotonically as the perturbation grows; rotation error RPE-R stays in a narrow ∼ 13–15 band, confirming that scale ambiguity manifests primarily as translation drift. Under occlusion, where each scene variant is internally 3D-consistent but visually different from the baseline, the 2D metrics respond: PCC GT remains at the baseline level ( 0.005≈0.002 ), correctly recognising each variant as geometrically consistent.
Figure 17: Multi-view geometric inconsistency of a 60-frame accumulated point cloud (DL3DV). Top: 15 of the 60 input frames (stride 4) used to build the cloud. Bottom: the accumulated point cloud, unprojected from the per-frame DA3 depth and DA3-estimated camera poses and shown from two viewpoints (top-down and front, in the reference-camera frame). Each frame’s depth is scored by reprojecting the global cloud into that view (z-buffer) and measuring the relative depth violation rel=(D−D^)/D^ ; points with rel>τ ( τ=0.15 ) are marked red , with opacity proportional to the degree of inconsistency max(rel−τ,0) (faint = borderline, solid = strongly inconsistent). Geometrically consistent points keep their RGB color.
Figure 18: Multi-view geometric inconsistency of a 60-frame accumulated point cloud (DL3DV).
Figure 19: Multi-view geometric inconsistency of a 60-frame accumulated point cloud (DL3DV).
Model
Paradigm
NFE
Native Res.
Per-view Time
OVIE
feed-forward + 16-scale search
∼ 1 ( × 16)
256
0.1 – 0.3 s
LagerNVS
feed-forward
∼ 1
512
0.2 – 0.5 s
ESC-Blender (rank 64)
diffusion
20
384
∼ 3 s
ESC-Mixed (rank 128)
diffusion
20
384
∼ 3 s
VIVID
diffusion + CFG ( g=1.5 )
32 ( ∼ 64)
384
3 – 4 s
GLD
cascade diffusion (L1/L2/L3)
multi-stage
—
5 – 15 s
Appendix
Table 7: Comparison of inference paradigms, total number of function evaluations (NFE), native resolution, and per-view latency. Per-view times are normalized to an H200 GPU and represent startup-corrected estimates. For our ESC models, we observe similar inference latency across the evaluated LoRA ranks (rank 64 for Blender; rank 128 for Mixed).
Table 8: Qualitative comparison of ESC variants on the synthetic Blender benchmark. Each row shows the source view (blue border) , the ground-truth target (green border) , and three ESC variants (orange border) : ESC-Shadow, ESC-PCC-BoN, and ESC-PCC-FT. PCC-aware selection and fine-tuning yield novel views that most faithfully reproduce the scene geometry.
Figure 20: Extended qualitative comparison on novel view synthesis (DL3DV and Tanks & Temples). For each scene we show the input source view ( blue border ), the renderings produced by the baselines ( gray border : PhotoNVS, VIVID, OVIE, GLD, LagerNVS), our ESC variants ( orange border : ESC-Shadow, ESC-Base, and the geometry-guided ESC-PCC-BoN (Best-of-N)), and the ground-truth target view ( green border ). Rows 1–9 are scenes from DL3DV, and the last two rows show the Church and Courthouse scenes from Tanks & Temples. Across datasets, our variants reproduce scene geometry and fine detail more faithfully than the baselines, with ESC-PCC-BoN consistently best matching the ground-truth target.
Figure 21: Extended qualitative comparison on novel view synthesis (RE10K). For each scene we show the input source view ( blue border ), the renderings produced by the baselines ( gray border : PhotoNVS, VIVID, OVIE, GLD, LagerNVS), our ESC variants ( orange border : ESC-Shadow, ESC-Base, and the geometry-guided ESC-PCC-BoN (Best-of-N)), and the ground-truth target view ( green border ). All rows are scenes from the RealEstate10K (RE10K) benchmark. Our variants reproduce indoor scene geometry and fine detail more faithfully than the baselines, with ESC-PCC-BoN consistently best matching the ground-truth target.
Figure 22: Best-of-N novel-view synthesis on DL3DV. Each row shows the source view (blue border) , ground truth (green border) , the PCC-selected best sample (orange border) , and three additional candidates (orange border) from the same Best-of-N pool. The first six rows are DL3DV scenes; the last row is the Barn scene from Tanks & Temples.
Figure 23: Additional Best-of-N novel-view synthesis. Each row shows the source view (blue border) , ground truth (green border) , the PCC-selected best sample (orange border) , and three additional candidates (orange border) from the same Best-of-N pool. The first six rows are RE10K scenes; the last row is the Truck scene from Tanks & Temples.
Table 10: RGB and depth visualization. For each of two scenes we show the source view (blue border) , the ground-truth target (green border) , and our prediction (orange border) , as an RGB row and the corresponding depth row. Our predictions closely match the ground-truth targets in both appearance and geometry.
Metric
ESC-shadow
ESC-Base
ESC-PCC-BoN
ESC-PCC-FT
Oracle
@130k iters
@100k iters
100k iters
@100k+30k iters
Synthesis Quality
PSNR ↑
31.90±3.38
30.73±3.69
31.59±3.29
33.65±3.87
⋅
SSIM ↑
0.9607±0.0178
0.9532±0.0259
0.9578±0.0176
0.9693±0.0170
⋅
Perceptual Quality
LPIPS ↓
0.0419±0.0319
0.0584±0.0566
0.0460±0.0324
0.0324±0.0301
⋅
Appendix
Table 11: Ablation of ESCvariants on the Blender benchmark. We compare four ESCconfigurations and an oracle across four metric groups: synthesis quality (PSNR, SSIM), perceptual quality (LPIPS, DINO similarity), geometric fidelity (PCC), and trajectory & view-prompt consistency (pose AUC at 5∘ / 15∘ / 30∘ , and relative pose errors RPE-R in degrees and RPE-T). Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) is better. ESC-Base uses a single generated sample; ESC-PCC-BoN selects the best of 20 samples per view by geometric consistency (PCC); ESC-PCC-FT additionally fine-tunes with the PCC objective; and ESC-shadow denotes a model variant we trained without depth conditioning and prediction. Oracle indicates the ground-truth label compared against the paired DA3 estimate on the source images. The best, second-best, and third-best results among the ESCvariants in each row are highlighted. PCC-aware selection and fine-tuning consistently improve perceptual quality and geometric fidelity, with ESC-PCC-FT achieving the best results across nearly all metrics. More qualitative results are shown in Table 8 .
Figure 24: Qualitative comparison of ESCvariants on a simulation environment. Panels show the source view ( blue border ), ground-truth ( green border ), and three ESCvariant generations ( orange border ). The bottom row magnifies the yellow dashed region to illustrate 3D geometric consistency using color-matched epipolar lines derived from the source keypoints. While the bottom-right corner of the pink(red) cube nearly touches the red(pink) epipolar line in the ground-truth, it sits nearly flush in ESC-PCC-FT, shows a minor pixel gap in ESC-PCC-BoN, and leaves a small but more prominent gap in ESC-shadow, establishing the order of geometric alignment quality across the variants in this example.
Current visual generation models are capable of producing high-quality content, yet they lack a coherent perception of the spatial structure. Existing generative novel view synthesis methods typically introduce explicit geometry priors, which enforce spatial consistency but inherently restrict generalization in large view changes. In contrast, recent interactive generative methods favor implicit scene modeling, offering greater flexibility at the cost of precise camera control and geometry consistency. In this paper, we propose MetaView, a diffusion-based monocular novel view synthesis framework that enables rendering under large view changes from a single image. Our key insight is to combine implicit geometry modeling with minimal yet essential explicit 3D cues: we incorporate implicit geometry priors from a feed-forward geometry perception network to regularize structure without imposing restrictive reconstruction pipelines, while leveraging metric depth to anchor the generation to a metric scale. This design allows MetaView to achieve both geometry consistency and precise controllability. Extensive experiments demonstrate that, under challenging monocular large viewpoint changes, MetaView significantly outperforms existing methods and exhibits superior generalization. Our code is publicly available at https://github.com/KlingAIResearch/MetaView.
Yufei Cai, Xuesong Niu, Hao Lu +3
Nanyang Technological University · Kolors Team, Kuaishou Technology · The Hong Kong University of Science and Technology (Guangzhou)
Novel view synthesis requires strong 3D geometric consistency and the ability to generate visually coherent images across diverse viewpoints. While recent camera-controlled video diffusion models show promising results, they often suffer from geometric distortions and limited camera controllability. To overcome these challenges, we introduce GeoNVS, a geometry-grounded novel-view synthesizer that enhances both geometric fidelity and camera controllability through explicit 3D geometric guidance. Our key innovation is the Gaussian Splat Feature Adapter (GS-Adapter), which lifts input-view diffusion features into 3D Gaussian representations, renders geometry-constrained novel-view features, and adaptively fuses them with diffusion features to correct geometrically inconsistent representations. Unlike prior methods that inject geometry at the input level, GS-Adapter operates in feature space, avoiding view-dependent color noise that degrades structural consistency. Its plug-and-play design enables zero-shot compatibility with diverse feed-forward geometry models without additional training, and can be adapted to other video diffusion backbones. Experiments across 9 scenes and 18 settings demonstrate state-of-the-art performance, achieving 11.3% and 14.9% improvements over SEVA and CameraCtrl, with up to 2x reduction in translation error and 7x in Chamfer Distance.
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL