cs.CVOct 14, 2026
SaveESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis
Organizations: FAIR at Meta · Mila – Québec AI Institute · Université de Montréal · Canada CIFAR AI Chair ·
Abstract
Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.
Figures & tables
Figure 1: Novel view synthesis highlights. Given a single input view, its corresponding depth map, and Plücker ray conditioning, our method generates novel views that remain 3D-consistent with the target across challenging datasets (DL3DV, RealEstate10K, T&T) and selective in-the-wild scenes. For the datasets, we illustrate a single novel view generation alongside the ground-truth target, whereas for the in-the-wild setting, we demonstrate two distinct novel view generations.
Figure 2: PCC computation pipeline. The source view (a) is from the original scene, while the target view (f) has been modified: a red sphere is added and a yellow sphere is removed. Top row ( , detecting missing geometry): the source depth (b) is unprojected to 3D and reprojected into the target camera frame (c); black regions indicate pixels outside the reprojection coverage. Where the yellow sphere was removed, target camera rays pass through and hit the cubes in the back instead of striking the sphere surface, producing large positive values. These inconsistent pixels are highlighted in red (e), with pixels exceeding the error threshold . Bottom row ( , detecting hallucinated geometry): The modified target depth is reprojected back into the source frame (h). The added red sphere reprojects into a region with no corresponding geometry in the source perspective, yielding large positive values and inconsistent pixels (j). The final PCC score of quantifies the overall geometric disagreement between the two views.
Figure 3: ESC Framework. The framework extends Qwen MM-DiT with depth information and Plücker rays conditioning by leveraging LoRA adapters and a Ray ControlNet. At inference time, we leverage PCC-based sampling to enforce improved generated scene geometry.
| Perceptual Quality | Geometry | |||
| Model | LPIPS | DINO-Sim | DreamSim | PCC |
| DL3DV Dataset | ||||
| PhotoNVS | ||||
| VIVID | ||||
| OVIE | ||||
| GLD | ||||
Table 1: Quantitative comparison on novel view synthesis across three datasets. We evaluate perceptual quality using LPIPS, DINO similarity, and DreamSim, and geometric consistency using PCC, on the DL3DV, RE10K, and Tanks & Temples datasets. Arrows indicate whether higher ( ) or lower ( ) is better. We compare our method (ESC) and its variants against prior baselines (PhotoNVS, VIVID, OVIE, GLD, LagerNVS), with results reported as mean standard deviation. The best, second-best, and third-best results in each column are highlighted. ESC consistently outperforms all baselines across perceptual metrics, and the geometry-guided ESC-PCC-BoN variant further improves geometric consistency. Models marked with were trained on the training split of the dataset. Models marked with were trained on a restricted subset comprising less than half of the full training split.
Figure 6: Metric correlation analyses. Per-view correlation between PCC and 2D metrics across both experiments. Each point represents a single (variant, view) pair. Red : scale jitter experiment—PCC correlates strongly with all 2D metrics ( for PSNR, for SSIM, for LPIPS), since geometric distortion simultaneously changes appearance. Blue : occlusion experiment—correlations weaken substantially ( for PSNR, for SSIM, for LPIPS), as views with large visual differences (high LPIPS, low PSNR) maintain low PCC because each scene is internally 3D-consistent. The strong-to-weak correlation shift between experiments demonstrates that PCC and 2D metrics measure fundamentally orthogonal properties.
Figure 7: Sensitivity of the DA3 PCC metric to controlled scale perturbations. Starting from a clean reference scene, we re-render each object with a random per-object scale jitter of 1–40% and measure (a) PCC, (b) pose AUC@3, 5, 15, 30, (c) relative rotation error (RPE-R), and (d) relative translation error (RPE-T), with depth and cameras predicted by DA3 and FG/BG-split affine alignment (no top-% pruning). With one exception, all evaluated metrics track geometric error monotonically alongside increasing jitter magnitude. Specifically, PCC rises from 0.07 at 0% jitter to 0.62 at 40%, AUC@30 falls from 0.84 to 0.51 (with AUC@3 collapsing from 0.67 to 0 by the 20% mark), and RPE-T increases from 0.77 to 12.35. RPE-R is the only metric that fails to reliably reflect the introduced noise, remaining largely stagnant and breaking monotonicity at 2%, 7%, and 30%.
Figure 8: Geometrically inconsistent point detections in a counterfactual scene. The real pair depicts a 3D-consistent scene in which all objects remain stationary as the camera moves between the two viewpoints. The edited pair shares the same source image (same left image), and its target image (right image) is rendered from the same camera viewpoint as in the consistent pair. However, one or more objects are displaced in the second’s target, producing a counterfactual scene that is incompatible with the 3D structure observed in source. By projecting the point cloud derived from each image into the other view, the PCC-based procedure identifies points whose projected geometry is inconsistent with the modified scene. These inconsistencies are highlighted as red and blue pixels, corresponding to inconsistencies detected by the forward and backward projections, respectively.
Figure 9: Effect of frame gap on model performance. Performance consistently degrades across all models as the frame gap increases. Nevertheless, our method outperforms all baselines at every frame-gap setting on all metrics except PSNR and SSIM, for which LagerNVS achieves the best results.
| Perceptual Quality | Geometry | |||
| Model | LPIPS | DINO-Sim | DreamSim | PCC |
| DL3DV Dataset | ||||
| ESC-Shadow | ||||
| ESC-Base | ||||
| ESC-PCC-BoN | ||||
| RE10K Dataset | ||||
Table 2: Ablation study of the ESC framework. Quantitative evaluation comparing ESC-shadow, ESC-base, and the full ESC-PCC-BoN pipeline across the DL3DV, RE10K, and Tanks & Temples datasets.
Figure 10: Correlation between DA3-derived and predicted PCC metrics.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: One Blender + Kubric scene rendered from all 15 grid viewpoints, with ground-truth disparity overlaid. Each tile shows the rendered RGB view; the bottom-right is the corresponding ground-truth depth converted to inverse-depth (disparity). Rows / columns follow the shared yaw pitch grid (yaws , pitches ).
| Source | Num. of Scenes Loaded | Weight | Min gap | Max gap |
|---|---|---|---|---|
| DL3DV-10K | 8,559 | |||
| RealEstate10K | 26,628 |
Table 3: Mixed training mixture. Two real-world sources are sampled in proportion to the listed weight . Min/Max gap is the temporal stride (in frames) between the source and target frame of a training pair; the wider RE10K range reflects its faster intra-scene camera motion.
| Dataset | Stride | Min Gap | Max Gap | Num. Pairs | Num. Scenes |
|---|---|---|---|---|---|
| DL3DV-10K | 28,780 | 54 | |||
| RealEstate10K | 18,084 | 1,991 | |||
| Tanks & Temples | 5,642 | 7 |
Table 4: Evaluation benchmark configurations, denoting the sequence frame stride, minimum/maximum frame separation gaps, total evaluated pairs, and scene counts.
| Hyper-parameter | Value |
|---|---|
| Base model | Qwen/Qwen-Image-Edit-2509 |
| Transformer backbone | linoyts/Qwen-Image-Edit-Rapid-AIO |
| Lightning weights | Qwen-Image-Edit-2511-Lightning-8steps-V1.0-bf16 |
| LoRA rank / | / ( / for Blender training) |
| LoRA target modules | q, k, v, out, MLP, mod, in, proj_out |
| LoRA learning rate |
Table 5: Mixed training configuration. Values reproduced verbatim from the released config.json (mixed-DA3 run).
Figure 12: Scale jitter experiment. Baseline (top) with no jitter, and two variants where each object is randomly scaled by 5% and 20%. All metrics degrade together: PCC increases (worse 3D consistency), while PSNR / SSIM / LPIPS decrease (worse visual similarity). Unlike the occlusion experiment, scale jitter is detectable by all metrics since it changes both geometry and appearance uniformly.
Figure 13: Occlusion experiment. Scene A (top) and two variants B 1 , B 2 where objects are added behind the anchor camera (View 01). Metrics: PCC / PSNR / SSIM / LPIPS . For Scene A, 2D metrics are perfect since each view is compared to itself. For Scene B variants, PCC remains unchanged ( Scene A) because each scene is internally 3D-consistent, while PSNR / SSIM / LPIPS correctly detect the visual differences from added objects.
Figure 14: Ground-truth versus DA3-estimated geometry for a two-view pair. We visualize three distinct 3D reconstructions of a two-view scene. Top two rows: The source and target RGB inputs alongside their corresponding ground-truth (GT) depth maps (from Blender) and scale-aligned DA3 depth predictions, all color-coded by distance. Bottom row (left to right): (1) The reference GT joint point cloud, obtained by unprojecting both GT depths using GT camera poses (PCC = 0.008). (2) A mixed reconstruction utilizing the source GT depth and the target DA3 depth. The target depth is affine-aligned to the GT scale using a foreground/background split (determined via Otsu’s thresholding on the pooled GT depth) to specifically isolate the error introduced by DA3 (PCC = 0.099). (3) The fully estimated point cloud, unprojected using DA3 depths and GT cameras for both views(PCC = 0.203).
Figure 15: Projection Consistency Constraint (PCC ) on DL3DV. Each column shows a source–target view pair: the top two rows are the source and target RGB frames, and the bottom two rows show the fused two-view point cloud (top-down and front) built from the GT source depth and DA3-estimated target depth (FG/BG affine-aligned), reprojected under the ground-truth poses. Consistent points keep their RGB color; red/blue highlight forward/backward reprojection inconsistencies, and the per-column PCC score (lower is better) quantifies them.
Figure 16: Projection Consistency Constraint (PCC ) on re10k. Each column shows a source–target view pair: the top two rows are the source and target RGB frames, and the bottom two rows show the fused two-view point cloud (top-down and front) built from the GT source depth and DA3-estimated target depth (FG/BG affine-aligned), reprojected under the ground-truth poses. Consistent points keep their RGB color; red/blue highlight forward/backward reprojection inconsistencies, and the per-column PCC score (lower is better) quantifies them.
| 2D image metrics | 3D / camera pose metrics | |||||||
|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PCC | PCC | AUC@5 | RPE-R | RPE-T | |
| Baseline | 1.000 | 0.000 | 0.002 | 0.074 | 0.743 | 12.79 | 0.77 | |
| Jitter 1% | 42.1 | 0.996 | 0.004 | 0.004 | 0.095 | 0.700 | 13.08 | 0.79 |
| Jitter 2% | 39.1 | 0.990 | 0.008 | 0.006 | 0.123 | 0.686 | 12.93 | 0.93 |
| Jitter 5% | 34.6 | 0.976 | 0.021 | 0.015 | 0.232 | 0.486 | 13.23 | 2.22 |
| Jitter 10% | 31.1 | 0.962 | 0.043 | 0.029 | 0.346 | 0.329 | 13.87 | 4.12 |
Table 6: Mean metrics across views for the scale-jitter and occlusion experiments. Under scale jitter, 2D quality (PSNR/SSIM/LPIPS) and the DA3-based geometry/pose metrics (PCC , AUC@5, RPE-T) all degrade monotonically as the perturbation grows; rotation error RPE-R stays in a narrow 13–15 band, confirming that scale ambiguity manifests primarily as translation drift. Under occlusion, where each scene variant is internally 3D-consistent but visually different from the baseline, the 2D metrics respond: PCC remains at the baseline level ( ), correctly recognising each variant as geometrically consistent.
Figure 17: Multi-view geometric inconsistency of a 60-frame accumulated point cloud (DL3DV). Top: 15 of the 60 input frames (stride 4) used to build the cloud. Bottom: the accumulated point cloud, unprojected from the per-frame DA3 depth and DA3-estimated camera poses and shown from two viewpoints (top-down and front, in the reference-camera frame). Each frame’s depth is scored by reprojecting the global cloud into that view (z-buffer) and measuring the relative depth violation ; points with ( ) are marked red , with opacity proportional to the degree of inconsistency (faint = borderline, solid = strongly inconsistent). Geometrically consistent points keep their RGB color.
Figure 18: Multi-view geometric inconsistency of a 60-frame accumulated point cloud (DL3DV).
Figure 19: Multi-view geometric inconsistency of a 60-frame accumulated point cloud (DL3DV).
| Model | Paradigm | NFE | Native Res. | Per-view Time |
| OVIE | feed-forward + 16-scale search | 1 ( 16) | – s | |
| LagerNVS | feed-forward | 1 | – s | |
| ESC-Blender (rank 64) | diffusion | 3 s | ||
| ESC-Mixed (rank 128) | diffusion | 3 s | ||
| VIVID | diffusion + CFG ( ) | ( 64) | – s | |
| GLD | cascade diffusion (L1/L2/L3) | multi-stage | — | – s |
Table 7: Comparison of inference paradigms, total number of function evaluations (NFE), native resolution, and per-view latency. Per-view times are normalized to an H200 GPU and represent startup-corrected estimates. For our ESC models, we observe similar inference latency across the evaluated LoRA ranks (rank 64 for Blender; rank 128 for Mixed).
Table 8: Qualitative comparison of ESC variants on the synthetic Blender benchmark. Each row shows the source view (blue border) , the ground-truth target (green border) , and three ESC variants (orange border) : ESC-Shadow, ESC-PCC-BoN, and ESC-PCC-FT. PCC-aware selection and fine-tuning yield novel views that most faithfully reproduce the scene geometry.
Figure 20: Extended qualitative comparison on novel view synthesis (DL3DV and Tanks & Temples). For each scene we show the input source view ( blue border ), the renderings produced by the baselines ( gray border : PhotoNVS, VIVID, OVIE, GLD, LagerNVS), our ESC variants ( orange border : ESC-Shadow, ESC-Base, and the geometry-guided ESC-PCC-BoN (Best-of-N)), and the ground-truth target view ( green border ). Rows 1–9 are scenes from DL3DV, and the last two rows show the Church and Courthouse scenes from Tanks & Temples. Across datasets, our variants reproduce scene geometry and fine detail more faithfully than the baselines, with ESC-PCC-BoN consistently best matching the ground-truth target.
Figure 21: Extended qualitative comparison on novel view synthesis (RE10K). For each scene we show the input source view ( blue border ), the renderings produced by the baselines ( gray border : PhotoNVS, VIVID, OVIE, GLD, LagerNVS), our ESC variants ( orange border : ESC-Shadow, ESC-Base, and the geometry-guided ESC-PCC-BoN (Best-of-N)), and the ground-truth target view ( green border ). All rows are scenes from the RealEstate10K (RE10K) benchmark. Our variants reproduce indoor scene geometry and fine detail more faithfully than the baselines, with ESC-PCC-BoN consistently best matching the ground-truth target.
Figure 22: Best-of-N novel-view synthesis on DL3DV. Each row shows the source view (blue border) , ground truth (green border) , the PCC-selected best sample (orange border) , and three additional candidates (orange border) from the same Best-of-N pool. The first six rows are DL3DV scenes; the last row is the Barn scene from Tanks & Temples.
Figure 23: Additional Best-of-N novel-view synthesis. Each row shows the source view (blue border) , ground truth (green border) , the PCC-selected best sample (orange border) , and three additional candidates (orange border) from the same Best-of-N pool. The first six rows are RE10K scenes; the last row is the Truck scene from Tanks & Temples.
Table 10: RGB and depth visualization. For each of two scenes we show the source view (blue border) , the ground-truth target (green border) , and our prediction (orange border) , as an RGB row and the corresponding depth row. Our predictions closely match the ground-truth targets in both appearance and geometry.
| Metric | ESC-shadow | ESC-Base | ESC-PCC-BoN | ESC-PCC-FT | Oracle |
|---|---|---|---|---|---|
| @130k iters | @100k iters | 100k iters | @100k+30k iters | ||
| Synthesis Quality | |||||
| PSNR | |||||
| SSIM | |||||
| Perceptual Quality | |||||
| LPIPS | |||||
Table 11: Ablation of ESCvariants on the Blender benchmark. We compare four ESCconfigurations and an oracle across four metric groups: synthesis quality (PSNR, SSIM), perceptual quality (LPIPS, DINO similarity), geometric fidelity (PCC), and trajectory & view-prompt consistency (pose AUC at / / , and relative pose errors RPE-R in degrees and RPE-T). Arrows indicate whether higher ( ) or lower ( ) is better. ESC-Base uses a single generated sample; ESC-PCC-BoN selects the best of 20 samples per view by geometric consistency (PCC); ESC-PCC-FT additionally fine-tunes with the PCC objective; and ESC-shadow denotes a model variant we trained without depth conditioning and prediction. Oracle indicates the ground-truth label compared against the paired DA3 estimate on the source images. The best, second-best, and third-best results among the ESCvariants in each row are highlighted. PCC-aware selection and fine-tuning consistently improve perceptual quality and geometric fidelity, with ESC-PCC-FT achieving the best results across nearly all metrics. More qualitative results are shown in Table 8 .
Figure 24: Qualitative comparison of ESCvariants on a simulation environment. Panels show the source view ( blue border ), ground-truth ( green border ), and three ESCvariant generations ( orange border ). The bottom row magnifies the yellow dashed region to illustrate 3D geometric consistency using color-matched epipolar lines derived from the source keypoints. While the bottom-right corner of the pink(red) cube nearly touches the red(pink) epipolar line in the ground-truth, it sits nearly flush in ESC-PCC-FT, shows a minor pixel gap in ESC-PCC-BoN, and leaves a small but more prominent gap in ESC-shadow, establishing the order of geometric alignment quality across the variants in this example.
More in this topic
Novel View Synthesis
Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.
See all 282 papers