Multi-view or video diffusion models have recently emerged as state-of-the-art approaches for novel-view synthesis from sparse image collections. While they can produce high quality images, they tend to hallucinate unrealistic spatial layouts when synthesizing views outside the input views (scene extrapolation). We propose to address this challenge for outdoor scenes by using satellite images as an additional information source to condition the multi-view generative model. We introduce a new satellite-conditioned multi-view diffusion model that combines conditioning from a satellite image and street-level input images when generating novel-views. Our cross-attention based conditioning allows us to retain pretrained multi-view diffusion priors to generate seamless, realistic, novel views from a handful of source views, while also adhering to the scene layout as seen from satellite images. Our model does not use 3D scene reconstruction, and only relies on posed street and satellite views. Our experiments on outdoor datasets show that our model is better than both satellite-conditioned or street-view conditioned baselines on novel-view synthesis, especially on challenging cases where target views have very little overlap with source views. Project page: https://nianticspatial.github.io/groundcontrol
Figures & tables
Figure 1: GroundControl builds on recent novel-view synthesis (NVS) diffusion models, and enables more accurate synthesis of novel views by additionally conditioning on a top-down satellite image. When compared to the NVS diffusion model SEVA [ 3 ] (left), our model has better fidelity to the scene’s spatial layout.
Street cond.
Ortho sat.
MV cons.
Public code
Intermediate representation
Output modality
[ 71 ]
N
N
N
N
Mesh
Panorama
[ 72 ]
N
N
Y
N
Mesh
Pose-controlled
[ 19 ]
N
N
Y
N
Splat
Pose-controlled
[ 37 ]
N
N
N
N
Voxels
Panorama
[ 59 ]
N
Y
N
N
Warp
Panorama
[ 50 ]
N
Y
N
Y
Volume
Panorama
Table 1: Compared with other related works, GroundControl enables street-level conditioning and orthorectified satellite views. We control views via pose conditioning, and outputs are multi-view consistent. Abbreviations: Street cond. = street-level conditioning; Ortho sat. = orthorectified satellite; MV cons. = multi-view consistency. * [ 11 ] accepts map input but not satellite images; † we will release code and models upon acceptance.
Figure 2: GroundControl incorporates satellite images as an efficient conditioning signal for pose-controllable multi-view diffusion models. (a) Our inputs include a DINOv3 encoded satellite image, known ground-view image(s) Xinp with associated camera parameters Cinp , and target camera parameters Ctgt corresponding to the views to be synthesized. GroundControl then generates output images Xtgt for each target camera. (b) A key contribution is the integration of satellite information into the SEVA architecture [ 3 ] . (c) Our ray-guided attention sub-block fuses satellite features with noisy latent embeddings via multi-head cross-attention.
Figure 3: Our ray-guided cross-attention attends to tokens of the satellite image that are along the 3D ray of the street-view token.
nuScenes [ 6 ]
Metropolis [ 44 ]
PSNR ↑
SSIM ↑
LPIPS ↓
DINO ↓
SEG ↑
IoU road ↑
IoU bdg ↑
PSNR ↑
SSIM ↑
LPIPS ↓
DINO ↓
SEG ↑
IoU road ↑
IoU bdg ↑
Street-View only
SEVA
11.334
0.308
0.616
6.140
0.492
0.425
0.182
10.884
0.315
0.586
5.654
0.551
0.581
0.291
Satellite only
Sat2Density [ 50 ]
10.679
0.301
0.714
6.657
0.507
–
–
9.074
0.346
0.743
6.775
0.537
–
–
Street-view + Satellite
Table 2: Comparison with baseline methods on test datasets that match our training domain , i.e . driving datasets. Our model improves over the baseline systems. Semantic IoU includes vehicles in both the road and building masks.
DL3DV [ 41 ]
PSNR ↑
SSIM ↑
LPIPS ↓
DINO ↓
SEG ↑
TSED ↓
%Consistent ↑
#Corrs. ↑
Sat2Density [ 50 ]
10.204
0.232
0.768
7.116
0.451
N/A
0.0
0
ViewCrafter* [ 77 ]
13.350
0.260
0.547
5.230
0.541
3845.57
2.02
332.62
SEVA [ 3 ]
14.912
0.360
0.407
3.534
0.674
22.75
62.99
287
Ours
15.170
0.372
0.414
3.653
0.674
18.35
65.56
290
Table 3: Comparison with baseline methods on DL3DV, a dataset which is out of distribution with respect to our training data , as it has large-viewpoint NVS with sparse inputs. We use the geolocalized dataset split from [ 64 ] . We use a total of 39 sequences with 3 input views and 37 test views per sequence. *ViewCrafter is incompatible with our input-target view split since it only generates a continuous trajectory, and therefore uses a different continuous input-target configuration.
Figure 4: Qualitative results on in-distribution dataset: Metropolis Top 4 rows show satellite images along with corresponding input and target views. Final two rows show synthesis of target views using GroundControl and SEVA. GroundControl is significantly better than SEVA at adhering to the layout of the scene.
Figure 5: Qualitative results on in-distribution datasets: Metropolis and nuScenes. Each row shows, from left: The satellite view S ; an example ground-level ‘seed’ input view Xinp ; a ground truth target view Xtgt ; then predictions from GroundControl, [ 3 ] and [ 50 ] . Satellite images have input camera frustums plotted in red , and the target frustum in green . † We crop Sat2Density results to make pinhole images from their panoramic outputs; see supplementary for their full panoramic output.
Figure 6: Qualitative results on the DL3DV benchmark: Despite being far from the distribution of street scenes, we find that GroundControl learns to leverage the satellite image when generating regions without sufficient overlap from sparse input views ( e.g . buildings in the background). † We resample Sat2Density panoramas to make pinhole images; see supplementary for their full panoramic output. Red triangles in satellite view are inputs, green is the shown target view.
PSNR ↑
DINO ↓
SEG ↑
Satellite view as input view
11.312
5.862
0.558
W/o ray guided cross-attention (RGCA)
11.675
5.583
0.573
W/o Ray and timestep embed. in RGCA query
11.372
5.309
0.579
UNet decoder conditioning
11.291
5.400
0.572
W/o separate CFG
12.706
5.144
0.608
Ours
12.809
5.247
0.605
Table 4: Ablation study on the Metropolis dataset.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure I: PSNR performance vs. image overlap measured by the number of correspondences on Metropolis dataset . GroundControl obtains better performance compared to the SEVA baseline across all overlap regimes.
nuScenes
DL3DV
offset (m,°)
PSNR ↑
SSIM ↑
DINO ↓
Seg ↑
PSNR ↑
SSIM ↑
DINO ↓
Seg ↑
SEVA
11.334
0.308
6.140
0.492
14.912
0.360
3.534
0.674
0,0
11.880
0.331
6.145
0.498
15.170
0.372
3.653
0.674
5,5
11.518
0.311
6.156
0.490
15.131
0.372
3.679
0.670
10,10
11.509
0.311
6.156
0.490
15.124
0.371
3.682
0.669
Appendix
Table I : Robustness to pose misalignment: Performance degradation is mild with alignment offsets.
Views
Model
Straight
Turns
PSNR ↑
LPIPS ↓
DINO ↓
Seg. ↑
PSNR ↑
LPIPS ↓
DINO ↓
Seg. ↑
1
RealCamI2V
11.57
0.508
4.482
0.627
10.28
0.599
5.672
0.557
SEVA
12.35
0.505
4.718
0.611
10.48
0.616
5.968
0.523
Ours
12.73
0.518
4.687
0.623
11.12
0.612
5.787
0.548
2
Veo 3.1
14.81
0.395
3.245
0.707
11.99
0.538
5.068
0.574
SEVA
12.99
0.472
4.427
0.633
11.29
0.545
5.196
0.571
Appendix
Table II : Comparison to image-to-video diffusion models on the Metropolis dataset for a video generation task (continuous trajectories). We divide our test set into straight car trajectories vs turns. Our model is slightly better than RealCamI2V in both cases, while being comparable to Veo on turns, with better semantic-feature metrics.
Dataset
p100
p75
p50
p25
p10
p0
nuScenes
2.714
0.850
0.527
0.221
-0.095
-0.817
Metropolis
3.444
2.396
1.962
1.449
0.941
0.258
Appendix
Table III : Distribution (percentile values) of per-scene (PSNRours−PSNRSEVA) . These show that satellite conditioning improves generations in most cases on nuScenes and Metropolis.
PSNR ↑
DINO ↓
SEG ↑
Flipped satellite
12.017
5.782
0.513
Mismatched satellite
12.052
5.591
0.530
Ours
12.809
5.247
0.605
Appendix
Table IV: Impact of corrupted satellite imagery: using flipped satellite images or mismatched satellite images degrades performance.
Figure II: Multi-view generations on the Metropolis dataset: These are multi-frame generations from a single diffusion process with 7 input views (shown in red on the satellite images). We show target views from 5 consecutive timesteps ( green in the satellite images). Our model’s generations are multi-view consistent, and adheres to the spatial layout information in the satellite image.
Figure III: Qualitative results on the Metropolis dataset. We show results of our satellite-conditioned model on sequences of length 21 frames. This includes 7 input frames (from the front camera) and 14 target frames (from the left and right cameras at the corresponding timestamps). The SEVA model without satellite conditioning underperforms in this sparse input-view regime which has little overlap with the target views.
Figure IV: Qualitative results on nuScenes dataset These show cases with very little overlap between input views ( red in satellite view) and the target views ( green in satellite view). Our model’s generations adhere more closely to the layout of the scene, since these are conditioned on the satellite view.
Figure V: Qualitative results with multiple noise samples : We compare variations in the generations from SEVA to that of our model, when starting the denoising process with different noise samples. In regions that are non-overlapping with the input view, SEVA’s generations vary more, and are inconsistent with the scene layout. Since our generations are conditioned on satellite imagery, they show less variance in the generated layout, while still generating diverse samples.
Figure VI: Varying satellite CFG on a scene from Metropolis Increasing the satellite CFG strengthens the adherence to the satellite image, but higher values trade off against image quality. The input views on the top row correspond to the red frustums in the satellite image, and the predicted views shown in the second row correspond to the green frustum.
Figure VII: Qualitative ablation results on NuScenes. Here we compare three alternatives: a model with no satellite input (No sat)̇; a model with satellite input but no ray-based cross-attention (no ray CA), and finally our model. Columns on the left show the satellite view (S) with the input views in red and target view in green , a representative input view (input) and the ground truth (GT) corresponding to this target view. We note that our model tends to have better fidelity to the ground truth layout.
Figure VIII: Visualization of Sat2Density results on nuScenes, complementing Fig. 5 . The predicted panorama is 128 × 512. Each row shows (left) the panorama prediction and below it a highlight showing the region used to generate the cropped image, and (right) the ground-truth target image.
Figure IX: Visualization of Satellite attention. Top row: input images; second row: satellite features, satellite ray samples aligned with target image, target image and generated image; third row: attention maps according to the Z value along the ray.
Figure X: Top row: some input views from our 3-view DL3DV split. Bottom row: a test view, showing the ground truth, ours, and LongLRM.
Figure XI: Veo 3.1 qualitative results on the Metropolis dataset with 3 input frames. We generate 2 separate videos with Veo to use the 3 conditioning frames, but GroundControl generates the target views in a single pass. Veo produces highly photorealistic videos, since its model size and training data are larger by orders of magnitude. Unlike our model, it can also handle dynamic objects (including the hood of the car the camera is mounted on). However, its camera trajectory may not necessarily follow the required target poses. It can also frequently hallucinate scene content in incorrect locations (as highlighted in the image).
Dataset
MV
Pano
#Scenes
Contrib
Satellite source
Argoverse 2 [ 67 ]
Yes
No
700
10%
Google + ArcGIS
nuScenes [ 6 ]
Yes
No
700
13%
ArcGIS
Metropolis [ 44 ]
Yes
No
75
7%
Original dataset
VIGOR [ 84 ]
Yes*
Yes
52105 †
70%
Azure Maps
Appendix
Table V : Training datasets. * We pair nearby panoramas to create a multiview version of the dataset. † Unlike other datasets, the number of scenes in VIGOR corresponds to the number of panoramas. MV = is multi-view? Pano = is panorama? Contrib = training contribution. We train jointly on all datasets.
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Kangjie Chen, Xiangyu Li, Dongbin Zhang +12
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
We present Cross-View Splatter, a feed-forward method that predicts pixel-aligned Gaussian splats for outdoor scenes captured at ground level AND by satellite. Faithful reconstructions require good camera coverage, but ground imagery is time-consuming and hard to capture at scale for large outdoor scenes. Fortunately, satellite imagery can provide a global geometric prior that is easy to access via public APIs. Cross-View Splatter fuses orthorectified satellite views with GPS-tagged ground photos to predict Gaussian splats in a unified 3D coordinate frame. By aligning ground and bird's-eye feature representations, our model improves scene coverage and novel-view synthesis, compared to ground imagery alone. We train on curated georeferenced datasets and paired satellite-terrain data, mined from open mapping services. We evaluate our method on a new benchmark for novel-view synthesis with georeferenced imagery allowing comparison to prior state-of-the-art methods. Our code and data preparation will be available at https://nianticspatial.github.io/cross-view-splatter/.
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL