Organizations: Huawei Hilbert Research Center · Technical University of Munich, Munich, Germany · Wuhan University, Wuhan, China · ETH Zurich, Zurich, Switzerland
Mesh models have become increasingly accessible for numerous cities; however, the lack of realistic textures restricts their application in virtual urban navigation and autonomous driving. To address this, this paper proposes MeSS (Meshbased Scene Synthesis) for generating high-quality, styleconsistent outdoor scenes with city mesh models serving as the geometric prior. While image and video diffusion models can leverage spatial layouts (such as depth maps or HD maps) as control conditions to generate street-level perspective views, they are not directly applicable to 3D scene generation. Video diffusion models excel at synthesizing consistent view sequences that depict scenes but often struggle to adhere to predefined camera paths or align accurately with rendered control videos. In contrast, image diffusion models, though unable to guarantee cross-view visual consistency, can produce more geometry-aligned results when combined with ControlNet. Building on this insight, our approach enhances image diffusion models by improving cross-view consistency. The pipeline comprises three key stages: first, we generate geometrically consistent sparse views using Cascaded Outpainting ControlNets; second, we propagate denser intermediate views via a component dubbed AGInpaint; and third, we globally eliminate visual inconsistencies (e.g., varying exposure) using the GCAlign module. Concurrently with generation, a 3D Gaussian Splatting (3DGS) scene is reconstructed by initializing Gaussian balls on the mesh surface. Our method outperforms existing approaches in both geometric alignment and generation quality. Once synthesized, the scene can be rendered in diverse styles through relighting and style transfer techniques. project page: https://albertchen98.github.io/mess/
Figures & tables
Figure 1: Starting from textureless urban meshes, our MeSS synthesizes high-quality Gaussian Splatting scenes with realistic appearance. After synthesis, these Gaussian scenes can be further rendered into stylized videos.
Figure 2: Overview of the MeSS pipeline. Bottom: given a city mesh and a camera path, we render control signals (depth, semantics, normals) at each viewpoint. Left (Stage I): key views are generated autoregressively—starting from the last pose with CtrlNet-S , then outpainting backwards via CtrlNet-N with warped reference images—and instantiated as Gaussian surfels on the mesh surface to form a 3DGS field. Right (Stage II): intermediate frames rendered from the Gaussian field are refined by Appearance Guided Inpainting, densifying the field with each inpainted view.
Figure 3: (a) Resample (left) vs. AGInpaint (right): AGInpaint inpaints slim regions more faithfully. (b) Without (left) and with (right) GCAlign: GCAlign harmonizes the seams caused by exposure differences.
Table 4
Figure 4: Qualitative comparison with scene-generation methods. Please zoom in to inspect details.
Figure 5: Zero-shot generalization to the real LoD3 mesh TUM2Twin [ 54 ] , no retraining. Top: mesh-rendered control signals—colormapped depth, semantics, normals, and the grayscale depth given to the video baseline. Middle: street-level views generated by MeSS along the camera path; facades follow the LoD3 geometry and remain consistent across views. Bottom: Cosmos-Transfer2.5 (CT2.5), a video-diffusion ControlNet conditioned on the same mesh geometry, preserves only a coarse layout with incorrect facade detail.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Wall time
Peak VRAM
Stage I
3 min 00 s
5.6 GB
Stage II
≈ 17 min 10 s
≈ 24.7 GB
End-to-end
≈ 20 min 10 s
≈ 24.7 GB
Appendix
Table 3: Per-stage wall-clock and peak VRAM on a single RTX PRO 6000 for a 200-frame / 200 m trajectory.
MEt3R ↓
KPM ↑
PSNR ↑
SSIM ↑
LPIPS ↓
VistaDream †
0.155
1310
12.49
0.327
0.568
MeSS (Ours)
0.129
1382
13.92
0.495
0.371
Appendix
Table 4: Cross-view consistency and held-out novel-view synthesis against the matched baseline VistaDream † (12 shared trajectories).
Figure 6: Silhouettes on novel views. The ellipses highlight the silhouette regions—areas of low rendered opacity in the Gaussian field—that AGInpaint is tasked to fill.
Figure 7: Architecture of our two cascaded CtrlNets. Left: CtrlNet-S takes three geometric conditions (semantic, depth, normal) concatenated along the channel dimension. Right: CtrlNet-N additionally takes a reference image encoded by the frozen VAE; the latent features are element-wise added to the geometric condition features before injection into the ControlNet. Snowflake and flame icons denote frozen and trainable components, respectively.
Figure 8: Text2Tex trial on a crossroad mesh from the City Sample Project.
Figure 9: Stylized rendering results with Cosmos-Transfer1 and TC-Light. Please zoom in to check for details.
Multi-view diffusion models have shown strong performance in scenes with strong geometric priors and sparse semantics, such as indoor rooms or simple outdoor environments (e.g., fields, courtyards). However, they often fail to maintain cross-view consistency under camera rotation, especially in structurally complex urban environments. Without explicit modeling of spherical correspondence across views, existing approaches tend to produce object duplication, structural distortion, and layout inconsistency. To address this limitation, we propose StreetDiff, a multi-view diffusion framework that explicitly enforces cross-view alignment during denoising. StreetDiff introduces a Panorama--Perspective Synergy design to decouple global layout reasoning from local detail synthesis, and incorporates a Panorama Alignment Module (PAM) that establishes spherical-projection-based attention constraints across views. By injecting structured alignment constraints without modifying the diffusion backbone, our framework achieves robust cross-view coherence in challenging urban street scene generation tasks. In addition, we construct Street360, a large-scale HDR multi-view urban panorama dataset. Extensive experiments demonstrate that StreetDiff significantly improves structural consistency and visual fidelity compared to prior multi-view diffusion generation methods.
Qi Zhang, Yanyifan Wang, Weiyuan Zhang +1
College of Computer Science and Software Engineering, Shenzhen University, China
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
Synthesizing large-scale, explorable, and geometrically accurate 3D urban scenes is a challenging yet valuable task for immersive and embodied applications. The challenge lies in the lack of large-scale and high-quality real-world 3D scans for training generalizable generative models. In this paper, we take an alternative route to create large-scale 3D scenes by leveraging readily available satellite imagery for realistic coarse geometry and open-domain diffusion models for high-quality close-up appearance synthesis. We propose Skyfall-GS, a novel hybrid framework that synthesizes immersive city-block scale 3D urban scenes by combining satellite reconstruction with diffusion refinement, eliminating the need for costly 3D annotations, and also featuring real-time, immersive 3D exploration. We tailor a curriculum-driven iterative refinement strategy to progressively enhance geometric completeness and photorealistic texture. Extensive experiments demonstrate that Skyfall-GS provides improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches. Project page: https://skyfall-gs.jayinnn.dev/
Jie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai +6
National Yang Ming Chiao Tung University · UIUC · University of Zaragoza +1