Organizations: Singapore–MIT Alliance for Research and Technology Centre (SMART), Singapore · Department of Urban Planning, Tsinghua University, Beijing, China · Department of Urban Studies and Planning, Massachusetts Institute of Technology, Cambridge, MA, USA · College of Urban and Environmental Sciences, Peking University, Beijing, China · Department of Urban and Regional Planning, University of Florida, Gainesville, FL, USA · Stanford Institute for Human-Centered Artificial Intelligence, Stanford University, Stanford, USA
Urban development has been a defining force in human history, shaping cities for centuries. However, past studies mostly analyze such development as predictive tasks, failing to reflect its generative nature. Therefore, this study designs a multimodal generative AI framework to envision sustainable urban development at a global scale. By integrating prompts and geospatial controls, our framework can generate high-fidelity, diverse, and realistic urban satellite imagery across the 500 largest metropolitan areas worldwide. It enables users to specify urban development goals, creating new images that align with them while offering diverse scenarios whose appearance can be controlled with text prompts and geospatial constraints. It also facilitates urban redevelopment practices by learning from the surrounding environment. Beyond visual synthesis, we find that it encodes and interprets latent representations of urban form for global cross-city learning, successfully transferring styles of urban environments across a global spatial network. The latent representations can also enhance downstream prediction tasks such as carbon emission prediction. Further, human expert evaluation confirms that our generated urban images are comparable to real urban images. Overall, this study presents innovative approaches for accelerated urban planning and supports scenario-based planning processes for worldwide cities.
Figures & tables
Figure 1: Spatial heterogeneity of population density and building volume density for global 500 cities. a. City-level global population density. b. Distribution of population density across 500 cities, sorted in ascending order. c. City-level global building volume density. d. Distribution of building volume density across 500 cities, sorted in ascending order. In a and c , the city-level densities are divided into five qualitative levels, where warm (orange and red) and cool colors (green and slight green) refer to high and low densities, respectively. In b and d , each dot corresponds to a city and is color-coded by continent (Europe, Asia, North America, South America, Oceania, and Africa). Representative cities, including Boston, Stockholm, Kigali, Singapore, and Hong Kong, are highlighted for reference.
Figure 2: Multi-modal conditional counterfactual synthesis of urban satellite imagery. a . Generated images respond to counterfactual template texts. The first row shows three generated images from the density metric prompt, and the second row shows three generated images from the land use prompt. b . Generated images respond to free-form texts. c . Urban redevelopment with the image inpaint model. d . Generative fidelity, precision and diversity performance of our framework.
Figure 3: Cross-city learning through generative urban visualization. a . Generated satellite images across cities from the source city (e.g., Hong Kong, Lusaka). b . Performance variation across regions. c . Human evaluation of our framework by urban experts. 42 urban experts (from the United States, Singapore, China, etc.; from both academia and industry) were recruited to rate (1-10 scores) for 136 questions. The generated urban images are comparable to those of real urban images.
Figure 4: Our framework quantifies urban visuals in latent space. a . Our framework has learned a distinguishable visual representation for each city. other city’s results can be found in SI. b . Cross-city transfer via prompt modification from “Stockholm” to “Stockholm from Kigali” yields hybrid satellite imagery, with latent features (black points) located between Kigali and Stockholm clusters. c . Performance comparison of predicted versus true global fossil fuel carbon emission for ResNet and DINOv3, using original data (top) and generative augmentation (bottom).
Extended Data Fig. 1 Modeling diagram of our framework.
Extended Data Fig. 2 Multi-modal (text-image) conditional counterfactual synthesis of urban satellite imagery in Kigali and Stockholm cities. a . Generated images respond to counterfactual texts in Kigali City. b . Generated images respond to counterfactual texts in Stockholm City. The first row shows three generated images from the density metric prompt, and the second row shows three generated images from the land use prompt.
Extended Data Fig. 3 Multi-modal (text-image) conditional counterfactual synthesis of urban satellite imagery in Chicago and Tampa cities. a . Generated images respond to counterfactual texts in Chicago City. b . Generated images respond to counterfactual texts in Tampa City. The first row shows three generated images from the density metric prompt, and the second row shows three generated images from the land use prompt.
Extended Data Fig. 4. Examples of reader study. a . An example question for urban experts to evaluate generated urban image in the urban density dimension. b . An example question for urban experts to evaluate generated urban redevelopment image.
Figure S1 : Relationship between Density Metrics
Figure S2 : Generated urban imagery responds to prompts and constraints. a . Generated images respond to different environmental constraints and text b . Generated imagery is diverse under the same control. RPD: residential population density; RVC: residential volume per capita; BVD: building volume density; BCR: building coverage ratio; Land use ratios: residential, commercial, industrial, etc.
Models
Steps
Params
Fidelity
Diversity
Precision
FID ↓
PSNR ↑
SSIM ↓
FSIM ↓
SSIM ↑
FSIM ↑
LPIPS ↓
MS-SSIM ↑
Baselines
Original SD (No finetuning)
0
1.4B
86.5
10.56±0.9
–
–
0.0927±0.008
0.5556±0.024
0.7162±0.045
0.0978±0.007
ControlNet
60k
1.4B
76.5
10.94±2.2
0.1847±0.13
0.5802±0.06
0.1087±0.07
0.5603±0.04
0.5623±0.038
0.1179±0.044
Few Steps
UCN+CLIP
20k
1.5B
72.5
10.58±2.3
0.1665±0.08
0.5560±0.04
0.1235±0.06
0.5545±0.035
0.5808±0.041
0.1120±0.048
Table S1: Quantitative assessment of image fidelity, diversity, and precision metrics
Experiment
BCR
BVD
MAE ↓
R2↑
MAE ↓
R2↑
UrbanControlNet
171.525
0.619
6.186
0.871
Table S2 : Prediction performance of a pre-trained CNN on Dtest and on synthetic SAIs generated using the models listed in the experiment column
Figure S3 : Text-constraint-conditional counterfactual synthesis of urban satellite imagery . a . Generated images respond to counterfactual density inputs. b . Generated images respond to free-form texts.
Figure S4 : Multi-source pseudo-Siamese network in UrbanControlNet.
Figure S5 : Numeral-aware network in UrbanControlNet.
Figure S6 : Model performance (R 2 ) distribution across more than 100 cities globally.
Model
R2(log)↑
R2↑
MAE ↓
RMSE ↓
ResNet
0.830
0.610
22.04
60.82
ResNet+TradAug
0.837
0.630
21.74
57.72
ResNet+AutoAug [ 8 ]
0.842
0.626
21.45
58.53
ResNet+SARandAug [ 36 ]
0.834
0.634
21.69
57.79
ResNet+GA
0.879
0.696
18.42
52.86
Swin Transformer
0.874
0.690
19.01
53.73
Table S3 : Performance comparison of different models for global fossil fuel carbon emission prediction.
3D urban generation from satellite imagery is an important task for scalable digital twins and real-world simulation environments. Existing approaches primarily rely on scene-level generation paradigms, which often require large-scale 3D city assets and struggle with controllability, geographic alignment, and realistic appearance grounding in real-world urban environments. To address these limitations, we present Sat2RealCity, a grounded urban generation framework that leverages object-level 3D generative priors for scalable city synthesis from satellite imagery. Our framework decomposes cities into geographically grounded building entities, enabling the reuse of pretrained object-level 3D generative priors while preserving real-world spatial structures. Supported by our constructed BuildVerse3D dataset, (1) we introduce an OpenStreetMap (OSM)-guided spatial grounding strategy to inject geospatial constraints into the 3D generation process; (2) we design an appearance-guided controllable generation mechanism for realistic architectural appearance and regional style consistency; and (3) we construct an MLLM-powered semantic pipeline for regional appearance understanding and semantic-aware appearance synthesis. Extensive experiments demonstrate that Sat2RealCity achieves strong geographic alignment, regional stylistic consistency, and plausible urban asset synthesis compared with existing urban generation and 3D asset generation approaches.
Street-view imagery (SVI) is widely used to quantify key indicators of urban environment, such as green- ery, sky, or road view indices. However, existing studies largely focus on measuring current streetscapes and rarely support the generation of alternative and non-existing urban scenarios, which is a core task in geospatial disciplines such as urban planning and design. To address this gap, we propose a gener- ative multimodal AI framework that synthesizes alternative streetscapes conditioned on targeted visual metrics, enabling direct visual exploration of urban scenarios. We first construct a multimodal dataset that aligns SVIs with textual descriptions, segmentation maps, road masks, and quantitative metrics of visual elements in Chicago and Orlando. Using this dataset, we demonstrate that diffusion models can produce realistic and semantically consistent streetscape imagery while responding to both textual and imagery controls. Our quantitative evaluations show that incorporating visual controls can improve semantic consistency, reducing the LPIPS index by approximately 6% while maintaining global visual realism. In addition, overall semantic consistency increases by 23.7% in Orlando and 46.4% in Chicago, as measured by the mIoU index, with class-wise gains exceeding even 100% improvement for building view indices. Streetscape generation can be controlled in a fine-grained manner by both visual and textual prompts, and when textual and visual controls conflict, imagery controls consistently dominate, indicating a clear control hierarchy and the importance of further developing visual controls for urban scene generation. Overall, this work establishes an important benchmark for streetscape generation us- ing SVIs and diffusion models, and illustrates how generative AI can serve as a practical, scalable, and controllable approach for urban scenario exploration.
Yuzhou Chen, Yuebing Liang, Lingqian Hu +4
Department of Urban and Regional Planning, University of Florida · Singapore-MIT Alliance for Research and Technology Centre (SMART) · Department of Landscape Architecture and Urban Planning, Texas A&M University +1
Urban Building Energy Modeling plays a critical role in achieving the United Nations' Sustainable Development Goals 7 and 11. Although existing studies based on satellite imagery and deep learning have achieved remarkable progress, many challenges exist: most existing studies are inherently predictive, failing to reflect the generative nature of urban planning; although generative AI and diffusion models have seen explosive growth in satellite imagery, they lack the urban functional generation (e.g., energy layer); third, aligned high-quality high-resolution building energy data with satellite imagery is limited and scarce. Here we propose SENSE (Satellite-based ENergy Synthesis for Sustainable Environment), a unified generative UBEM framework that jointly synthesizes realistic urban satellite imagery and aligned high-quality building energy consumption and height maps. By conditioning on road networks and urban density metrics, SENSE, based on a controllable diffusion model, leverages the knowledge learned by large vision models to generate urban building energy consumption and height information (annotations) in the latent space. Experiments across four cities (New York City, Boston, Lyon, Busan) demonstrate that SENSE achieves high visual fidelity and strong physical consistency, satisfying the ASHRAE standard metric. Experiments demonstrate that SENSE can generate enough annotated synthetic data using less than 20% labeled energy data, boosting downstream prediction performance by 10% IoU. Compared to SOTA urban energy prediction methods, SENSE significantly reduced prediction error (reduced 3%-11% NMBE and 1%-9% CVRMSE). This study offers an energy-efficiency urban planning and physical generation solution for urban science, energy science and building science. The dataset and code: https://huggingface.co/datasets/skl24/MUSE and https://github.com/kailaisun/GenAI4Urban-Energy/.
Kailai Sun, Mingyi He, Heye Huang +5
SMART, Singapore · Massachusetts Institute of Technology Cambridge, MA, USA · University of Florida Gainesville, Florida, USA