WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation

Wenhua Wu; Huai Guan; Zhe Liu; Hesheng Wang

cs.CVFeb 25, 2026

WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation

Authors: Wenhua Wu, Huai Guan, Zhe Liu, Hesheng Wang

Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China

Abstract

Editable high-fidelity 4D scenes are crucial for autonomous driving, as they can be applied to end-to-end training and closed-loop simulation. However, existing reconstruction methods are primarily limited to replicating observed scenes and lack the capability for diverse weather simulation. While image-level weather editing methods tend to introduce scene artifacts and offer poor controllability over the weather effects. To address these limitations, we propose WeatherCity, a novel framework for 4D urban scene reconstruction and weather editing. Specifically, we leverage a text-guided image editing model to achieve flexible editing of image weather backgrounds. To tackle the challenge of multi-weather modeling, we introduce a novel weather Gaussian representation based on shared scene features and dedicated weather-specific decoders. This representation is further enhanced with a content consistency optimization, ensuring coherent modeling across different weather conditions. Additionally, we design a physics-driven model that simulates dynamic weather effects through particles and motion patterns. Extensive experiments on multiple datasets and various scenes demonstrate that WeatherCity achieves flexible controllability, high fidelity, and temporal consistency in 4D reconstruction and weather editing. Our framework not only enables fine-grained control over weather conditions (e.g., light rain and heavy snow) but also supports object-level manipulation within the scene. Codes are released at https://github.com/IRMVLab/WeatherCity.

Figures & tables

Figure 2: Overview of WeatherCity. Our framework comprises four main modules. First, the image editing module employs a text-guided video editing foundation model to adapt image weather background. Second, the scene representation module introduces a weather-aware Gaussian representation based on shared features and multi-weather decoders, which disentangles geometric-textural attributes from weather-specific appearances, thereby ensuring structural consistency across varying meteorological conditions. Subsequently, we construct RGB and content losses for consistency optimization. Finally, a physics-driven dynamic weather simulation mechanism is designed to achieve flexible and controllable editing of diverse dynamic weather effects.

Figure 3: Qualitative results of multi-weather editing on the Waymo Open Dataset. Our method produces realistic and consistent weather editing effects while supporting dynamic weather simulation, whereas all baselines exhibit significant scene distortion.

Figure 4: Qualitative results of multi-weather editing on the nuScenes dataset. Our method produces realistic and consistent weather editing effects while supporting dynamic weather simulation, whereas all baselines exhibit significant scene distortion.

Method	Waymo Open Dataset			nuScenes Dataset
Method	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$
ControlNet [ 47 ]	0.634	0.238	0.695	0.656	0.228	0.811
TurboEdit [ 11 ]	0.830	0.220	0.801	0.782	0.250	0.829
FRESCO [ 45 ]	0.720	0.213	0.824	0.710	0.224	0.855
Qwen-Image [ 39 ]	0.785	0.279	0.843	0.804	0.279	0.902
WeatherCity (Ours)	0.872	0.303	0.915	0.870	0.302	0.968

Table 1: Quantitative comparison on Waymo Open Dataset and nuScenes Dataset.

\uparrow

means higher is better. WeatherCity achieves state-of-the-art performance across all metrics, outperforming other methods by a large margin.

Figure 5: Visualization of object editing results. The text prompt is ”Remove all vehicles except the red and white ones in the center and change the weather to snowy”.

Method	Speed (FPS) $\uparrow$
ControlNet [ 47 ]	0.033
TurboEdit [ 11 ]	0.097
FRESCO [ 45 ]	0.142
WeatherCity (Ours)	25.67

Table 2: Comparison of runtime measured in FPS.

Figure 6: Visualization of ablation study.

Method	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$
a. Baseline	0.735	0.276	0.891
b. w/o WGS	0.781	0.212	0.894
c. w/o $\mathcal{L}_{cc}$	0.817	0.289	0.916
WeatherCity	0.880	0.320	0.943

Table 3: Ablation study results. The results validate the effectiveness of each design.

Figure 7: Qualitative comparison on dynamic weather effect.

Weather	Prompt
Rainy	Please strictly maintain the original composition, all scene contents (including ground, buildings, vegetation, cars, background, pedestrians, etc.), their positions, and the original artistic style. Convert the scene to a rainy setting. Requirements: The image should be clear and realistic; the sky must be overcast with dark clouds; the ground should be wet with puddles and reflections; the lighting should be dim, and the overall tone should be cool to create a rainy atmosphere. Do NOT include: sunny weather, blue skies and white clouds, sunlight, dry ground, any elements not present in the original image, trees that are not in the original, distorted visuals, deformed subjects, or incorrect proportions.
Snowy	Please strictly maintain the original composition, all scene contents (including ground, buildings, vegetation, cars, background, pedestrians, etc.), their positions, and the original artistic style. Convert the scene to a snowy setting. Requirements: The image should be clear and realistic; the sky should be overcast with falling snowflakes; the ground should be naturally covered with snow; do not add any extra vegetation; the lighting should be soft, and the overall tone should be cool. If the original image contains green leaves, please turn them into snow-covered bare branches. Do NOT include: sunny weather, warm tones, sunlight, melting snow, elements unrelated to the original image, elements not present in the original image, distorted visuals, deformed subjects, or incorrect proportions.
Foggy	Please strictly maintain the original composition, all scene contents (including ground, buildings, vegetation, cars, background, pedestrians, etc.), their positions, and the original artistic style. Convert the scene to a foggy setting. Requirements: The image should be clear while maintaining realistic atmospheric fog; the sky should appear overcast; the scene should be filled with natural, soft fog that reduces visibility in the distance; lighting should be diffused and soft, with an overall cool tone. Do NOT include: sunny weather, warm tones, sunlight, dry and clear air, elements unrelated to the original image, elements not present in the original image, deformed subjects, or incorrect proportions.
Snowy & vehicle removal	Please remove all vehicles in the image except for the white and red ones, and then transform the scene into snowy weather. Requirements: The image should be clear and realistic; the sky should be overcast with falling snowflakes; the ground should be naturally covered with snow; do not add any extra vegetation; the lighting should be soft, and the overall tone should be cool. If the original image contains green leaves, please turn them into snow-covered bare branches. Do NOT include: sunny weather, warm tones, sunlight, melting snow, elements unrelated to the original image, distorted visuals, deformed subjects, or incorrect proportions.

Table 4: Prompts used for Qwen-Image and all baseline methods under each weather condition.

Figure 8: Qualitative comparison of rainy weather on the Waymo Open Dataset. Our model excels at capturing complex lighting interactions and environmental changes induced by rain, providing a higher level of physical realism.

Figure 9: Qualitative comparison of snowy weather on the Waymo Open Dataset. As shown, WeatherCity generates highly realistic snow accumulation on vehicles and roads while avoiding the texture distortions common in other approaches.

Figure 10: Qualitative comparison of foggy weather on the Waymo Open Dataset. WeatherCity demonstrates superior depth-consistent haze rendering, effectively preserving the semantic layout of the original scene compared to baseline methods.

Figure 11: Qualitative comparison of rainy weather on the nuScenes Dataset. Our method accurately simulates realistic wet surface reflections and rain streaks, significantly outperforming ControlNet and FRESCO in terms of visual fidelity.

Figure 12: Qualitative comparison of snowy weather on the nuScenes Dataset. Our approach produces the most visually plausible winter scenes with natural white-out effects, surpassing the consistency and quality of competing editing frameworks.

Figure 13: Qualitative comparison of foggy weather on the nuScenes Dataset. WeatherCity achieves a natural atmospheric haze that smoothly obscures distant objects, exhibiting fewer artifacts than TurboEdit or other baselines.

Figure 14: Qualitative comparison of rainy weather on temporal consistency. The baselines suffer from scene deformation and erratic fluctuations in weather effects, exhibiting noticeable inter-frame flickering. In contrast, our approach enables temporally consistent weather editing.

Figure 15: Qualitative comparison of snowy weather on temporal consistency. The baselines suffer from scene deformation and erratic fluctuations in weather effects, exhibiting noticeable inter-frame flickering. In contrast, our approach enables temporally consistent weather editing.

Figure 16: Qualitative comparison of foggy weather on temporal consistency. The baselines suffer from scene deformation and erratic fluctuations in weather effects, exhibiting noticeable inter-frame flickering. In contrast, our approach enables temporally consistent weather editing.

Method	Rainy			Snowy			Foggy
Method	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$	CLIP-DS $\uparrow$
ControlNet [ 47 ]	0.654	0.228	0.713	0.615	0.261	0.677	0.225
TurboEdit [ 11 ]	0.843	0.233	0.825	0.816	0.228	0.787	0.221
FRESCO [ 45 ]	0.721	0.209	0.852	0.719	0.253	0.797	0.177
Qwen-Image [ 39 ]	0.813	0.248	0.845	0.757	0.310	0.840	0.251
WeatherCity (Ours)	0.898	0.300	0.931	0.847	0.330	0.899	0.278

Table 5: Comparison on Waymo Open Dataset.

\uparrow

means higher is better.

Method	Rainy			Snowy			Foggy
Method	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$	CLIP-DS $\uparrow$
ControlNet [ 47 ]	0.703	0.250	0.810	0.609	0.234	0.812	0.201
TurboEdit [ 11 ]	0.806	0.225	0.848	0.758	0.261	0.811	0.266
FRESCO [ 45 ]	0.726	0.220	0.863	0.694	0.239	0.847	0.213
Qwen-Image [ 39 ]	0.823	0.256	0.891	0.785	0.302	0.913	0.264
WeatherCity (Ours)	0.880	0.272	0.977	0.860	0.333	0.960	0.301

Table 6: Comparison on nuScenes Dataset.

\uparrow

means higher is better.

Figure 17: Qualitative comparison with ClimateNeRF on Waymo Open Dataset. ClimateNeRF is limited to static modeling and static weather effect editing, while WeatherCity produces more photorealistic results and additionally supports dynamic weather particles, such as falling snowflakes.

Method	Snowy			Foggy	FPS $\uparrow$
Method	CLIP-S $\uparrow$	CLIP-DS $\uparrow$	Sem-CS $\uparrow$	CLIP-DS $\uparrow$	FPS $\uparrow$
ClimateNeRF [ 22 ]	0.807	0.294	0.905	0.269	0.032
WeatherCity (Ours)	0.847	0.341	0.941	0.280	25.67

Table 7: Comparison on Waymo Open Dataset scene 788.

\uparrow

means higher is better.

Explore similar work

AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing
Mar 27, 2026Tianyu Liu, Weitao Xiong, Kunming Luo +4Video Editing Image Editing
The Hong Kong University of Science and Technology · Xiamen University · Meituan-M17, Hong Kong
Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis
Jun 27, 2026Chenghao Qian, Nedko Savov, Lingdong Kong +7Synthetic Data Augmentation Physical Consistency in Video Generation
University of Leeds, UK · INSAIT, Sofia University “St. Kliment Ohridski” · National University of Singapore, Singapore +2
RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models
Aug 3, 2026Yuwei Ning, Liangzhi Wang, Yi Xiao +5Synthetic-to-Real Domain Adaptation World Models for Autonomous Driving
1Sun Yat-sen University · 2Shenzhen Loop Area Institute

Mar 27, 2026J/K move · Enter open · S save

AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing

Tianyu Liu, Weitao Xiong, Kunming Luo, Manyuan Zhang+3

The Hong Kong University of Science and Technology · Xiamen University · Meituan-M17, Hong Kong

Video Editing Image Editing Physical Consistency in Video Generation Autonomous Driving Scenario Generation

Open paper AskPDF arXiv

Generative video models have significantly advanced the photorealistic synthesis of adverse weather for autonomous driving; however, they consistently demand massive datasets to learn rare weather scenarios. While 3D-aware editing methods alleviate these data constraints by augmenting existing video footage, they are fundamentally bottlenecked by costly per-scene optimization and suffer from inherent geometric and illumination entanglement. In this work, we introduce AutoWeather4D, a feed-forward 3D-aware weather editing framework designed to explicitly decouple geometry and illumination. At the core of our approach is a G-buffer Dual-pass Editing mechanism. The Geometry Pass leverages explicit structural foundations to enable surface-anchored physical interactions, while the Light Pass analytically resolves light transport, accumulating the contributions of local illuminants into the global illumination to enable dynamic 3D local relighting. Extensive experiments demonstrate that AutoWeather4D achieves comparable photorealism and structural consistency to generative baselines while enabling fine-grained parametric physical control, serving as a practical data engine for autonomous driving.

Abstract

Explore similar work

AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing

Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis

RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models