cs.CVMar 27, 2026
SaveAutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing
Organizations: The Hong Kong University of Science and Technology · Xiamen University · Meituan-M17, Hong Kong
Abstract
Generative video models have significantly advanced the photorealistic synthesis of adverse weather for autonomous driving; however, they consistently demand massive datasets to learn rare weather scenarios. While 3D-aware editing methods alleviate these data constraints by augmenting existing video footage, they are fundamentally bottlenecked by costly per-scene optimization and suffer from inherent geometric and illumination entanglement. In this work, we introduce AutoWeather4D, a feed-forward 3D-aware weather editing framework designed to explicitly decouple geometry and illumination. At the core of our approach is a G-buffer Dual-pass Editing mechanism. The Geometry Pass leverages explicit structural foundations to enable surface-anchored physical interactions, while the Light Pass analytically resolves light transport, accumulating the contributions of local illuminants into the global illumination to enable dynamic 3D local relighting. Extensive experiments demonstrate that AutoWeather4D achieves comparable photorealism and structural consistency to generative baselines while enabling fine-grained parametric physical control, serving as a practical data engine for autonomous driving.
Figures & tables
Figure 1: AutoWeather4D: Weather & Time-of-Day Control for Driving Videos . AutoWeather4D enables fine-grained control over weather (rain, snow, fog) and time-of-day (dawn, noon, night) in driving videos. The red zoom-in box showcases realistic snow accumulation, while the blue zoom-in box highlights rain-induced road wetness and ripples. See supplementary videos for dynamic visualizations.
| Method | Env light/shadow control | Extra light source | Weather change | Feed-forward | 4D dynamic scene | Tuning-free | Open-source |
| Cosmos-Transfer2.5 [ 1 ] | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| WAN-FUN 2.2 [ 56 ] | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| Ditto [ 2 ] | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| WeatherWeaver [ 28 ] | ✓ | ✓ | ✓ | ||||
| WeatherDiffusion [ 80 ] | ✓ | ✓ | |||||
| SceneCrafter [ 81 ] | ✓ | ✓ | ✓ |
Table 1: Comparison of state-of-the-art weather and time-of-day synthesis models for autonomous driving. We evaluate existing paradigms across key capabilities: Env light/shadow control (arbitrarily adjusting environment light directions and correcting shadows), Extra light source (precisely adding and controlling local lights within driving scenes), and Weather change (synthesizing appearances under diverse weather conditions). Additionally, we compare their architectural and practical properties, including whether they are Feed-forward (requiring no per-scene optimization), applicable to 4D dynamic scenes, Tuning-free (requiring no extra datasets for weight fine-tuning).
Figure 2: Overview of our framework. The pipeline formulates physically-grounded video editing for multi-weather and time-of-day synthesis. We first extract explicit G-buffers from the input video: metric depth via feed-forward 4D reconstruction, alongside intrinsic material properties (normal , metallic , albedo , roughness ) via an inverse renderer. The scene modifications are analytically resolved through the G-Buffer Dual-Pass Editing : (1) The Geometry Pass physically modulates to instantiate explicit weather mechanics (e.g., snow, rain, ground wetness); (2) The Light Pass executes parametric illumination control, independently synthesizing detected local light sources and global environmental lighting (e.g., dawn, noon, blue hours) to reflect atmospheric and temporal shifts. Finally, the deterministic rendered sequence is processed by the VidRefiner . This terminal refiner synthesizes real-world sensor nuances while preserving the classical shading cues and explicit scene dynamics resolved in the dual-pass stages.
| S.A. | Night | Fog | Rain | Snow | |
| Time (s) | 128.1 | 167.1 | 170.9 | 2.2 | 67.6 |
Table 2: Running time (seconds) per video for different tasks and weather conditions on a NVIDIA V100 GPU. S.A.: Semantic Annotation, which is computed once and shared by all four weather.
Figure 3: Qualitative Comparisons of AutoWeather4D on Waymo Weather/ Time-of-day Conversions: Validating Physically Plausible and Fine-Grained Control for Autonomous Driving.
Figure 4: Qualitative Comparisons with domain-specific architectures: Validating spatially anchoring weather effects. DR: DiffusionRenderer [ 27 ] .
| Model | CLIP Score (↑) | Vehicle 3D Detection IoU (↑) | Vehicle CLIP cosine similarity (↑) | Human Evaluation (↑) |
|---|---|---|---|---|
| Video-P2P | 0.2448 | - | - | 0 |
| Ditto | 0.2532 | 0.805 | 0.769 | 0.425 |
| Cosmos-Transfer2.5 | 0.2558 | 0.913 | 0.837 | 0.580 |
| WAN-FUN 2.2 | 0.2577 | 0.888 | 0.794 | 0.668 |
| Ours | 0.2586 | 0.915 | 0.871 | 0.826 |
Table 3: Quantitative evaluation of weather and time-of-day conversions on the Waymo dataset . All results are derived from 120 source videos, each converted into four common adverse conditions (rain, snow, fog, night) with 57 frames per video. This results in a total of 27,360 frames evaluated. “-" indicates omitted due to frame cropping. All metrics are averaged across all frames, where red indicates the best performance.
Figure 5: Ablation of 4D reconstruction. (a) Integer-quantized depth priors [ 27 ] induce severe spatial discretization and aliasing during local relighting. (b) The deployed feed-forward 4D reconstruction establishes a continuous floating-point manifold, enforcing smooth, artifact-free illumination gradients.
| Setting | ACDC mIoU(↑) | ACDC mAcc(↑) | DarkZurich mIoU(↑) | DarkZurich mAcc(↑) |
|---|---|---|---|---|
| w/o augmentation | 49.20 | 60.72 | 23.92 | 38.29 |
| w/ cosmos | 49.66 (+0.93%) | 62.31 (+2.62%) | 23.93 (+0.04%) | 39.52 (+3.21%) |
| w/ ours | 49.81 (+1.24%) | 62.52 (+2.96%) | 24.09 (+0.71%) | 39.73 (+3.76%) |
Table 4: AutoWeather4D for data augmentation in adverse weather semantic segmentation.
Figure 6: Global Illumination Editing driven by HDR environment map .
Figure 7: Explicit Parameterized Control for Physically Consistent Weather and Time-of-Day Video Editing .
| Model | Depth si-RMSE |
|---|---|
| Video-P2P [ 29 ] | 0.511 |
| Ditto [ 2 ] | 0.632 |
| Cosmos-Transfer2.5 [ 1 ] | 0.225 |
| WAN-FUN 2.2 [ 56 ] | 0.267 |
| Ours | 0.247 |
Table 5: Depth alignment evaluation. Depth si-RMSE ( ) averaged over 480 generated videos.
| Model | Edge F1 |
|---|---|
| Video-P2P [ 29 ] | 0.104 |
| Ditto [ 2 ] | 0.043 |
| Cosmos-Transfer2.5 [ 1 ] | 0.321 |
| WAN-FUN 2.2 [ 56 ] | 0.187 |
| Ours | 0.129 |
Table 6: Edge alignment evaluation. Edge F1 scores (↑) for different methods averaged over 480 generated videos.
| Model | FVD Score | DOVER Score |
|---|---|---|
| Video-P2P [ 29 ] | 1687.4 | 0.194 |
| Ditto [ 2 ] | 2024.2 | 0.345 |
| Cosmos-Transfer2.5 [ 1 ] | 808.7 | 0.358 |
| WAN-FUN 2.2 [ 56 ] | 1029.5 | 0.448 |
| Ours | 886.8 | 0.370 |
Table 7: Perceptual metric evaluation. Temporal consistency (FVD) and visual quality (DOVER) averaged over 480 generated videos. Higher DOVER scores indicate better visual quality, while lower FVD scores indicate better temporal consistency.
| ID | Core Verification Point | Experimental Configuration Design |
|---|---|---|
| ① | 4D reconstruction can effectively prevent aliasing in local light rendering | Experiment A: Compare the effect with and without the 4D reconstruction module in Fig. 7 . |
| ② | Inverse rendering + Env light Editing can effectively manipulate shadows according to HDR changes | Experiment B: Compare shadow manipulation results with and without the combination of Inverse rendering + Env light Editing in Fig. 8 . |
| ③ | Geometry pass editing can effectively handle snow/rain weather editing based on prompts | Experiment C: Compare rain editing results with/without Geometry pass editing, and snow editing results with/without Geometry pass editing in Fig. 9 . |
| ④ | Light pass editing can effectively handle fog/night weather editing | Experiment D: Compare fog editing results with/without Light pass editing, and night editing results with/without Light pass editing in Fig. 10 . |
| ⑤ | VidRefiner can effectively remove noise in G-Buffer | Experiment E: Compare noisy G-Buffer processing results with and without VidRefiner in Fig. 11 . |
Table 8: Ablation Study on Single Module Effectiveness
Figure 7: Experiment A: Compare the effect with and without the 4D reconstruction module.
Figure 8: Experiment B: Compare shadow manipulation results with and without the combination of Inverse rendering + Env light Editing.
Figure 9: Experiment C: Compare rain editing results with/without Geometry pass editing, and snow editing results with/without Geometry pass editing.
Figure 10: Experiment D: Compare fog editing results with/without Light pass editing, and night editing results with/without Light pass editing.
Figure 11: Experiment E: Compare noisy G-Buffer processing results with and without VidRefiner.
| ID Pair | Core Verification Point | Experimental Configuration Design |
|---|---|---|
| ① + ⑤ | Whether VidRefiner enhances fidelity of anti-aliased local light rendering | Experiment F: Compare local light rendering results of “4D Recon alone” vs. “4D Recon + VidRefiner” in Fig. 12 . |
| ② + ⑤ | Whether VidRefiner preserves shadow manipulation accuracy while enhancing fidelity | Experiment G: Compare shadow editing results of “Inverse Renderer+Env Light Editing alone” vs. “the combination + VidRefiner” in Fig. 13 . |
| ③ + ⑤ | Whether VidRefiner maintains weather effect realism and improves frame fidelity | Experiment H: Compare snow/rain editing results of “Geometry Pass Editing alone” vs. “Geometry Pass Editing + VidRefiner” in Fig. 14 . |
| ④ + ⑤ | Whether VidRefiner enhances fidelity of fog/night lighting edits | Experiment I: Compare fog/night editing results of “Light Pass Editing alone” vs. “Light Pass Editing + VidRefiner” in Fig. 15 . |
Table 9: Ablation Study on Module Synergies (VidRefiner + Functional Modules)
Figure 12: Experiment F: Compare 4D Reconstruction editing results with/without VidRefiner
Figure 13: Experiment G: Compare Inverse Render + Light Pass editing results with/without VidRefiner.
Figure 14: Experiment H: Compare rain Geometry Pass editing results with/without VidRefiner, and snow Geometry Pass editing results with/without VidRefiner.
Figure 15: Experiment I: Compare fog Light Pass editing results with/without VidRefiner, and night Light Pass editing results with/without VidRefiner.
| ID Combination | Core Verification Point | Experimental Configuration |
|---|---|---|
| ①+②+⑤ | Anti-aliasing + shadow manipulation + fidelity | Not feasible: Anti-aliasing (4D reconstruction) acts primarily on light rendering, with no direct relevance to shadow manipulation. |
| ①+③+⑤ | Anti-aliasing + snow/rain editing + fidelity | Not feasible: Anti-aliasing (4D reconstruction) targets light rendering artifacts, which are irrelevant to snow/rain geometry edits. |
| ①+④+⑤ | Anti-aliasing + fog/night editing + fidelity | Redundant (Duplicate of Experiment F): Exact same setup as Experiment F (Tab. 9 ); results are reused to avoid redundant testing. |
| ②+③+⑤ | Shadow manipulation + snow/rain editing + fidelity | Not feasible: Snow/rain scenarios typically involve overcast conditions, where shadows are indistinct or absent, making shadow manipulation irrelevant. |
| ②+④+⑤ | Shadow manipulation + fog/night editing + fidelity | Not feasible: Night/fog conditions obscure direct light sources (e.g., sunlight), resulting in minimal or no distinct shadows to manipulate. |
| ③+④+⑤ | Snow/rain + fog/night editing + fidelity | Experiment J: Compare results of “Geometry Pass Editing + Light Pass Editing alone” vs. “the combination + VidRefiner” in Fig. 16 . |
Table 10: Ablation Study on Module Synergies (VidRefiner + 2-Module Subsets)
Figure 16: Experiment J: Compare Geometry Pass Editing + Light Pass editing results with/without VidRefiner
| ID Combination | Core Verification Point | Experimental Configuration |
|---|---|---|
| ①+②+③+⑤ | Anti-aliasing + shadow manipulation + snow/rain editing + fidelity | Not feasible: Anti-aliasing (via 4D reconstruction) targets light rendering artifacts and is irrelevant to shadow manipulation or snow/rain geometry edits. |
| ①+②+④+⑤ | Anti-aliasing + shadow manipulation + fog/night editing + fidelity | Not feasible: Night/fog conditions obscure direct light sources (e.g., sunlight), resulting in minimal distinct shadows, making shadow manipulation irrelevant. |
| ①+③+④+⑤ | Anti-aliasing + snow/rain + fog/night editing + fidelity | Redundant (Duplicate of Experiment J): Exact same setup as Experiment J (Tab. 10 ); results are reused to avoid redundant testing. |
| ②+③+④+⑤ | Shadow manipulation + snow/rain + fog/night editing + fidelity | Not feasible: Night/fog conditions eliminate distinct shadows, rendering the shadow manipulation module irrelevant to combined snow/rain and fog/night weather edits. |
Table 11: Ablation Study on Module Synergies (VidRefiner + 3-Module Subsets)
Figure 17: Comparing the individual contributions of each component in the rain-related Geometry Pass Editing.
Figure 18: Comparing the individual contributions of each component in the snow-related Geometry Pass Editing.
Figure 19: Comparing the individual illuminant contributions of Light Pass Editing.
Figure 20: Ablation of VidRefiner strength across different values.
Figure 21: Case study on error tolerance under extreme conditions. We visualize a challenging extreme-low-light scenario to illustrate the error-compensation mechanism of our decoupled architecture. Due to high sensor ISO in the raw input (a), the feed-forward extraction yields severely flawed intrinsic components, including a catastrophic depth failure in the sky (b) and high-frequency noise in the normal/roughness maps (c, e). Naive explicit forward rendering inherently propagates these errors, leading to structural collapse (f). AutoWeather4D mitigates this through compensations: First, explicit boundary priors (e.g., sky-masking) anchor the macroscopic geometry (g); however, the intended physical weather edits on the road are shattered into high-frequency artifacts by the underlying normal noise. Second, rather than naive blurring, the generative VidRefiner treats this noisy render as a conditioning signal (h). It leverages diffusion priors to semantically absorb the intrinsic artifacts while harmonizing the fragmented weather elements into photorealistic wet-surface reflections.
Figure 22: Qualitative comparison results. As highlighted by the red boxes, baseline models erroneously inherit hard shadows from the source video, whereas our method successfully avoids retaining these spurious shadow artifacts.
Figure 23: Qualitative comparison results. AutoWeather4D selectively synthesizes explicit headlight cones for moving vehicles, reliably maintaining the unlit appearance of the stationary parked cars. In contrast, existing baselines typically rely on global tone shifts or unstructured darkening, making it challenging to achieve such state-aware local lighting.
Figure 24: Qualitative comparison results. As highlighted by the red boxes, existing baselines erroneously inherit hard directional shadows from the source video, whereas our method successfully avoids this artifact and injects plausible local headlight illumination. Furthermore, the blue boxes demonstrate that Gaussian-Splatting-based methods (e.g., WeatherEdit) struggle to reconstruct dynamic moving objects from monocular videos, resulting in severe motion ghosting. In contrast, our approach preserves the structural integrity of dynamic vehicles.
Figure 25: Qualitative comparison results. As highlighted by the red boxes, several baselines erroneously inherit hard directional shadows from the source video.
Figure 26: Qualitative comparison demonstrating the advantages of decoupled G-buffers over 4D Gaussian Splatting (WeatherEdit). (Red boxes) AutoWeather4D synthesizes explicit surface interactions (e.g., geometry-anchored puddles with dynamic ripples). In contrast, WeatherEdit struggles to apply fine-grained, geometry-aware physical modifications to the road surface. (Blue boxes) Our feed-forward G-buffer formulation maintains the structural integrity of fast-moving vehicles, effectively mitigating the motion ghosting observed in the optimization-based 4DGS baseline.
Figure 27: Limitation in handling light-emitting objects. To maintain 3D reconstruction stability, our G-buffer design deliberately omits a specific channel for self-illuminating objects. Consequently, active light sources like traffic lights (red boxes) are modeled as high-albedo reflective surfaces. As a result, they experience unintended darkening when the global sunlight is reduced during sunny-to-rainy translation.