MeteoVerse: Unified Weather-Controllable Video World Model

Renlong Wu; Guanqiao Wang; Xuan Shang; Yin Hanming; Xiaoxiao Sheng; Tianyu Huang; Hui Li; Wangmeng Zuo

cs.CVSep 29, 2026

MeteoVerse: Unified Weather-Controllable Video World Model

Authors: Renlong Wu, Guanqiao Wang, Xuan Shang, Yin Hanming, Xiaoxiao Sheng, Tianyu Huang, Hui Li, Wangmeng Zuo

Organizations: Harbin Institute of Technology · Huawei

Abstract

Video world models aim to predict future content from an observed scene while following prescribed camera motion. Real-world scene evolution is determined not only by changes in viewpoint and object dynamics, but also by environmental conditions such as weather, which can substantially alter scene appearance and visibility. Modeling such realistic weather evolution is challenging because the required weather modification depends jointly on the observed and desired weather states. Depending on their relation, the model may need to preserve, introduce, or remove a weather effect. Existing video world models typically leave this weather transition implicit, forcing the generation backbone to infer weather evolution together with scene dynamics and camera motion, which leads to imprecise weather control. To address this limitation, we propose MeteoVerse, a unified weather-controllable video world model that generates future videos from a single sunny or adverse-weather image, conditioned on a weather-free scene description, a target-weather instruction, and a camera trajectory. Rather than conditioning only on the desired weather, MeteoVerse explicitly estimates the observed and target weather states and represents the required weather transition. A transition-aware mixture of weather experts then translates this transition into category-specific residual weather features, unifying weather preservation, introduction, and removal while enabling fine-grained control over introduced weather intensity. We further construct the MeteoVerse dataset with over 50K real-world weather video clips, generated sunny counterparts, disentangled scene and weather descriptions, weather-intensity annotations, and camera trajectories. Extensive experiments demonstrate substantially improved weather controllability while retaining competitive scene consistency and camera-control performance.

Figures & tables

Figure 1: Overview of MeteoVerse, which supports weather preservation, introduction, and removal from sunny or adverse-weather inputs under prescribed camera trajectories.

Figure 2: Overview of MeteoVerse. The weather-state predictor estimates the observed and target weather states and derives the required weather transition. MeteoMoE modulates the rain, snow, and fog experts according to the corresponding transition components, fuses their responses through self-attention, and residually injects the resulting transition-aware features into the DiT backbone.

Figure 3: Construction of MeteoVerse dataset. We collect over 50K 81-frame real-world rain, snow, and fog clips with disentangled scene and weather descriptions, weather-intensity annotations, camera trajectories, and pseudo-paired sunny counterparts. The sunny and adverse-weather observations are then recombined to provide supervision for weather preservation, introduction, and removal.

Method	VBench-I2V Evaluation							Camera Evaluation		Weather Evaluation
Method	Overall Score $\uparrow$	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
Weather Preservation
Uni3C	86.53	95.80	94.15	99.21	30.50	50.87	63.86	0.755	0.102	88.00	74.99	76.80
GEN3C	86.22	96.89	94.34	99.50	23.50	50.11	60.46	0.331	0.073	85.50	72.30	74.90
VerseCrafter	85.01	95.20	92.68	99.17	23.00	50.86	66.21	5.926	0.128	90.75	77.02	78.60
NeoVerse	85.73	92.83	92.15	99.15	38.00	48.76	61.93	0.678	0.054	85.75	63.38	68.70

Table 1: Quantitative comparison across Weather Preservation , Weather Introduction , and Weather Removal . Overall Score is reported only for Weather Preservation, as intentional weather changes in Weather Introduction and Weather Removal make the aggregate input-consistency score less appropriate. The best and second-best results are highlighted in bold and underlined , respectively.

Figure 4: Visual comparisons on weather introduction and removal. More results are in the Suppl .

Methods	VBench-I2V Evaluation						Camera Evaluation		Weather Evaluation
Methods	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
Target-weather Prompt $p_{\mathrm{w}}$	92.60	93.14	98.87	48.50	51.88	61.30	4.084	0.115	80.85	63.98	73.63
Target State $\mathbf{s}_{\mathrm{tar}}$	93.96	93.29	98.78	58.33	51.92	61.76	4.372	0.189	66.03	58.73	64.07
Transition State $\mathbf{s}_{\mathrm{trans}}$ (Ours)	95.24	94.11	98.92	46.17	50.99	64.07	3.366	0.121	81.00	69.38	76.67

Table 2: Ablation on weather-control representations. All metrics are equally averaged over Weather Preservation, Weather Introduction, and Weather Removal.

Figure 5: Visual results of fine-grained weather control. More results are in the Suppl .

Methods	VBench-I2V Evaluation						Camera Evaluation		Weather Evaluation
Methods	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
Global Transition Conditioning	93.53	92.98	98.76	64.17	51.90	67.51	4.291	0.095	58.33	54.75	58.42
Shared Weather Expert	93.60	93.09	98.80	62.33	51.69	66.30	4.370	0.114	65.00	58.96	63.74
w/o Backbone LoRA	94.31	93.56	98.90	45.50	51.60	66.83	4.396	0.141	52.02	51.55	53.82
Full MeteoMoE (Ours)	95.24	94.11	98.92	46.17	50.99	64.07	3.366	0.121	81.00	69.38	76.67

Table 3: Ablation on MeteoMoE and backbone adaptation. All metrics are equally averaged over Weather Preservation, Weather Introduction, and Weather Removal.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Methods	VBench-I2V Evaluation						Camera Evaluation		Weather Evaluation
Methods	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
GT States	94.35	93.65	98.87	52.33	50.68	63.26	3.695	0.120	82.00	70.18	77.56
Predicted States (Ours)	95.24	94.11	98.92	46.17	50.99	64.07	3.366	0.121	81.00	69.38	76.67

Appendix

Table 4: Effect of weather-state prediction on downstream generation. All metrics are equally averaged over Weather Preservation, Weather Introduction, and Weather Removal.

Explore similar work

Jun 18, 2026cs.CV

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls the scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object controls with consistent scene structure while transferring scenes into diverse target weather states, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at https://xiangchenyin.github.io/Holo-World/

Xiangchen Yin, Wenzhang Sun, Jiahui Yuan +6

University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · Li Auto.

Video World Models Camera-Conditioned Video Generation Camera Weather Prediction

Ask PDF

Jun 27, 2026cs.CV

Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis

Weather synthesis aims to add weather effects to input videos while preserving scene identity, structure, and motion. The key limitation of existing methods is the lack of diversity in weather appearance and effective control over weather dynamics (e.g., temporal evolution and particle motion). Most approaches rely on text prompts, which are inherently underspecified and often fail to produce detailed weather characteristics. Additionally, general-purpose video editors optimized for clean and aesthetic outputs tend to suppress heavy weather phenomena, making dense particle effects difficult to generate. To address these, we propose a Semantic-Aware, Physics-Informed, and Geometry-Grounded framework that steers an off-the-shelf video editor to synthesize diverse global appearances and detailed particle dynamics. We factorize the synthesis into three conditional signals, so that each provides a distinct and stable source of guidance: semantics specifies what the weather should look like, dynamics governs how it evolves over time, and geometry determines where it should appear in the scene. Specifically, we introduce (1) semantic-aware appearance anchoring to establish the target appearance from scene semantics and user input; (2) physics-informed dynamic simulation to generate particle effects by simulating a Gaussian-represented particle field under gravity, wind, and turbulence; and (3) geometry-grounded video synthesis to align the simulated particles with target scene geometry and synthesize the final video. Experiments demonstrate that our method produces diverse, physically and visually realistic weather effects. Furthermore, we show that our synthesized data significantly improves the robustness of autonomous driving semantic segmentation under adverse weather conditions. Project page: https://jumponthemoon.github.io/w-crafter/.

Chenghao Qian, Nedko Savov, Lingdong Kong +7

University of Leeds, UK · INSAIT, Sofia University “St. Kliment Ohridski” · National University of Singapore, Singapore +2

Indoor Scene Generation Synthesis Autonomous Driving Motion

Ask PDF

Jul 28, 2026cs.CV

Wonder: Video World Model Done Better

We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration. Given an image or a conditional video, Wonder constructs a playable world where users can navigate interactively by moving the camera, discovering unseen regions, and revisiting previously observed areas in real time and over a long-term horizon. Achieving this capability requires a system-level co-design of control method, memory mechanism, and training strategy. We introduce a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues, allowing the model to interpret camera motion directly as visual evidence. To support fast and precise memory retrieval over a growing generation context, we propose an efficient sparse attention-based memory mechanism, enabling the model to selectively attend to a small set of relevant context tokens at inference time, regardless of actual context length. We further develop several techniques to rectify the self-forcing-style distillation pipeline, improving the student model's ability to respect control signals, as well as maintaining diverse generation modes and long-term memory from the teacher. Together, these components enable Wonder to synthesize diverse, minute-scale videos at 16 FPS while preserving coherent geometry, appearance, and dynamics across long rollouts. Beyond image-to-video generation, Wonder naturally supports video-conditioned generation, allowing existing dynamic scenes to be re-shot in real time.

Jiacong Xu, Hanwen Jiang, Zhixin Shu +3

Adobe Research · Johns Hopkins University

Video World Models

Ask PDF