MeteoVerse: Unified Weather-Controllable Video World Model

Renlong Wu; Guanqiao Wang; Xuan Shang; Yin Hanming; Xiaoxiao Sheng; Tianyu Huang; Hui Li; Wangmeng Zuo

cs.CVSep 29, 2026

MeteoVerse: Unified Weather-Controllable Video World Model

Authors: Renlong Wu, Guanqiao Wang, Xuan Shang, Yin Hanming, Xiaoxiao Sheng, Tianyu Huang, Hui Li, Wangmeng Zuo

Organizations: Harbin Institute of Technology · Huawei

Abstract

Video world models aim to predict future content from an observed scene while following prescribed camera motion. Real-world scene evolution is determined not only by changes in viewpoint and object dynamics, but also by environmental conditions such as weather, which can substantially alter scene appearance and visibility. Modeling such realistic weather evolution is challenging because the required weather modification depends jointly on the observed and desired weather states. Depending on their relation, the model may need to preserve, introduce, or remove a weather effect. Existing video world models typically leave this weather transition implicit, forcing the generation backbone to infer weather evolution together with scene dynamics and camera motion, which leads to imprecise weather control. To address this limitation, we propose MeteoVerse, a unified weather-controllable video world model that generates future videos from a single sunny or adverse-weather image, conditioned on a weather-free scene description, a target-weather instruction, and a camera trajectory. Rather than conditioning only on the desired weather, MeteoVerse explicitly estimates the observed and target weather states and represents the required weather transition. A transition-aware mixture of weather experts then translates this transition into category-specific residual weather features, unifying weather preservation, introduction, and removal while enabling fine-grained control over introduced weather intensity. We further construct the MeteoVerse dataset with over 50K real-world weather video clips, generated sunny counterparts, disentangled scene and weather descriptions, weather-intensity annotations, and camera trajectories. Extensive experiments demonstrate substantially improved weather controllability while retaining competitive scene consistency and camera-control performance.

Figures & tables

Figure 1: Overview of MeteoVerse, which supports weather preservation, introduction, and removal from sunny or adverse-weather inputs under prescribed camera trajectories.

Figure 2: Overview of MeteoVerse. The weather-state predictor estimates the observed and target weather states and derives the required weather transition. MeteoMoE modulates the rain, snow, and fog experts according to the corresponding transition components, fuses their responses through self-attention, and residually injects the resulting transition-aware features into the DiT backbone.

Figure 3: Construction of MeteoVerse dataset. We collect over 50K 81-frame real-world rain, snow, and fog clips with disentangled scene and weather descriptions, weather-intensity annotations, camera trajectories, and pseudo-paired sunny counterparts. The sunny and adverse-weather observations are then recombined to provide supervision for weather preservation, introduction, and removal.

Method	VBench-I2V Evaluation							Camera Evaluation		Weather Evaluation
Method	Overall Score $\uparrow$	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
Weather Preservation
Uni3C	86.53	95.80	94.15	99.21	30.50	50.87	63.86	0.755	0.102	88.00	74.99	76.80
GEN3C	86.22	96.89	94.34	99.50	23.50	50.11	60.46	0.331	0.073	85.50	72.30	74.90
VerseCrafter	85.01	95.20	92.68	99.17	23.00	50.86	66.21	5.926	0.128	90.75	77.02	78.60
NeoVerse	85.73	92.83	92.15	99.15	38.00	48.76	61.93	0.678	0.054	85.75	63.38	68.70

Table 1: Quantitative comparison across Weather Preservation , Weather Introduction , and Weather Removal . Overall Score is reported only for Weather Preservation, as intentional weather changes in Weather Introduction and Weather Removal make the aggregate input-consistency score less appropriate. The best and second-best results are highlighted in bold and underlined , respectively.

Figure 4: Visual comparisons on weather introduction and removal. More results are in the Suppl .

Methods	VBench-I2V Evaluation						Camera Evaluation		Weather Evaluation
Methods	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
Target-weather Prompt $p_{\mathrm{w}}$	92.60	93.14	98.87	48.50	51.88	61.30	4.084	0.115	80.85	63.98	73.63
Target State $\mathbf{s}_{\mathrm{tar}}$	93.96	93.29	98.78	58.33	51.92	61.76	4.372	0.189	66.03	58.73	64.07
Transition State $\mathbf{s}_{\mathrm{trans}}$ (Ours)	95.24	94.11	98.92	46.17	50.99	64.07	3.366	0.121	81.00	69.38	76.67

Table 2: Ablation on weather-control representations. All metrics are equally averaged over Weather Preservation, Weather Introduction, and Weather Removal.

Figure 5: Visual results of fine-grained weather control. More results are in the Suppl .

Methods	VBench-I2V Evaluation						Camera Evaluation		Weather Evaluation
Methods	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
Global Transition Conditioning	93.53	92.98	98.76	64.17	51.90	67.51	4.291	0.095	58.33	54.75	58.42
Shared Weather Expert	93.60	93.09	98.80	62.33	51.69	66.30	4.370	0.114	65.00	58.96	63.74
w/o Backbone LoRA	94.31	93.56	98.90	45.50	51.60	66.83	4.396	0.141	52.02	51.55	53.82
Full MeteoMoE (Ours)	95.24	94.11	98.92	46.17	50.99	64.07	3.366	0.121	81.00	69.38	76.67

Table 3: Ablation on MeteoMoE and backbone adaptation. All metrics are equally averaged over Weather Preservation, Weather Introduction, and Weather Removal.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Methods	VBench-I2V Evaluation						Camera Evaluation		Weather Evaluation
Methods	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	Weather Alignment $\uparrow$	VLM Evaluation $\uparrow$	User Study $\uparrow$
GT States	94.35	93.65	98.87	52.33	50.68	63.26	3.695	0.120	82.00	70.18	77.56
Predicted States (Ours)	95.24	94.11	98.92	46.17	50.99	64.07	3.366	0.121	81.00	69.38	76.67

Appendix

Table 4: Effect of weather-state prediction on downstream generation. All metrics are equally averaged over Weather Preservation, Weather Introduction, and Weather Removal.

Explore similar work

Holo-World: Unified Camera, Object and Weather Control for Video World Model
Jun 18, 2026Xiangchen Yin, Wenzhang Sun, Jiahui Yuan +6Video World Models Camera-Conditioned Video Generation
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · Li Auto.
Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis
Jun 27, 2026Chenghao Qian, Nedko Savov, Lingdong Kong +7Indoor Scene Generation Synthesis
University of Leeds, UK · INSAIT, Sofia University “St. Kliment Ohridski” · National University of Singapore, Singapore +2
Wonder: Video World Model Done Better
Jul 28, 2026Jiacong Xu, Hanwen Jiang, Zhixin Shu +3Video World Models
Adobe Research · Johns Hopkins University

Jun 18, 2026, cs.CVJ/K move · Enter open · S save

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Xiangchen Yin, Wenzhang Sun, Jiahui Yuan, Zijie Liu+5

University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · Li Auto.

Video World Models Camera-Conditioned Video Generation Camera Weather Prediction

Open paper AskPDF arXiv

Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls the scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object controls with consistent scene structure while transferring scenes into diverse target weather states, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at https://xiangchenyin.github.io/Holo-World/

Abstract

Explore similar work

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis

Wonder: Video World Model Done Better