AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecasting
Authors: Jing Wu, Yang Liu, Lin Zhang, Junbo Zeng, Jiabin Wang, Zi Ye, Guowen Li, Shilei Cao, +8 more
Organizations: Sun Yat-sen University, Zhuhai, China · The Chinese University of Hong Kong, Hong Kong, China · Jiangxi Science and Technology Normal University, Nanchang, China · China Meteorological Administration, Beijing, China · Huawei Technologies Co., Ltd, China · Guangdong-Hong-Kong-Macao Greater Bay Area Weather Research Center for Monitoring Warning and Forecasting, China · National Supercomputing Center in Shenzhen, Shenzhen, China · Tsinghua University, Shenzhen, China
Accurate weather forecasting is more than grid-wise regression: it must preserve coherent synoptic structures and physical consistency of meteorological fields, especially under autoregressive rollouts where small one-step errors can amplify into structural bias. Existing physics-priors approaches typically impose global, once-for-all constraints via architectures, regularization, or NWP coupling, offering limited state-adaptive and sample-specific controllability at deployment. To bridge this gap, we propose Agent-Guided Cross-modal Decoding (AGCD), a plug-and-play decoding-time prior-injection paradigm that derives state-conditioned physics-priors from the current multivariate atmosphere and injects them into forecasters in a controllable and reusable way. Specifically, We design a multi-agent meteorological narration pipeline to generate state-conditioned physics-priors, utilizing MLLMs to extract various meteorological elements effectively. To effectively apply the priors, AGCD further introduce cross-modal region interaction decoding that performs region-aware multi-scale tokenization and efficient physics-priors injection to refine visual features without changing the backbone interface. Experiments on WeatherBench demonstrate consistent gains for 6-hour forecasting across two resolutions (5.625 degree and 1.40625 degree) and diverse backbones (generic and weather-specialized), including strictly causal 48-hour autoregressive rollouts that reduce early-stage error accumulation and improve long-horizon stability.
Figures & tables
Figure 1: Global static physics-priors vs. State-conditioned physics-priors: proposed AGCD injects cached state-conditioned physics-priors at decoding time.
Figure 2: The overview of the proposed AGCD.
Figure 3: Structure of Cross-Modal Interaction .
Method
T2m [K]
10m Wind [m/s]
Z500 [m 2 /s 2 ]
T850 [K]
5.625 ∘
1.40625 ∘
5.625 ∘
1.40625 ∘
5.625 ∘
1.40625 ∘
5.625 ∘
1.40625 ∘
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
ViT
1.6859
0.9554
1.3570
0.9710
0.5490
0.9723
0.5946
0.9668
131.01
0.9929
80.80
0.9970
1.21
0.9736
0.91
0.9852
ViT+AGCD
1.2601
0.9768
1.2450
0.9754
0.4781
0.9788
0.5600
0.9695
113.07
0.9951
75.90
0.9976
0.98
0.9830
0.86
0.9871
CaiT
1.8747
0.9317
1.5200
0.9658
0.6051
0.9674
0.6420
0.9622
149.95
0.9917
104.60
0.9953
1.51
0.9516
1.06
0.9829
CaiT+AGCD
1.8703
0.9450
1.4700
0.9682
0.5993
0.9678
0.6210
0.9638
132.80
0.9938
96.80
0.9960
1.41
0.9635
0.99
0.9846
Table 1: 6-hour forecasting results on WeatherBench at two resolutions. AGCD consistently improves RMSE and ACC across backbones. RMSE ↓ and ACC ↑ .
Figure 4: Qualitative comparison of 6 hour weather forecasting with Pangu and Pangu+AGCD (AGCD) on 1.40625 ∘ data across multiple variables. (a) Initial fields at time t . (b) Ground-truth targets at t+6 h. (c) Predictions from the vanilla Pangu. (d) Error maps from the vanilla Pangu. (e) Predictions from Pangu with our AGCD. (f) Error maps from Pangu with our AGCD. Error maps visualize Pred − GT.
Figure 5: Autoregressive rollout comparison between Pangu and Pangu+AGCD up to 48 hours (6 hour steps).
Text setting
Z500
T850
T2m
10m Wind
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
Vision-only (no text)
131.01
0.9929
1.21
0.9736
1.6859
0.9554
0.5490
0.9723
Matched (ours)
113.07
0.9951
0.98
0.9830
1.2601
0.9768
0.4781
0.9788
Shuffled (mismatch)
136.40
0.9922
1.24
0.9730
1.7120
0.9550
0.5650
0.9718
Empty (null prompt)
134.80
0.9924
1.23
0.9732
1.7050
0.9552
0.5600
0.9720
Table 2: Semantic relevance controls. All settings keep the visual backbone (ViT) and CRID identical; only the text input is modified.
Table 3: Ablation on MMNP generation strategies (same CRID and backbone (ViT)).
Figure 7: Ablation on variable-specific description agents in MMNP (ViT backbone; AI and E fixed). We report latitude-weighted RMSE (bars; lower is better) and ACC (line; higher is better) on Z500, T850, T2m, and 10m wind.
Enabled A⋅
Z500
T850
T2m
10m wind
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
RMSE ↓
ACC ↑
No MMNP
131.01
0.9929
1.21
0.9736
1.6859
0.9554
0.5490
0.9723
At2m
130.62
0.9929
1.16
0.9752
1.5438
0.9620
0.5352
0.9735
At2m,10m
130.12
0.9930
1.13
0.9768
1.4820
0.9650
0.5249
0.9745
At2m,10m,t850
129.66
0.9930
1.10
0.9782
1.4187
0.9678
0.5146
0.9755
At2m,10m,t850,z500
113.07
0.9951
0.98
0.9830
1.2601
0.9768
0.4781
0.9788
Table 4: Incremental ablation on enabling variable-specific description agents in MMNP (ViT backbone; AI and E fixed). RMSE ↓ / ACC ↑ for 6 hour forecasts.
Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.
Manmeet Singh, Somnath Luitel, Prabhjot Singh +3
Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.
Recent probabilistic weather forecasters train stochastic predictors with the continuous ranked probability score (CRPS) to generate each ensemble member in a single forward pass. These models learn the predictive distribution from the forecast context alone, which becomes difficult at longer forecast horizons where uncertainty is high. To learn the predictive distribution more effectively, we introduce auxiliary conditional denoising tasks that predict the same future state from the context and its corrupted version, which provides partial future information that can reduce prediction ambiguity. Building on distributional diffusion models, we learn the conditional distributions of these tasks with a single stochastic predictor by minimizing a proper scoring rule across noise levels. At inference, the predictor can still generate each ensemble member in a single forward pass at the fully corrupted endpoint. Standard CRPS training is recovered as the endpoint-only special case of our formulation, so our framework extends existing CRPS-based forecasters with only additional conditioning inputs. Controlled experiments show that the auxiliary tasks improve one-step forecasting across architectures, with larger gains at longer forecast horizons. The gains extend to high-dimensional global weather forecasting under both training from scratch and fine-tuning, along with improved calibration and potential benefits for generalization under distribution shift.
Existing machine-learning weather forecasting models rely on predetermined and fixed autoregressive timesteps. The choice of model timestep involves a fundamental trade-off: shorter timesteps (e.g. 1 to 6 hours) finely resolve atmospheric dynamics within the diurnal cycle but increase error accumulation for a given forecast horizon, while longer timesteps (e.g. 24 hours) reduce error accumulation but limit the usability of short-range forecasts where sub-daily predictability is high. In this work, we present GEM-3, a probabilistic global weather model that addresses this trade-off through explicit multi-timestep inference. With a single set of trained weights, the model timestep can be configured at inference time to balance predictability and usability across a broad forecast horizon. Additionally, we find that mixed-timestep training consistently improves rollout stability relative to timestep-specialist models. Under the hood, GEM-3 is a lightweight neighborhood-attention transformer with ~134M parameters on an equirectangular grid with a number of architectural advancements beyond its predecessor GEM-2. The result is a practical forecasting system that couples near-SOTA medium-range probabilistic skill, stable extended-range rollouts, efficient training and inference, and decision-relevant diagnostics.