AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecasting

Jing Wu; Yang Liu; Lin Zhang; Junbo Zeng; Jiabin Wang; Zi Ye; Guowen Li; Shilei Cao

cs.AIMar 16, 2026

AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecasting

Authors: Jing Wu, Yang Liu, Lin Zhang, Junbo Zeng, Jiabin Wang, Zi Ye, Guowen Li, Shilei Cao, +8 more

Organizations: Sun Yat-sen University, Zhuhai, China · The Chinese University of Hong Kong, Hong Kong, China · Jiangxi Science and Technology Normal University, Nanchang, China · China Meteorological Administration, Beijing, China · Huawei Technologies Co., Ltd, China · Guangdong-Hong-Kong-Macao Greater Bay Area Weather Research Center for Monitoring Warning and Forecasting, China · National Supercomputing Center in Shenzhen, Shenzhen, China · Tsinghua University, Shenzhen, China

Abstract

Accurate weather forecasting is more than grid-wise regression: it must preserve coherent synoptic structures and physical consistency of meteorological fields, especially under autoregressive rollouts where small one-step errors can amplify into structural bias. Existing physics-priors approaches typically impose global, once-for-all constraints via architectures, regularization, or NWP coupling, offering limited state-adaptive and sample-specific controllability at deployment. To bridge this gap, we propose Agent-Guided Cross-modal Decoding (AGCD), a plug-and-play decoding-time prior-injection paradigm that derives state-conditioned physics-priors from the current multivariate atmosphere and injects them into forecasters in a controllable and reusable way. Specifically, We design a multi-agent meteorological narration pipeline to generate state-conditioned physics-priors, utilizing MLLMs to extract various meteorological elements effectively. To effectively apply the priors, AGCD further introduce cross-modal region interaction decoding that performs region-aware multi-scale tokenization and efficient physics-priors injection to refine visual features without changing the backbone interface. Experiments on WeatherBench demonstrate consistent gains for 6-hour forecasting across two resolutions (5.625 degree and 1.40625 degree) and diverse backbones (generic and weather-specialized), including strictly causal 48-hour autoregressive rollouts that reduce early-stage error accumulation and improve long-horizon stability.

Figures & tables

Figure 1: Global static physics-priors vs. State-conditioned physics-priors: proposed AGCD injects cached state-conditioned physics-priors at decoding time.

Figure 2: The overview of the proposed AGCD.

Figure 3: Structure of Cross-Modal Interaction .

Method	T2m [K]				10m Wind [m/s]				Z500 [m 2 /s 2 ]				T850 [K]
	5.625 ∘		1.40625 ∘		5.625 ∘		1.40625 ∘		5.625 ∘		1.40625 ∘		5.625 ∘		1.40625 ∘
	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$
ViT	1.6859	0.9554	1.3570	0.9710	0.5490	0.9723	0.5946	0.9668	131.01	0.9929	80.80	0.9970	1.21	0.9736	0.91	0.9852
ViT+AGCD	1.2601	0.9768	1.2450	0.9754	0.4781	0.9788	0.5600	0.9695	113.07	0.9951	75.90	0.9976	0.98	0.9830	0.86	0.9871
CaiT	1.8747	0.9317	1.5200	0.9658	0.6051	0.9674	0.6420	0.9622	149.95	0.9917	104.60	0.9953	1.51	0.9516	1.06	0.9829
CaiT+AGCD	1.8703	0.9450	1.4700	0.9682	0.5993	0.9678	0.6210	0.9638	132.80	0.9938	96.80	0.9960	1.41	0.9635	0.99	0.9846

Table 1: 6-hour forecasting results on WeatherBench at two resolutions. AGCD consistently improves RMSE and ACC across backbones. RMSE

\downarrow

and ACC

\uparrow

Figure 4: Qualitative comparison of 6 hour weather forecasting with Pangu and Pangu+AGCD (AGCD) on 1.40625 ∘ data across multiple variables. (a) Initial fields at time $t$ . (b) Ground-truth targets at $t{+}6$ h. (c) Predictions from the vanilla Pangu. (d) Error maps from the vanilla Pangu. (e) Predictions from Pangu with our AGCD. (f) Error maps from Pangu with our AGCD. Error maps visualize Pred $-$ GT.

Figure 5: Autoregressive rollout comparison between Pangu and Pangu+AGCD up to 48 hours (6 hour steps).

Text setting	Z500		T850		T2m		10m Wind
Text setting	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$
Vision-only (no text)	131.01	0.9929	1.21	0.9736	1.6859	0.9554	0.5490	0.9723
Matched (ours)	113.07	0.9951	0.98	0.9830	1.2601	0.9768	0.4781	0.9788
Shuffled (mismatch)	136.40	0.9922	1.24	0.9730	1.7120	0.9550	0.5650	0.9718
Empty (null prompt)	134.80	0.9924	1.23	0.9732	1.7050	0.9552	0.5600	0.9720

Table 2: Semantic relevance controls. All settings keep the visual backbone (ViT) and CRID identical; only the text input is modified.

Figure 6: Relevance case study: State-consistent priors yield targeted error reductions.

Text generator	Z500		T850		T2m		10m Wind
Text generator	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$
Single-agent ( $A_{I}$ only)	123.80	0.9937	1.05	0.9802	1.3900	0.9688	0.5070	0.9756
Multi-agent w/o evaluator ( $\{A_{V_{i}}\}{+}A_{I}$ )	118.20	0.9944	1.01	0.9819	1.3200	0.9742	0.4920	0.9773
Full MMNP (ours) ( $\{A_{V_{i}}\}{+}A_{I}{+}E$ )	113.07	0.9951	0.98	0.9830	1.2601	0.9768	0.4781	0.9788

Table 3: Ablation on MMNP generation strategies (same CRID and backbone (ViT)).

Figure 7: Ablation on variable-specific description agents in MMNP (ViT backbone; $A_{I}$ and $E$ fixed). We report latitude-weighted RMSE (bars; lower is better) and ACC (line; higher is better) on Z500, T850, T2m, and 10m wind.

Enabled $A_{\cdot}$	Z500		T850		T2m		10m wind
	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$
No MMNP	131.01	0.9929	1.21	0.9736	1.6859	0.9554	0.5490	0.9723
$A_{\text{t2m}}$	130.62	0.9929	1.16	0.9752	1.5438	0.9620	0.5352	0.9735
$A_{\text{t2m,10m}}$	130.12	0.9930	1.13	0.9768	1.4820	0.9650	0.5249	0.9745
$A_{\text{t2m,10m,t850}}$	129.66	0.9930	1.10	0.9782	1.4187	0.9678	0.5146	0.9755
$A_{\text{t2m,10m,t850,z500}}$	113.07	0.9951	0.98	0.9830	1.2601	0.9768	0.4781	0.9788

Table 4: Incremental ablation on enabling variable-specific description agents in MMNP (ViT backbone;

A_{I}

and

E

fixed). RMSE

\downarrow

/ ACC

\uparrow

for 6 hour forecasts.

Setting	Z500		T850		T2m		10m Wind
Setting	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$	RMSE $\downarrow$	ACC $\uparrow$
Vision-only (no CRID)	131.01	0.9929	1.21	0.9736	1.6859	0.9554	0.5490	0.9723
+ Text, w/o Region tokens	124.70	0.9936	1.09	0.9786	1.4200	0.9681	0.5150	0.9750
+ Text, w/o HopfieldPool	115.60	0.9948	1.00	0.9824	1.3000	0.9758	0.4860	0.9780
+ Text, w/o CMG gating	118.90	0.9943	1.02	0.9818	1.3400	0.9748	0.4930	0.9775
Full CRID (ours)	113.07	0.9951	0.98	0.9830	1.2601	0.9768	0.4781	0.9788

Table 5: Ablation on CRID components.

Explore similar work

AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions
Aug 25, 2026Manmeet Singh, Somnath Luitel, Prabhjot Singh +3Forecasting Benchmarks Language Model Generation Evaluation
Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.
Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecasting
Sep 29, 2026Joonhyeong Park, Giung Nam, Hyungi Lee +3Probabilistic Forecasting Proper Scoring Rules
KAIST · Kookmin University · New York University
Timestep-Conditioned Transformers for Global Weather Forecasting
Aug 6, 2026Sam Levang, Fran Bartolic, Ty Dickinson +3Time Series Forecasting Probabilistic Forecasting
Salient

Aug 25, 2026J/K move · Enter open · S save

AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions

Manmeet Singh, Somnath Luitel, Prabhjot Singh, Manraaj Banga+2

Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.

Forecasting Benchmarks Language Model Generation Evaluation Weather Forecasting Scientific Reasoning in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.

Abstract

Explore similar work

AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions

Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecasting

Timestep-Conditioned Transformers for Global Weather Forecasting