Organizations: Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.
Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.
Figures & tables
Figure 1: AFDBench station network: 13 NWS offices spanning 7 U.S. climate regions. Blue circles = training offices (393–888 samples each); red squares = held-out offices for geographic evaluation (Section 5.5 ). Marker labels show WFO code and sample count.
Model
Params
Met-Align
Style
Ground
Human NWS Expert
—
100.0
1.000
1.000
AFDBench (GRPO)
7B
13.73
0.619
0.940
AFDBench (SFT)
7B
14.05
0.318
0.881
Qwen2.5-7B-Inst.
7B
13.77 ∗
0.331 ∗
—
Mistral-7B-v0.3
7B
12.95 ∗
0.327 ∗
—
Hermes-3-Llama-3.1
8B
12.84 ∗
0.338 ∗
—
Table 1: AFDBench results on 1,033 held-out samples (BOX + MRX offices). GRPO nearly doubles Style-Align and improves Input-Grounding while Met-Align is bounded by single-timestep input. Zero-shot baselines evaluated on 200-sample subset.
Figure 3: GRPO training dynamics over 3,349 steps. (a) Per-component reward means: temperature ( Rtemp ) saturates at 1.0 immediately, synoptic ( Rsyn ) holds near 0.8, format ( Rfmt ) improves from 0.12 to 0.30. (b) Generation entropy remains stable near 0.85, indicating no mode collapse.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
SFT
GRPO
Base model
Qwen2.5-7B-Instruct
SFT checkpoint
LoRA rank ( r )
16
16
LoRA alpha ( α )
16
16
Learning rate
2×10−4
1×10−5
LR schedule
Cosine
Cosine
Batch size
1
1
Appendix
Table 2: Complete training configurations for SFT and GRPO phases.
Recent advances in visual-language models (VLMs) have led to significant improvements in a plethora of complex multimodal tasks like image captioning, report generation, and visual perception. However, generating text from meteorological data is highly challenging because the atmosphere is a chaotic system that is rapidly changing at various spatial and temporal scales. Given the complexity of atmospheric phenomena, it is critical to verifiably quantify the effectiveness of existing VLMs on weather forecasting data. In this work, we present SynopticBench, a high-quality dataset consisting of 1,367,041 text samples of Area Forecast Discussions created by the National Weather Service over the continental United States paired to images of 500mb geopotential height, 2 meter temperature, and 850mb wind velocity in weather forecasts. We also present Synoptic Phenomena Alignment and Coverage Evaluation (SPACE), a novel evaluation framework that can be used to effectively estimate the quality of text descriptions of synoptic weather phenomena. Extensive experiments on generating forecast discussions using state-of-the-art VLMs show the sensitivity of existing evaluation metrics in this domain and enable further exploration into synoptic weather and climate text generation.
Timothy B. Higgins, Antonios Mamalakis, Chirag Agarwal
Department of Environmental Sciences, University of Virginia, Charlottesville, 22903, Virginia, United States. · School of Data Science, University of Virginia, Charlottesville, 22903, Virginia, United States.
Accurate weather forecast reporting enables individuals and communities to better plan daily activities and agricultural operations. However, the current reporting process primarily relies on manual analysis of multi-source data, which leads to information overload and reduced efficiency. With the development of multimodal large language models (MLLMs), leveraging data-driven models to analyze and generate reports in the weather forecasting domain remains largely underexplored. In this work, we propose the Weather Forecasting Report (WFR) task and construct the first instruction-tuning dataset for this task, named~\DatasetNameL, which covers 31 cities in America and 8 weather aspects. Based on this corpus, we develop the first model, \ModelNameL, specialized in generating weather forecast reports. Evaluation across multiple metrics on our dataset shows that \ModelNameL~ consistently outperforms leading closed-source MLLMs, particularly on structurally complex weather aspects. We further analyze its performance across diverse geographic regions and weather aspects. \ModelNameL~ demonstrates strong transferability across different regions, highlighting its zero-shot generalization capability. \ModelNameL~offers valuable insight for developing MLLMs specialized in weather report generation. .
Zinan Zheng, Yang Liu, Nuo Chen +3
Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong · HY Founda- tion Model Team, Tencent +2
The development of practical (multimodal) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce K-MetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geo-cultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/datasets/soyeonbot/K-MetBench .