SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future

Timothy B. Higgins; Antonios Mamalakis; Chirag Agarwal

cs.CLApr 7, 2026

SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future

Authors: Timothy B. Higgins, Antonios Mamalakis, Chirag Agarwal

Organizations: Department of Environmental Sciences, University of Virginia, Charlottesville, 22903, Virginia, United States. · School of Data Science, University of Virginia, Charlottesville, 22903, Virginia, United States.

Abstract

Recent advances in visual-language models (VLMs) have led to significant improvements in a plethora of complex multimodal tasks like image captioning, report generation, and visual perception. However, generating text from meteorological data is highly challenging because the atmosphere is a chaotic system that is rapidly changing at various spatial and temporal scales. Given the complexity of atmospheric phenomena, it is critical to verifiably quantify the effectiveness of existing VLMs on weather forecasting data. In this work, we present SynopticBench, a high-quality dataset consisting of 1,367,041 text samples of Area Forecast Discussions created by the National Weather Service over the continental United States paired to images of 500mb geopotential height, 2 meter temperature, and 850mb wind velocity in weather forecasts. We also present Synoptic Phenomena Alignment and Coverage Evaluation (SPACE), a novel evaluation framework that can be used to effectively estimate the quality of text descriptions of synoptic weather phenomena. Extensive experiments on generating forecast discussions using state-of-the-art VLMs show the sensitivity of existing evaluation metrics in this domain and enable further exploration into synoptic weather and climate text generation.

Figures & tables

Dataset	Meteorological Data	Text Size	Spatial Scale	Target
GridRad-Severe murphy_development_2023	RS	x	SC, MC, MS	Extreme Events
SEVIR veillette_sevir_2020	RS	x	MS	Extreme Events
WeatherQA ma_weatherqa_2024	Reanalysis	8,000	MS	Extreme Events
ClimateIQA chen_climateiqa_2025	Reanalysis	762,120	MS, Synoptic	Extreme Events
CLLMate li_cllmate_2024	Reanalysis	41,000	MS	Extreme Events
RadarQA he_radarqa_2025	RS	69,000	Mesoscale	All Weather

Table 1: Comparison between SynopticBench and existing work using multimodal datasets for atmospheric data. Here, we present the type of meteorological data (RS

\to

Remote Sensing; NWP

\to

Numerical Weather Prediction), the number of text samples, the relevant spatial scales (SC

\to

Single-cell; MC

\to

Multi-cell; MS

\to

Mesoscale), and the target of each study. SynopticBench is the first large-scale benchmark that generates text from NWP and contains 1,367,041 text samples

Figure 1: An example case of a single sample from the training set (top panel). Each training sample image has a yellow box indicating the location of the discussion. The example answer is a filtered AFD. All of the locations used in the discussions are shown in the bottom panel. The format of the training samples is also shown, with 117 AFDs matched to each forecast

Figure 2: Several examples of matching large- (green), medium- (purple), and small-scale (orange) location keywords. Blue lines indicate the potential matches that these locations would make if found in the predicted or reference text

Model	Bertscore	ROUGE-L	METEOR	F1	Gemini-2.5-Flash
LLaVA-v1.5-7B-base	.7143 $\mathbin{\scriptstyle\pm.0001}$	.1091 $\mathbin{\scriptstyle\pm.0001}$	.1327 $\mathbin{\scriptstyle\pm.0001}$	.1471 $\mathbin{\scriptstyle\pm.0002}$	.1565 $\mathbin{\scriptstyle\pm.0031}$
LLaVA-v1.5-13B-base	.7192 $\mathbin{\scriptstyle\pm.0001}$	.1086 $\mathbin{\scriptstyle\pm.0001}$	.1376 $\mathbin{\scriptstyle\pm.0001}$	.1498 $\mathbin{\scriptstyle\pm.0002}$	.1896 $\mathbin{\scriptstyle\pm.0032}$
LLaMA-3.2-11B-Vision-base	.7051 $\mathbin{\scriptstyle\pm.0001}$	.1043 $\mathbin{\scriptstyle\pm.0002}$	.1212 $\mathbin{\scriptstyle\pm.0002}$	.1313 $\mathbin{\scriptstyle\pm.0002}$	.2623 $\mathbin{\scriptstyle\pm.0041}$
Qwen2.5-VL-7B-base	.7136 $\mathbin{\scriptstyle\pm.0001}$	.1083 $\mathbin{\scriptstyle\pm.0001}$	.1447 $\mathbin{\scriptstyle\pm.0001}$	.1591 $\mathbin{\scriptstyle\pm.0003}$	.2566 $\mathbin{\scriptstyle\pm.0035}$
LLaVA-v1.5-7B-LoRA	.7814 $\mathbin{\scriptstyle\pm.0002}$	.1629 $\mathbin{\scriptstyle\pm.0003}$	.1353 $\mathbin{\scriptstyle\pm.0003}$	.1943 $\mathbin{\scriptstyle\pm.0004}$	.2426 $\mathbin{\scriptstyle\pm.0048}$
LLaVA-v1.5-13B-LoRA	.7858 $\mathbin{\scriptstyle\pm.0002}$	.1663 $\mathbin{\scriptstyle\pm.0003}$	.1407 $\mathbin{\scriptstyle\pm.0003}$	.2010 $\mathbin{\scriptstyle\pm.0004}$	.2472 $\mathbin{\scriptstyle\pm.0052}$

Table 2: Traditional metrics for base models, finetuned models, and baselines are shown. We calculate the mean and standard error of the mean Bertscore, ROUGE-L, METEOR, F1, and LLM-judge scores across samples within the test set. The range of possible scores for all metrics is 0 to 1, with 1 being a perfect score

Model	$s_{s}^{\text{loc}}$	$s_{m}^{\text{loc}}$	$r_{c}^{\text{loc}}$	$s_{s}^{\text{agg}}$	$s_{m}^{\text{agg}}$	$r_{c}^{\text{agg}}$
LLaVA-v1.5-7B-base	.0976 $\mathbin{\scriptstyle\pm.0008}$	.4592 $\mathbin{\scriptstyle\pm.0056}$	.2158 $\mathbin{\scriptstyle\pm.0013}$	.4177 $\mathbin{\scriptstyle\pm.0008}$	.5039 $\mathbin{\scriptstyle\pm.0056}$	.9467 $\mathbin{\scriptstyle\pm.0013}$
LLaVA-v1.5-13B-base	.1553 $\mathbin{\scriptstyle\pm.0007}$	.4404 $\mathbin{\scriptstyle\pm.0061}$	.3615 $\mathbin{\scriptstyle\pm.0012}$	.4698 $\mathbin{\scriptstyle\pm.0007}$	.5044 $\mathbin{\scriptstyle\pm.0061}$	.9317 $\mathbin{\scriptstyle\pm.0012}$
LLaMA-3.2-11B-Vision-base	.1194 $\mathbin{\scriptstyle\pm.0008}$	.6296 $\mathbin{\scriptstyle\pm.00013}$	.1936 $\mathbin{\scriptstyle\pm.0012}$	.6492 $\mathbin{\scriptstyle\pm.0008}$	.6690 $\mathbin{\scriptstyle\pm.0013}$	.9648 $\mathbin{\scriptstyle\pm.0012}$
Qwen2.5-VL-7B-base	.0985 $\mathbin{\scriptstyle\pm.0018}$	.5072 $\mathbin{\scriptstyle\pm.0032}$	.1950 $\mathbin{\scriptstyle\pm.0031}$	.5831 $\mathbin{\scriptstyle\pm.0018}$	.6221 $\mathbin{\scriptstyle\pm.0032}$	.9279 $\mathbin{\scriptstyle\pm.0031}$
LLaVA-v1.5-7B-LoRA	.2235 $\mathbin{\scriptstyle\pm.0015}$	.6575 $\mathbin{\scriptstyle\pm.0023}$	.3364 $\mathbin{\scriptstyle\pm.0018}$	.6486 $\mathbin{\scriptstyle\pm.0015}$	.6714 $\mathbin{\scriptstyle\pm.0023}$	.9635 $\mathbin{\scriptstyle\pm.0018}$
LLaVA-v1.5-13B-LoRA	.2604 $\mathbin{\scriptstyle\pm.0015}$	.6601 $\mathbin{\scriptstyle\pm.0021}$	.3900 $\mathbin{\scriptstyle\pm.0018}$	.6410 $\mathbin{\scriptstyle\pm.0015}$	.6629 $\mathbin{\scriptstyle\pm.0021}$	.9648 $\mathbin{\scriptstyle\pm.0018}$

Table 3: SPACE was used for synoptic pressure system evaluation for base models, finetuned models, and baselines. We calculate the mean and standard error of the SPACE-local and SPACE-aggregate scores (

s_{s}

), match scores (

s_{m}

), and coverage ratios (

r_{c}

) within the test set. The range of possible scores for all metrics is 0 to 1, with 1 being a perfect score

Figure 3: Two cases demonstrating differences between Space scores and traditional skill metrics. The reference text samples are filtered NWS AFDs and the prediction text samples are generated from the finetuned version of LLaVA-v1.5-7B. The terms in bold are used to compute Space scores for pressure systems. Sentences that are irrelevant to the Space scores are shown in red

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$s_{s}^{\text{loc}}$	$s_{m}^{\text{loc}}$	$r_{c}^{\text{loc}}$	$s_{s}^{\text{agg}}$	$s_{m}^{\text{agg}}$	$r_{c}^{\text{agg}}$
LLaVA-v1.5-7B-base	.0173 $\mathbin{\scriptstyle\pm.0028}$	.8278 $\mathbin{\scriptstyle\pm.0526}$	.0207 $\mathbin{\scriptstyle\pm.0031}$	.3447 $\mathbin{\scriptstyle\pm.0028}$	.5602 $\mathbin{\scriptstyle\pm.0526}$	.6017 $\mathbin{\scriptstyle\pm.0031}$
LLaVA-v1.5-13B-base	.0793 $\mathbin{\scriptstyle\pm.0048}$	.9068 $\mathbin{\scriptstyle\pm.0125}$	.0862 $\mathbin{\scriptstyle\pm.0051}$	.4229 $\mathbin{\scriptstyle\pm.0008}$	.6053 $\mathbin{\scriptstyle\pm.0056}$	.6843 $\mathbin{\scriptstyle\pm.0013}$
LLaMA-3.2-11B-Vision-base	.0671 $\mathbin{\scriptstyle\pm.0025}$	.7294 $\mathbin{\scriptstyle\pm.0075}$	.0886 $\mathbin{\scriptstyle\pm.0030}$	.3300 $\mathbin{\scriptstyle\pm.0025}$	.5683 $\mathbin{\scriptstyle\pm.0075}$	.5709 $\mathbin{\scriptstyle\pm.0030}$
Qwen2.5-VL-7B-base	.0491 $\mathbin{\scriptstyle\pm.0032}$	.8330 $\mathbin{\scriptstyle\pm.0155}$	.0577 $\mathbin{\scriptstyle\pm.0035}$	.3302 $\mathbin{\scriptstyle\pm.0032}$	.5478 $\mathbin{\scriptstyle\pm.0155}$	.5878 $\mathbin{\scriptstyle\pm.0035}$
LLaVA-v1.5-7B-LoRA	.1863 $\mathbin{\scriptstyle\pm.0056}$	.8839 $\mathbin{\scriptstyle\pm.0075}$	.2077 $\mathbin{\scriptstyle\pm.0060}$	.4679 $\mathbin{\scriptstyle\pm.0056}$	.5965 $\mathbin{\scriptstyle\pm.0075}$	.7718 $\mathbin{\scriptstyle\pm.0060}$
LLaVA-v1.5-13B-LoRA	.1862 $\mathbin{\scriptstyle\pm.0063}$	.8697 $\mathbin{\scriptstyle\pm.0085}$	.2098 $\mathbin{\scriptstyle\pm.0067}$	.4209 $\mathbin{\scriptstyle\pm.0063}$	.5870 $\mathbin{\scriptstyle\pm.0085}$	.7043 $\mathbin{\scriptstyle\pm.0067}$

Appendix

Table 4: SPACE was used for synoptic temperature system evaluation for base models, finetuned models, and baselines. We calculate the mean and standard error of the SPACE-local and SPACE-aggregate scores (

s_{s}

), match scores (

s_{m}

), and coverage ratios (

r_{c}

) within the test set. The range of possible scores for all metrics is 0 to 1, with 1 being a perfect score

Explore similar work

AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions
Aug 25, 2026Manmeet Singh, Somnath Luitel, Prabhjot Singh +3Forecasting Benchmarks Language Model Generation Evaluation
Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.
WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation
May 8, 2026Zinan Zheng, Yang Liu, Nuo Chen +3Multimodal Large Language Models Weather Forecasting
Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong · HY Founda- tion Model Team, Tencent +2
WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models
Mar 18, 2026Wanjun Du, Zifeng Yuan, Tingting Chen +3VLM Robustness VLM Reasoning
Beijing Jiaotong University · National University of Singapore · Monash University

Aug 25, 2026J/K move · Enter open · S save

AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions

Manmeet Singh, Somnath Luitel, Prabhjot Singh, Manraaj Banga+2

Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · RediMinds Inc., Southfield, MI, USA.

Forecasting Benchmarks Language Model Generation Evaluation Weather Forecasting Scientific Reasoning in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.

Abstract

Explore similar work

AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions

WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation

WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models