A Station-Based Evaluation of Machine Learning-based Weather Forecasting Models in Northern Norway

Siyan Chen; Lars Uebbing; Eirik Mikal Samuelsen; Georgios Leontidis; Arnt-Børre Salberg; Sébastien Lefèvre; Robert Jenssen; Kristoffer Wickstrøm

physics.ao-phAug 25, 2026

A Station-Based Evaluation of Machine Learning-based Weather Forecasting Models in Northern Norway

Authors: Siyan Chen, Lars Uebbing, Eirik Mikal Samuelsen, Georgios Leontidis, Arnt-Børre Salberg, Sébastien Lefèvre, Robert Jenssen, Kristoffer Wickstrøm

Organizations: UiT The Arctic University of Norway · The Norwegian Meteorological Institute · Norwegian Computing Center · University of South Brittany ·

Abstract

Recent machine learning weather prediction (MLWP) models have demonstrated remarkable forecasting skill on global reanalysis-based benchmarks. However, their performance remains unclear in challenging environments such as Northern Norway, where narrow fjords and rapidly changing weather result in highly variable local wind conditions. In this case study, we evaluate FourCastNet3 (FCN3), GraphCast, and ECMWF High Resolution Forecast (HRES) for wind speed forecasting using multi-year station observations from Northern Norway, focusing on their relative performance, generalization beyond the training period, and performance under high-wind conditions. Our results show that HRES slightly outperforms FCN3 and GraphCast, with an overall RMSE of 2.89 $\mathrm{m\,s^{-1}}$ , compared to 2.96 $\mathrm{m\,s^{-1}}$ for FCN3 and 2.94 $\mathrm{m\,s^{-1}}$ for GraphCast. Notably, the MLWP models maintain comparable performance beyond their respective training periods, with no clear evidence of noticeable degradation. FCN3 performs best under high-wind conditions, although all models substantially underestimate strong winds. Our findings suggest that MLWP has become competitive with NWP for local wind, but further refinements are still needed to capture complex terrain better.

Figures & tables

Figure 1: Mean RMSE of wind speed forecasts from FCN3, GraphCast, and HRES at lead times from +6 to +72 hours with shaded bands indicating

\pm 1

standard error, calculated using station observations as the reference. Results are aggregated across all 12 stations and all evaluation years from 2016 to 2022.

Figure 2: Locations of the 12 meteorological stations in Northern Norway, grouped by terrain category. Grid lines indicate $0.25^{\circ}\times 0.25^{\circ}$ cells.

$N=366{,}585$
Model	RMSE $\downarrow$	nRMSE $\downarrow$	Bias $(\to 0)$	nBias $(\to 0)$	Corr.
HRES	$\bm{2.89\,(0.75)}$	$\bm{53.06\,(15.00)}$	$-1.21\,(0.85)$	$-21.29\,(15.76)$	$\bm{0.685(0.088)}$
FCN3	$2.96\,(0.71)$	$54.27\,(14.20)$	$\bm{-0.85\,(1.04)}$	$\bm{-15.63\,(19.32)}$	$0.648(0.088)$
GraphCast	$2.94\,(0.86)$	$53.43\,(14.61)$	$-1.26\,(0.99)$	$-21.40\,(16.68)$	$0.682(0.095)$

Table 1: Model performance evaluated against station observations and ERA5 across all stations during 2016–2022.

Panel of Figure 3 — Figure 3: RMSE performance of FCN3, GraphCast, and HRES across forecast lead times, stations, and years.

Models	Diff.	95% CI	Sig.
FCN3–GC	0.029	[-0.113, 0.157]	No
FCN3–HRES	0.073	[0.005, 0.135]	Yes
GC–HRES	0.044	[-0.074, 0.164]	No

Table 2: Pairwise station-level RMSE differences from 10,000 bootstrap resamples (GC: GraphCast).

Figure 4: Model Performance on ETS by Beaufort force category

$N=40{,}247$
Model	RMSE $\downarrow$	nRMSE $\downarrow$	Bias $(\to 0)$	nBias $(\to 0)$	Corr.
HRES	$6.05\,(1.88)$	$45.59\,(14.97)$	$-5.50\,(2.07)$	$-41.51\,(16.69)$	$0.480\,(0.156)$
FCN3	$\mathbf{5.77\,(1.96)}$	$\mathbf{43.42\,(15.45)}$	$\mathbf{-4.97\,(2.39)}$	$\mathbf{-37.58\,(18.75)}$	$0.464\,(0.121)$
GraphCast	$6.01\,(1.93)$	$45.14\,(14.75)$	$-5.43\,(2.16)$	$-40.82\,(16.58)$	$\bm{0.488\,(0.12)}$

Table 3: High-wind performance across all stations during 2016–2022.

Panel of Figure 5 — Figure 5: Scatter plots of observed versus predicted wind speeds at four selected stations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Time (hours)	Memory (GB)
FCN3	9.05–9.50	66–68
GraphCast	13.99–14.51	58–61

Appendix

Table 4: Computational requirements for the FCN3 and GraphCast experiments. Execution times and peak memory usage are reported for individual annual tasks.

Force	Description	Wind speed (m/s)
0	Calm	0.0–0.5
1	Light air	0.5–1.6
2	Light breeze	1.6–3.4
3	Gentle breeze	3.4–5.5
4	Moderate breeze	5.5–8.0
5	Fresh breeze	8.0–10.8

Appendix

Table 5: Beaufort wind force scale.

Station ID	Station	Latitude	Longitude	Elevation	Terrain label
SN91740	Sørkjosen Lufthavn	69.7900	20.9520	2.76	Fjord
SN95350	Banak	70.0600	24.9790	6.54	Fjord
SN90720	Måsvik	69.9900	18.6940	11.04	Fjord
SN90490	Tromsø-Langnes	69.6767	18.9133	8.97	Fjord
SN87110	Andøya	69.3070	16.1310	4.35	Marine Coast
SN96400	Slettnes Fyr	71.0890	28.2170	9.11	Marine Coast

Appendix

Table 6: Geographical and terrain characteristics of the 12 observation stations.

$N=366{,}948$
Model	RMSE $\downarrow$	nRMSE $\downarrow$	Bias $(\to 0)$	nBias $(\to 0)$	Corr.
HRES	$1.31,(0.57)$	$27.17,(4.93)$	$-0.32,(0.58)$	$-4.95,(9.46)$	$0.853(0.036)$
FCN3	$1.24,(0.40)$	$26.13,(2.67)$	$\bm{0.04,(0.10)}$	$\bm{0.81,(2.17)}$	$0.846(0.036)$
GraphCast	$\bm{1.14,(0.37)}$	$\bm{24.34,(3.80)}$	$-0.37,(0.46)$	$-5.30,(9.49)$	$\bm{0.890(0.034)}$

Appendix

Table 7: Model performance evaluated against ERA5 across all stations during 2016–2022.

Model	$N$	RMSE $\downarrow$	nRMSE $\downarrow$	Bias $(\to 0)$	nBias $(\to 0)$
CARRA1	121,955	2.41 (0.57)	44.67 (14.20)	-0.28 (0.69)	-3.78 (10.46)
ERA5	122,615	2.71 (0.74)	50.19 (16.54)	-0.89 (1.02)	-16.47 (18.98)

Appendix

Table 8: CARRA1 and ERA5 performance against station observations.

Figure 7: Scatter plot for Skrova Fyr.

Figure 8: Scatter plot for Bardufoss.

Panel of Figure 9 — Figure 9: Scatter plots of observed and predicted wind speeds at stations.

Figure 10: Station-Wise RMSE / nRMSE / Bias / nBias against Station Observations

Panel of Figure 11 — Figure 11: Zoomed-in views of the local surroundings of the 12 stations.

Weather Forecasting

Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.

See all 172 papers

Skillful high-resolution weather forecasting independent of physical models
May 27, 2026Pengcheng Zhao, Siqi Xiang, Weixin Jin +10Weather Forecasting
1Microsoft Corporation. · University of Cambridge, Cambridge, United Kingdom. · 3The Alan Turing Institute, London, United Kingdom.
Multi-site Learning for Forecasting Day-Ahead Wind-Power Production for Onshore Wind Farms in Great Britain with Deep Neural Networks
Oct 18, 2026James Kean, Aidan O'SullivanWind Power Forecasting Time Series Forecasting
UCL Energy Institute, University College London
Varda-single-1.0: deterministic data-driven weather forecasting at 1 km resolution over Switzerland's complex topography
Oct 1, 2026Alberto Pennino, Francesco Zanetta, Michele Cattaneo +28Weather Forecasting
Federal Office of Meteorology and Climatology MeteoSwiss · Swiss Data Science Center (SDSC), ETH Zürich · Center for Climate Systems Modeling (C2SM), ETH Zürich +4

May 27, 2026J/K move · Enter open · S save

Skillful high-resolution weather forecasting independent of physical models

Pengcheng Zhao, Siqi Xiang, Weixin Jin, Zekun Ni+9

1Microsoft Corporation. · University of Cambridge, Cambridge, United Kingdom. · 3The Alan Turing Institute, London, United Kingdom.

Weather Forecasting

Open paper AskPDF arXiv

Accurate and timely weather forecasts are critical for high-impact decisions in modern society. Machine-learning-based weather prediction is emerging as an alternative for producing initial conditions, forecasts, and even both in end-to-end systems. These methods deliver predictions faster and often with higher skill than traditional numerical weather prediction (NWP). However, even end-to-end models typically rely on NWP-generated reanalyses for supervision, thereby inheriting the biases and resolution limitations of those NWPs, and limiting adaptation to settings where suitable reanalysis products are unavailable, infrequently updated, or expensive to produce. Here we introduce ObsCast, a regional system that generates both analysis and predictions, without using any NWP-derived data in either training or inference, while still achieving state-of-the-art performance in short-term high-resolution regional modeling. Over the contiguous United States and Europe, ObsCast outperforms operational NWP for near-surface variables through 18 h and produces skillful precipitation forecasts. It provides a simpler and more adaptable route to build and refine regional forecasting services directly from local observations, without the need to develop complex and costly traditional forecasting pipelines.

Abstract

Weather Forecasting

Explore similar work

Skillful high-resolution weather forecasting independent of physical models

Multi-site Learning for Forecasting Day-Ahead Wind-Power Production for Onshore Wind Farms in Great Britain with Deep Neural Networks

Varda-single-1.0: deterministic data-driven weather forecasting at 1 km resolution over Switzerland's complex topography