EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

Yiming Sun; Yuan Zhang; Qi Cheng; Yingda Fan; Runlong Yu; Yiqun Xie; Xiaowei Jia

cs.AIOct 26, 2026

EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

Authors: Yiming Sun, Yuan Zhang, Qi Cheng, Yingda Fan, Runlong Yu, Yiqun Xie, Xiaowei Jia

Organizations: Rutgers University, New Brunswick · Apple Inc. · University of Alabama · University of Maryland, College Park

Abstract

Machine learning models are now widely used to predict environmental variable dynamics in domains like hydrology, climate science, agriculture, and ecology. These models are usually compared on fixed benchmark datasets using a single average accuracy score. As models improve, their scores become very close when averaged over large regions and long time periods, so the benchmark can no longer tell them apart, and a single number hides where and when a model fails. Three failures are common across environmental domains: models get worse as the climate shifts (non-stationarity), models fail at unmonitored sites (spatial extrapolation), and models miss rare but important regimes such as floods and droughts. We present EvoBench, a benchmark framework that shows where a model fails, tracks failures over time, and evolves to focus on model capability gaps. We apply EvoBench to daily streamflow prediction as a case study (CAMELS-Evolve, about 9{,}000 U.S. basins, 1985--2024) and compare an operational physical model (NWM), a conceptual reference (HBV), and several data-driven models. No model wins on every testing scenarios (e.g., in terms of average accuracy, performance at low flows and unmonitored basins). Also we observe that all data-driven models under-predict floods. Additionally, we show that targeted fix based on the reported failures could mitigate weaknesses the benchmark identified in certain scenarios. EvoBench turns model evaluation from a saturated ranking into a moving map of model limitations.

Figures & tables

Figure 1. Pipeline of the EvoBench framework.

Model	NSE	KGE ′	log-NSE	peak%bias	skill/HBV
LSTM	0.674	0.711	0.595	$-$ 32.2	$+$ 0.368
GRU	0.665	0.688	0.545	$-$ 30.6	$+$ 0.379
NWM	0.644	0.694	0.563	$-$ 13.0	$+$ 0.332
EA-LSTM	0.633	0.689	0.521	$-$ 28.9	$+$ 0.337
TCN	0.595	0.619	0.465	$-$ 42.1	$+$ 0.273
Transformer	0.558	0.498	$-$ 0.910	$-$ 27.0	$+$ 0.170

Table 1. Head-to-head on the core group (162 basins, WY2011–2022), median over basins. The best value in each column is in bold ; the order changes by metric.

Model	NSE	KGE ′	log-NSE	peak%bias	skill/HBV
LSTM	0.637	0.653	0.606	$-$ 31.2	$+$ 0.391
GRU	0.630	0.661	0.477	$-$ 29.3	$+$ 0.373
EA-LSTM	0.586	0.595	0.502	$-$ 34.4	$+$ 0.314
NWM	0.562	0.664	0.468	$-$ 13.6	$+$ 0.261
TCN	0.525	0.567	0.325	$-$ 44.2	$+$ 0.208
Transformer	0.517	0.529	$-$ 0.337	$-$ 31.2	$+$ 0.186

Table 2. Head-to-head at larger scale (extended, 1,139 basins), median over basins. The LSTM, GRU, and EA-LSTM surpass NWM on NSE, while NWM keeps the best KGE ′ and flood (peak-day) bias.

Model	NSE gauged	NSE ungauged	spatial gap
LSTM	0.674	0.484	0.191
GRU	0.665	0.499	0.165
EA-LSTM	0.633	0.529	0.105
TCN	0.595	0.420	0.175
Transformer	0.558	0.306	0.252
PatchTST	0.465	0.216	0.250

Table 3. Prediction at unmonitored basins (core), using leave-6-cluster-out testing. The EA-LSTM transfers best despite a lower score on monitored basins.

Figure 2. 2-D failure grid: LSTM percent bias by aridity and flow regime (extended). Low flows are over-predicted (red, left) and peaks are under-predicted (blue, right). The two errors cancel in a single NSE.

	Core		Extended
Model	overall	arid	overall	arid	snow
LSTM	0.679	0.410	0.644	0.480	0.505
GRU	0.667	0.359	0.632	0.476	0.469
NWM	0.642	0.409	0.585	0.381	0.265
EA-LSTM	0.622	0.367	0.610	0.418	0.375
TCN	0.593	0.264	0.517	0.329	0.294

Table 4. Five-year sliding rounds: median NSE averaged over the five rounds, for the core group (162 basins) and the extended group (1,139 basins). Every data-driven model is retrained to each round’s cutoff. Best per column in bold .

Figure 3. Five-year sliding rounds (core): overall NSE for each model over time. The LSTM and GRU stay above NWM in nearly every round, while weaker models swing between rounds; HBV is a flat reference. NWM’s rounds 4–5 cover only part of the window (record ends January 2023).

Figure 4. Fixing a diagnosed weakness (LSTM, core, fixed split): arid- and snow-cell NSE, mean

\pm

standard deviation over three seeds. The low-flow loss does not help, while over-sampling lifts both the arid and the snow cell.

Model	(1) baseline	(2) targeted	(3) targeted+memory
LSTM	0.680	0.682	0.690
EA-LSTM	0.629	0.625	0.637
GRU	0.668	0.660	0.679
Transformer	0.503	0.484	0.484
TCN	0.579	0.575	0.569
DLinear	0.412	0.407	0.408

Table 5. Overall NSE (mean over five sliding rounds, core) for each baseline under the three conditions. Best per row in bold .

Model	(1) baseline	(2) targeted	(3) targeted+memory
LSTM	0.457	0.451	0.446
EA-LSTM	0.361	0.401	0.422
GRU	0.360	0.442	0.469
Transformer	0.043	0.038	$-$ 0.017
TCN	0.240	0.249	0.294
DLinear	$-$ 0.408	$-$ 0.369	$-$ 0.274

Table 6. Arid-cell NSE (mean over five rounds, core), where the intervention is aimed. The intervention helps most models, most clearly the GRU and EA-LSTM; the LSTM, already the strongest on arid basins, and the Transformer do not gain.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Component	Streamflow (this paper)	Other domains
Reference model	per-basin HBV	process model (e.g. TEM for CH 4 , chemical-transport for air quality)
Failure cells	basin attributes $\times$ flow period	site attributes $\times$ season/regime
Spatial gap	leave-cluster-out basins (PUB)	leave-region-out monitoring sites
Evolving loop	5-year sliding water-year window	sliding window at the task’s cadence
Plausibility check	water balance ( $Q/P\!\leq\!1$ )	mass/energy budget closure

Appendix

Table 7. EvoBench components are domain-general. The streamflow case study instantiates each; the final column sketches analogues in other environmental-modeling tasks.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 394 papers

ClimateBench v2.0: Probabilistic Climate Model Benchmarking
Oct 3, 2026Duncan Watson-Parris, Willa Tobin, Aytaç Paçal +32Climate Modeling Benchmark Design
Scripps Institution of Oceanography, University of California San Diego · Halicioglu Data Science Institute, University of California San Diego · University of Tübingen, connAIx Research School for Applied AI, Heilbronn, Germany +23
fev-bench: A Realistic Benchmark for Time Series Forecasting
Sep 30, 2025Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5Benchmark Design Time Series Forecasting
AWS · Keystone AI · Prior Labs +2
Earth Observation Foundation Models for Terrestrial Ecohydrology: From Representation Learning to Process Inference
Aug 15, 2026Yi Yu, Jian Peng, Yucheng Lin +2Geospatial Foundation Models Hydrology
Precision Agriculture, Hydrology and Geoinformation Science Laboratory, School of Life and Environmental Sciences, The University of Sydney, Eveleigh, NSW, Australia · Department of Remote Sensing, Helmholtz Centre for Environmental Research–UFZ, Leipzig, Germany · Institute for Earth System Science and Remote Sensing, Leipzig University, Leipzig, Germany +3

Oct 3, 2026J/K move · Enter open · S save

ClimateBench v2.0: Probabilistic Climate Model Benchmarking

Duncan Watson-Parris, Willa Tobin, Aytaç Paçal, Manuel Schlund+31

Scripps Institution of Oceanography, University of California San Diego · Halicioglu Data Science Institute, University of California San Diego · University of Tübingen, connAIx Research School for Applied AI, Heilbronn, Germany+23

Climate Modeling Benchmark Design Forecasting Benchmarks Probabilistic Forecasting

Open paper AskPDF arXiv

We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in projecting mid-century regional temperature and precipitation changes. The protocol is designed to evaluate any physics-based, data-driven, or hybrid climate model on equal footing using a common set of observational and out-of-distribution tests. We define three tiers of evaluation. Tier I establishes physical credibility through entry-ticket tests of energy conservation, coupled (co-)variability, and basic forced responses. Tier II scores models against post-2015 observations of surface temperature, precipitation, radiative fluxes, sea ice, and key modes of variability using fair CRPS as the primary probabilistic score, complemented by distributional and ensemble-consistency diagnostics. Tier III tests out-of-distribution generalization through paleoclimate simulations spanning the Last Interglacial, Last Glacial Maximum, and Mid-Holocene, and through perfect-model experiments in which data-driven models must predict the future climate of existing Earth system models from historical data alone. We reserve all observational data after 2015 for testing, and submissions must include multiple ensemble members to enable probabilistic evaluation. This reservation exploits a new opportunity provided by the decade of observations accumulated since the end of the CMIP6 historical experiment, which constitutes an out-of-sample record of forced climate change (and internal variability) for the current generation of models, and we quantify, in an idealized setting, the information it carries about mid-century warming. We provide the evaluation code, observational reference datasets, and perfect-model training data as an open benchmark to drive measurable progress in climate projection across all modeling approaches.

Abstract

Benchmark Design

Explore similar work

ClimateBench v2.0: Probabilistic Climate Model Benchmarking

fev-bench: A Realistic Benchmark for Time Series Forecasting

Earth Observation Foundation Models for Terrestrial Ecohydrology: From Representation Learning to Process Inference