HydroTuring: A physics probe benchmark testing mass, energy, and momentum conservation in hydrologic models
Organizations: Flood Lab, Department of Civil, Environmental and Architectural Engineering, and Institute of Arctic and Alpine Research, University of Colorado Boulder, Boulder, CO, USA · South China University of Technology, China · Department of Electrical Engineering, Columbia University, New York, NY, USA · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, China ·
Abstract
Hydrologic models, including deep learning systems now deployed for operational flood forecasting, are evaluated primarily by how closely their simulated hydrographs match observed streamflow. However, high hydrograph skill does not guarantee physical consistency: in a synthetic calibration experiment, a conceptual model that loses 15% of its water input to an unreported sink still reaches a Nash-Sutcliffe efficiency of 0.93, and no shared benchmark tests whether models obey fundamental conservation laws. Here we introduce HydroTuring (v0.1.0), a black-box benchmark of 37 physics probes that test mass (24 probes), energy (7) and momentum (6) conservation in hydrologic models. Each probe runs a controlled experiment on forcing generated at run time from a recorded seed and applies binary pass or fail criteria validated against 18 physically exact and 59 deliberately broken reference models. Across 32 evaluated models (3 exact references, 7 AI-based models and 22 process-based models), only the 3 exact references pass every probe among the 28 models scored on 5 or more probes. Process-based and AI-based models pass similar shares of their scored probes (76% and 74%), and a pre-trained flood-forecasting LSTM fails 6 of the 11 runoff-only probes it can be scored on. Furthermore, 49% of all 1,184 model-probe pairs remain unscored because models omit the required state or flux variables, while the scored failures occur in perturbation responses and state bounds rather than long-term water-budget closure. HydroTuring provides a reproducible smoke test for model development, peer review and operational deployment under an open community protocol.
Figures & tables
| Benchmark | Quantity evaluated | Test cases | Physical laws tested |
|---|---|---|---|
| CAMELS, Caravan ( Newman et al., 2015 ; Addor et al., 2017 ; Kratzert et al., 2023 ) | Streamflow skill in observed basins | Public observed records | Not tested |
| WeatherBench 2 ( Rasp et al., 2024 ) | Forecast skill against reanalysis and observations | Public data | Not tested as laws |
| PDEBench ( Takamoto et al., 2022 ) | Accuracy of learned PDE solvers | Public simulated datasets | Partly, through the error of conserved quantities |
| Agents’ Last Exam ( Sun and others, 2026 ) | Agent performance on professional workflows | Tasks with verifiable outcomes, extended continuously | Not applicable |
| CheckList ( Ribeiro et al., 2020 ) | Behavior of NLP models | Test templates | Not applicable |
| HydroTuring | Whether a model conserves mass, energy and momentum | Generated at run time from a seed, with hidden tolerances | Mass, energy and momentum, one law per probe |
| Law | Family | Probes | Representative experiment | Representative criterion and limit |
|---|---|---|---|---|
| Mass | Closure | 4 | 10 years of generated weather, 5 seeds | Cumulative residual 5% of precipitation |
| Mass | Invariance | 5 | Same weather at 1 min, 1 h and 1 d | Runoff volumes agree within 10% |
| Mass | Response | 6 | Largest storm scaled by 2, 5 and 10 | Added runoff share between 0.1 and 1.05 |
| Mass | Dynamics | 5 | 2 years without rain | Runoff and stores only decline |
| Mass | Groundwater and human | 4 | 380 mm irrigation withdrawal | Budget difference accounts for the volume within 5% |
| Energy | Single budget | 4 | Hourly bare-surface budget | Residual within 5% or 2 W m -2 |
| Name | Units | Physical meaning |
| pr | mm d -1 | Precipitation, echoed exactly as the forcing gave it |
| evspsbl | mm d -1 | Evapotranspiration, including sublimation; net of dew and frost where the model nets them |
| mrro | mm d -1 | Total runoff |
| dis | m 3 s -1 | River discharge |
| gwex | mm d -1 | Declared exchange with the outside: regional groundwater, inter-basin transfer |
| snm | mm d -1 | Liquid delivered from the snow module to the ground, rain passed through included, and not an outflow beside mrro |
| Verdict | Reason | Scored | Meaning |
|---|---|---|---|
| PASS | OK | yes | Every criterion on every seed is within its limit |
| FAIL | VIOLATION | yes | At least 1 criterion on 1 seed is outside its limit |
| N/A | INCOMPLETE | no | The model does not report the variables that the probe needs |
| N/A | INCOMPATIBLE | no | The model and the probe disagree on the time step, on an input, or on paired-perturbation support |
| – | ERROR | no | The adapter or the machinery failed, which is operational and makes the run exit with an error code |
| Criterion | Question | Limit |
|---|---|---|
| closure | Does precipitation minus evaporation, runoff and the change of every store stay near zero? | Cumulative residual at most 5% of cumulative precipitation |
| et_plausible | Does evaporation stay at or below the demand? | No step exceeds the potential evaporation supplied |
| state_bounds | Does every reported store stay in its physical range? | No store outside its range |
| non_degenerate | Does the model produce runoff that responds to rain? | Runoff ratio between 0.02 and 0.98 and coefficient of variation at least 0.1 |
| forcing_fidelity | Does the model echo the forcing that it was given? | Relative tolerance of |
| Probe | Verdict | Measured quantity |
|---|---|---|
| resolution-invariance | FAIL | Runoff differs by 73.8% of precipitation between hourly and daily records of the same month (limit 10%) |
| extreme-rain | FAIL | Across the ladder up to 10 times the largest storm, 35 of 526 mm of added rain run off (0.07). The exact bucket returns 0.56 to 1.00 |
| dry-down | FAIL | 2 rainless years yield a near-constant 0.3 mm d -1 that does not decay (coefficient of variation 0.02, minimum 0.1) |
| warming-response | FAIL | Runoff falls by 0.088 per unit of added demand (limit 0.10). The exact bucket falls by 0.20 |
| phase-counterfactual | FAIL | Runoff differs by % of the rain when snow falls as rain (limit 5%) |
| response-nonnegativity | FAIL | Runoff falls 0.18 mm d -1 below the control after 120 mm of rain is added |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Name | Group | Units | Physical meaning |
|---|---|---|---|
| pr | flux | mm d -1 | Precipitation, echoed exactly as the forcing gave it |
| evspsbl | flux | mm d -1 | Evapotranspiration, including sublimation; net of dew and frost where the model nets them |
| mrro | flux | mm d -1 | Total runoff |
| dis | flux | m 3 s -1 | River discharge |
| gwex | flux | mm d -1 | Declared exchange with the outside: regional groundwater, inter-basin transfer |
| gw_sw_exchange | flux | mm d -1 | Net river-aquifer exchange inside the control volume, not added to gwex or counted in closure |
| Probe | Question the probe asks | Scored criteria | Seeds |
|---|---|---|---|
| Mass: closure | |||
| catchment-closure | Catchment water balance closure under synthetic forcing | closure , et_plausible | 5 |
| snowpack-mass-closure | Internal snowpack water-balance closure | closure , snowpack_response | 5 |
| extreme-event-closure | Event-wise water balance closure under extreme rainfall | closure , event_water_closure | 5 |
| ungauged-basin-closure | Water balance integrity across declared catchment attribute domains | closure , et_plausible | 12 |
| Mass: invariance | |||
| Probe | Authors |
|---|---|
| mass/catchment-closure | Zhi Li |
| mass/snowpack-mass-closure | Jinlong Hu |
| mass/extreme-event-closure | Taiqi Lian |
| mass/ungauged-basin-closure | Shunan Zhou |
| mass/resolution-invariance | Zhi Li |
| mass/area-invariance | Zhi Li |
| Model | Proposed by |
|---|---|
| lstm | Zhi Li (CU Boulder) |
| dhbv2 | Zhi Li (CU Boulder) |
| wflow_sbm | Yuanhang Liu (Independent Researcher) |
| summa | Yuanhang Liu (Independent Researcher) |
| cwatm | Yuanhang Liu (Independent Researcher) |
| lisflood | Yuanhang Liu (Independent Researcher) |
| Model | Probes requiring a pass |
|---|---|
| reference_bucket | 25 |
| flex_lumped | 25 |
| sacsma_snow17 | 25 |
| flex_topo | 24 |
| reference_coupled | 3 |
| reference_saint_venant | 2 |
| Reference model | Defect built in | Probe | Criterion |
|---|---|---|---|
| abstraction_blind | The exact bucket, blind to a prescribed human abstraction. | human-abstraction | human_abstraction |
| air_emitter | Negative control for energy/radiation-consistency . | radiation-consistency | radiative_identity |
| anticipating | The reference bucket with its runoff smoothed over a centered 7-day window, so every reported value carries 3 days that have not happened yet. | causality | causality |
| area_leak | The bucket losing a share of its runoff that grows with the catchment area it is given: 0% at 25 km 2 and 50% at 2,500 km 2 . | area-invariance | invariance |
| calendar | The bucket model with a recession that drifts with the calendar year. | time-origin-invariance | invariance |
| cheater | Solves for storage as whatever balances the budget. | antecedent-monotonicity , catchment-closure , precipitation-counterfactual , spinup-cycle-invariance , ungauged-basin-closure | antecedent_monotonicity , counterfactual_response , state_bounds |
| Tolerance (% of precipitation) | Evaluated models failing (of 22) | reference_leaky |
| 0.01 | 4 | fails |
| 0.1 | 2 | fails |
| 0.5 | 1 | fails |
| 1 | 1 | fails |
| 2 | 1 | fails |
| 5 (adopted) | 0 | fails |
| Probe | Model | Seeds passed | Criterion trips or measured range |
| catchment-closure | reference_bucket | 20/20 | largest residual of precipitation |
| catchment-closure | flex_lumped | 20/20 | largest residual of precipitation |
| catchment-closure | sacsma_snow17 | 20/20 | largest residual of precipitation |
| catchment-closure | reference_leaky | 0/20 | closure 20 of 20 |
| catchment-closure | reference_cheater | 0/20 | state_bounds 20 of 20 |
| catchment-closure | reference_degenerate | 0/20 | non_degenerate 20, et_plausible 18 of 20 |
| Model | Class | All | Mass | Energy | Momentum |
|---|---|---|---|---|---|
| flex_lumped | Physical reference | 25/25 | 19/19 | 1/1 | 5/5 |
| sacsma_snow17 | Physical reference | 25/25 | 20/20 | 1/1 | 4/4 |
| flex_topo | Physical reference | 24/24 | 19/19 | 1/1 | 4/4 |
| mcp1 | AI-based | 18/20 | 17/19 | 1/1 | – |
| mcp3 | AI-based | 17/21 | 16/19 | 1/1 | 0/1 |
| mcp2 | AI-based | 15/20 | 15/19 | 0/1 | – |