Session 13 · Learning from data

From one river to 665

The Leaf River is one of 665 catchments for which dCREST was calibrated and its parameters released. This last session zooms out: where across the United States the model works and where it fails, how it compares with other models on hundreds of rivers, how to use the released parameters, and how far to trust them.

Time about 35 minutesNeeds Sessions 10 and 11
Watch this session first: a 6-minute narrated video with English subtitles. The page below goes further and lets you try each step yourself.

You will be able to

The CAMELS-US catchments

CAMELS (Newman et al., 2015; Addor et al., 2017) is a set of 671 catchments across the contiguous United States, chosen because their rivers are little affected by dams, diversions or cities, so that their flow reflects weather and landscape. For each it provides daily basin-average weather, daily river flow and dozens of catchment attributes. Of the 671, seven were dropped because their outline could not be matched to the reported drainage area, and one gauge outside the original list was kept, leaving 665. Every one was calibrated exactly as the Leaf River was in Session 11: the same sixteen parameters, ranges, training period, six runs and selection rule.

ExploreTest NSE in 665 catchments
catchmenthover over a dot

One dot per catchment, at its gauge, coloured by test NSE (1989–1999, Daymet forcing, released parameters). The large dot is the Leaf River.

The pattern is clear even without numbers: skill is highest in the wet Southeast, the Appalachians and the Pacific Northwest, and lowest in the Great Plains and the dry Southwest.

How good, and where

Test NSE of all 665 catchments. The red line marks the Leaf River. Values below −0.5 are counted in the leftmost bar.
Test NSE against aridity (potential evaporation divided by precipitation; above 1, the air could evaporate more water than falls). Logarithmic horizontal axis; the line joins the medians of four aridity classes.

Across all 665 catchments the median test NSE is 0.662; 80% exceed 0.5 and 14 (2.1%) fall below zero. The Leaf River's 0.835 ranks 31st: it is better than 95% of the catchments, which is one reason it makes a clear teaching case. Skill falls steadily with dryness:

Aridity (PET / P)catchmentsmedian test NSEshare below 0
below 0.8 (humid, like the Leaf River at 0.76)2600.700.4%
0.8 to 1.22300.670.4%
1.2 to 21200.545.0%
above 2 (arid)540.359.3%

Snow matters less in this summary (median 0.68 where under 10% of precipitation falls as snow, 0.64 where over 30% does), but the paper shows that in snowy catchments dCREST trails the HBV model by a clear margin, and that replacing its simple degree-day snow (Session 3) by the more detailed Snow-17 model does not close the gap.

Why dry places are hard

Every model in the CAMELS benchmark is weakest in arid catchments. Sabino Creek near Tucson, Arizona (USGS 09484000, aridity 2.9) shows why. Compare its years with the Leaf River's:

Each dot is one water year: precipitation against river flow, both in mm. Dashed lines mark runoff ratios of 0.1, 0.25, 0.5 and 1 (the share of the year's precipitation that left as river flow). Water years with a complete flow record: Leaf River 1981–2014, Sabino Creek 1990–2014.

On the Leaf River the share of the year's precipitation that reaches the river varies between 0.13 and 0.51, and wetter years reliably bring more flow. On Sabino Creek it varies between 0.02 and 0.97. In water year 1991, 593 mm of precipitation produced 447 mm of flow; in 2000, 470 mm produced 8 mm; in 2006 the measured flow equalled 97% of the gridded precipitation. In such catchments runoff depends on the timing and intensity of a few storms, and partly on rain that basin-average gridded data do not capture. No single set of parameters can fit all those years: dCREST's training NSE there is 0.20 and its test NSE 0.33. Users in dry regions should check the per-catchment scores shipped with the parameters before relying on a set.

Against other models

The CAMELS benchmark compares models on 531 of the catchments over the same test decade, with Maurer precipitation. 528 of them are in dCREST's set; some benchmark models cover fewer, and each paired difference uses only the catchments both models have. The table is the paper's (Table 4), abbreviated.

ModelCalibrationmedian test NSEdCREST minus model (median per catchment)
dCREST v1.0best of 6 runs on 1999–20080.659—
dCREST v1.0, mean of its 6 runsas above, averaged0.672—
SAC-SMA + Snow-17 (A)best of about 10 runs; calibrated 1980–1995, six test years in-sample0.664−0.005
SAC-SMA + Snow-17 (B)single run, calibrated 1999–20080.608+0.042
HBV, upper benchmarkmean of 100 calibrated runs0.679−0.005
mHMper catchment0.665+0.001
FUSE (three structures)per catchment0.625–0.655+0.009 to +0.042
VICper catchment0.554+0.084
HBV, lower benchmarkmean of 1,000 uncalibrated runs0.418+0.216
LSTM (regional)one network for all catchments, 8 seeds0.760−0.081

dCREST sits within the band of per-catchment conceptual models: clearly above the protocol-matched SAC-SMA (B), level with mHM, slightly below HBV's average of 100 runs (and level with it when dCREST's own six runs are averaged). SAC-SMA A's small lead is confined to the six test years inside its own calibration period; on the four test years outside it, dCREST leads A by 0.036. The regional LSTM, a neural network trained on all catchments at once, remains clearly better. On a single river these differences vanish into noise (Session 10); across 528 catchments they are measurable. The paper's conclusion is modest and specific: the CREST structure performs at the level of SAC-SMA, HBV and mHM.

Using the released parameters

The parameters are released with the dcrest Python package (PyTorch), one file per forcing: Daymet for 665 catchments, Maurer for 528, and a variant with an extra soil store. Each file holds the chosen parameters with their training and test scores and provenance; every individual calibration run is released alongside, in camels/members/. The browser model in this course reproduces the package to about 10−14 mm/day (Session 9), but for real work use the package:

pip install -e .                       # from a checkout of github.com/Skyan1002/dCREST

import torch, dcrest
# P, T, E: daily precipitation (mm), mean air temperature (°C) and potential evaporation (mm),
# tensors of shape (days, catchments), columns in the order given by subset=
params, meta = dcrest.load_parameters("camels/dCREST_CAMELS_v1.0_daymet665.npz",
                                      subset=["02472000"])
cfg = dcrest.config_from_meta(meta)    # the settings the parameters were fitted under
Q = dcrest.simulate(P, T, E, params, cfg)   # (days, catchments), mm/day
Two rules that prevent silently wrong results

Run the model over three spin-up years before the period you care about (Session 9), and use forcing consistent with the set: Daymet-calibrated parameters with Daymet-like precipitation.

How far to trust each parameter

A parameter set can reproduce the river well while individual values are poorly determined (Session 11). The paper tested this for every parameter: it calibrated all catchments a second time with SCE-UA and asked whether the two methods rank the catchments the same way (Spearman rank correlation across catchments; 1 means the same order).

Does a second calibration reproduce the values? Rank correlation across 665 catchments between the released (Adam) values and an independent SCE-UA calibration (bars), between Adam runs (circles, median of 15 pairs) and between two SCE-UA seeds (diamonds). Paper, Table 1.

Seven parameters are reproduced reasonably well by the other method (rank correlation 0.53–0.70): the snow threshold and melt factor, the soil capacity wm, the evaporation multiplier ke, the loss fraction lf, the aquifer capacity hmaxaq and the unit-hydrograph shape nshape. Two are in between (0.44): qc and kmult. Seven are not (0.10–0.39): fc, gwe, b, ksoil, ft, im and gwc. (The snow parameters were a flat plain on the Leaf River, where it hardly snows; across all catchments, many of them snowy, they are among the most reproducible.) Those fit the river equally well under either method but take different values; their released values reflect how the search was done as much as the data. And even the reproducible ones are effective values: their relations with measured catchment attributes such as soil depth or forest cover are weak (rank correlations of 0.30 or less). Use the parameters to simulate rivers; treat them with caution as descriptions of soils or aquifers.

Limitations

In Python

The 665-catchment summary, from the same file the map uses.

Check your understanding

The Leaf River scores 0.84. Is that what you should expect from dCREST in a new catchment?

No. The Leaf River beats 95% of the catchments. The median is 0.66, and in arid catchments it is far lower. Check the per-catchment scores, or those of catchments similar to yours.

Why is a dry catchment like Sabino Creek hard for any lumped model with fixed parameters?

The fraction of precipitation that becomes river flow swings enormously from year to year (from 0.02 to 0.97), depending on a few intense storms that a basin-average daily series smooths out. No single parameter set fits both kinds of year.

dCREST beats SAC-SMA B by 0.04 in median but trails SAC-SMA A by 0.005. Which comparison is fair?

B was calibrated on the same training period as dCREST, so its test scores are out of sample like dCREST's. A was calibrated on years that include six of the ten test years, so part of its score is in-sample; on the four test years outside its calibration period, dCREST leads A.

Why must you call config_from_meta?

Each parameter set is valid only with the configuration it was fitted under (the unit-hydrograph limit and the parameter ranges). A default configuration can reject the set or simulate a different river.

Can the released ksoil values be used as a map of soil drainage rates?

Not on their own. ksoil is among the parameters a second calibration method does not reproduce (rank correlation 0.28), and all parameters relate only weakly to measured soil attributes. They are effective values for reproducing river flow.

What to remember

The end of the course

You have followed one river through a whole model: its water budget (Session 1), the five stores and how they pass water on each day (Sessions 2–8), the full simulation and its check against the package (Session 9), how to judge it (Session 10), how its parameters were found (Session 11), what gradients reveal (Session 12), and how it all scales to a continent (this session). The glossary collects every variable and parameter. The code is at github.com/Skyan1002/dCREST.

References

Addor, N., Newman, A. J., Mizukami, N., and Clark, M. P.: The CAMELS data set: catchment attributes and meteorology for large-sample studies, Hydrology and Earth System Sciences, 21, 5293–5313, doi:10.5194/hess-21-5293-2017, 2017.

Beven, K.: A manifesto for the equifinality thesis, Journal of Hydrology, 320, 18–36, doi:10.1016/j.jhydrol.2005.07.007, 2006.

Clark, M. P., Slater, A. G., Rupp, D. E., Woods, R. A., Vrugt, J. A., Gupta, H. V., Wagener, T., and Hay, L. E.: Framework for Understanding Structural Errors (FUSE): A modular framework to diagnose differences between hydrological models, Water Resources Research, 44, W00B02, doi:10.1029/2007WR006735, 2008.

Duan, Q., Sorooshian, S., and Gupta, V.: Effective and efficient global optimization for conceptual rainfall-runoff models, Water Resources Research, 28, 1015–1031, doi:10.1029/91WR02985, 1992.

Duan, Q., Sorooshian, S., and Gupta, V. K.: Optimal use of the SCE-UA global optimization method for calibrating watershed models, Journal of Hydrology, 158, 265–284, doi:10.1016/0022-1694(94)90057-4, 1994.

Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F.: Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling, Journal of Hydrology, 377, 80–91, doi:10.1016/j.jhydrol.2009.08.003, 2009.

Kingma, D. P., and Ba, J.: Adam: A Method for Stochastic Optimization, 3rd International Conference on Learning Representations (ICLR), doi:10.48550/arXiv.1412.6980, 2015.

Knoben, W. J. M., Freer, J. E., and Woods, R. A.: Technical note: Inherent benchmark or not? Comparing Nash–Sutcliffe and Kling–Gupta efficiency scores, Hydrology and Earth System Sciences, 23, 4323–4331, doi:10.5194/hess-23-4323-2019, 2019.

Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G.: Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets, Hydrology and Earth System Sciences, 23, 5089–5110, doi:10.5194/hess-23-5089-2019, 2019.

Liang, X., Lettenmaier, D. P., Wood, E. F., and Burges, S. J.: A simple hydrologically based model of land surface water and energy fluxes for general circulation models, Journal of Geophysical Research, 99, 14415–14428, doi:10.1029/94JD00483, 1994.

Maurer, E. P., Wood, A. W., Adam, J. C., Lettenmaier, D. P., and Nijssen, B.: A long-term hydrologically based dataset of land surface fluxes and states for the conterminous United States, Journal of Climate, 15, 3237–3251, doi:10.1175/1520-0442(2002)015<3237:ALTHBD>2.0.CO;2, 2002.

Mizukami, N., Rakovec, O., Newman, A. J., Clark, M. P., Wood, A. W., Gupta, H. V., and Kumar, R.: On the choice of calibration metrics for "high-flow" estimation using hydrologic models, Hydrology and Earth System Sciences, 23, 2601–2614, doi:10.5194/hess-23-2601-2019, 2019.

Nash, J. E., and Sutcliffe, J. V.: River flow forecasting through conceptual models part I — A discussion of principles, Journal of Hydrology, 10, 282–290, doi:10.1016/0022-1694(70)90255-6, 1970.

Newman, A. J., Clark, M. P., Sampson, K., Wood, A., Hay, L. E., Bock, A., Viger, R. J., Blodgett, D., Brekke, L., Arnold, J. R., Hopson, T., and Duan, Q.: Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance, Hydrology and Earth System Sciences, 19, 209–223, doi:10.5194/hess-19-209-2015, 2015.

Newman, A. J., Mizukami, N., Clark, M. P., Wood, A. W., Nijssen, B., and Nearing, G.: Benchmarking of a Physically Based Hydrologic Model, Journal of Hydrometeorology, 18, 2215–2225, doi:10.1175/JHM-D-16-0284.1, 2017.

Oudin, L., Hervieu, F., Michel, C., Perrin, C., Andréassian, V., Anctil, F., and Loumagne, C.: Which potential evapotranspiration input for a lumped rainfall–runoff model? Part 2—Towards a simple and efficient potential evapotranspiration model for rainfall–runoff modelling, Journal of Hydrology, 303, 290–306, doi:10.1016/j.jhydrol.2004.08.026, 2005.

Seibert, J., Vis, M. J. P., Lewis, E., and van Meerveld, H. J.: Upper and lower benchmarks in hydrological modelling, Hydrological Processes, 32, 1120–1125, doi:10.1002/hyp.11476, 2018.

Sorooshian, S., Duan, Q., and Gupta, V. K.: Calibration of rainfall-runoff models: Application of global optimization to the Sacramento Soil Moisture Accounting Model, Water Resources Research, 29, 1185–1194, doi:10.1029/92WR02617, 1993.

Thornton, P. E., Running, S. W., and White, M. A.: Generating surfaces of daily meteorological variables over large regions of complex terrain, Journal of Hydrology, 190, 214–251, doi:10.1016/S0022-1694(96)03128-9, 1997 (the method behind Daymet).

Wang, J., Hong, Y., Li, L., Gourley, J. J., Khan, S. I., Yilmaz, K. K., Adler, R. F., Policelli, F. S., Habib, S., Irwn, D., Limaye, A. S., Korme, T., and Okello, L.: The coupled routing and excess storage (CREST) distributed hydrological model, Hydrological Sciences Journal, 56, 84–98, doi:10.1080/02626667.2010.543087, 2011.