From one river to 665
The Leaf River is one of 665 catchments for which dCREST was calibrated and its parameters released. This last session zooms out: where across the United States the model works and where it fails, how it compares with other models on hundreds of rivers, how to use the released parameters, and how far to trust them.
You will be able to
- describe the CAMELS-US catchments and the community benchmark;
- say where dCREST is skilful and where it is not, and why dry catchments are hard;
- place dCREST among other models without over-reading one number;
- load and run the released parameters correctly;
- say which parameters can be trusted as values and which only as part of a set;
- list the model's main limitations.
The CAMELS-US catchments
CAMELS (Newman et al., 2015; Addor et al., 2017) is a set of 671 catchments across the contiguous United States, chosen because their rivers are little affected by dams, diversions or cities, so that their flow reflects weather and landscape. For each it provides daily basin-average weather, daily river flow and dozens of catchment attributes. Of the 671, seven were dropped because their outline could not be matched to the reported drainage area, and one gauge outside the original list was kept, leaving 665. Every one was calibrated exactly as the Leaf River was in Session 11: the same sixteen parameters, ranges, training period, six runs and selection rule.
The pattern is clear even without numbers: skill is highest in the wet Southeast, the Appalachians and the Pacific Northwest, and lowest in the Great Plains and the dry Southwest.
How good, and where
Across all 665 catchments the median test NSE is 0.662; 80% exceed 0.5 and 14 (2.1%) fall below zero. The Leaf River's 0.835 ranks 31st: it is better than 95% of the catchments, which is one reason it makes a clear teaching case. Skill falls steadily with dryness:
| Aridity (PET / P) | catchments | median test NSE | share below 0 |
|---|---|---|---|
| below 0.8 (humid, like the Leaf River at 0.76) | 260 | 0.70 | 0.4% |
| 0.8 to 1.2 | 230 | 0.67 | 0.4% |
| 1.2 to 2 | 120 | 0.54 | 5.0% |
| above 2 (arid) | 54 | 0.35 | 9.3% |
Snow matters less in this summary (median 0.68 where under 10% of precipitation falls as snow, 0.64 where over 30% does), but the paper shows that in snowy catchments dCREST trails the HBV model by a clear margin, and that replacing its simple degree-day snow (Session 3) by the more detailed Snow-17 model does not close the gap.
Why dry places are hard
Every model in the CAMELS benchmark is weakest in arid catchments. Sabino Creek near Tucson, Arizona (USGS 09484000, aridity 2.9) shows why. Compare its years with the Leaf River's:
On the Leaf River the share of the year's precipitation that reaches the river varies between 0.13 and 0.51, and wetter years reliably bring more flow. On Sabino Creek it varies between 0.02 and 0.97. In water year 1991, 593 mm of precipitation produced 447 mm of flow; in 2000, 470 mm produced 8 mm; in 2006 the measured flow equalled 97% of the gridded precipitation. In such catchments runoff depends on the timing and intensity of a few storms, and partly on rain that basin-average gridded data do not capture. No single set of parameters can fit all those years: dCREST's training NSE there is 0.20 and its test NSE 0.33. Users in dry regions should check the per-catchment scores shipped with the parameters before relying on a set.
Against other models
The CAMELS benchmark compares models on 531 of the catchments over the same test decade, with Maurer precipitation. 528 of them are in dCREST's set; some benchmark models cover fewer, and each paired difference uses only the catchments both models have. The table is the paper's (Table 4), abbreviated.
| Model | Calibration | median test NSE | dCREST minus model (median per catchment) |
|---|---|---|---|
| dCREST v1.0 | best of 6 runs on 1999–2008 | 0.659 | — |
| dCREST v1.0, mean of its 6 runs | as above, averaged | 0.672 | — |
| SAC-SMA + Snow-17 (A) | best of about 10 runs; calibrated 1980–1995, six test years in-sample | 0.664 | −0.005 |
| SAC-SMA + Snow-17 (B) | single run, calibrated 1999–2008 | 0.608 | +0.042 |
| HBV, upper benchmark | mean of 100 calibrated runs | 0.679 | −0.005 |
| mHM | per catchment | 0.665 | +0.001 |
| FUSE (three structures) | per catchment | 0.625–0.655 | +0.009 to +0.042 |
| VIC | per catchment | 0.554 | +0.084 |
| HBV, lower benchmark | mean of 1,000 uncalibrated runs | 0.418 | +0.216 |
| LSTM (regional) | one network for all catchments, 8 seeds | 0.760 | −0.081 |
dCREST sits within the band of per-catchment conceptual models: clearly above the protocol-matched SAC-SMA (B), level with mHM, slightly below HBV's average of 100 runs (and level with it when dCREST's own six runs are averaged). SAC-SMA A's small lead is confined to the six test years inside its own calibration period; on the four test years outside it, dCREST leads A by 0.036. The regional LSTM, a neural network trained on all catchments at once, remains clearly better. On a single river these differences vanish into noise (Session 10); across 528 catchments they are measurable. The paper's conclusion is modest and specific: the CREST structure performs at the level of SAC-SMA, HBV and mHM.
Using the released parameters
The parameters are released with the dcrest Python package (PyTorch), one file per forcing: Daymet for 665 catchments, Maurer for 528, and a variant with an extra soil store. Each file holds the chosen parameters with their training and test scores and provenance; every individual calibration run is released alongside, in camels/members/. The browser model in this course reproduces the package to about 10−14 mm/day (Session 9), but for real work use the package:
pip install -e . # from a checkout of github.com/Skyan1002/dCREST
import torch, dcrest
# P, T, E: daily precipitation (mm), mean air temperature (°C) and potential evaporation (mm),
# tensors of shape (days, catchments), columns in the order given by subset=
params, meta = dcrest.load_parameters("camels/dCREST_CAMELS_v1.0_daymet665.npz",
subset=["02472000"])
cfg = dcrest.config_from_meta(meta) # the settings the parameters were fitted under
Q = dcrest.simulate(P, T, E, params, cfg) # (days, catchments), mm/day
- Always build the configuration from the file with
config_from_meta. A parameter set is valid only with the unit-hydrograph limit and parameter ranges it was fitted under; a default configuration either rejects the set as out of bounds or gives a different river. - Line up parameters with forcing columns. The file holds all catchments in its own order; pass
subset=with your gauge identifiers in the order of your forcing columns.
Run the model over three spin-up years before the period you care about (Session 9), and use forcing consistent with the set: Daymet-calibrated parameters with Daymet-like precipitation.
How far to trust each parameter
A parameter set can reproduce the river well while individual values are poorly determined (Session 11). The paper tested this for every parameter: it calibrated all catchments a second time with SCE-UA and asked whether the two methods rank the catchments the same way (Spearman rank correlation across catchments; 1 means the same order).
Seven parameters are reproduced reasonably well by the other method (rank correlation 0.53–0.70): the snow threshold and melt factor, the soil capacity wm, the evaporation multiplier ke, the loss fraction lf, the aquifer capacity hmaxaq and the unit-hydrograph shape nshape. Two are in between (0.44): qc and kmult. Seven are not (0.10–0.39): fc, gwe, b, ksoil, ft, im and gwc. (The snow parameters were a flat plain on the Leaf River, where it hardly snows; across all catchments, many of them snowy, they are among the most reproducible.) Those fit the river equally well under either method but take different values; their released values reflect how the search was done as much as the data. And even the reproducible ones are effective values: their relations with measured catchment attributes such as soil depth or forest cover are weak (rank correlations of 0.30 or less). Use the parameters to simulate rivers; treat them with caution as descriptions of soils or aquifers.
Limitations
- Lumped and daily. One value per store for the whole catchment, one step per day. Rain falling on part of a catchment, or a flood rising within hours, is averaged away.
- Dry and snowy catchments. Skill is lowest where it is arid (Sabino Creek above), and dCREST trails HBV where snow dominates, for reasons not yet identified.
- Tuned for NSE. The objective weighs floods heavily; low flows can be poor even when NSE is high (the Leaf River's drought flows were five times too low, Session 10).
- Equifinality. Equally good fits differ in their parameters and in their test scores (Session 11); about half the parameters are not reproduced by a different calibration method.
- Stationarity. Parameters fitted on one decade are applied to another; when a catchment's behaviour shifts (the Leaf River's wetter 1990s, Session 10), the error shows up as bias.
- Tied to their configuration and forcing. A parameter set is valid only with the configuration and the kind of forcing it was fitted with.
In Python
The 665-catchment summary, from the same file the map uses.
Check your understanding
The Leaf River scores 0.84. Is that what you should expect from dCREST in a new catchment?
No. The Leaf River beats 95% of the catchments. The median is 0.66, and in arid catchments it is far lower. Check the per-catchment scores, or those of catchments similar to yours.
Why is a dry catchment like Sabino Creek hard for any lumped model with fixed parameters?
The fraction of precipitation that becomes river flow swings enormously from year to year (from 0.02 to 0.97), depending on a few intense storms that a basin-average daily series smooths out. No single parameter set fits both kinds of year.
dCREST beats SAC-SMA B by 0.04 in median but trails SAC-SMA A by 0.005. Which comparison is fair?
B was calibrated on the same training period as dCREST, so its test scores are out of sample like dCREST's. A was calibrated on years that include six of the ten test years, so part of its score is in-sample; on the four test years outside its calibration period, dCREST leads A.
Why must you call config_from_meta?
Each parameter set is valid only with the configuration it was fitted under (the unit-hydrograph limit and the parameter ranges). A default configuration can reject the set or simulate a different river.
Can the released ksoil values be used as a map of soil drainage rates?
Not on their own. ksoil is among the parameters a second calibration method does not reproduce (rank correlation 0.28), and all parameters relate only weakly to measured soil attributes. They are effective values for reproducing river flow.
What to remember
- dCREST was calibrated identically on 665 CAMELS-US catchments; median test NSE 0.66, highest in humid regions, lowest in arid ones.
- Across 528 catchments it performs at the level of SAC-SMA, HBV and mHM; a regional LSTM is clearly better.
- Load sets with
load_parametersandconfig_from_meta; line up catchments withsubset=. - About half the parameters are reproducible between calibration methods; all are effective values, not measurements.
The end of the course
You have followed one river through a whole model: its water budget (Session 1), the five stores and how they pass water on each day (Sessions 2–8), the full simulation and its check against the package (Session 9), how to judge it (Session 10), how its parameters were found (Session 11), what gradients reveal (Session 12), and how it all scales to a continent (this session). The glossary collects every variable and parameter. The code is at github.com/Skyan1002/dCREST.
References
Addor, N., Newman, A. J., Mizukami, N., and Clark, M. P.: The CAMELS data set: catchment attributes and meteorology for large-sample studies, Hydrology and Earth System Sciences, 21, 5293–5313, doi:10.5194/hess-21-5293-2017, 2017.
Beven, K.: A manifesto for the equifinality thesis, Journal of Hydrology, 320, 18–36, doi:10.1016/j.jhydrol.2005.07.007, 2006.
Clark, M. P., Slater, A. G., Rupp, D. E., Woods, R. A., Vrugt, J. A., Gupta, H. V., Wagener, T., and Hay, L. E.: Framework for Understanding Structural Errors (FUSE): A modular framework to diagnose differences between hydrological models, Water Resources Research, 44, W00B02, doi:10.1029/2007WR006735, 2008.
Duan, Q., Sorooshian, S., and Gupta, V.: Effective and efficient global optimization for conceptual rainfall-runoff models, Water Resources Research, 28, 1015–1031, doi:10.1029/91WR02985, 1992.
Duan, Q., Sorooshian, S., and Gupta, V. K.: Optimal use of the SCE-UA global optimization method for calibrating watershed models, Journal of Hydrology, 158, 265–284, doi:10.1016/0022-1694(94)90057-4, 1994.
Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F.: Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling, Journal of Hydrology, 377, 80–91, doi:10.1016/j.jhydrol.2009.08.003, 2009.
Kingma, D. P., and Ba, J.: Adam: A Method for Stochastic Optimization, 3rd International Conference on Learning Representations (ICLR), doi:10.48550/arXiv.1412.6980, 2015.
Knoben, W. J. M., Freer, J. E., and Woods, R. A.: Technical note: Inherent benchmark or not? Comparing Nash–Sutcliffe and Kling–Gupta efficiency scores, Hydrology and Earth System Sciences, 23, 4323–4331, doi:10.5194/hess-23-4323-2019, 2019.
Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G.: Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets, Hydrology and Earth System Sciences, 23, 5089–5110, doi:10.5194/hess-23-5089-2019, 2019.
Liang, X., Lettenmaier, D. P., Wood, E. F., and Burges, S. J.: A simple hydrologically based model of land surface water and energy fluxes for general circulation models, Journal of Geophysical Research, 99, 14415–14428, doi:10.1029/94JD00483, 1994.
Maurer, E. P., Wood, A. W., Adam, J. C., Lettenmaier, D. P., and Nijssen, B.: A long-term hydrologically based dataset of land surface fluxes and states for the conterminous United States, Journal of Climate, 15, 3237–3251, doi:10.1175/1520-0442(2002)015<3237:ALTHBD>2.0.CO;2, 2002.
Mizukami, N., Rakovec, O., Newman, A. J., Clark, M. P., Wood, A. W., Gupta, H. V., and Kumar, R.: On the choice of calibration metrics for "high-flow" estimation using hydrologic models, Hydrology and Earth System Sciences, 23, 2601–2614, doi:10.5194/hess-23-2601-2019, 2019.
Nash, J. E., and Sutcliffe, J. V.: River flow forecasting through conceptual models part I — A discussion of principles, Journal of Hydrology, 10, 282–290, doi:10.1016/0022-1694(70)90255-6, 1970.
Newman, A. J., Clark, M. P., Sampson, K., Wood, A., Hay, L. E., Bock, A., Viger, R. J., Blodgett, D., Brekke, L., Arnold, J. R., Hopson, T., and Duan, Q.: Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance, Hydrology and Earth System Sciences, 19, 209–223, doi:10.5194/hess-19-209-2015, 2015.
Newman, A. J., Mizukami, N., Clark, M. P., Wood, A. W., Nijssen, B., and Nearing, G.: Benchmarking of a Physically Based Hydrologic Model, Journal of Hydrometeorology, 18, 2215–2225, doi:10.1175/JHM-D-16-0284.1, 2017.
Oudin, L., Hervieu, F., Michel, C., Perrin, C., Andréassian, V., Anctil, F., and Loumagne, C.: Which potential evapotranspiration input for a lumped rainfall–runoff model? Part 2—Towards a simple and efficient potential evapotranspiration model for rainfall–runoff modelling, Journal of Hydrology, 303, 290–306, doi:10.1016/j.jhydrol.2004.08.026, 2005.
Seibert, J., Vis, M. J. P., Lewis, E., and van Meerveld, H. J.: Upper and lower benchmarks in hydrological modelling, Hydrological Processes, 32, 1120–1125, doi:10.1002/hyp.11476, 2018.
Sorooshian, S., Duan, Q., and Gupta, V. K.: Calibration of rainfall-runoff models: Application of global optimization to the Sacramento Soil Moisture Accounting Model, Water Resources Research, 29, 1185–1194, doi:10.1029/92WR02617, 1993.
Thornton, P. E., Running, S. W., and White, M. A.: Generating surfaces of daily meteorological variables over large regions of complex terrain, Journal of Hydrology, 190, 214–251, doi:10.1016/S0022-1694(96)03128-9, 1997 (the method behind Daymet).
Wang, J., Hong, Y., Li, L., Gourley, J. J., Khan, S. I., Yilmaz, K. K., Adler, R. F., Policelli, F. S., Habib, S., Irwn, D., Limaye, A. S., Korme, T., and Okello, L.: The coupled routing and excess storage (CREST) distributed hydrological model, Hydrological Sciences Journal, 56, 84–98, doi:10.1080/02626667.2010.543087, 2011.