Session 10 · Putting it together

Is the model any good?

Session 9 showed that the model in your browser is the real model. This session asks whether the real model is any good, that is, how close its river is to the Leaf River. You will build the two scores hydrologists use most, see what each one rewards and what it ignores, and put the Leaf River's score next to simple baselines and other published models.

Time about 40 minutesNeeds Sessions 1 and 9
Watch this session first: a 9-minute narrated video with English subtitles. The page below goes further and lets you try each step yourself.

You will be able to

Look before you score

Everything in this session uses the test decade, water years 1990–1999 (1 October 1989 to 30 September 1999). The model's parameters were tuned on a different period, 1 October 1999 to 30 September 2008, and never saw the test decade's river flows; why that matters is the subject of a later section. A score squeezes ten years of river into one number, so look at the river first.

Try itThe test decade, one water year at a time

Top: observed and simulated river flow. Bottom: the daily error, model minus observation; below zero means the model is too low. Drag across a chart to zoom, double-click to zoom out.

Three patterns repeat from year to year. The model's rises and falls come on the right days. Many of its flood peaks are too low; the largest flood of the decade, on 1990-02-17, reached 53.9 mm/day at the gauge and 29.4 mm/day in the model. And in long dry spells the model's river falls further than the real one.

One event stands apart. Between 11 and 15 July 1993, 122 mm of rain fell on a summer soil that was 75% full. The model's canopy caught about 51 mm of it (much of which evaporated again), its soil took in about 42 mm, and only 29 mm ran off; its river rose to 5.5 mm/day; the real river rose to 25.4 mm/day. In the 34-year record, every other summer storm that brought 100 mm or more in five days raised the river to at most 11 mm/day. Perhaps the rain was heavier than the gridded data say, or fell mostly near the river; the model sees only the catchment average and cannot tell. A model can only be as good as the rain it is given.

Every day of the test decade, model against observation. Both axes are logarithmic: the labelled lines are factors of ten apart. A perfect model would put every point on the dashed line. Points below the line are days on which the model is too low.

The scatter shows the same two weaknesses. At high flows the cloud sits a little below the line (peaks too low). At the lowest flows, below about 0.3 mm/day, it bends far below the line: in droughts the model's river is several times too small.

Errors, one day at a time

The error on day \(t\) is the difference between the model and the observation:

\[ e_t = Q_{\text{sim},t} - Q_{\text{obs},t} \]

Over the 3652 days of the test decade there are 3652 errors. Three ways to summarise them:

SummaryFormulaLeaf River, test decadeWhat it says
bias (mean error)\(\overline{e} = \frac{1}{n}\sum e_t\)−0.189 mm/daythe model delivers 12% less water than the river
mean absolute error\(\frac{1}{n}\sum |e_t|\)0.487 mm/daya typical day's error, ignoring its sign
root-mean-square error\(\sqrt{\frac{1}{n}\sum e_t^2}\)1.261 mm/daylike the mean absolute error, but a few large errors weigh much more

Is an error of 1.26 mm/day large? That depends on the river. For a river whose flow hardly changes it would be enormous; for one that swings between 0.1 and 50 mm/day it is modest. A score needs a yardstick that comes from the river itself.

The Nash–Sutcliffe efficiency

Nash and Sutcliffe (1970) supplied the yardstick: compare the model with the laziest possible prediction, the river's average flow, every day of the decade. The Nash–Sutcliffe efficiency is

\[ \text{NSE} \;=\; 1 \;-\; \frac{\sum_t \left(Q_{\text{sim},t} - Q_{\text{obs},t}\right)^2}{\sum_t \left(Q_{\text{obs},t} - \overline{Q}_{\text{obs}}\right)^2} \]

The numerator is the model's squared error. The denominator is the squared error of the lazy prediction \(\overline{Q}_{\text{obs}}\), the observed average.

For the Leaf River test decade the model's squared errors add up to 5809 (mm/day)²; the lazy prediction's add up to 35228. The model removes all but 16.5% of the lazy prediction's error, so

\[ \text{NSE} = 1 - \frac{ 5809 }{ 35228 } = 0.835 \]
NSEMeaning
1perfect: the model matches the river every day
between 0 and 1better than predicting the average flow every day; the share of the average's squared error that the model removes
0no better than predicting the average flow every day
below 0worse than that; there is no lower limit
A detail that matters

The lazy prediction uses the average of the very decade being scored, which nobody knows in advance. A forecaster who predicted the training decade's average flow every day would score −0.02. NSE = 0 is a low bar, but not a fair forecast.

Where the squared error comes from

Squaring makes big errors dominate. An error of 20 mm/day on a flood day counts as much as an error of 0.2 mm/day on ten thousand dry days. The chart sorts the days from the largest error to the smallest and adds up their share of the total.

LookA few days carry most of the error
Worst daysobservedmodelerrorshare of all squared error

The worst 1% of days (37 days in ten years) produce 64% of the squared error, and the worst 5% produce 88%. In 31 of those 37 days the model was too low. Days when the observed flow was among the highest tenth contribute 86%. NSE is mostly a score for floods. A model can get the floods right and the droughts badly wrong and still score well, which is what happens here (see the flow duration curve below).

The Kling–Gupta efficiency

Gupta et al. (2009) showed that the squared error can be split into three parts that answer three separate questions:

PartFormulaQuestionLeaf River, test
correlation \(r\)Pearson correlation of the two seriesdo the ups and downs happen at the right times?0.918
variability ratio \(\alpha\)\(\sigma_{\text{sim}} / \sigma_{\text{obs}}\)are the swings the right size?0.861
bias ratio \(\beta\)\(\mu_{\text{sim}} / \mu_{\text{obs}}\)is the total amount of water right?0.878

\(\sigma\) is the standard deviation (the typical size of the swings) and \(\mu\) the mean. A perfect model has \(r = \alpha = \beta = 1\).

The Kling–Gupta efficiency is one minus the distance from that perfect point:

\[ \text{KGE} = 1 - \sqrt{(r-1)^2 + (\alpha-1)^2 + (\beta-1)^2} = 0.797 \]

Read the parts for the Leaf River: the timing is good (\(r\) = 0.92); the model's swings are 86% the size of the real ones; and it delivers 88% of the river's water. All three fall short of 1; the swing size and the volume a little more than the timing.

The same three parts sit inside NSE. Writing the squared error in terms of means, standard deviations and correlation gives

\[ \text{NSE} = 2\alpha r - \alpha^2 - \left(\frac{\mu_{\text{sim}} - \mu_{\text{obs}}}{\sigma_{\text{obs}}}\right)^2 \]

For a given timing \(r\), this is largest when \(\alpha = r\). Since \(r\) is always below 1, a model tuned to maximise NSE learns to make its swings a little too small, which means floods that are too low. The Leaf River model was tuned on NSE, and on its training period it has \(r\) = 0.936 and \(\alpha\) = 0.934: almost exactly equal, as the formula predicts. Part of the "peaks too low" pattern above is the objective, not the physics.

NSE and KGE are different scales

For NSE, predicting the average flow scores 0. For KGE the same lazy prediction scores about −0.41 (Knoben et al., 2019), so a KGE of 0 is not "no better than the average". Compare NSE with NSE and KGE with KGE.

Break the model on purpose

The quickest way to understand a score is to damage a simulation in a known way and watch the score respond. The widget changes the model's river after it has been computed: it can move it in time, scale its volume, or smooth it with a moving average. The model itself is not re-run.

Try itDamage the simulation, watch the scores

Scores are for the whole test decade, with the change from the model as computed in brackets; the chart shows one water year. In the bars, hollow circles mark the model as computed. In the chart, grey is the model as computed and red the damaged version.

Things to try:

Low flows: the flow duration curve

To see the droughts that NSE ignores, sort the decade's daily flows from highest to lowest and plot each flow against the percentage of days on which it was exceeded. This flow duration curve forgets the dates and shows only how often each flow occurs: the left end is floods, the middle is ordinary days, the right end is drought.

Flow duration curves, test decade. The vertical axis is logarithmic. Where the two curves separate, the model gets the frequency of those flows wrong.
Flow exceeded on …observed (mm/day)model (mm/day)
1% of days (floods)15.5814.11
5% of days6.365.91
50% of days (the median)0.530.42
95% of days (drought)0.160.03
99% of days0.110.02

The high end agrees well. The low end does not: the flow the real river exceeds on 95% of days is 0.16 mm/day, the model's is 0.03, about five times smaller. In Session 7 terms, the model's aquifer runs down further in a long drought than the real catchment's storage does. NSE barely notices, because the squared errors on those days are tiny. An NSE computed on the logarithm of flow, which treats every factor of two equally whether the flow is large or small, is 0.63, against 0.84 for ordinary NSE.

Key idea

A model is good for a purpose. The Leaf River parameters were tuned on NSE, so they are good for floods and ordinary flows. If low flows matter (water supply, river ecology), the model should be tuned and judged with a score that sees them.

Training and test

Any model with enough adjustable parameters can be made to fit a record well, including the record's measurement errors and its accidents. A fit says little about whether the model has learned how the catchment works. The test is to tune it on one period and judge it on another. This course follows the protocol shared by most CAMELS studies (Newman et al., 2017; Kratzert et al., 2019):

Two runs, two decades. The tuning in Session 11 sees only the observed flows of the training period. Spin-up years (Session 9) are simulated but never scored. The training run's spin-up uses the 1996–1999 weather but none of the observed river flows.
WindowNSEKGE\(r\)\(\alpha\)\(\beta\)rain (mm/yr)river (mm/yr)runoff ratio
training, 1999–20080.8760.9080.9360.9341.00514754000.27
test, 1989–19990.8350.7970.9180.8610.87815525640.36

The training scores are higher, as they should be: the parameters were chosen to fit that period. Of the three parts, \(\beta\) changes most, from 1.005 to 0.878, and in KGE the swings and the volume contribute about equally to the drop. NSE, which weighs timing most, falls mainly because \(r\) fell: 83% of its drop comes from timing, 8% from the swings and 9% from the volume. The volume error has a clear cause. The two periods had different water balances: the test decade had 5% more rain than the training period but 41% more river flow, and its runoff ratio was 0.36 instead of 0.27. The parameters learned how much of the rain the river returned in the drier 2000s; in the 1990s the real catchment returned more. The yearly volume errors in the year viewer above tell the same story: the model is too dry in every year of the test decade.

This is why "10% more water" in the widget is not a fix. The 10% was read off the test decade's observed flows, which a model meant for the future would not have. Adjusting a model after looking at the test data turns the test into more training, and the score stops meaning anything.

On the Leaf River

A small training-to-test drop with a clear cause is a good result. It shows the model transfers to a different decade, and it locates the weakness (the volume when the climate shifts) precisely enough to act on.

Is 0.84 good?

A score means little alone. Two kinds of comparison give it meaning: simple baselines that need no model, and other models on the same river and decade.

LookBaselines on the test decade
PredictionNSE

The seasonal baseline is the average observed flow on each calendar day over the training period, smoothed over a month. It uses no rain at all. The chart compares it with the observed flow in one test year.

The seasonal cycle alone scores 0.09. The Leaf River is higher in winter than in summer (Session 1), but on any particular day its flow depends on the storms of the past few days, which only a model of the catchment's response can follow. Everything between 0.09 and 0.84 is what the model adds.

The CAMELS benchmark lets us compare with other published models on exactly the same river and decade. Those models were run with a different precipitation dataset (Maurer et al., 2002, instead of the Daymet data used in this course), so the fair comparison uses the dCREST parameters calibrated with Maurer forcing, which score 0.824 rather than 0.835. The glossary says briefly what each of the other models is.

ModelHow it was calibratedtest NSE

On this river every calibrated model scores between about 0.70 and 0.86, and the best conceptual models differ by a few hundredths, not much more than dCREST's own score changes (0.011) when the precipitation dataset changes. One catchment cannot rank models; the paper compares them across 528 catchments (Session 13). What one river can show is that dCREST behaves like an established conceptual model.

Two SAC-SMA calibrations

Two different SAC-SMA calibrations circulate for this benchmark. A (Newman et al., 2015) was calibrated on the first 15 years of record, 1980–1995, which include six of the ten test years; we pick, for each catchment, the best of its about ten runs on the 1999–2008 window, as for dCREST. B (as distributed by Kratzert et al., 2019, attributed to Newman et al., 2017) was calibrated on the training period like dCREST. A is the stronger reference but partly in-sample.

In Python

Every number in this session from scratch. The cell uses the stored run of the PyTorch package, which Session 9 showed is identical to the browser model.

Change one line and re-run: replace sim by 1.1 * sim after it is defined, or by np.roll(sim, 1) (one day late; the first day wraps around, which hardly matters), and watch which numbers move.

Check your understanding

A model predicts the test decade's average flow on every day. What is its NSE, and why is that not a fair forecast?

Exactly 0, by construction. It is not a fair forecast because nobody knows a decade's average flow before the decade happens; using the training period's average instead gives −0.02 here.

Why does a one-day timing error cost so much NSE?

The flood is counted twice: as missing water on the day it should have arrived and as extra water on the day it does arrive. Both errors are large and squared. On the Leaf River, delaying the whole simulation by one day lowers NSE from 0.84 to 0.68.

A model scores NSE = 0.84 but log-flow NSE = 0.63. What do you suspect?

Floods and ordinary flows are good, low flows are poor. Squared errors are dominated by high flows; logarithms weigh relative errors equally across the range. The flow duration curve confirms it: the model's drought flows are about five times too low.

Multiplying the simulation by 1.1 raises the test NSE. Why may you not do it?

The factor was found by looking at the test decade's observed flows. Using it turns the test into training, and the score no longer says how the model would do on years it has not seen.

Why do models tuned on NSE tend to produce floods that are too small?

Because NSE = 2αr − α² − (bias term)², which is largest at α = r. With imperfect timing (r < 1), the best NSE comes from swings smaller than the observed ones (α < 1).

What to remember