Is the model any good?
Session 9 showed that the model in your browser is the real model. This session asks whether the real model is any good, that is, how close its river is to the Leaf River. You will build the two scores hydrologists use most, see what each one rewards and what it ignores, and put the Leaf River's score next to simple baselines and other published models.
You will be able to
- compute a simulation's daily errors and summarise them as bias, mean absolute error and root-mean-square error;
- compute the Nash–Sutcliffe efficiency (NSE) and explain why it is mostly a score for floods;
- split the Kling–Gupta efficiency (KGE) into timing, variability and volume;
- read a flow duration curve and spot where a model fails at low flows;
- explain why a model is judged on years it was not tuned on;
- say whether an NSE of 0.84 is good, by comparing it with baselines and with other models.
Look before you score
Everything in this session uses the test decade, water years 1990–1999 (1 October 1989 to 30 September 1999). The model's parameters were tuned on a different period, 1 October 1999 to 30 September 2008, and never saw the test decade's river flows; why that matters is the subject of a later section. A score squeezes ten years of river into one number, so look at the river first.
Three patterns repeat from year to year. The model's rises and falls come on the right days. Many of its flood peaks are too low; the largest flood of the decade, on 1990-02-17, reached 53.9 mm/day at the gauge and 29.4 mm/day in the model. And in long dry spells the model's river falls further than the real one.
One event stands apart. Between 11 and 15 July 1993, 122 mm of rain fell on a summer soil that was 75% full. The model's canopy caught about 51 mm of it (much of which evaporated again), its soil took in about 42 mm, and only 29 mm ran off; its river rose to 5.5 mm/day; the real river rose to 25.4 mm/day. In the 34-year record, every other summer storm that brought 100 mm or more in five days raised the river to at most 11 mm/day. Perhaps the rain was heavier than the gridded data say, or fell mostly near the river; the model sees only the catchment average and cannot tell. A model can only be as good as the rain it is given.
The scatter shows the same two weaknesses. At high flows the cloud sits a little below the line (peaks too low). At the lowest flows, below about 0.3 mm/day, it bends far below the line: in droughts the model's river is several times too small.
Errors, one day at a time
The error on day \(t\) is the difference between the model and the observation:
Over the 3652 days of the test decade there are 3652 errors. Three ways to summarise them:
| Summary | Formula | Leaf River, test decade | What it says |
|---|---|---|---|
| bias (mean error) | \(\overline{e} = \frac{1}{n}\sum e_t\) | −0.189 mm/day | the model delivers 12% less water than the river |
| mean absolute error | \(\frac{1}{n}\sum |e_t|\) | 0.487 mm/day | a typical day's error, ignoring its sign |
| root-mean-square error | \(\sqrt{\frac{1}{n}\sum e_t^2}\) | 1.261 mm/day | like the mean absolute error, but a few large errors weigh much more |
Is an error of 1.26 mm/day large? That depends on the river. For a river whose flow hardly changes it would be enormous; for one that swings between 0.1 and 50 mm/day it is modest. A score needs a yardstick that comes from the river itself.
The Nash–Sutcliffe efficiency
Nash and Sutcliffe (1970) supplied the yardstick: compare the model with the laziest possible prediction, the river's average flow, every day of the decade. The Nash–Sutcliffe efficiency is
The numerator is the model's squared error. The denominator is the squared error of the lazy prediction \(\overline{Q}_{\text{obs}}\), the observed average.
For the Leaf River test decade the model's squared errors add up to 5809 (mm/day)²; the lazy prediction's add up to 35228. The model removes all but 16.5% of the lazy prediction's error, so
| NSE | Meaning |
|---|---|
| 1 | perfect: the model matches the river every day |
| between 0 and 1 | better than predicting the average flow every day; the share of the average's squared error that the model removes |
| 0 | no better than predicting the average flow every day |
| below 0 | worse than that; there is no lower limit |
The lazy prediction uses the average of the very decade being scored, which nobody knows in advance. A forecaster who predicted the training decade's average flow every day would score −0.02. NSE = 0 is a low bar, but not a fair forecast.
Where the squared error comes from
Squaring makes big errors dominate. An error of 20 mm/day on a flood day counts as much as an error of 0.2 mm/day on ten thousand dry days. The chart sorts the days from the largest error to the smallest and adds up their share of the total.
The worst 1% of days (37 days in ten years) produce 64% of the squared error, and the worst 5% produce 88%. In 31 of those 37 days the model was too low. Days when the observed flow was among the highest tenth contribute 86%. NSE is mostly a score for floods. A model can get the floods right and the droughts badly wrong and still score well, which is what happens here (see the flow duration curve below).
The Kling–Gupta efficiency
Gupta et al. (2009) showed that the squared error can be split into three parts that answer three separate questions:
| Part | Formula | Question | Leaf River, test |
|---|---|---|---|
| correlation \(r\) | Pearson correlation of the two series | do the ups and downs happen at the right times? | 0.918 |
| variability ratio \(\alpha\) | \(\sigma_{\text{sim}} / \sigma_{\text{obs}}\) | are the swings the right size? | 0.861 |
| bias ratio \(\beta\) | \(\mu_{\text{sim}} / \mu_{\text{obs}}\) | is the total amount of water right? | 0.878 |
\(\sigma\) is the standard deviation (the typical size of the swings) and \(\mu\) the mean. A perfect model has \(r = \alpha = \beta = 1\).
The Kling–Gupta efficiency is one minus the distance from that perfect point:
Read the parts for the Leaf River: the timing is good (\(r\) = 0.92); the model's swings are 86% the size of the real ones; and it delivers 88% of the river's water. All three fall short of 1; the swing size and the volume a little more than the timing.
The same three parts sit inside NSE. Writing the squared error in terms of means, standard deviations and correlation gives
For a given timing \(r\), this is largest when \(\alpha = r\). Since \(r\) is always below 1, a model tuned to maximise NSE learns to make its swings a little too small, which means floods that are too low. The Leaf River model was tuned on NSE, and on its training period it has \(r\) = 0.936 and \(\alpha\) = 0.934: almost exactly equal, as the formula predicts. Part of the "peaks too low" pattern above is the objective, not the physics.
For NSE, predicting the average flow scores 0. For KGE the same lazy prediction scores about −0.41 (Knoben et al., 2019), so a KGE of 0 is not "no better than the average". Compare NSE with NSE and KGE with KGE.
Break the model on purpose
The quickest way to understand a score is to damage a simulation in a known way and watch the score respond. The widget changes the model's river after it has been computed: it can move it in time, scale its volume, or smooth it with a moving average. The model itself is not re-run.
Things to try:
- One day late. NSE falls from 0.835 to 0.678. A flood shown one day late is counted twice: once as missing water on the day of the flood and once as extra water on the next day. Only \(r\) changes; \(\alpha\) and \(\beta\) stay put.
- 10% more water. NSE rises to 0.841 and KGE to 0.896, because the model is too dry in this decade. Only \(\beta\) changes, along with \(\alpha\) (scaling also scales the swings). Do not take this as a fix; the section on training and test explains why.
- Smoothing over a week. Peaks drop and spread, \(\alpha\) falls, and NSE falls with it.
Low flows: the flow duration curve
To see the droughts that NSE ignores, sort the decade's daily flows from highest to lowest and plot each flow against the percentage of days on which it was exceeded. This flow duration curve forgets the dates and shows only how often each flow occurs: the left end is floods, the middle is ordinary days, the right end is drought.
| Flow exceeded on … | observed (mm/day) | model (mm/day) |
|---|---|---|
| 1% of days (floods) | 15.58 | 14.11 |
| 5% of days | 6.36 | 5.91 |
| 50% of days (the median) | 0.53 | 0.42 |
| 95% of days (drought) | 0.16 | 0.03 |
| 99% of days | 0.11 | 0.02 |
The high end agrees well. The low end does not: the flow the real river exceeds on 95% of days is 0.16 mm/day, the model's is 0.03, about five times smaller. In Session 7 terms, the model's aquifer runs down further in a long drought than the real catchment's storage does. NSE barely notices, because the squared errors on those days are tiny. An NSE computed on the logarithm of flow, which treats every factor of two equally whether the flow is large or small, is 0.63, against 0.84 for ordinary NSE.
A model is good for a purpose. The Leaf River parameters were tuned on NSE, so they are good for floods and ordinary flows. If low flows matter (water supply, river ecology), the model should be tuned and judged with a score that sees them.
Training and test
Any model with enough adjustable parameters can be made to fit a record well, including the record's measurement errors and its accidents. A fit says little about whether the model has learned how the catchment works. The test is to tune it on one period and judge it on another. This course follows the protocol shared by most CAMELS studies (Newman et al., 2017; Kratzert et al., 2019):
| Window | NSE | KGE | \(r\) | \(\alpha\) | \(\beta\) | rain (mm/yr) | river (mm/yr) | runoff ratio |
|---|---|---|---|---|---|---|---|---|
| training, 1999–2008 | 0.876 | 0.908 | 0.936 | 0.934 | 1.005 | 1475 | 400 | 0.27 |
| test, 1989–1999 | 0.835 | 0.797 | 0.918 | 0.861 | 0.878 | 1552 | 564 | 0.36 |
The training scores are higher, as they should be: the parameters were chosen to fit that period. Of the three parts, \(\beta\) changes most, from 1.005 to 0.878, and in KGE the swings and the volume contribute about equally to the drop. NSE, which weighs timing most, falls mainly because \(r\) fell: 83% of its drop comes from timing, 8% from the swings and 9% from the volume. The volume error has a clear cause. The two periods had different water balances: the test decade had 5% more rain than the training period but 41% more river flow, and its runoff ratio was 0.36 instead of 0.27. The parameters learned how much of the rain the river returned in the drier 2000s; in the 1990s the real catchment returned more. The yearly volume errors in the year viewer above tell the same story: the model is too dry in every year of the test decade.
This is why "10% more water" in the widget is not a fix. The 10% was read off the test decade's observed flows, which a model meant for the future would not have. Adjusting a model after looking at the test data turns the test into more training, and the score stops meaning anything.
A small training-to-test drop with a clear cause is a good result. It shows the model transfers to a different decade, and it locates the weakness (the volume when the climate shifts) precisely enough to act on.
Is 0.84 good?
A score means little alone. Two kinds of comparison give it meaning: simple baselines that need no model, and other models on the same river and decade.
The seasonal cycle alone scores 0.09. The Leaf River is higher in winter than in summer (Session 1), but on any particular day its flow depends on the storms of the past few days, which only a model of the catchment's response can follow. Everything between 0.09 and 0.84 is what the model adds.
The CAMELS benchmark lets us compare with other published models on exactly the same river and decade. Those models were run with a different precipitation dataset (Maurer et al., 2002, instead of the Daymet data used in this course), so the fair comparison uses the dCREST parameters calibrated with Maurer forcing, which score 0.824 rather than 0.835. The glossary says briefly what each of the other models is.
| Model | How it was calibrated | test NSE |
|---|
On this river every calibrated model scores between about 0.70 and 0.86, and the best conceptual models differ by a few hundredths, not much more than dCREST's own score changes (0.011) when the precipitation dataset changes. One catchment cannot rank models; the paper compares them across 528 catchments (Session 13). What one river can show is that dCREST behaves like an established conceptual model.
Two different SAC-SMA calibrations circulate for this benchmark. A (Newman et al., 2015) was calibrated on the first 15 years of record, 1980–1995, which include six of the ten test years; we pick, for each catchment, the best of its about ten runs on the 1999–2008 window, as for dCREST. B (as distributed by Kratzert et al., 2019, attributed to Newman et al., 2017) was calibrated on the training period like dCREST. A is the stronger reference but partly in-sample.
In Python
Every number in this session from scratch. The cell uses the stored run of the PyTorch package, which Session 9 showed is identical to the browser model.
Change one line and re-run: replace sim by 1.1 * sim after it is defined, or by np.roll(sim, 1) (one day late; the first day wraps around, which hardly matters), and watch which numbers move.
Check your understanding
A model predicts the test decade's average flow on every day. What is its NSE, and why is that not a fair forecast?
Exactly 0, by construction. It is not a fair forecast because nobody knows a decade's average flow before the decade happens; using the training period's average instead gives −0.02 here.
Why does a one-day timing error cost so much NSE?
The flood is counted twice: as missing water on the day it should have arrived and as extra water on the day it does arrive. Both errors are large and squared. On the Leaf River, delaying the whole simulation by one day lowers NSE from 0.84 to 0.68.
A model scores NSE = 0.84 but log-flow NSE = 0.63. What do you suspect?
Floods and ordinary flows are good, low flows are poor. Squared errors are dominated by high flows; logarithms weigh relative errors equally across the range. The flow duration curve confirms it: the model's drought flows are about five times too low.
Multiplying the simulation by 1.1 raises the test NSE. Why may you not do it?
The factor was found by looking at the test decade's observed flows. Using it turns the test into training, and the score no longer says how the model would do on years it has not seen.
Why do models tuned on NSE tend to produce floods that are too small?
Because NSE = 2αr − α² − (bias term)², which is largest at α = r. With imperfect timing (r < 1), the best NSE comes from swings smaller than the observed ones (α < 1).
What to remember
- Look at the hydrographs before the scores.
- NSE = 1 − (model's squared error) / (squared error of predicting the average). 0 means no better than the average. It is dominated by floods.
- KGE measures the distance from perfect timing (\(r\)), swing size (\(\alpha\)) and volume (\(\beta\)); its scale differs from NSE's.
- Low flows need their own check: the flow duration curve, or scores on log flows.
- The honest score is on years not used for tuning. On the Leaf River the test decade also exposed a volume error (\(\beta\) = 0.88), because it returned more of its rain to the river than the training period did.
- A score needs a baseline. On one river, good conceptual models differ by hundredths.