Session 11 · Learning from data

Calibration

dCREST has sixteen parameters, and none of them can be measured in the field: they describe the whole catchment as the model sees it. They are found by making the model reproduce the observed river. This session does that by hand first, then shows how a computer does it: the classic SCE-UA search, first demonstrated on the Leaf River, and the gradient-based search that dCREST was built for.

Time about 50 minutesNeeds Sessions 9 and 10
Watch this session first: a 9-minute narrated video with English subtitles. The page below goes further and lets you try each step yourself.

You will be able to

Sixteen dials

Session 2 introduced the parameters. Some sound measurable: wm is a soil water capacity in millimetres, hmaxaq an aquifer capacity. But the model's wm is not the capacity of any soil sample. It is the value that makes one bucket behave like all the soils of a 1924 km² catchment together, with their depths, textures and slopes. Such effective parameters can only be found by asking which values make the simulated river look like the real one. That is calibration.

ParameterMeaningUnitsAllowed rangeDefaultLeaf River

Each parameter may take any value in its allowed range (its bounds). The default is a generic starting value, the same for every catchment. The last column is the released calibration for the Leaf River, used throughout this course.

Calibrate by hand

Calibration needs three things: a period to fit (the training period, 1 October 1999 to 30 September 2008, after three years of spin-up), a score to maximise (NSE on that decade, Session 10), and a way to change the parameters. You are the third. The widget starts from the defaults.

Try itSixteen dials, one score

Every move re-runs the model over twelve years (spin-up and training period) and scores the training period. Sliders for wide ranges are logarithmic; for im, fc and ksoil the left end is a small positive number rather than exactly zero.

With the defaults, the training NSE is 0.39. The Session 10 parts say what is wrong: the model returns 1.43 times the river's water (\(\beta\)), its swings are 60% of the real ones (\(\alpha\)), and its timing is poor (\(r\) = 0.66). Some hints, if you want them: more water must evaporate instead of draining to the aquifer (ksoil, gwc); the river responds more slowly than the defaults assume (kmult); the fast reservoir releases faster (qc).

You will find two things. First, no single dial gets far: the best any one parameter can do on its own, with the others at their defaults, is 0.52 (by moving gwc). Second, the dials interact. Press Reveal one dial at a time: it moves the dials to their Leaf River values one by one. Moving ksoil alone, to exactly its final value, drops the score from 0.39 to −0.01, because a soil that keeps its water needs routing that matches it. The score recovers only when kmult follows (0.70), and it even falls again when nshape moves before qc (0.78 → 0.72). The right value of each parameter depends on the others.

Key idea

Calibration is a search in sixteen dimensions at once. Changing one parameter at a time is slow and gets stuck, because the parameters compensate for each other. A good search moves them together.

The landscape

Think of every possible set of sixteen values as a place, and of its training NSE as the height of the ground there. Calibration looks for the highest point. Nobody can see a sixteen-dimensional landscape, but we can take a slice: hold fourteen parameters at their Leaf River values and vary two. The map below runs the model about a thousand times to draw one slice.

Try itA slice through the landscape
Search from the white cross (click the map to move it):

Colour: training NSE with the other fourteen parameters at their Leaf River values. Outlined cells are within 0.01 of the best value on this slice. Ring: the Leaf River values. Hover to read the map.

The five slices show the shapes a calibration has to cope with:

Ridges and plains are not faults of the optimiser. They are properties of the model and the data, and every search method meets them.

Searching without slopes: SCE-UA

Duan, Sorooshian and Gupta (1992) designed the shuffled complex evolution method, SCE-UA, for exactly this kind of landscape, and Sorooshian, Duan and Gupta (1993) demonstrated it by calibrating the Sacramento model on the Leaf River. It only ever asks for heights, never for slopes, so it works with any model. In outline:

  1. Scatter a population of points at random over the allowed ranges and score each one.
  2. Divide the population into groups called complexes.
  3. Let each complex evolve on its own for a while. Pick a few of its points, favouring the better ones, and try to improve the worst of them: move it through the centre of the others to the same distance on the far side. If that is no better, try halfway towards the centre; if that fails too, replace it with a random point.
  4. Shuffle: pool all complexes, sort by score and deal the points out again, so information found by one complex spreads to the others.
  5. Repeat until the population has gathered at one place or the budget of model runs is spent.

Press SCE-UA on the map above to watch it in two dimensions, with two complexes of five points each (Duan et al., 1994, recommend 2n + 1 points per complex for n parameters). The population closes in on the best region, then shrinks around it. On a ridge, notice how far apart its points can be while still scoring almost the same.

Searching with slopes: gradient descent and Adam

If you could feel the slope of the ground under your feet, you would simply walk uphill. The slope in every direction at once is the gradient: for sixteen parameters, sixteen numbers, each saying how fast the score changes when that parameter moves a little. Gradient descent takes a step in the direction that improves the score fastest, measures the slope again, and repeats. (By convention it descends the loss, here 1 − NSE, which is the same as climbing NSE.)

dCREST uses Adam (Kingma and Ba, 2015), a version of gradient descent with two additions. It keeps a running average of the recent gradients (momentum), so it rolls through small bumps instead of zig-zagging. And it divides each parameter's step by the typical size of that parameter's recent gradients, so parameters with very steep and very shallow slopes move at a similar pace. With a gradient \(g_t\) at step \(t\):

\[ m_t = 0.9\, m_{t-1} + 0.1\, g_t, \qquad v_t = 0.999\, v_{t-1} + 0.001\, g_t^2, \qquad \theta_{t+1} = \theta_t - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t} + 10^{-8}} \]

\(\hat m_t\) and \(\hat v_t\) are \(m_t\) and \(v_t\) corrected for starting at zero; \(\eta\), the learning rate, sets the step length.

Press Adam on the map to compare. In this demonstration the slope is measured by nudging each parameter and re-running the model (finite differences), which costs extra runs per step. dCREST obtains all sixteen slopes from one backward pass through the computation instead (Session 12).

What the two methods trade

Gradient descent needs far fewer model runs, but it climbs the nearest hill; on a landscape with several peaks, where it ends depends on where it starts. SCE-UA explores the whole range with a population and is less likely to get trapped, at the price of many more runs. The usual remedy for gradient descent is to restart it from several places, which dCREST does.

Staying inside the bounds

Gradient descent knows nothing about bounds; a step could carry lf below 0 or qc above 1, where the model makes no physical sense. dCREST never lets the optimiser touch the parameter directly. It works on an unbounded number \(u\) and maps it into the allowed range with the logistic function:

\[ \theta = \theta_{\min} + (\theta_{\max} - \theta_{\min})\,\sigma(u), \qquad \sigma(u) = \frac{1}{1 + e^{-u}} \]

Whatever value \(u\) takes, \(\theta\) stays strictly inside its range. The cost is that the slope seen by the optimiser is multiplied by \(d\theta/du = (\theta_{\max} - \theta_{\min})\,\sigma(u)\,[1 - \sigma(u)]\), which shrinks towards zero near either bound.

Try itThe logistic transform

The optimiser moves \(u\) freely; the model sees \(\theta\). Near a bound, a large change in \(u\) moves \(\theta\) very little.

Two consequences follow. A parameter can approach a bound but never reach it exactly. And a parameter close to a bound moves very slowly: within 0.1% of the range from a bound, \(|u|\) is above 6.9 and the slope reaching the optimiser is less than 0.4% of what it is mid-range.

All sixteen at once, in your browser

Now the real problem: Adam on all sixteen parameters, from the defaults, with the logistic transform. The slopes are again measured by nudging each parameter (17 model runs per step); on a desktop computer the 150 steps take a few seconds.

In about thirty steps Adam takes the training NSE from 0.39 past 0.8, and after all 150 steps, a few seconds of computing, it is near 0.86. The parameters do not all end at the released values, and some barely move at all; the next two sections show why both are normal. The real calibration below runs 2,048 steps and gets its slopes by automatic differentiation rather than by nudging.

The real calibration of the Leaf River

The released parameters came from the dCREST package's own calibration routine, with gradients from automatic differentiation. Its settings, used for all 665 catchments:

The trace below is a re-run of this procedure for the Leaf River alone, from the default starting values, recording every step.

Replay2,048 steps of Adam on the Leaf River

Top left: training NSE at each step and the best so far; dots mark the three restarts. Top right: the parameters at the chosen step. Bottom: water year 2003, simulated with the parameters of the chosen step.

A bug, found and fixed

"Store the parameters that produced the best score" has to happen before the step that moves them. Version 1.0.0 of the package stored them one step too late, so it kept the parameters one Adam step after the best ones. The released sets were made with 1.0.0. Tracking both versions of the record along the same calibration of all 665 catchments changed test NSE by a median of 1 × 10⁻⁵, by more than 0.001 in 26 catchments and by more than 0.01 in one; two repeated runs of the same configuration differ by more than 0.01 in 33 catchments. Version 1.0.1 fixes it, with a test that the returned parameters reproduce the returned score (to within 10−6). The released scores were always computed from the released parameters themselves, so they are consistent.

Six runs, one choice

Because gradient descent climbs the nearest hill, each catchment is calibrated six times: once from the defaults and five times from randomly perturbed starting values. The released set takes, for each catchment, the run with the best training score. The test decade plays no part in the choice.

RunStarttraining NSEtest NSE

Five of the six runs end within 0.002 of each other on the training period. One (run 4) stopped lower, at 0.839, on a different hill. The five near-equivalent runs score between 0.835 and 0.844 on the test decade. The chosen run (run 3) happens to be the lowest of them on the test decade: run 1 would have scored 0.844 instead of 0.835. But choosing by the test score would be peeking (Session 10); the honest choice accepts that equally good fits generalise slightly differently.

Where the equally good fits put each parameter. Each row is one parameter's allowed range (logarithmic for wide ranges). Circles: the six Adam runs (dark: the chosen one; light: the four other equally good runs; hollow: the run that stopped lower). Diamonds: two independent SCE-UA calibrations of the same model in the same ranges.

On the Leaf River, most parameters land in nearly the same place in all five equally good Adam runs: the soil (wm, b, ksoil), evaporation (ke), aquifer size (hmaxaq), losses (lf) and routing (qc, nshape, kmult). The two snow parameters scatter across their ranges: with almost no snow, the river's flow cannot pin them down (the flat plain on the landscape above), so they keep whatever values the search left them at. The baseflow parameters gwc and gwe and the tiny impervious fraction im also vary between runs.

SCE-UA agrees with Adam on the soil, evaporation and aquifer-size parameters, but takes a different route through the interflow: it puts fc near its upper bound, which switches the interflow cap off (Session 6), and fits as well with different baseflow and routing values. The same excess water reaches the river by a different path, and the training scores are almost the same. The paper found this in many catchments and reports, parameter by parameter, how far the released values can be trusted.

Adam and SCE-UA on the same problem

The paper calibrated all 665 catchments a second time with SCE-UA, changing nothing but the optimiser: the same ranges, loss, training data and selection rule. The chart shows the Leaf River.

Training NSE against computational effort, Leaf River. SCE-UA (eight complexes, two random seeds) is plotted against model runs. Adam's best-so-far is plotted against model-run equivalents: on the GPU used for the paper, one Adam step (a model run plus its gradient) cost as much as 7.5 model runs. Logarithmic horizontal axis.

On this river Adam reached a training NSE of 0.876 and SCE-UA 0.872 and 0.866 (two seeds) after 37,996 and 37,149 model runs. Across all 665 catchments (paper, Sect. 4.3) the two optimisers reach the same median test skill (NSE 0.662 for Adam, 0.661 for SCE-UA with eight complexes). Counted in model runs, Adam is three to four times cheaper. Counted in wall-clock time on one GPU, SCE-UA was faster, because it can evaluate dozens of candidate points in one batched run while each Adam step must also run the computation backwards. SCE-UA's parameters were also more reproducible from seed to seed. Neither optimiser produced a better model; they are two good ways to find the same level of fit.

In Python

A one-dimensional slice through the landscape, and the slope at its top. It runs the model eleven times over twelve years, which takes a few seconds in the browser.

Try another parameter: replace "kmult" and its range by "lf" and np.linspace(0, 0.3, 9), or by "ddf" and np.linspace(1, 8, 9). A flat line means the data cannot tell the values apart.

Check your understanding

Why can't wm simply be measured in a soil pit?

The model's wm is an effective value for a single bucket standing in for all the soils of a 1924 km² catchment. No sample measures that; it is defined by how the model behaves, so it is found by calibration.

Moving ksoil to its final calibrated value made the score worse. How can a "correct" value hurt?

It is correct only together with the other fifteen. With the default routing, a soil that drains slowly produces a river that responds in the wrong way; once kmult and the other routing parameters follow, the same ksoil is part of the best set.

What does a long ridge on the landscape mean for the parameter values you report?

Many combinations along the ridge fit almost equally well, so the data do not determine the individual values, only a combination of them. Values reported from such a ridge depend on where the search happened to stop.

Why does dCREST optimise \(u\) rather than the parameter itself?

The logistic transform maps any \(u\) into the allowed range, so the optimiser can take any step without producing a physically meaningless value. The price is that slopes shrink near the bounds.

Six runs reach nearly the same training NSE. Why not release the one with the best test score?

Because the test decade must not influence any decision; otherwise the test score would no longer measure performance on unseen years.

What to remember