Calibration
dCREST has sixteen parameters, and none of them can be measured in the field: they describe the whole catchment as the model sees it. They are found by making the model reproduce the observed river. This session does that by hand first, then shows how a computer does it: the classic SCE-UA search, first demonstrated on the Leaf River, and the gradient-based search that dCREST was built for.
You will be able to
- explain why the parameters are calibrated rather than measured;
- tune parameters by hand, and say why sixteen at once is hard;
- read a loss landscape: valleys, ridges and flat directions;
- describe how SCE-UA and gradient descent (Adam) search;
- explain the logistic transform that keeps parameters inside their bounds;
- read the Leaf River calibration: loss curve, restarts and parameter paths;
- explain why six runs are made and how one of them is chosen.
Sixteen dials
Session 2 introduced the parameters. Some sound measurable: wm is a soil water capacity in millimetres, hmaxaq an aquifer capacity. But the model's wm is not the capacity of any soil sample. It is the value that makes one bucket behave like all the soils of a 1924 km² catchment together, with their depths, textures and slopes. Such effective parameters can only be found by asking which values make the simulated river look like the real one. That is calibration.
| Parameter | Meaning | Units | Allowed range | Default | Leaf River |
|---|
Each parameter may take any value in its allowed range (its bounds). The default is a generic starting value, the same for every catchment. The last column is the released calibration for the Leaf River, used throughout this course.
Calibrate by hand
Calibration needs three things: a period to fit (the training period, 1 October 1999 to 30 September 2008, after three years of spin-up), a score to maximise (NSE on that decade, Session 10), and a way to change the parameters. You are the third. The widget starts from the defaults.
With the defaults, the training NSE is 0.39. The Session 10 parts say what is wrong: the model returns 1.43 times the river's water (\(\beta\)), its swings are 60% of the real ones (\(\alpha\)), and its timing is poor (\(r\) = 0.66). Some hints, if you want them: more water must evaporate instead of draining to the aquifer (ksoil, gwc); the river responds more slowly than the defaults assume (kmult); the fast reservoir releases faster (qc).
You will find two things. First, no single dial gets far: the best any one parameter can do on its own, with the others at their defaults, is 0.52 (by moving gwc). Second, the dials interact. Press Reveal one dial at a time: it moves the dials to their Leaf River values one by one. Moving ksoil alone, to exactly its final value, drops the score from 0.39 to −0.01, because a soil that keeps its water needs routing that matches it. The score recovers only when kmult follows (0.70), and it even falls again when nshape moves before qc (0.78 → 0.72). The right value of each parameter depends on the others.
Calibration is a search in sixteen dimensions at once. Changing one parameter at a time is slow and gets stuck, because the parameters compensate for each other. A good search moves them together.
The landscape
Think of every possible set of sixteen values as a place, and of its training NSE as the height of the ground there. Calibration looks for the highest point. Nobody can see a sixteen-dimensional landscape, but we can take a slice: hold fourteen parameters at their Leaf River values and vary two. The map below runs the model about a thousand times to draw one slice.
The five slices show the shapes a calibration has to cope with:
- A single peak (wm and b): the soil capacity is pinned to within about 15% and the curve shape to within a factor of two; everything else is clearly worse. These two are well determined by the river.
- A diagonal ridge (ke and lf): both remove water, one by evaporation and one as a loss, so more of one can be balanced by less of the other. Many combinations along the ridge score almost the same. Different parameter sets giving equally good fits is called equifinality (Beven, 2006).
- Another ridge (kmult and qc): a slower unit hydrograph can be offset by a faster-draining fast reservoir. Both shape the timing of each flood, so the river pins down a combination of them more tightly than either one.
- One pinned, one free (ksoil and hmaxaq): the score changes sharply with the soil drainage rate but hardly at all with the aquifer size, which could be anything from about 40 to 700 mm.
- A flat plain (tsnow and ddf): the Leaf River has almost no snow, so the whole map lies within 0.01 of the best. The data cannot determine these parameters at all.
Ridges and plains are not faults of the optimiser. They are properties of the model and the data, and every search method meets them.
Searching without slopes: SCE-UA
Duan, Sorooshian and Gupta (1992) designed the shuffled complex evolution method, SCE-UA, for exactly this kind of landscape, and Sorooshian, Duan and Gupta (1993) demonstrated it by calibrating the Sacramento model on the Leaf River. It only ever asks for heights, never for slopes, so it works with any model. In outline:
- Scatter a population of points at random over the allowed ranges and score each one.
- Divide the population into groups called complexes.
- Let each complex evolve on its own for a while. Pick a few of its points, favouring the better ones, and try to improve the worst of them: move it through the centre of the others to the same distance on the far side. If that is no better, try halfway towards the centre; if that fails too, replace it with a random point.
- Shuffle: pool all complexes, sort by score and deal the points out again, so information found by one complex spreads to the others.
- Repeat until the population has gathered at one place or the budget of model runs is spent.
Press SCE-UA on the map above to watch it in two dimensions, with two complexes of five points each (Duan et al., 1994, recommend 2n + 1 points per complex for n parameters). The population closes in on the best region, then shrinks around it. On a ridge, notice how far apart its points can be while still scoring almost the same.
Searching with slopes: gradient descent and Adam
If you could feel the slope of the ground under your feet, you would simply walk uphill. The slope in every direction at once is the gradient: for sixteen parameters, sixteen numbers, each saying how fast the score changes when that parameter moves a little. Gradient descent takes a step in the direction that improves the score fastest, measures the slope again, and repeats. (By convention it descends the loss, here 1 − NSE, which is the same as climbing NSE.)
dCREST uses Adam (Kingma and Ba, 2015), a version of gradient descent with two additions. It keeps a running average of the recent gradients (momentum), so it rolls through small bumps instead of zig-zagging. And it divides each parameter's step by the typical size of that parameter's recent gradients, so parameters with very steep and very shallow slopes move at a similar pace. With a gradient \(g_t\) at step \(t\):
\(\hat m_t\) and \(\hat v_t\) are \(m_t\) and \(v_t\) corrected for starting at zero; \(\eta\), the learning rate, sets the step length.
Press Adam on the map to compare. In this demonstration the slope is measured by nudging each parameter and re-running the model (finite differences), which costs extra runs per step. dCREST obtains all sixteen slopes from one backward pass through the computation instead (Session 12).
Gradient descent needs far fewer model runs, but it climbs the nearest hill; on a landscape with several peaks, where it ends depends on where it starts. SCE-UA explores the whole range with a population and is less likely to get trapped, at the price of many more runs. The usual remedy for gradient descent is to restart it from several places, which dCREST does.
Staying inside the bounds
Gradient descent knows nothing about bounds; a step could carry lf below 0 or qc above 1, where the model makes no physical sense. dCREST never lets the optimiser touch the parameter directly. It works on an unbounded number \(u\) and maps it into the allowed range with the logistic function:
Whatever value \(u\) takes, \(\theta\) stays strictly inside its range. The cost is that the slope seen by the optimiser is multiplied by \(d\theta/du = (\theta_{\max} - \theta_{\min})\,\sigma(u)\,[1 - \sigma(u)]\), which shrinks towards zero near either bound.
Two consequences follow. A parameter can approach a bound but never reach it exactly. And a parameter close to a bound moves very slowly: within 0.1% of the range from a bound, \(|u|\) is above 6.9 and the slope reaching the optimiser is less than 0.4% of what it is mid-range.
All sixteen at once, in your browser
Now the real problem: Adam on all sixteen parameters, from the defaults, with the logistic transform. The slopes are again measured by nudging each parameter (17 model runs per step); on a desktop computer the 150 steps take a few seconds.
In about thirty steps Adam takes the training NSE from 0.39 past 0.8, and after all 150 steps, a few seconds of computing, it is near 0.86. The parameters do not all end at the released values, and some barely move at all; the next two sections show why both are normal. The real calibration below runs 2,048 steps and gets its slopes by automatic differentiation rather than by nudging.
The real calibration of the Leaf River
The released parameters came from the dCREST package's own calibration routine, with gradients from automatic differentiation. Its settings, used for all 665 catchments:
- Loss: 1 − NSE on the training period. (Across many catchments the observed variance in the NSE formula is floored at 0.1 mm²/day² so that a few nearly flat rivers cannot dominate; for the Leaf River the floor never applies.)
- 2,048 Adam steps in four phases of 512. The first phase starts from the starting values; each of the other three restarts from the best parameters found so far, nudged by random noise that shrinks from restart to restart (0.6, 0.4 and 0.25 standard deviations of the unbounded \(u\) introduced above). The learning rate starts each phase at 0.05, 0.03, 0.02 and 0.012 respectively and decays smoothly to a tenth of that, and Adam's running averages start afresh.
- Keep the best: at every step, if the score just computed is the best yet, store the parameters that produced it. The released parameters are the best ones seen, not the last ones (but see the box below).
The trace below is a re-run of this procedure for the Leaf River alone, from the default starting values, recording every step.
"Store the parameters that produced the best score" has to happen before the step that moves them. Version 1.0.0 of the package stored them one step too late, so it kept the parameters one Adam step after the best ones. The released sets were made with 1.0.0. Tracking both versions of the record along the same calibration of all 665 catchments changed test NSE by a median of 1 × 10⁻⁵, by more than 0.001 in 26 catchments and by more than 0.01 in one; two repeated runs of the same configuration differ by more than 0.01 in 33 catchments. Version 1.0.1 fixes it, with a test that the returned parameters reproduce the returned score (to within 10−6). The released scores were always computed from the released parameters themselves, so they are consistent.
Six runs, one choice
Because gradient descent climbs the nearest hill, each catchment is calibrated six times: once from the defaults and five times from randomly perturbed starting values. The released set takes, for each catchment, the run with the best training score. The test decade plays no part in the choice.
| Run | Start | training NSE | test NSE |
|---|
Five of the six runs end within 0.002 of each other on the training period. One (run 4) stopped lower, at 0.839, on a different hill. The five near-equivalent runs score between 0.835 and 0.844 on the test decade. The chosen run (run 3) happens to be the lowest of them on the test decade: run 1 would have scored 0.844 instead of 0.835. But choosing by the test score would be peeking (Session 10); the honest choice accepts that equally good fits generalise slightly differently.
On the Leaf River, most parameters land in nearly the same place in all five equally good Adam runs: the soil (wm, b, ksoil), evaporation (ke), aquifer size (hmaxaq), losses (lf) and routing (qc, nshape, kmult). The two snow parameters scatter across their ranges: with almost no snow, the river's flow cannot pin them down (the flat plain on the landscape above), so they keep whatever values the search left them at. The baseflow parameters gwc and gwe and the tiny impervious fraction im also vary between runs.
SCE-UA agrees with Adam on the soil, evaporation and aquifer-size parameters, but takes a different route through the interflow: it puts fc near its upper bound, which switches the interflow cap off (Session 6), and fits as well with different baseflow and routing values. The same excess water reaches the river by a different path, and the training scores are almost the same. The paper found this in many catchments and reports, parameter by parameter, how far the released values can be trusted.
Adam and SCE-UA on the same problem
The paper calibrated all 665 catchments a second time with SCE-UA, changing nothing but the optimiser: the same ranges, loss, training data and selection rule. The chart shows the Leaf River.
On this river Adam reached a training NSE of 0.876 and SCE-UA 0.872 and 0.866 (two seeds) after 37,996 and 37,149 model runs. Across all 665 catchments (paper, Sect. 4.3) the two optimisers reach the same median test skill (NSE 0.662 for Adam, 0.661 for SCE-UA with eight complexes). Counted in model runs, Adam is three to four times cheaper. Counted in wall-clock time on one GPU, SCE-UA was faster, because it can evaluate dozens of candidate points in one batched run while each Adam step must also run the computation backwards. SCE-UA's parameters were also more reproducible from seed to seed. Neither optimiser produced a better model; they are two good ways to find the same level of fit.
In Python
A one-dimensional slice through the landscape, and the slope at its top. It runs the model eleven times over twelve years, which takes a few seconds in the browser.
Try another parameter: replace "kmult" and its range by "lf" and np.linspace(0, 0.3, 9), or by "ddf" and np.linspace(1, 8, 9). A flat line means the data cannot tell the values apart.
Check your understanding
Why can't wm simply be measured in a soil pit?
The model's wm is an effective value for a single bucket standing in for all the soils of a 1924 km² catchment. No sample measures that; it is defined by how the model behaves, so it is found by calibration.
Moving ksoil to its final calibrated value made the score worse. How can a "correct" value hurt?
It is correct only together with the other fifteen. With the default routing, a soil that drains slowly produces a river that responds in the wrong way; once kmult and the other routing parameters follow, the same ksoil is part of the best set.
What does a long ridge on the landscape mean for the parameter values you report?
Many combinations along the ridge fit almost equally well, so the data do not determine the individual values, only a combination of them. Values reported from such a ridge depend on where the search happened to stop.
Why does dCREST optimise \(u\) rather than the parameter itself?
The logistic transform maps any \(u\) into the allowed range, so the optimiser can take any step without producing a physically meaningless value. The price is that slopes shrink near the bounds.
Six runs reach nearly the same training NSE. Why not release the one with the best test score?
Because the test decade must not influence any decision; otherwise the test score would no longer measure performance on unseen years.
What to remember
- Parameters are effective, catchment-scale values; calibration chooses them so the simulated river matches the observed one on the training period.
- Parameters interact. One-at-a-time tuning stalls; the search has to move them together.
- Landscapes have peaks, ridges (equifinality) and flat directions (parameters the data cannot determine).
- SCE-UA searches with a population and needs only scores; Adam follows the gradient and needs far fewer runs but restarts.
- The logistic transform keeps parameters in bounds and makes them sluggish near the bounds.
- Six runs, the best on training; equally good fits can still differ in their parameters and in their test scores.