Interactive comment on “ Technical note : Improving the AWAT filter with interpolation schemes for advanced processing of high resolution data

Detailed comments: 1. P2 L20-23: need not to be discussed in this paper, but suggestion for future work: have you looked at routines used to process GNSS (GPS, GLONASS) time series, where identifying steps is challenging as well? 2. P2 L25: “if the signal strength is high” . . .”noise is high”: could you quantify? 3. P3 L4: the flux is zero: but what happens if the distance between anchors is reduced? 4. P5 l14: what


Introduction
Precipitation (P (L T −1 )) and evapotranspiration (ET (L T −1 )) have to be precisely known to answer many questions regarding water, solute and energy fluxes in the soilplant atmosphere continuum.In several simulation studies, the precise values for P and ET are required only as daily averages (e.g.Schelle et al., 2012).However, in other cases the diurnal course of P and ET must be known, e.g.whether root water uptake shall be simulated with a physically based model (Javaux et al., 2008;Couvreur et al., 2012) or macropore flow due to heavy but short precipitation events be simulated under realistic conditions (Malone et al., 2004;Mc-Grath et al., 2008).
Today, weighing lysimeter measurements with a high mass and temporal resolution yield the most precise values for both P and ET.This is since systematic as well as random errors are largely eliminated, the former due to their installation height exactly at ground surface and the latter due to the relatively large size in comparison to other devices.The high temporal resolution of the measurement is required to distinguish between P and ET, which might follow each other even in small time intervals.
The mass resolution of the lysimeter can be as high as 0.01 mm for modern weighing systems (von Unold and Fank, 2008) and can even be used for dew fall measurements (Meissner et al., 2007).With such high resolutions, small disturbances, e.g.due to wind, are visible in the data as noise (Nolz et al., 2013) and must be eliminated before the data can be interpreted (Fank, 2013;Schrader et al., 2013;Peters et al., 2014).Moreover, the disturbance, and thus the accuracy, of the system depends on wind speed and is therefore not constant but time variable.After elimination of the measurement noise with appropriate filter routines each increase A. Peters et al.: Improving the AWAT filter with interpolation schemes in system mass is interpreted as precipitation and each decrease as evapotranspiration.
As already suggested by Fank (2013) and Schrader et al. (2013), such filter routines can be carried out in two steps.First a smoothing routine (for example a simple moving average) with a certain window width w (T) is applied and second all changes of the smoothed data smaller than a predefined threshold value δ (L) are discarded.The second step is mandatory to avoid small changes in the smoothed data being interpreted as P and ET.Schrader et al. (2013) showed that there are no "ideal" values for w or δ within a longer time interval because at some events small values for w and δ are required, whereas at other events high values for w or δ are required to get the maximum information content from the data.
Therefore, Peters et al. (2014) suggested the so-called AWAT (Adaptive Window Adaptive Threshold) filter.The innovation in the AWAT filter consists in the variability of w and δ, which are adjusted according to the characteristics of the measured data.If the signal strength is high (e.g.due to precipitation), w gets small, and if signal strength is low, w gets large.Similarly, if noise is high, δ gets large, and if it is low, δ gets small.The AWAT filter was successfully applied in recent studies (Gebler, et al., 2015;Hannes et al., 2015;Hoffmann et al., 2016).
The threshold approach makes sure that significant weight changes are separated from insignificant changes and leads to a step-like course of the calculated cumulative upper boundary flux (see Fig. 6 in Schrader et al., 2013, or Figs. 6 and7 in Peters et al., 2014).The points in time at which the steps occur can be called anchor points and all other points are mere interpolated data.
ET and P are given as the first derivatives of the cumulative upper boundary flux and are commonly required as the mean for an application-specific time interval.Since the time span between two anchor points is usually much smaller than 1 day, the step interpolation scheme gives fairly good results if only daily resolution is required.However, if the required time interval for the upper boundary flux is much smaller than the time span between the anchor points (e.g. 1 h or even 10 min), the step interpolation yields unrealistic values: at time intervals between two subsequent anchor points the calculated flux is zero.If a time interval comprises one anchor point, the calculated flux is large.Moreover, the magnitude of the flux depends on the length of the chosen time interval since the step occurs immediately.Using such data will probably lead to erroneous simulations and also to numerical problems due to abrupt changes in the boundary conditions, with high fluxes alternating with no fluxes.
Note that the step scheme with the abrupt changes directly reflects the resolution of the system.If no further assumptions about the underlying process are justified, this is the maximum information which can be derived from the measuring set-up.However, many flux processes at the interface between the soil-plant system and the atmosphere, such as ET or dew fall, are known to be rather smooth and continuous than abrupt.
The aim of this contribution is (i) to show the impact of the step interpolation scheme on calculated fluxes for different time intervals and (ii) to improve the AWAT filter by eliminating the above mentioned problems using linear or cubic Hermitian spline interpolation schemes between the anchor points.This leads to a smoothing of the steps but guarantees that the cumulated fluxes are still exactly the same as in the original approach.
2 Material and methods

Lysimeter set-up
The measurements were conducted at the Berlin-Marienfelde (52.396731 • N, 13.367524 • E) lysimeter station.The lysimeter was a so-called grass-reference lysimeter with simulated groundwater depth at 1.3 m.It was 1.5 m deep with a surface area of 1 m 2 .A lever-arm counterbalance system was combined with a laboratory scale, which resulted in an overall resolution of the system of 100 g, which corresponds to approximately 0.1 mm for the upper boundary fluxes.The outflow/inflow of water at the lower boundary was directly recorded with a scale with a resolution of 5 g.The data were logged in a 1 min time interval.
The soil material was a packed silt loam taken from a Haplic Phaeozem, which ensures good capillary connection between groundwater level and the root system.The 20 cm bottom layer consisted of fully water saturated gravel.The 12 cm high grass on the lysimeters was a mixture of Lolium perenne, Festuca arundinacea and Poa pratensis, three coolseason grass species with large rooting depths.

Data processing
The data for this study were recorded from 1 January to 5 August 2014 (Fig. 1).Between 2 and 8 April no data were available due to malfunction of the lysimeter scale.In order to evaluate the interpolation schemes, we focussed on three time intervals: (i) 16-17 February 2014, representing very low evaporation rates, (ii) 30-31 Mai 2014, representing high evaporation rates, and (iii) 7 July 2014 between 13:30 and 15:30, representing the start of a heavy rainfall event.
Note that the "sawtooth" shape of the first subplot is caused by the two scales with different resolution.If outflow at the lower boundary occurs, each 5 g outflow is recorded in the data, leading to an apparent increase in cumulative outflow.If approximately 100 g flew out, the lysimeter scale records an apparent decrease in cumulative outflow again of 100 g.This is repeated and sometimes superimposed by a real signal like ET or P .

Threshold and interpolation schemes
The complete filter scheme is given in detail in Peters et al. (2014) and is therefore not explained here.The filter was applied using a minimum window width of 1 min, a maximum window width of 31 min, a minimum threshold value of 0.1 mm, and a maximum threshold value of 0.24 mm.

Step interpolation scheme
After the moving average (MA) is calculated, the threshold routine distinguishes between significant and insignificant mass changes starting with the first value of the MA at t = 0, which might be called the first anchor point ap 0 .This value is kept for all subsequent time steps until the difference between the corresponding value of the MA and the anchor point ap 0 is greater than the threshold value δ.Then, the new value is the next anchor point ap 1 (see Fig. 2 for illustration).This leads to a stepwise course of the calculated cumulative upper boundary flux.
All values between the anchor points can be regarded as interpolated values, whereas the anchor points coincide exactly with the MA.This procedure guarantees that small oscillations, which occur even after smoothing the data, will not be regarded as real mass changes and thus interpreted as evapotranspiration or precipitation.

Linear and spline interpolation schemes
In order to prevent the problems discussed above, which arise from the step scheme for the upper boundary flux, alternative interpolation schemes can be used.The simplest way is to calculate a linear interpolation between two subsequent anchor points.An alternative is the use of piecewise Hermitian splines (Fritsch and Carlson, 1980), which smooth the time course of the upper flux but do not oscillate like simple splines.Cubic Hermitian splines are frequently used in soil hydrology, e.g. for the description of hydraulic functions (Iden and Durner, 2007) or for temporal interpolation of mea-sured values in evaporation experiments (Peters and Durner, 2008;Peters et al., 2015).In contrast to the linear interpolation scheme, the spline interpolation yields a smooth curve at the anchor points and is thus even continuously differentiable.
Such interpolation schemes reflect smooth processes with small changes in small time intervals like evapotranspiration.However, for abrupt changes like rain events, such an interpolation might smooth the data too much and thus lead to unrealistic results again.If, for example, a heavy rain event occurs directly after a longer time with neither evapotranspiration nor precipitation, two subsequent anchor points might comprise a long time interval and have very different mass values.Then, the new interpolation schemes would yield a low rain intensity for a prolonged time instead of no flux in most of the time interval and a strong rain at the end.This problem is solved by only allowing the interpolations outlined above for mass decreases (evapotranspiration) or if the mass increase from one to the other anchor point is less than a defined value, e.g.aδ, where a must be greater than 1.The latter allows very small precipitation events like dew fall to be smoothed as well.Thus, the step interpolation between two anchor points is kept only if the mass change M > aδ, which comprises all sorts of medium to strong precipitation events.We refer to δ when selecting this scheme because δ defines the resolution of the system so that mass changes larger than δ between two anchor points indicate strong signals, which are typical for precipitation events.The parameter a must be larger than 1 but should not be too large to prevent medium precipitation from being smoothed unfavourably.We chose a = 1.1 heuristically, meaning that the mass difference must be at least 10 % larger than the system resolution at the specific time.As stated above, the step interpolation scheme directly reflects the resolution of the measurement system and is therefore the final part of a mere data evaluation process.Using the suggested two interpolation schemes is the first step towards data interpretation.The linear interpolation scheme as well as the cubic Hermitian spline interpolation routine of Fritsch and Carlson (1980) were implemented in the AWAT code (Peters et al., 2014).In this study all three interpolation schemes (steps, linear, splines) with a = 1.1 for the linear and spline interpolations are applied and compared.In order to test the importance of the rain correction, we additionally applied the linear and spline interpolation schemes without rain correction, setting a to the very high value of 9999 (linear * , spline * ).This guaranteed that the criterion M > aδ is never met.
The fluxes were calculated for time intervals of 1 day, 1 h, and 10 min.The calculated evapotranspiration rates for the three different schemes and time intervals were then compared for the two time spans at 16-17 February 2014 and 30-31 May 2014.The performance of the different schemes, including linear * and spline * , with respect to precipitation following a time with low fluxes was compared for the time span on 7 July 2014 between 13:30 and 15:30.Finally, the biases of the different schemes were compared for the complete data set by analysing the residuals between filtered and measured data.

Definition of bias term
The time series of observations (O) can be decomposed as signal and noise: where R are the unknown real values and N is the noise.Then the filtered and interpolated time series F (as described above) is given by where MA is the moving average time series.By definition the bias of where E is the linear expected value operator.Considering Eq. ( 1) yields Note that the bias of the first filter step (MA) is given by E(N ) = 0 means that wind and other disturbing factors do not have any significant systematic effects, and E(MA) − E(O) = 0 means that the MA does not lead to systematic deviations between smoothed data and observations.The latter is only given for (i) very small signals, i.e. if the real values (R) in the time window w are very similar, or (ii) if w is small, which is the case for the AWAT filter when signals are strong.Thus these assumptions are reasonable and allow one to use the distribution of residuals between the mere MA and raw data as a reference for the distribution of residuals between interpolated data and raw data.

Effect on temporal course of cumulative upper flux
Figure 2 shows the raw data together with the original filter scheme (step) as well as the results of the two other interpolation schemes (linear, spline) for 2 days with low (left) and high (right) evapotranspiration rates.On 16 and 17 February, the evapotranspiration rates were approximately 0. original routine leads to larger differences between interpolated and MA smoothed values.The differences increase with increasing time between two anchor points and with increasing time from the last anchor point.Moreover, this interpolation scheme leads to single, very high changes at the steps and no fluxes during the other time periods, which is especially problematic at low evapotranspiration rates, e.g. at night (see the step in the upper subplot in Fig. 2, right) or in winter (Fig. 2, left), where the continuously low ET fluxes of several hours are lumped into one single step.
Both the linear and spline interpolations lead to smoothed cumulative fluxes, closer to the MA values (Fig. 2).The differences between linear and spline interpolated cumulative fluxes are negligible except that the spline interpolation leads to slightly more smoothing.The different schemes will have an influence on calculated fluxes for small time intervals, as will be shown next.

One-day versus one-hour intervals
If the required temporal resolution is only 1 day, the original AWAT filter routine with step interpolation yields sufficient results, since the time intervals between two anchor points are much smaller than 1 day.The resulting evapotranspiration rates are shown as grey bars in Fig. 3.However, if the required resolution is 1 h, the original step interpolation scheme yields very unrealistic fluxes, especially if potential ET is low (e.g. during night time, or in winter).If a step occurs within an interval, the calculated flux is high, otherwise the flux is zero (Fig. 3, top).The calculated ET reaches a maximum of 15 mm d −1 in May and approximately 2.5 mm d −1 in February.
The linear (Fig. 3, centre) or spline (Fig. 3, bottom) interpolation schemes lead to smooth and more realistic evapotranspiration prediction.During day time both schemes yield comparable results.However, during night time, the linear scheme predicts small constant ET between two anchor points, whereas the spline scheme predicts a decreasing course until the inflection point between two anchor points is reached, followed by increasing ET again.

Ten-minute intervals
The unrealistic prediction of ET with the original scheme is even more pronounced if the required time interval gets smaller.For an interval of 10 min, the calculated ET can get as high as 35 mm d −1 in May and is still 15 mm d −1 in February or even zero during day time in May (Fig. 4,top).Thus, the fluxes occur not only erratically, but the magnitude of the fluxes within one time interval also depends on the selected time interval.This is avoided by the linear or spline interpolation schemes, where the maximum fluxes have roughly the same magnitude for either 1 h or 10 min intervals (Figs. 3 and 4, centre and bottom).Thus, the proposed interpolation schemes allow a more realistic simulation with very high temporal resolution of upper boundary fluxes using lysimeter data, which is important for many physically based studies.Moreover, since precipitation might occur suddenly with very high fluxes in very short time intervals, selecting such small intervals is important for many simulation studies regarding a realistic expression of precipitation.Only with the new interpolation schemes can such precipitation events be described in combination with evapotranspiration events within the same temporal resolution.

Analysing residuals
Figure 5 shows the frequency distribution of the residuals between filtered and measured data.The blue bars show the residuals for the case without a threshold value, i.e. for the sole MA, and are thus the same for all three compared schemes.These residuals are symmetrically distributed with a zero mean, which is expected from a moving average with relatively small window widths, ranging from 1 to 31 min.Thus, if the raw data are regarded as being unbiased, the MA can also be regarded as unbiased.
Applying the original step interpolation scheme (Fig. 5, left, red bars) yields a bias towards negative values with a mean of −0.035 mm.This tendency towards negative values is explained by the fact that this interpolation scheme sticks to the mass values at the old anchor points until the threshold is reached, leading to overestimations of precipitation and underestimations of evapotranspiration periods, with the latter exceeding the former (Peters et al., 2014).Note that applying filters with fixed w and δ yields even greater biases (see Fig. 8 in Peters et al., 2014).
The simple linear interpolation scheme (Fig. 5, centre) leads to a more than three-fold smaller bias of 0.01 mm, with a slight tendency towards positive values.The spline scheme (right) even leads to a slightly smaller deviation.Thus, the linear and spline interpolation schemes are not only superior for the selected time spans in February and May, but also for the complete measured period.The additional computational burden is only minor for any interpolation scheme in comparison with the preceding AWAT filtering.Thus, we suggest always using the spline scheme. .Raw data of a period of evapotranspiration followed by a precipitation event on 7 July 2014.Anchor p: anchor point; MA: moving average; steps: original step interpolation scheme; linear: linear interpolation scheme; spline: cubic Hermitian spline interpolation scheme; linear * : linear interpolation scheme without precipitation correction; spline * : cubic Hermitian spline interpolation scheme without precipitation correction.

Effect on rain events
If a relatively strong precipitation event follows a prolonged period with no significant flux, the mere interpolation schemes without rain correction smooth such an event in an unrealistic manner (linear * and spline * in Fig. 6).The heuristic selection criterion determines that the step interpolation is kept for time intervals between two anchor points if M > 1.1δ (linear and spline).This prevents unfavourable smoothing at the beginning of rain events.

Summary and conclusions
The original step interpolation scheme of the threshold routine of the AWAT yields unrealistic fluxes with abrupt Hydrol.Earth Syst. Sci., 20, 2309-2315, 2016 www.hydrol-earth-syst-sci.net/20/2309/2016/ changes for short time intervals.This is most pronounced when real fluxes are small and therefore the distance between two anchor points is similar to or larger than the chosen time interval.This is problematic if highly resolved boundary conditions are needed for e.g.physically based simulations of water and energy fluxes in the soil-plant atmosphere system.Improving the filter by the proposed interpolation schemes solves this problem, leading to smoothed values, which are more realistic, especially for evapotranspiration events.Moreover, the spline scheme allows even a continuous differentiation and thus any temporal resolution for the predicted fluxes.A simple heuristic selection criterion, which separates medium to strong precipitation from all other events, prevents such precipitations from being smoothed in an unfavourable way.Thus, upper boundary conditions for physically based simulations with very short time intervals can now be automatically derived from precision lysimeters.
In this study, we used a counterbalance weighing system with approximately 0.1 mm resolution.Modern lysimeters resting on weighing cells (von Unold and Fank, 2008) can have a resolution of up to 0.01 mm.Then, the problems of the step interpolation scheme are less pronounced but still present, specifically at times with low fluxes.Thus, the proposed solution is important, especially for lysimeters with limited resolution, which are still often used, but is also favourable for systems with higher resolution.
Note that the results and conclusions regarding the interpolation schemes hold also for filters with fixed window widths and threshold values (e.g.Fank, 2013;Schrader et al., 2013).

Figure 1 .
Figure 1.Raw data for the cumulative upper boundary flux of a grass covered lysimeter in Berlin-Marienfelde, Germany.The data of the three selected time intervals on 16-17 February 2014, 30-31 May 2014, and 7 July 2014 between 13:30 and 15:30 are given in the three subplots.Note that the time and flux intervals for the three intervals are different in the subplots.

Figure 2 .
Figure 2. Raw data of two evapotranspiration events, filtered with the original AWAT filter (steps) and linear as well as spline interpolation schemes.Left: low evapotranspiration on 16-17 February 2014; right: high evapotranspiration rates on 30-31 May 2014.
if we assume E(N ) = 0 and E(MA) − E(O) = 0 leads to b MA = 0 and b F = E F (MA) − E(O).

Figure 3 .
Figure2shows the raw data together with the original filter scheme (step) as well as the results of the two other interpolation schemes (linear, spline) for 2 days with low (left) and high (right) evapotranspiration rates.On 16 and 17 February, the evapotranspiration rates were approximately 0.35 mm d −1 , whereas the rates were approximately 5 mm d −1 at the end of May.By definition, the anchor points coincide with the MA, whereas the step interpolation of the

Figure 4 .
Figure 4. Derived potential evapotranspiration rates from data shown in Fig. 2 with temporal resolutions of 1 day or 10 min, respectively.Steps: original step interpolation scheme; linear: linear interpolation scheme; spline: cubic Hermitian spline interpolation scheme.Note different scales on ordinates for the step scheme between Figs. 4 and 3.

Figure 5 .
Figure 5. Relative residual frequency distribution for the complete data set and the different interpolation schemes.Blue bars indicate residuals between original and filtered data for the cases with mere smoothing, omitting the threshold values; red bars indicate cases with threshold values and subsequent interpolation.The broad bars at plot edges comprise all residuals greater than 0.25 or smaller than −0.25 mm.Steps: original step interpolation scheme; linear: linear interpolation scheme; spline: cubic Hermitian spline interpolation scheme.
Figure6.Raw data of a period of evapotranspiration followed by a precipitation event on 7 July 2014.Anchor p: anchor point; MA: moving average; steps: original step interpolation scheme; linear: linear interpolation scheme; spline: cubic Hermitian spline interpolation scheme; linear * : linear interpolation scheme without precipitation correction; spline * : cubic Hermitian spline interpolation scheme without precipitation correction.