2.1 Overview

A virtual hydrological evaluation involves the comparison of simulated streamflow statistics to virtual-observed streamflow statistics (Fig. 1c), defined as the following.

• Simulated streamflow is streamflow produced by the hydrological model by inputting simulated rainfall at a given site.

• Virtual-observed streamflow is streamflow produced by the hydrological model by inputting observed rainfall at the same given site.

The virtual framework undertakes a relative assessment of the simulated and observed rainfall after its transformation by the same hydrological model to provide insight into the performance of the SRM. Because the hydrological evaluation is a relative comparison of the observed and simulated rainfall, it is important that all other model parameters and extraneous variables (e.g. potential evapotranspiration) relating to the hydrological model are kept the same for the simulation of both virtual-observed and simulated streamflow. It is also important that the selected hydrological model is fit for purpose so that it can simulate the streamflow characteristics of interest. This selection process should also ensure that the hydrological model is compatible with the tested rainfall model and the objectives of the test (i.e. a distributed hydrological model would not be selected to evaluate a single-site rainfall model for deficiencies).

The virtual hydrological evaluation framework is best used to augment and complement existing evaluation methods, rather than act as a replacement. The three evaluation frameworks could work together as follows, where (i) observed-rainfall evaluation identifies any deficiencies in the SRM prior to any hydrological considerations; (ii) the virtual hydrological framework identifies which of these rainfall deficiencies impact on the key predictions of interest, that is, simulated streamflow; and (iii) observed-streamflow evaluation provides a final validation. Therefore, together they enable a more focused approach to identify opportunities for improvement of a SRM. This is because the ultimate goal of the SRM modelling process remains the same: to match observed streamflow for a catchment of interest.

Figure 2Virtual hydrological evaluation procedure.

Download

The formal implementation of the virtual hydrological evaluation framework is summarised in Fig. 2. It uses a series of steps to identify poor performing sites (i.e. specific locations in space, for example, the location of a rainfall gauge), then poor performing time periods, and then key deficiencies in the SRM for those sites and time periods. It combines both observed rainfall-evaluation and virtual hydrological evaluation. The virtual hydrological evaluation includes two different types of tests, an “integrated test” that isolates issues for a given site and “unit tests” that isolate issues for specific time periods. This enables the diagnosis of the key deficiencies in the simulated rainfall. The following sections explain the three steps in turn.

2.2 Step 1 – identify poor performing sites

The first step focuses on using integrated tests to identify poor performing sites for further evaluation. Following the selection of a primary streamflow characteristic of interest and a suitable hydrological model, integrated tests are conducted for each rainfall site (described below in Sect. 2.2.2). The results of the integrated tests are then used to identify sites that are poor performing, according to the systematic application of quantitative performance criteria (see Sect. 2.2.3), for the primary streamflow characteristic.

2.2.3 Identify poor performing sites using the CASE framework

The integrated test results aim to identify the sites that are poor performing for the primary streamflow characteristic. Model performance is categorised using a CASE framework approach as “good”, “fair”, or “poor” following Bennett et al. (2018). The quantitative tests for each performance category are provided in Table 2 alongside an illustration of each in Fig. 3. The quantitative tests proceed by comparing the statistics of the virtual-observed streamflow against those calculated from replicates of the simulated streamflow. Performance was categorised as “good” if the selected statistic for the virtual-observed streamflow fell within the 90 % limits of the statistic calculated from the simulated streamflow replicates (Fig. 3, case i), as “fair” if the virtual-observed statistic fell outside the 90 % limits of the simulated streamflow replicates but within the 99.7 % limits (Fig. 3, case ii), and as “poor” if otherwise (Fig. 3, case iii).

Table 2CASE performance classification criteria. Adapted from Bennett et al. (2018).

Download Print Version

Figure 3Illustration of performance classification: case (i) shows “good” performance, case (ii) shows “fair” performance, and case (iii) shows “poor” performance. Adapted from Bennett et al. (2018).

2.3 Step 2 – identify poor performing time periods

The second framework step is to identify poor performing time periods by conducting a detailed analysis of the integrated test results and comparing these results with an observed-rainfall evaluation at the monthly scale.

Evaluating monthly total flows is a valuable test of SRM performance as the production of monthly total flow volumes relies on the integration of many daily rainfall characteristics (amount, duration, persistence). For each of the poor performing sites, each of these statistics for each month are categorised as “good”, “fair”, and “poor” using the CASE framework. See Sect. 2.2.3 for further explanation of the categorisation procedure. This enables the identification of poor performing time periods from the perspective of the virtual hydrological evaluation.

Poor performance in reproducing virtual-observed streamflow is then contrasted against an observed-rainfall evaluation so that specific poor performing time periods can be identified for further investigation in Step 3. By contrasting CASE performance categories (“good”, “fair”, and “poor”) for observed-rainfall evaluation against virtual-observed streamflow evaluation, poor performing time periods from both rainfall and streamflow perspectives can be identified. This comparison between the observed-rainfall evaluation and the virtual hydrological evaluation (integrated test) can be summarised graphically (e.g. see Fig. 7, Sect. 4.2).

2.4 Step 3 – identify sources of poor performance

The third step of the framework is to identify sources of poor performance in streamflow according to deficiencies in the simulated rainfall. Step 2 identifies the poor performing time periods from a streamflow perspective. However, due to catchment “memory”, the poor performance in streamflow could be due to deficiencies in the simulated rainfall from a range of potential influencing months during or prior to the poor performing time period. For example, poor streamflow performance in an evaluated month may be due to the influence of (i) rainfall deficiencies mostly in the same month (i.e. concurrent influencing months), (ii) rainfall deficiencies over a contiguous block of months including and preceding the evaluated month (i.e. prior and concurrent influencing months), or (iii) rainfall deficiencies in a preceding month more so than in the evaluated month (i.e. prior influencing months). The integrated test cannot isolate which influencing months produce these deficiencies. Therefore, the unit test is designed to enable the identification of sources of poor performance in streamflow. The sources of poor performance are described in terms of which influencing months exhibit key deficiencies in simulated rainfall and therefore which SRM components should be improved.

2.4.1 Unit test procedure

The unit test investigates the impact of simulated rainfall in a given influencing month on the production of streamflow in an evaluated month of interest. This is achieved by splicing observed and simulated rainfall into a single time series which is used to produce simulated streamflow.

Figure 4Schematic of (a) the method of constructing a unit test by embedding simulated months in an observed time series and (b) the error profile produced when using the integrated and unit tests for the evaluated time period of June (t=6) (boxplot whiskers indicate the 90 % limits of the simulated streamflow replicates). For the unit test the errors in the evaluated period (t) are calculated as the difference between $Q_{\left(k\right)}^{\mathrm{spl}}$ and $Q_{\left(t\right)}^{\mathrm{vo}}$. For the integrated test the errors are calculated as the difference between Q^sim and $Q_{\left(t\right)}^{\mathrm{vo}}$.

Download

Following Fig. 4a, consider the time series of observed, R^obs, and simulated, R^sim, daily rainfall for each year (and replicate) at a given site. Figure 4a illustrates the embedding of simulated rainfall $R_k^{\mathrm{sim}}$ in an influencing month, k, within observed rainfall $R_m^{\mathrm{obs}}$ for all other months $m\in \left\{\mathrm{1},\mathrm{\dots },\mathrm{12}\mathrm{|}m\ne k\right\}$. The resulting spliced rainfall time series $R_{\left(k\right)}^{\mathrm{spl}}$ is denoted with respect to the influencing month, k, and has the same length as the corresponding observed, R^obs, and simulated, R^sim, time series.

$$\begin{array}{}\text{(3)}& R_{\left(k\right)}^{\mathrm{spl}}=\bigcup _{m=\mathrm{1}}^{\mathrm{12}}\left\{\begin{array}{ll}{R}_m^{\mathrm{sim}};& m=k\\ R_m^{\mathrm{obs}};& m\ne k\end{array}\right.\end{array}$$

For example, if June (k=6) is selected as the influencing month, each year of the spliced time series, $R_{\left(\mathrm{6}\right)}^{\mathrm{spl}}$, would be composed as follows:

$$\begin{array}{}\text{(4)}& R_{\left(\mathrm{6}\right)}^{\mathrm{spl}}=\left\{R_{\mathrm{1}}^{\mathrm{obs}},\mathrm{\dots },R_{\mathrm{5}}^{\mathrm{obs}},R_{\mathrm{6}}^{\mathrm{sim}},R_{\mathrm{7}}^{\mathrm{obs}},\mathrm{\dots },R_{\mathrm{12}}^{\mathrm{obs}}\right\}.\end{array}$$

The ensemble of $k=\mathrm{1},\mathrm{\dots },\mathrm{12}$ spliced rainfall time series $R_{\left(k\right)}^{\mathrm{spl}}$ for all influencing months and additional inputs (e.g. potential evapotranspiration) indicated by “…” is transformed according to a hydrological model g[] to produce an ensemble of simulated streamflow, $Q_{\left(k\right)}^{\mathrm{spl}}$. This procedure is repeated for all simulated rainfall replicates.

$$\begin{array}{}\text{(5)}& Q_{\left(k\right)}^{\mathrm{spl}}=g\left[R_{\left(k\right)}^{\mathrm{spl}},\mathrm{\dots }\right]\end{array}$$

By construction, the spliced rainfall is identical to the observed rainfall for all months other than the influencing month, so any errors in streamflow statistics can be attributed to the influencing month free from other factors.

The full set of spliced rainfall (e.g. spliced rainfall for each month designated as the influencing month $R_{\left(k\right)}^{\mathrm{spl}};k=\mathrm{1},\mathrm{\dots },\mathrm{12}$) is input to the hydrological model. This step is repeated for all available replicates of the spliced time series. The results of the unit test and the integrated test (Steps 1–2) are then investigated and compared by selecting each month as the evaluated time period in turn as well as other key time periods (e.g. annual).

4.1 Step 1 – identify poor performing sites

To undertake Step 1, annual total flow volumes were designated as the primary streamflow characteristic to narrow the number of sites investigated. Following the selection of the primary streamflow characteristic and selection of the hydrological model, GR4J, integrated tests were undertaken to evaluate the simulated rainfall at the 22 sites. The annual total flow distribution was used to give a broad indication of performance. This step categorised 10 of the 22 sites as “poor” and 12 as “good”, which is in strong contrast to earlier evaluation efforts using observed-rainfall evaluation (Bennett et al., 2018) that categorised the majority of sites and statistics as “good” (see Sect. 2.2.3 for category definitions).

The 10 sites categorised as “poor” are the focus of subsequent virtual hydrological evaluation framework steps. These “poor” performing sites are indicated by the triangles in Fig. 5.

Figure 7Integrated test, comparing observed-rainfall evaluation (a) with the virtual hydrologic evaluation (b). Comparison of daily and aggregate (“Agg.”) rainfall statistics against aggregate flow statistics for individual months and years, where means are denoted as “m” and standard deviations as “SD”.

Download

4.2 Step 2 – identify poor performing time periods

The poor performing sites identified in Step 1 were then compared in terms of both an observed-rainfall evaluation and virtual hydrological evaluation via an integrated test. Figure 7 graphically summarises this comparison, with each row presenting monthly or annual performance of the following statistics:

• simulated daily rainfall statistics (mean (m) daily amounts, standard deviation (SD) of daily amounts, mean number of wet days (nwet) and the standard deviation of the number of wet days);

• aggregate rainfall statistics (mean and standard deviation of total rainfall); and

• aggregate streamflow statistics (mean and standard deviation of total flow).

The first to fourth columns of Fig. 7 summarise the observed-rainfall evaluation and the fifth and sixth of Fig. 7 summarise the virtual hydrological evaluation. The first column of Fig. 7 indicates that of the poor performing sites, the SRM exhibited “good” performance in simulating daily rainfall means and standard deviations as well as the mean number of wet days for all sites and months and at an annual level according to the observed-rainfall evaluation. Each of the three statistics presented in the first column are assessed separately but are presented together to avoid repetition. Whereas the second column indicates that there is mixed performance across sites and months in simulating the variability in the number of wet days (SD(nwet)). Likewise, the third and fourth columns indicate overall “good” performance in simulating mean monthly totals and mixed performance in simulating the monthly or annual total standard deviations (SD(total)). Whereas the virtual hydrological evaluation (fifth and sixth) columns show mostly “good” performance in all months other than those in the “wettest” or “wetting-up” periods.

A clear trend from Fig. 7 is the contrast in performance between the observed-rainfall evaluation and the virtual hydrological evaluation. One contrast is that in the driest months (December, January, February) “poor” performance in simulating rainfall (based on observed-rainfall evaluation) did not necessarily translate to “poor” performance in simulating streamflow (based on virtual hydrological evaluation). For example, examining the first row of Fig. 7, the observed-rainfall evaluation shows that in January the SRM's ability to simulate variability in the number of wet days, SD(nwet), was “poor” for all sites. However, in contrast the virtual hydrological evaluation shows that most sites had “good” performance in simulating the January distribution of monthly total flow (i.e. m(total) and SD(total)).

A second contrast is that “good” performance in the observed-rainfall evaluation does not necessarily translate to “good” performance for the virtual hydrological evaluation, particularly for months in the “wettest” and “wetting-up” periods. For example, in Fig. 7 the rows summarising June and August show large percentages of “poor” sites in the virtual hydrological evaluation of monthly total flow. This deficiency would have been difficult to infer using the observed-rainfall evaluation due to the 100 % “good” performance of m(total) rainfall and “good/fair” performance of SD(total) rainfall in these months.

Likewise, by examining the bottom row of Fig. 7 that summarises annual performance, it can be seen that the observed-rainfall evaluation shows unbiased mean annual total, m(total), rainfall (100 % “good”) and yet the mean annual total flows showed only 10 % of sites as “good”. Discussion of the unit tests in the following section will investigate reasons why apparently “good” rainfall can yield “poor” flow.

4.3 Step 3 – identify sources of poor performance

To undertake Step 3, unit tests were run to evaluate the source of deficiencies in poor performing time periods. The results of these test were compared against integrated tests in terms of their relative errors. From this comparison the source and type of key deficiencies in the simulated rainfall that lead to poor performance in simulated streamflow were identified. A comparison of the virtual-observed flow duration curve for the poor performing time periods and the flow duration curves resulting from unit tests for key influencing months was also undertaken to illustrate the impact of these key deficiencies on the daily flow duration curve.

Figure 8Lobethal, Site 12 (a) observed-rainfall evaluation mean monthly total rainfall, (b) observed-rainfall evaluation standard deviation of monthly total rainfall, (c) virtual hydrological evaluation (integrated test) mean monthly total streamflow, (d) virtual hydrological evaluation (integrated test) standard deviation of monthly total streamflow. Boxplot whiskers indicate the 90 % limits of the simulated streamflow or rainfall replicates.

Download

Here, four examples of the different types of key deficiencies are illustrated using two locations, Site 12 and Site 10 (see Figs. 8 to 13). For completeness these results are presented together with the results of the observed-rainfall evaluation (panels a and b of Figs. 8 and 11).

4.3.1 Streamflow errors mostly originate from rainfall model deficiencies in the evaluated month

A common case for streamflow errors is that they originate from rainfall in the same month. This case can be illustrated using Site 12 in Fig. 8 where left-side panels show results for the mean and right-side panels show the standard deviation and where panels (a) and (b) summarise the observed-rainfall evaluation, panels (c) and (d) summarise the integrated test. From panels (a) and (b), the simulated monthly rainfall is generally unbiased, but from panels (c) and (d) the mean and standard deviation of the simulated streamflow is lower than the virtual-observed flow from June to September. Here, September is selected as an illustrative case for an application of the unit test in Fig. 9 since it shows biased flow.

Figure 9Lobethal, Site 12 (a) unit test error in mean monthly flow (September), (b) unit test error in standard deviation of monthly flow (September), (c) unit test September flow duration curve when August and September are selected as influencing months (top 10 % of flow days shown). Boxplot whiskers indicate the 90 % limits of the simulated streamflow replicates.

Download

Taking September as the evaluated month (t=9), Fig. 9a and b compare the unit tests for all 12 influencing months (yellow and blue striped boxplots) with the integrated test (blue shaded boxplot) in terms of the error in the simulated flow. When the influencing month is September (i.e. the September rainfall is “spliced” into the observed record, k=9) the resultant error is greatest and closest to the error for the integrated test for both the mean monthly total flow (Fig. 9a) and standard deviation of monthly total flow (Fig. 9c). For the example of the standard deviation, when the influencing month is July (i.e. July rainfall is spliced into the observed record) the median error is less than 2 %, whereas when September is taken as the influencing month the median error is approximately 16 % (Fig. 9b). Therefore, to improve September flows, September rainfall should be improved in preference to all other months.

This need to improve September in preference to preceding months is also illustrated via Fig. 9c where the September daily flow duration curves are shown for the cases where August (orange shading) and September (blue shading) are the influencing months compared against the virtual-observed September flow duration curve (purple dots). Where August is selected as the influencing month, the virtual-observed flow duration curve largely sits inside the 90 % limits of the flow duration curves resulting from the unit testing procedure. Whereas, the virtual-observed flow duration curve is located outside the 90 % limits of the unit test flow duration curve when September is taken as the influencing month. Thereby providing further evidence that, to improve September flows, September rainfall should be improved in preference to other months.

Analysing other sites and months suggests that over 50 % of the evaluations correspond to this case, and they typically occur in spring and summer months when the catchment is drying out.

4.3.2 Streamflow errors originate from rainfall model deficiencies over a contiguous block of months

An illustration of the case where streamflow errors originate from rainfall model deficiencies over a contiguous block of months is provided by Site 12, where July is selected as the evaluated month. Comparison of the July performance in the integrated and unit tests (Fig. 10a and b) demonstrates that the errors in July streamflow do not originate in the July rainfall alone (unlike the case for September – see Sect. 4.3.1). Although the largest percentage error in flow is attributable to July (a median error of 8 % in mean monthly total flow and 25 % in the standard deviation of monthly total flow when the influencing month is July) a significant proportion of the error for July streamflow originates in prior months. June and May rainfall have a significant influence on the July flow with percentage errors of up to 15 % in July flow when June or May are the influencing month. Therefore, to improve July flows, it is not just the July rainfall that should be improved, but also the preceding 2 months.

Figure 10Lobethal, Site 12 (a) unit test error in mean monthly total flow (July), and (b) unit test error in standard deviation of monthly total flow (July), (c) July flow duration curve when June and July are selected as influencing months in unit test (top 10 % of flow days shown). Boxplot whiskers indicate the 90 % limits of the simulated streamflow replicates.

Download

This need to improve July and preceding months is also illustrated via Fig. 10c where the July daily flow duration curves are shown for the cases where June (orange shading) and July (blue shading) are the influencing months compared against the virtual-observed July flow duration curve (purple dots). For both cases the virtual-observed flow duration curve is located outside the 90 % limits of the flow duration curves resulting from the unit testing procedure.

Typically, “wetting-up” and “wettest” months fall in this case where streamflow errors originate from rainfall model deficiencies over a contiguous block of months, approximately 40 % of the site/month combinations.

4.3.3 Streamflow errors originate from rainfall model deficiencies in a preceding month more so than evaluated month

An example of the case where the largest contribution to streamflow errors arises from rainfall deficiencies in a preceding month is provided by Site 10, where July is selected as the evaluated month. July is selected as an illustrative case for application of the unit test since it shows biased flow (see Fig. 11c and d), but did not show any bias in the simulated rainfall (see Fig. 11a and b).

Figure 11Happy Valley, Site 10 (a) observed-rainfall evaluation mean monthly total rainfall, (b) observed-rainfall evaluation standard deviation of monthly total rainfall, (c) virtual hydrological evaluation (integrated test) mean monthly total streamflow (d) virtual hydrological evaluation (integrated test) standard deviation of monthly total streamflow. Boxplot whiskers indicate the 90 % limits of the simulated streamflow or rainfall replicates.

Download

Figure 12Happy Valley, Site 10 (a) unit test error in mean monthly flow (July), (b) unit test error in standard deviation of monthly flow (July), and (c) July flow duration curve when June and July are selected as influencing months in the unit test (top 10 % of flow days shown). Boxplot whiskers indicate the 90 % limits of the simulated streamflow replicates.

Download

The largest contributor to error in July flow is not July rainfall but June rainfall (Fig. 12a and b). That is, the largest errors occur when there is observed rainfall for July spliced with simulated rainfall for June. In contrast, simulated July rainfall spliced with observed rainfall in other months yields a smaller median error. This deficiency in June rainfall can also be seen in an examination of the July flow duration curves (Fig. 12c) where the virtual-observed flow duration curve sits within the 90 % limits of the simulated flow duration curve where July is designated as the influencing month, whereas when June is designated as the influencing month the virtual-observed flow duration curve sits outside the 90 % limits for a number of the higher flow days.

While improving the July rainfall will improve the simulation of July flow, a more significant improvement will be obtained by focusing on improving the June rainfall. The category where streamflow errors originate from rainfall model deficiencies in a preceding month represents about 10 % of the evaluated site/month combinations (i.e. those identified in Step 2).

4.3.4 Influence of monthly rainfall on annual flow volumes

While annual simulated rainfall was unbiased, annual simulated streamflow was biased. An illustration of how errors in annual total streamflow arise from deficiencies in simulated rainfall is shown for Site 10. Figure 13a and b show that when the months of May to August are assessed as the influencing month they produce the largest errors in distribution of annual total flow for Site 10. Splices of other months do not significantly degrade the simulation of total annual flow. This deficiency can also be seen via an examination of the flow duration curve (Fig. 13c) in which the virtual-observed flow duration curve is located outside portions of the simulated flow duration curves where May or June are designated as the influencing month. Improvements to the simulation of annual total flow will therefore come from improving the SRM in the “wetting-up” and wettest months of the seasonal cycle (May to August). This insight from the use of unit testing would be difficult to obtain using other evaluation strategies (further discussed in Sect. 5.2).

Figure 13Happy Valley, Site 10 (a) unit test error in mean annual total flow, and (b) unit test error in standard deviation of annual total flow, and (c) annual flow duration curve when May and June are selected as influencing months in unit test (top 10 % of flow days shown). Boxplot whiskers indicate the 90 % limits of the simulated streamflow replicates.

Download

5.1 The importance of streamflow-based evaluation

Streamflow arises from the integration of rainfall processes (e.g. rainfall amounts, occurrences and wet-dry patterns) over a catchment. Features of the catchment, such as catchment storage, thresholds and nonlinearities in the rainfall–streamflow response function, can either act to amplify or dampen the rainfall errors at different times of the year. These behaviours were clearly identified and demonstrated in Step 2 of the virtual hydrological evaluation framework that compares observed-rainfall evaluation and virtual hydrological evaluation (see Sect. 4.2).

In terms of amplification, the elasticity of the rainfall–streamflow response (Chiew, 2006) suggests that catchments can have strong sensitivities to discrepancies in rainfall. Given that the rainfall elasticity of streamflow to rainfall is a factor of 2 to 3.5 (Chiew, 2006), using the principles of error propagation (Ang and Tang, 2007), assuming linearity it follows that a 10 % error in mean/standard deviation of rainfall could potentially be amplified to 20 %–35 % error in the mean/standard of streamflow. This estimate represents a lower-bound of the potential amplification, since the non-linear nature of the rainfall-runoff transformation will likely produce a larger potential amplification of errors. This indicates that streamflow-based evaluation of rainfall models provides a stronger test than observed-rainfall evaluation in terms of the sensitivity of the statistics. For example, Fig. 7 shows that July rainfall statistics were classified as “good”, yet despite this, the streamflow response was “poor” (see Sect. 4.2 for further discussion). It could be argued that the rainfall results presented in Fig. 7 were classified as “good” because the observed-rainfall evaluation was limited, but the evaluation was methodical and used a comprehensive range of daily and monthly statistics (Bennett et al., 2018). While many rainfall statistics were preserved (means, standard deviation, extremes, marginal distributions of daily rainfall) the rainfall–streamflow response of the catchment exposes that there are deficiencies in the rainfall model not clearly identified by the observed-rainfall evaluation (Bennett et al., 2018).

In terms of dampened influence, catchment storages and high evapotranspiration can also act to suppress errors in the rainfall simulations. For example, Fig. 7 showed that the variability in the number of wet days, SD(nwet), was “poor” for all sites in January, yet this did not result in “poor” streamflow. The high potential evapotranspiration in January indicates that the majority of rainfall in January is converted into actual evapotranspiration yielding little streamflow. Hence, any errors in rainfall do not noticeably impact on January streamflow.

It is clear that streamflow-based evaluation is beneficial in addition to conventional observed-rainfall evaluation.

5.2 The benefits of the virtual evaluation framework

A benefit of virtual hydrological evaluation is that it is a relative measure of performance, where the hydrological model is a common factor in the construction of virtual-observed and simulated streamflow. This enables discrepancies in the streamflow to be identified in terms of SRM features. In contrast, observed-streamflow evaluation is typically hampered by difficulties in separating the impact of data errors, hydrological model predictive performance from the errors in the SRM. A further benefit is the ability to undertake streamflow-based evaluation at any site where rainfall is observed and simulated. This enables insights into the SRM performance for simulating streamflow on a site-by-site basis.

The use of a virtual hydrological framework for evaluation provides the unique opportunity to develop innovative tests that can target specific aspects of the SRM. This paper introduces an innovative unit test that was used as a method for isolating the influence of rainfall in a month (i.e. the influencing month) on streamflow in an evaluated month while excluding the possibility of deficiencies from other rainfall months. The test enables a procedure for targeting months that are influential in terms of streamflow production rather than interpreting model performance based on blunt evaluation of rainfall or streamflow.

This unit test provides added value over and above the integrated test because it identifies which are the influencing months that have deficiencies in the modelled rainfall that produce poor streamflow predictions. For example, Sect. 4.3.2 illustrated that while the integrated test identified that there was poor streamflow in July for Site 12, the unit test was able to identify that the simulated rainfall in the prior influencing months of both May and June (Fig. 10) made significant contributions (10 %–15 % errors) to July's poor streamflow. A second example is shown in the influence of monthly rainfall on the errors in annual flow volumes in Sect. 4.3.4. If the modeller had focussed on improving the SRM by focusing on months with the highest contribution to annual total flow, July to September would have been identified as important, whereas the unit test identifies a different focus (May–August). The unit tests in Sect. 4.3.4 show that May and June combined make up 13 % of the total annual flow volume (Fig. 11c). However, they contribute to 11 % of the error in the mean annual total flow (Fig. 13a) and 24 % error in the standard deviation (Fig. 13b). By contrast, September is a high-flow month contributing 21 % of the annual total flow, but only 2 % error in the mean and 6 % error in the standard deviation. Without the unit test, it would have been less clear that the “wetting-up” months such as May and June were a more important focus for SRM improvement than a high-flow month such as September.

5.3 Identifying key deficiencies in the rainfall properties of the SRM

The previous section highlighted that simulated rainfall in the “wetting-up” period May–June was a key cause of errors in the streamflow. Returning to the observed-rainfall evaluation of the SRM (see Fig. 7) enables identification of the rainfall properties that are likely to be the cause of these errors in the streamflow. In the May–June months, the mean and standard deviation of rainfall amounts on wet days and the mean number of wet days show 100 % “good” performance, while the standard deviation of the number of wet days shows a high proportion of sites with “poor” performance (50 % of sites in May and 40 % of sites in June). This identifies that rainfall amounts on wet days are reproduced well, but there is not enough variability in the rainfall occurrences in the key months of May–June. The observed-rainfall evaluation identifies that this is a common problem for this SRM, with 8 months having some proportion of “poor” sites for the standard deviation of the number of wet days – similar results were found in Bennett et al. (2018). However, the virtual hydrological evaluation identified that it is the rainfall occurrences in 2 key months (May and June) that will provide the most benefit in terms of virtual hydrological evaluation. This provides clear guidance on the rainfall properties that need to be improved for this SRM.

5.4 Limitations and future research

The virtual hydrological framework for SRM evaluation provides an opportunity for further improvements in the future to substantially augment the framework's diagnostic capabilities, including the following.

• i.

  Using multiple, well-tested hydrological models – a potential limitation of the virtual hydrologic evaluation framework is that it is reliant on the use of a single hydrological model. Hydrological model structural errors may potentially skew interpretation of the SRM evaluation if the hydrological model poorly represents the catchment processes. To reduce these impacts the steps taken in this study included (a) using a well-tested hydrological model that has demonstrated good performance on a wide range of catchments (e.g. the GR4J model has been widely tested – see Perrin et al., 2003 and Coron et al., 2012); and (b) calibrating and evaluating the hydrological model on a catchment close to the observed rainfall sites to ensure it provided sufficiently good performance (e.g. GR4J was calibrated to the Onkaparinga catchment – see Westra et al., 2014b, a). Future research will use multiple, well-tested hydrological models with sufficiently good performance to reduce the reliance on a single hydrological model and to ensure the identification of SRM deficiencies is not dependent on a single hydrological model.

• ii.

  Comparison of SRMs – this framework can be extended to provide more direct guidance on which rainfall features (in terms of components of the SRM) should be modified to improve streamflow performance. This can be done by comparing multiple rainfall model variants (parametrically, or via bootstrap techniques) which are designed to have contrasting features of a key characteristic (e.g. intermittency, rainfall correlation). Such an approach was undertaken by Evin et al. (2018) using an observed-rainfall evaluation approach. If the SRMs have monthly/seasonal autocorrelation (these were not significant for the rainfall in the Onkaparinga catchment), the unit testing approach would need to be extended by conditionally sampling the simulated rainfall in a manner that preserves monthly correlations.

• iii.

  Evaluation of temporal non-stationarity – this framework can be extended to evaluate the impact of non-stationarity on SRM model performance by applying it on a selected non-stationary period. Care would be needed in the selection of statistics to identify model performance (since the performance in different sub-periods could be masked when evaluating an overall period). A related issue is that the hydrological model should provide adequate performance across the range of non-stationary climate forcings to which it is subjected.

• iv.

  Evaluation of spatial performance – there are multiple opportunities to develop tests for spatial performance, including (a) repeating the integrated test for all sites, and for catchment average rainfall means it would be possible to diagnose whether specific locations or the spatial dependence cause poor reproduction of streamflow statistics; (b) developing a spatial unit test (which is analogous to the temporal unit test but extended to space) where different combinations of sites are “spliced” in the construction of catchment average rainfall – to evaluate the impact of “mixed” performance in the SRMs between sites on the catchment average rainfall; and (c) these spatial unit tests could be used to evaluate stochastic weather generators more generally as well as spatially distributed rainfall generators – though these would require a spatially distributed hydrological model.

• v.

  Evaluation of SRMs at subdaily timescales – this extension to evaluate SRMs at other timescales provides opportunities for the development of further unit test variants that focus on a range of different aspects of subdaily rainfall (intensity, duration, persistence) and its impact on streamflow characteristics of interest (e.g. subdaily flow peaks). This extension would require the use of a well-tested continuous subdaily hydrological model capable of simulating the relevant streamflow characteristics.