An educational model for ensemble streamflow simulation and uncertainty analysis

Abstract. This paper presents the hands-on modeling toolbox, HBV-Ensemble, designed as a complement to theoretical hydrology lectures, to teach hydrological processes and their uncertainties. The HBV-Ensemble can be used for in-class lab practices and homework assignments, and assessment of students' understanding of hydrological processes. Using this modeling toolbox, students can gain more insights into how hydrological processes (e.g., precipitation, snowmelt and snow accumulation, soil moisture, evapotranspiration and runoff generation) are interconnected. The educational toolbox includes a MATLAB Graphical User Interface (GUI) and an ensemble simulation scheme that can be used for teaching uncertainty analysis, parameter estimation, ensemble simulation and model sensitivity. HBV-Ensemble was administered in a class for both in-class instruction and a final project, and students submitted their feedback about the toolbox. The results indicate that this educational software had a positive impact on students understanding and knowledge of uncertainty in hydrological modeling.


Introduction
Rainfall-runoff models have been used to describe nonlinear hydrological processes, predict extreme events and assess the impacts of potential changes in future climates and/or land use. Numerous physical, conceptual, and statistical models have been used for modeling rainfall-runoff processes (e.g., Singh and Woolhiser, 2002;Beven, 2001;Bergström, 1995;Wheater et al., 1993). Given the importance of water resources and the significance of hydrologic extremes on human livelihood and society, educating students on various aspects of the hydrological cycle is very important. However, reliable rainfall-runoff modeling and flood management entails a strong background in the hydrological cycle and modeling, which students may not have.
The United States National Research Council has also stressed the need for an improve hydrology curriculum, specifically in the areas of hydrologic modeling and data analysis (e.g., NRC, 2000NRC, , 1991Wagener et al., 2012). In a report by the Consortium for Universities for the Advancement of Hydrologic Science (CUAHSI), the potential role of hydrologic models in transforming the way hydrology is taught and communicated to students is emphasized (CUAHSI, 2007).
Recent research on engineering and science education suggests that students acquire a better knowledge of hydrological processes and their uncertainties when exposed to novel educational techniques (e.g., student centered methods) as a complement to traditional lecture-driven classes (see Thompson et al., 2012 and references therein). Wagener et al. (2010) argue that the changing demands on hydrology offers an unprecedented opportunity to advance hydrology education. Recent advances in simulation models, graphical user interface developments and physical models provide opportunities for improving existing hydrology curriculum (see Shaw and Walter, 2012;Habib et al., 2012;Pathirana et al., 2012;Seibert and Vis, 2012a;Rusca et al., 2012;Rodhe, 2012;. Hydrologic models can be used to teach complex hydrological processes by providing tools for hands-on projectbased learning. Thompson et al. (2012) review recent theoretical developments in engineering and science education research that are relevant to teaching hydrological processes.

A. AghaKouchak et al.: Ensemble streamflow simulation and uncertainty analysis
In a recent study,  introduced HBV-EDU which is a hands-on modeling tool developed for students to help them learn the fundamentals of hydrological processes, parameter estimation and model calibration. HBV-EDU provides an application-oriented learning environment that introduces the interconnected hydrological processes through the use of a simplified conceptual hydrologic model. Using HBV-EDU, students can practice conceptual thinking in solving hydrology problems. Using a detailed course survey,  showed that students were more inspired by handson application-oriented teaching methods (e.g., using models) than by purely theoretical lecture driven classes. Seibert and Vis (2012b) presented HBV-light which is also a userfriendly conceptual model, especially useful for teaching hydrological modeling and uncertainty estimation. The model includes different functionalities such as automatic calibration and Monte Carlo simulations designed for teaching advanced hydrology classes and research projects.
Like HBV-EDU, most hydrologic models used for both teaching and research are deterministic, providing the best simulation based on estimated parameters (e.g., Beven, 2001;Young, 2002). However, quantification of uncertainties associated with hydrologic models are fundamental for risk assessment and decision making. To accomplish this, ensemble streamflow simulation can be used for uncertainty analysis, risk assessment and probabilistic analysis of flood forecasts (Beven, 2008;Wood et al., 2002;Georgakakos et al., 2004;Vrugt et al., 2008). For example, using ensemble streamflow simulations, one can derive the probability of the water level exceeding a certain extreme threshold. Also, the effect of the uncertainty in observations, model representations of hydrological processes, and global climate studies has been highlighted in numerous studies (Bell and Moore, 2000;Goodrich et al., 1995;Obled et al., 1994;AghaKouchak et al., 2012).
The concepts of ensemble simulation and uncertainty analysis are typically covered in hydrology classes only theoretically. Several models and tools have been used for estimation of uncertainty of hydrologic models and for teaching purposes (e.g., Rainfall-Runoff Modelling Toolbox -RRMT, Wagener et al., 2004;HBV-light;Seibert and Vis, 2012b;GLUE Software, GLUEWIN;Beven and Binley, 1992). We hypothesize that the students would gain a better knowledge of model uncertainty using educational simulation tools and techniques. There are different approaches to uncertainty estimation including statistical methods, and physical and non-statistical methods (see Beven and Kimberlain, 2009;Montanari et al., 2009). This study builds upon the previous model (HBV-EDU) and provides an educational software for teaching ensemble simulation and uncertainty analysis using a statistical approach. The modeling toolbox, named HBV-Ensemble, provides an ensemble of streamflow simulations based on randomly selected parameters that satisfy a certain objective function. The aim of HBV-Ensemble is both to teach both hydrological processes and uncertainty estimation. HBV-Ensemble can be employed for in-class lab practices and assignments as well as assessment of students' understanding of hydrological processes. We anticipate that the presented educational toolbox to encourage students to learn more about the fundamentals of hydrology, ensemble simulation and uncertainty analysis. Notice that an ensemble is often described as simulations from different models. In this paper, an ensemble is defined as multiple simulations using different sets of parameters (e.g., Beven and Freer, 2001;Renard et al., 2010;Wagener, 2003;Murphy et al., 2004;Piani et al., 2005).
The paper is organized into five sections. After this introduction, the model concept and methodology are briefly introduced. In the third section, an example application of the toolbox is presented. The fourth section is devoted to the students feedback. Finally, the last section summarizes the results and conclusions.

HBV model
The proposed model is based on the a modified version of HBV hydrologic model (Bergström, 1995). The model is originally developed by the Swedish Meteorological and Hydrological Institute. Various versions of the model are now available that vary in complexity and utility features. The hydrological model used in HBV-Ensemble is a modified version of the HBV presented in Hundecha and Bárdossy (2004) and . The HBV-Ensemble consists of five main modules: (1) snowmelt and snow accumulation; (2) soil moisture and effective precipitation; (3) evapotranspiration; (4) runoff response; (5) ensemble simulation. A detailed discussion on the HBV model is provided in this Special Issue (see Seibert and Vis, 2012b) as well as in Hundecha and Bárdossy (2004) and . For this reason, only a brief overview of the model is presented here.
In this model, observed precipitation partitions into rainfall and snow based on observed temperature. As long as the temperature remains below the melting threshold snow accumulates, and for temperatures above the melting threshold snow melts (see Seibert and Vis, 2012b for the governing equations). This approach is known as the degreeday method. The combination of rainfall and snowmelt will then be partitioned into direct (surface) runoff and infiltration based on the soil moisture condition.
In HBV-Ensemble, the actual evapotranspiration is derived based on the long-term monthly potential evapotranspiration, adjusted for temperature deviation from the longterm monthly mean temperature (AghaKouchak and . The runoff response module of the model includes two conceptual reservoirs, where the upper reservoir models the near surface flow and the lower reservoir simulates the base flow (groundwater flow). A constant percolation rate is used to connect the reservoirs. The upper reservoir has two outlets for estimation of the near surface flow and interflow, whereas the lower reservoir has one outlet for simulation of the baseflow. The total surface water (runoff) would then be derived as the sum of the outflows from both reservoirs.

Ensemble simulation module
HBV-Ensemble provides an educational software for teaching ensemble simulation and uncertainty analysis. In HBV-Ensemble a range of model parameters are sampled using the Monte Carlo technique and all simulations that satisfy the objective function will be accepted as one realization in the ensemble output. A common objective function is the Nash-Sutcliffe coefficient (Nash and Sutcliffe, 1970): where ; and n = number of time steps. The model parameters of HBV-Ensemble include: degree-day factor; field capacity; shape coefficient; evapotranspiration adjustment parameter; permanent wilting point; near surface flow, interflow and baseflow constants; percolation storage constant; and threshold water level for near surface flow. For a detailed discussion on the parameters, the reader is referred to Seibert and Vis (2012b) and AghaKouchak and . The procedure to generate an ensemble of streamflow simulations is as follows: 1. Select reasonable upper and lower bounds for the model parameters mentioned above based on expert knowledge, available data or literature.
2. Draw random samples of parameters from the above range (e.g., 1000 sets of randomly selected parameters) using the Generalized Likelihood Uncertainty Estimation (GLUE; Beven and Binley, 1992 -see GLUE demonstration software available through the Lancaster University for more information).
3. Run HBV-Ensemble with all parameter combinations obtained from the previous time step.
4. Accept simulations (ensemble members) and parameter sets that satisfy a certain objective function (for example, Nash-Sutcliffe coefficient (NSC) above 0.7, or root mean square error below an acceptable threshold). Each accepted simulation will then be a member in the final ensemble. Alternatively, one can select the best simulations (e.g., top 100) that lead to a root mean square error below an acceptable threshold.
5. The bounds of the final streamflow ensemble (maximum and minimum bounds) describe the uncertainties in streamflow simulation due to uncertainties in model parameters.
6. Finally, the model provides a deterministic simulation which is based on the set of parameters that lead to the best value of the objective function.
It should be noted that the above steps are built-in functions in HBV-Ensemble and undergraduate students are not expected to do all the steps on their own. Figure 1 illustrates the HBV-Ensemble Graphical User Interface (GUI). In panel a, the user can specify the upper and lower bounds of the parameters (see the first column in panel a). The initial values, such as the initial state of soil moisture, can be entered using panel b. Panel c can be used to load the input data. The required input data include precipitation, temperature, long-term monthly evapotranspiration and temperature. Using panel d, the user can select the objective function (e.g., root mean square error, Nash-Sutcliffe coefficient and correlation coefficient). The number of Monte Carlo runs (randomly sampled parameters) can be specified using panel e. Finally, the performance measure value for the simulation with the best performance value will appear in panel f. Figure 2 presents sample input precipitation (top), temperature (middle) and simulated ensemble streamflow (bottom). In Fig. 2 (bottom), the solid red line represents the observed runoff. The gray lines show the uncertainty limits for all the acceptable parameter value sets using 1000 simulations. In Fig. 2 (bottom) the solid black line displays the simulation from the best-estimate parameter-value set. One can see that in this approach, in addition to runoff, estimates of upper and lower bounds (gray lines) provide measures of uncertainty.

Application
It should be noted that this educational toolbox produces other variables besides runoff, including time series of snow accumulation, soil moisture, evapotranspiration, and upper and lower reservoir water levels. For instance, Fig. 3 displays sample outputs derived using panel g in Fig. 1.
The presented hydrologic modeling toolbox can be used for both in-class lab practices and homework assignments to test the extent of the students' understanding of hydrological processes. An executable version of this modeling toolbox is also available for students who are not familiar with MAT-LAB, which is used to develop this hands-on toolbox. Having this modeling toolbox, students can easily change the parameters and see the effects on simulated streamflow promptly. The toolbox can also be used for teaching sensitivity analysis by changing one parameter at a time and observing the effect of the parameter on model output. Furthermore, HBV-Ensemble can be used for a lab practice or homework on the  effects of initial values on streamflow simulation. For example, one can run the model with different initial values of soil moisture and compare the output hydrographs (as shown in Fig. 4). Using this particular exercise, student will find out that the initial values will have a significant impact on the model outputs at the beginning of the simulations. However, the effects of the initial values diminish over time in the longterm simulations.
The visualizations provided by HBV-Ensemble can help students investigate "what-if" scenarios for model parameters, initial conditions and objective functions. The toolbox allows students to learn about the impact of the number of Monte Carlo runs on the output ensemble. Furthermore, students can alter the choice of objective function and evaluate its impact on the output ensemble. In addition to the choice of objective function, students can learn more about uncertainty by changing the behavioral thresholds (e.g., NSC > 0.6, NSC > 0.7) and observing the effects on the uncertainty bounds.

Students feedback and discussion
The previous version of the toolbox (Excel spreadsheet version) was used at the University of Louisiana at Lafayette (ULL) in Spring 2009 and students' feedback were reported in AghaKouchak and . The presented educational toolbox has been administered at the University of California, Irvine (UCI) in Winter Quarters 2011 and 2012 (Watershed Modeling CEE173-273). Students learned the fundamentals of the HBV model concept and used the MAT-LAB GUI, shown in Fig. 1 for their final project (hydrologic modeling for a watershed in California). In the following, the feedback from UCI students who used the MATLAB GUI are presented.
The Watershed Modeling course includes theoretical instructions and several homework assignments and projects. The objective of the course is to introduce hydrologic modeling tools and techniques to students. It should be noted that for undergraduate students, the Watershed Modeling class is an upper division course, and participants are required to have passed Hydrology. This means that undergraduate students who were exposed to this educational toolbox had already some background in hydrology.
In this course, the students are first exposed to the theory of the HBV model concept including calculations of snowmelt, snow accumulation, soil moisture, effective precipitation, evapotranspiration, and runoff. During theoretical presentations of the course, with the help of the instructor, students perform all the calculations for a hydrologic modeling exercise in the class using an Excel spreadsheet. The reason for using a spreadsheet is to ensure students learn the calculations and how modeling works in general. This part of the course is designed to teach basics of hydrologic modeling, and does not include model calibration, validation and uncertainty. Once the students learn the fundamentals of hydrologic modeling, the HBV-Ensemble, which includes parameter sampling and calibration module, is presented in the class. With several homework assignments students practice model calibration, sensitivity analysis, the effects of initial conditions on model simulations, etc. For the final project, students are required to simulate the streamflow for a watershed in California and submit a detailed project report.
In 2011 and 2012, a total of 60 students completed the project from which 56 students participated in an anonymous survey designed to gauge students' learning gains. The survey was administered once the students learned about the processes of HBV and how the toolbox works, but prior to completing the final project. Table 1 summarizes the survey questions. The first ten questions (Q1-Q10) aimed to gauge students' learning gains as a result of using the presented education toolbox. The last four questions (Q11-Q14) aimed to understand which aspects of this teaching tool contributed to students' learning gains. Figure 5 presents students' responses on their learning gains using a five-point ranking scale where: 1 = no gains; 2 = a little gain; 3 = moderate gain; 4 = good gain; and 5 = great gain. Figure 5 (top panel) displays the mean and confidence intervals (here defined as ± 3 × the standard error) of student responses for each question. Figure 5 (bottom panel) shows the boxplots of the students' responses. On each box in Fig. 5 (bottom panel), the central red mark refers to the median, while the box edges are the 25th (25q) and 75th (75q) quantiles of the data. In the figure, the outliers, defined as data points larger than 75q + 1.5 (75q − 25q) or smaller than 25q − 1.5 (75q − 25q) (see McGill et al., 1978;Velleman and Hoaglin, 1981), are marked with a plus sign. The whiskers in Fig. 5 (bottom panel) represent the range of data points not considered outliers. Figure 5 indicates that this educational software had a positive impact on students understanding and knowledge of hydrological processes. Note that the students were asked to evaluate their learning gains as a result of their work with this education toolbox in the class (see Table 1). However, the authors acknowledge that evaluating students' responses and associating them to only the educational toolbox and not to the combination of instruction and model used was not possible in  Table 1); 1 = no gains; 2 = a little gain; 3 = moderate gain; 4 = good gain; and 5 = great gain. the current study. Furthermore, we acknowledge that a positive student feedback does not necessarily prove the success of a teaching strategy and/or an educational approach, and it may only reflect a lumped assessment of the course.
It is worth repeating that the theoretical aspects of hydrologic modeling were introduced prior to using HBV-Ensemble. The authors recommend using this educational toolbox after students are introduced to theoretical hydrology. Students, especially undergraduate students, without basic knowledge of hydrology may not be able to benefit from this educational toolbox. Currently, efforts are underway to improve HBV-Ensemble by providing additional tools for teaching purposes. One example would be providing dotty plots, representing the best model parameterization and the parameter surface for each parameter versus the performance measure.

Conclusions
This study presents a modeling toolbox, HBV-Ensemble, designed for teaching hydrological processes, uncertainty analysis, parameter estimation and model sensitivity to parameters and initial conditions. This modeling toolbox has been used in an upper level watershed modeling class at the University of California, Irvine, and the students' feedback have been positive as shown in Fig. 5.
HBV-Ensemble offers an educational tool that can help students in understanding complex and interconnected hydrological processes. HBV-Ensemble exposes students to practical modeling skills that are necessary for their future As a result of your work with this education toolbox in the class, what gains did you make in each of the followings?

Q1
Hydrologic modeling in general Q2 Water budget analysis Q3 Rainfall-runoff processes, their mathematical formulations and the required calculations to estimate the flood resulting from a given precipitation event Q4 The effect of evapotranspiration on rainfall-runoff processes, its mathematical formulation and the required calculations Q5 The effect of soil moisture on rainfall-runoff processes, its mathematical formulation and the required calculations Q6 Model calibration and ensemble simulation Q7 Sensitivity analysis Q8 Differences between empirical and physically-based parameters Q9 Enthusiasm for the subject of hydrologic modeling and analysis Q10 Confidence in performing hydrologic modeling How each of the following aspects and attributes of the developed teaching tool contributed to your learning gains? Q11 The use of a practical case study with actual data Q12 The use of hands-on calculations in the lecture Q13 The fact that you could change the model parameters and see their effects Q14 The requirement of a hydrologic modeling project using this hands-on toolbox.
careers in hydrology. The presented modeling toolbox provides the opportunity for students to investigate "what-if" scenarios for initial conditions, parameters, objective functions, etc., and practice experiential learning.
We recommend using HBV-Ensemble after students are introduced to theoretical aspects of hydrologic modeling. The toolbox can be used at the conclusion of an undergraduate hydrology class after the students have been already exposed to the fundamental processes; in such settings, the tool can serve as an add-on value for early introduction of advanced concepts on model uncertainty and ensemble predictions. HBV-Ensemble can be used for in-class lab practices and homework assignments to improve students' understanding of hydrological processes. Instructors, students and interested readers can request a free copy of HBV-Ensemble for educational purposes.