Hydrology and Earth System Sciences www.hydrol-earth-syst-sci.net 1027-5606 1607-7938 13 2 2009 10.5194/hess-13-141-2009 http://www.hydrol-earth-syst-sci.net/13/141/2009/ http://www.hydrol-earth-syst-sci.net/13/141/2009/hess-13-141-2009.html http://www.hydrol-earth-syst-sci.net/13/141/2009/hess-13-141-2009.pdf 141 153 2009-02-05 The european flood alert system EFAS – Part 2: Statistical skill assessment of probabilistic and deterministic operational forecasts J. C. Bartholmes jens.bartholmes@ec.europa.eu J. Thielen M. H. Ramos S. Gentilini EC, Joint Research Centre, Institute for Environment and Sustainability, Via Fermi 2749, 21027 Ispra (VA), Italy now at: EC, Joint Research Centre, Scientific EC Work Programme Unit, Square de Meeus 8, 1050 Brussels, Belgium now at: CEMAGREF, Parc de Tourvoie 44, 92163 Antony Cedex, France Since 2005 the European Flood Alert System (EFAS) has been producing probabilistic hydrological forecasts in pre-operational mode at the Joint Research Centre (JRC) of the European Commission. EFAS aims at increasing preparedness for floods in trans-national European river basins by providing medium-range deterministic and probabilistic flood forecasting information, from 3 to 10 days in advance, to national hydro-meteorological services. <br><br> This paper is Part 2 of a study presenting the development and skill assessment of EFAS. In Part 1, the scientific approach adopted in the development of the system has been presented, as well as its basic principles and forecast products. In the present article, two years of existing operational EFAS forecasts are statistically assessed and the skill of EFAS forecasts is analysed with several skill scores. The analysis is based on the comparison of threshold exceedances between proxy-observed and forecasted discharges. Skill is assessed both with and without taking into account the persistence of the forecasted signal during consecutive forecasts. <br><br> Skill assessment approaches are mostly adopted from meteorology and the analysis also compares probabilistic and deterministic aspects of EFAS. Furthermore, the utility of different skill scores is discussed and their strengths and shortcomings illustrated. The analysis shows the benefit of incorporating past forecasts in the probability analysis, for medium-range forecasts, which effectively increases the skill of the forecasts. Atger, F.: Verification of intense precipitation forecasts from single models and ensemble prediction systems, Nonlin. Processes Geophys., 8, 401–417, 2001. Baldwin, M. E. and Kain, J. S.: Examining the sensitivity of various performance measures, 17th Conf. on Probability and Statistic in the Atmospheric Sciences, 84th AMS Annual Meeting, Seattle, WA, 2.9, 1–8 January, 2004. Bartholmes, J. and Todini, E.: Coupling meteorological and hydrological models for flood forecasting, Hydrol. Earth Syst. Sci., 9, 333–346, 2005. Bartholmes, J., Thielen, J., and Ramos, M. H.: Quantitative analyses of EFAS forecasts using different verification (skill) scores, in: The benefit of probabilistic flood forecasting on European scale, edited by: Thielen, J. , EUR 22560 EN, 58–79, 2006. Bradley, A. A., Schwartz, S. S., and Hashino, T.: Distributions-Oriented Verification of Ensemble Streamflow Predictions, J. Hydrometeorol., 5(3), 532–545, 2004. Brier, G. W.: Verification of forecasts expressed in terms of probability, Mon. Weather Rev., 78, 1–3, 1950. Clark, M. P. and Hay, L. E.: Use of Medium-Range Numerical Weather Prediction Model Output to Produce Forecasts of Streamflow, J. Hydrometeorol., 5(1), 15–32, 2004. De Roo, A., Wesseling, C. G., and Van Deurssen, W. P. A.: Physically based river basin modelling within a GIS: the LISFLOOD model, Hydrol. Process., 4(11–12), 1981–1992, 2000. EEA, European Environment Agency: Mapping the impacts of recent natural disasters and technological accidents in Europe, \textitEnvironmental issue report No. 35, published by European Environment Agency, Copenhagen, 2003, p 47, 2003. Flueck, J. A.: A study of some measures of forecast verification, Preprints, 10th Conf. on Probability and Statistics in Atmospheric Sciences, Edmonton, AB, Canada, Amer. Meteor. Soc., 69–73, 1987. Franz, K. J., Hartmann, H. C., Sorooshian, S., and Bales, R.: Verification of National Weather Service Ensemble Streamflow Predictions for Water Supply Forecasting in the Colorado River Basin, J. Hydrometeorol., 4(6), 1105–1118, 2003. Gandin, L. S. and Murphy, A.: Equitable skill scores for categorical forecasts, Mon. Weather Rev., 120, 361–370, 1992. Gilbert, G. F.: Finley's tornado predictions, Am. Meteorol. J., 1, 166–172, 1884. Göber, M., Wilson, C. A., Milton, S. F., and Stephenson, D. B.: Fairplay in the verification of operational quantitative precipitation forecasts, J. Hydrol., 288, 225–236, 2004. Gordon, N. D. and Shaykewich, J. E.: Guidelines on Performance Assessment of Public Weather Services – Geneva: WMO, 2000, (WMO/TD 1023), 2000. Hamill, T. M. and Jura, J.: Measuring forecast skill: is it real skill or is it the varying climatology?, Q. J. Roy. Meteorol. Soc., 132, 2905–2923, 2006. Hanssen, A. W. and Kuipers, W. J. A.: On the relationship between the frequency of rain and various meteorological parameters, Meded. Verh., 81, 2–15, 1965. Heidke, P.: Berechnung des Erfolges und der Gute der Windstarkevorhersagen im Sturmwarnungsdienst, Geogr. Ann., 8, 301–349, 1926. Legg, T. P. and Mylne, K. R.: Early warnings of severe weather from ensemble forecast information, Weather and Forecasting, 19, 891–906, 2004. Kalas, M., Ramos M. H., Thielen, J., and Babiakova, G.: Evaluation of the medium-range European flood forecasts for the March–April 2006 flood in the Morava River, J.Hydrol. Hydromech., 56(2), 116–132, 2008. Marzban, C.: Scalar measures of performance in rare-event situations, Weather and Forecasting, 13, 753–763, 1998. Mason, I.: Dependence of the critical success index on sample climate and threshold probability, Aust. Met. Mag., 37, 75–81, 1989. Ebert, E. and McBride, J. L.: Methods for verifying quantitative precipitation forecasts: application to the BRMC LAPS model 24-hour precipitation forecasts, BRMC Techniques Development Report No 2, 87 pp., 1997. McBride, J. L. and Ebert, E.: Verification of quantitative precipitation forecasts from operational Numerical weather prediction models over Australia, Weather and Forecasting, 15, 103–121, 2000. Mullusky, M., Demargne, J., Welles, E., Wu, L., and Schaake, J.: Hydrologic applications of short and medium range ensemble forecasts in the NWS Advanced Hydrologic Prediction Services (AHPS), AMS 16th Conference on Numerical Weather Prediction Symposium on Forecasting the Weather and Climate of the Atmosphere and Ocean, Seattle, Paper J11.5, 2004. Murphy, A. H. and Epstein, E. S.: A note on probability forecasts and "hedging", J. Appl. Meteor., 6, 1002–1004, 1967. Murphy, A. H. and Winkler, R. L.: A general framework for forecast verification, Mon. Weather. Rev., 115, 1330–1338, 1987. Murphy, A. H.: The Finlay Affair. A signal event in the history of forecast verification, Weather and Forecasting, 11, 3–20, 1996. Murphy, A. H.: Forecast verification, in: Economic Value of Weather and Forecasts, edited by: Katz, R. W. and Murphy, A. H., 19–74, Cambridge: Cambridge University Press, 1997. Nash, J. E. and Sutcliffe, J. V.: River flow forecasting through conceptual models Part~1 – A discussion of principles, J. Hydrol., 10(3), 282–290, 1970. Pappenberger F., Scipal, K., Buizza, R.: Hydrological aspects of meteorological verification, Atmos. Sci. Lett., 9(2), 43–52, 2008. Pedemonte L., Corazza, M., Sacchetti, D., Trovatore, E., and Buzzi, A.: Verification of limited-area models precipitation forecasts during the map-sop, Proceedings ICAM/MAP 2005 Zadar, Croatia, 23–27~May, 2005. Peirce, C. S.: The numerical measure of success of predictions, Science, 4, 453–454, 1884. Roulin, E.: Skill and relative economic value of medium-range hydrological ensemble predictions, Hydrol. Earth Syst. Sci., 11, 725–737, 2007. Roulin, E. and Vannitsem, S.: Skill of Medium-Range Hydrological Ensemble Predictions, J. Hydrometeorol., 6(5), 729–744, 2005. Schaefer, J. T.: The Critical Success Index as an Indicator of Warning Skill, Weather and Forecasting, 5, 570–575, 1990. Stanski, H. R., Wilson, L. J., and Burrows, W. R.: Survey of common verification methods in meteorology, Technical Report No 8 Geneva: WMO, 1989 (WMO/TD 358), 1989. Stephenson, D. B.: Use of the "odds ratio" for diagnosing forecast skill, Weather and Forecasting, 15, 221–232, 2000. Thielen, J., Bartholmes, J., Ramos, M.-H., and de Roo, A.: The European Flood Alert System - Part 1: Concept and development, Hydrol. Earth Syst. Sci., 13, 125–140, 2009. Van Der Knijff, J. and De Roo, A.: LISFLOOD – distributed water balance and flood simulation model, User manual; EUR Report EUR 22166 EN, 2006. Wilson, C.: Review of current methods and tools for verification of numerical forecasts of precipitation, COST-717 Working document WDF_01_200109_1, www.smhi.se/cost717/, 2001. Wilks, D. S.: Statistical Methods in the Atmospheric Sciences, Academic Press, San Diego,CA, 265–267, 1995. Younis, J., Ramos, M. H., and Thielen, J.: EFAS forecasts for the March–April 2006 flood in the Czech part of the Elbe river basin – a case study, Atmos. Sci. Lett., 9, 88–94, doi:10.1002/asl.179., 2008.