Hydrology and Earth System Sciences www.hydrol-earth-syst-sci.net 1027-5606 1607-7938 11 4 2007 10.5194/hess-11-1267-2007 http://www.hydrol-earth-syst-sci.net/11/1267/2007/ http://www.hydrol-earth-syst-sci.net/11/1267/2007/hess-11-1267-2007.html http://www.hydrol-earth-syst-sci.net/11/1267/2007/hess-11-1267-2007.pdf 1267 1277 2007-05-03 Verification tools for probabilistic forecasts of continuous hydrological variables F. Laio francesco.laio@polito.it S. Tamea Dipartimento di Idraulica, Trasporti ed Infrastrutture Civili, Politecnico di Torino, Torino, Italy In the present paper we describe some methods for verifying and evaluating probabilistic forecasts of hydrological variables. We propose an extension to continuous-valued variables of a verification method originated in the meteorological literature for the analysis of binary variables, and based on the use of a suitable cost-loss function to evaluate the quality of the forecasts. We find that this procedure is useful and reliable when it is complemented with other verification tools, borrowed from the economic literature, which are addressed to verify the statistical correctness of the probabilistic forecast. We illustrate our findings with a detailed application to the evaluation of probabilistic and deterministic forecasts of hourly discharge values. Abramson, B. and Clemen, C.: Probability forecasting, Int. J. Forecast., 11, 1–4, 1995. Berkovitz, J.: Testing density forecasts, with applications to risk management, J. Bus. Econ. Stat., 19, 465–474, 2001. Beven, K.: A manifesto for the equifinality thesis, J. Hydrol., 320, 18–36, \doi10.1016/j.jhydrol.2005.07.007, 2006. Box, G. E P. and Jenkins, G M.: Time series analysis: forecasting and control, Holden Day, San Francisco, CA, 1970. Chatfield, C.: Prediction intervals for time series forecasting, in: Principles of forecasting: a handbook for researchers and practitioners, edited by Armstrong, J., pp. 475–494, Kluwer Academic Publishers, Norwell, MA, 2001. Christoffersen, P F.: Evaluating interval forecast, Int. Econ. Rev., 39, 841–862, 1998. Cleveland, W S. and Loader, C L.: Smoothing by Local Regression: Principles and Methods, in: Statistical Theory and Computational Aspects of Smoothing, edited by: Haerdle, W. and Schimek, M G., 10–49, Springer, New York, 1996. D'Agostino, R B. and Stephens, A M., eds.: Goodness-of-fit techniques, Dekker, New York, 1986. De~Gooijer, J G. and Zerom, D.: Kernel-based multistep-ahead predictions of the US short-term interest rate, J. Forecast., 19, 335–353, 2000. Diebold, F X., Gunther, T A., and Tay, A S.: Evaluating density forecasts with applications to financial risk management, Int. Econ. Rev., 39, 863–883, 1998. Epstein, E S.: A scoring system for probability forecasts of ranked categories, J. Appl. Meteorol., 8, 985–987, notes and correspondence, 1969. Fan, J. and Gijbels, I.: Local polynomial modelling and its applications, Chapman and Hall, London, UK, 1996. Farmer, J D. and Sidorowich, J J.: Predicting chaotic time series, Phys. Rev. Lett., 59, 845–848, 1987. Ferraris, L., Rudari, R., and Siccardi, F.: The uncertainty in the prediction of flash floods in the Northern Mediterranean Environment, J. Hydrometeorol., 3, 714–727, 2002. Gangopadhyay, S., Clark, M., and Rajagopalan, B.: Statistical downscaling using K-nearest neighbors, Water Resour. Res., 41, \doi10.1029/2004WR003444, 2005. Georgakakos, K P., Seo, D.-J., Gupta, H., Schaake, J., and Butts, M B.: Towards the characterization of streamflow simulation uncertainty through multimodel ensembles, J. Hydrol., 298, 222–241, \doi10.1016/j.jhydrol.2004.03.037, 2004. Goodman, L A. and Kruskal, W H.: Measures of association for cross classifications, J. Am. Stat. Assoc., 49, 732–764, 1954. Hersbach, H.: Decomposition of the continuous ranked probability score for ensemble prediction systems, Weather and Forecasting, 15, 559–569, 2000. Jolliffe, I T. and Stephenson, D B., eds.: Forecast verification: a practitioner's guide in atmospheric science, Wiley, New York, USA, 2003. Kendall, M G. and Stuart, A.: The advanced theory of statistics, Griffin Press, London, 2nd ed., 1977. Krzysztofowicz, R.: The case for probabilistic forecasting in hydrology, J. Hydrol., 249, 2–9, 2001. Laio, F., Ridolfi, L., and Tamea, S.: Probabilistic prediction of real-world time series: a local regression approach, Geophys. Res. Lett., 34, L03403, \doi10.1029/2006GL028776, 2007. Montanari, A.: Large sample behaviors of the generalized likelihood uncertainty estimation (GLUE) in assessing the uncertainty of rainfall-runoff simulations, Water Resour. Res., 41, W08406, \doi10.1029/2004WR003826, 2005. Montanari, A. and Brath, A.: A stochastic approach for assessing the uncertainty of rainfall-runoff simulations, Water Resour. Res., 40, W01106, \doi10.1029/2003WR002540, 2004. Murphy, A H.: Measures of the utility of probabilistic predictions in cost-loss ratio decision situation in which knowledge of the cost-loss ratio is incomplete, J. Appl. Meteorol., 8, 863–873, 1969. Murphy, A H.: The ranked probability score and the probability score: a comparison, Mon. Weather Rev., 98, 917–924, 1970. Murphy, A H.: A note on the ranked probability score, J. Appl. Meteorol., 10, 155–156, 1971. Noceti, P., Smith, J., and Hodges, S.: An evaluation of tests of distributional forecasts, J. Forecast., 22, 447–455, 2003. Palmer, T N.: Predicting uncertainty in forecasts of weather and climate, Rep. Progr. Phys., 63, 71–116, 2000. Porporato, A. and Ridolfi, L.: Nonlinear analysis of river flow time sequences, Water Resour. Res., 33, 1353–1367, 1997. Regonda, S., Rajagopalan, B., Lall, U., Clark, M., and Moon, Y.: Local polynomial method for ensemble forecast of time series, Nonlin. Proc. Geophys., 12, 397–406, 2005. Richardson, D S.: Economic value and skill, in: Forecast verification: a practitioner's guide in atmospheric science, edited by: Jolliffe, I T. and Stephenson, D B., Wiley, New York, USA, 2003. Roulin, E.: Skill and relative economic value of medium-range hydrological ensemble predictions, Hydrol. Earth Syst. Sci., 11, 725–737, 2007. Siccardi, F., Boni, G., Ferraris, L., and Rudari, R.: A hydrometeorological approach for probabilistic flood forecast, J. Geophys. Res., 110, D05101, \doi10.1029/2004JD005314, 2005. Tamea, S., Laio, F., and Ridolfi, L.: Probabilistic nonlinear prediction of river flows, Water Resour. Res., 41, W09421, \doi10.1029/2005WR004136, 2005. Tay, A S. and Wallis, K F.: Density forecasting: a survey, J. Forecast., 19, 235–254, 2000. Todini, E.: Role and treatment of uncertainty in real-time flood forecasting, Hydrol. Process., 18, 2743–2746, 2004. Wilks, D S.: Statistical methods in the atmospheric sciences, Academic press, 1995.