Hydrology and Earth System Sciences www.hydrol-earth-syst-sci.net 1027-5606 1607-7938 10 6 2006 10.5194/hess-10-957-2006 http://www.hydrol-earth-syst-sci.net/10/957/2006/ http://www.hydrol-earth-syst-sci.net/10/957/2006/hess-10-957-2006.html http://www.hydrol-earth-syst-sci.net/10/957/2006/hess-10-957-2006.pdf 957 965 2006-12-14 Optimal estimator for assessing landslide model performance J. C. Huang S. J. Kao sjkao@gate.sinica.edu.tw Research Center for Environmental Changes, Academia Sinica, Taipei, Taiwan The commonly used success rate (SR) in evaluating cell-based landslide model performance is based on the ratio of successfully predicted landslide sites over total actual landslide sites without considering the performance in predicting stable cells. We proposed a modified SR (MSR), in which the performance of stable cell prediction is included. The advantage of MSR is to avoid over- and under-prediction while upholding the stable sensitivity throughout all simulated cases. Stochastic analyses are conducted by using artificial landslide maps and simulations with a full range of performances (from worst to perfect) in both stable and unstable cell predictions. Stochastic analyses reveal mathematical responses of estimators to various model results in calculating performance. The Kappa method, which is commonly used for satellite image analysis, is improper for landslide modeling giving inconsistent performance when landslide coverage changes. To examine differences among SR and MSR in real model application, we applied the SHALSTAB model onto a mountainous watershed in Taiwan. Case study shows that stable and unstable cell predictions are inter-exclusive in SHALSTAB model. The optimal estimator should compromise landslide over- and under-prediction. According to our 4000 simulations, the best simulation generated by MSR projects 83 hits over 131 actual landslide sites while the unstable cells cover only 16% of the studied watershed. By contrast, despite the fact that the best simulation deduced from SR projects 120 hits over 131 actual landslide sites, this high performance is only obtained when unstable cells cover an incredibly high landslide cover (~75%) of the entire watershed exhibiting a significant landslide over-prediction. Ayalew, L. and Yamagishi, H.: The application of GIS-based logistic regression for landslide susceptibility mapping in the Kakuda-Yahiko Mountains, Central Japan, Geomorphology, 65, 15–31, 2005. Borga, M., Fontana, G. D., and Marchi, L.: Shallow landslide hazard assessment using a physically based model and digital elevation data, Environ. Geol., 32, 81–88, 1998. Borga, M., Dalla Fontana, G., Gregoretti, C., and Marchi, L.: Assessment of shallow landsliding by using a physically based model of hillslope stability, Hydrol. Processes, 16, 2833–2851, 2002. Carrara, A., Cardinali, M., Guzzetti, F., and Reichenbach, P.: GIS technology in mapping landslide hazard, in: Geographical Information Systems in Assessing Natural Hazards, edited by: Carrara, A. and Guzetti, F., Kluwer, Dordrecht, Netherlands, pp. 135–175, 1995. Cheng, Y. L.: Study on Risk Analysis of Slopes Consideration Spatial Variability – A Case Study at the Lisan, Master Thesis, National Chunghsing University, 2003. Casadei, M., Dietrich, W. E., and Miller, N. L.: Testing a model for predicting the timing and location of shallow landslide initiation in soil-mantled landscapes, Earth Surf. Processes Landf., 28, 925–950, 2003. Dietrich, W. E., Reiss, R., Hsu, M. L., and Montgomery, D. R.: A process-based model for colluvial soil depth and shallow landsliding using digital elevation data, Hydrol. Processes, 9, 383–400, 1995. Dietrich, W. E. and Montgomery, D. R.: SHALSTAB: A digital terrain model for mapping shallow landslide potential, http://socrates.berkeley.edu/~geomorph/shalstab, 1998. Duan, J. and Grant, G. E.: Shallow landslide delineation for steep forest watersheds based on topographic attributes and probability analysis, in: Terrain Analysis – Principles and Applications, edited by: Wilson, J. P. and Gallant, J. C., John Wiley & Sons, New York, pp. 311–329, 2000. Hsu, M. L.: A Grid-Based Model for Predicting Shallow Landslides: A Case Study in Linkou, Taipei. Eos, Transaction, American Geophysical Union, 79(24), W25, 1998. Huang, J. C., Kao S. J., Hsu, M. L., and Lin, J. C.: Stochastic procedure to extract and to integrate landslide susceptibility maps: An example of mountainous watershed in Taiwan, Nat. Hazards Earth Syst. Sci., 6, 803–815, 2006. Industrial Technology Research Institute: The management of Dai-Jia Reservoir Watershed-the 4th technique report, The management committee of Dai-Jia Reservoir (in Chinese), 1998. Lee, S.: Application and cross-validation of spatial logistic multiple regression for landslide susceptibility analysis, Geosciences, 9(1), 63–71, 2005. Longley, P. A., Goodchild, M. F., and Rhind, D. W.: Geographic Information Systems and Sciences, New York, Wiley, 2001. Montgomery, D. R. and Dietrich, W. E.: A physically based model for the topographic control on the shallow landsliding, Water Resour. Res., 30, 1153–1171, 1994. Pack, R. T., Tarboton, D. G., and Goodwin, C. N.: The SINMAP Approach to Terrain Stability Mapping, Congress of the International Association of Engineering Geology, Vancouver, British Columbia, Canada, 21–25 September 1998. Pack, R. T., Tarboton, D. G., and Goodwin, C. N.: Assessing terrain stability in a GIS using SINMAP, in 15th annual GIS conference, Vancouver, British Columbia, 19–22 February 2001. Schuster, R. L. and Krizek, R. J.: Landslide: analysis and control, Transportation Research Board Special Report, 176, 1–10, 1978. Sidle, R. C., Pearce, A. J., and O'Loughlin, C. L.: Hillslope stability and landuse, Washington, DC, American Geophysical Union, Water Resour. Monogr., 11, 140 pp, 1985. Viera, A. J. and Garrett, J. M.: Understanding interobserver agreement: the Kappa statistics, Fam. Med., 37(5), 360–363, 2005. Zhou, Q. and Liu, X.: Error assessment of grid-based flow routing algorithms used in hydrological models, Int. J. Geogr. Inf. Sci., 16(8), 819–842, 2002. Zhou, G., Esaki, T., Mitani, Y., Xie, M., and Mori, J.: Spatial probabilistic modeling of slope failure using an integrated GIS Monte Carlo simulation approach, Eng. Geol., 68, 373–386, 2003.