Dear authors,

The two reviewers made a number of valid points, for which I am very thankful. Unfortunately, the third reviewer did not find time to review the paper. It appears that the paper is of interest, but has a general deficiency that motivated a few of the points of critique raised by the reviewers. This is the lack of clarity about what is considered satisfactory model performance and what the data actually says, independently of the authors' opinion. Reviewer #1 criticised that hard facts are not sufficiently separated from opinions, whereas Reviewer #2 disagreed on several occasions with the authors on what may be considered satisfactory model performance. I concur with both reviewers and I believe that the paper is only publishable in HESS if these points are clarified, if facts are clearly separated from opinions and if all conclusions are supported by verifiable facts. I suggest to either separate the discussion from the results section, or to more clearly separate opinions and recommendations from the presentation of the results and hard facts in the "Results and discussion" section. In addition, I suggest for the conclusions section to limit the list of conclusions to those that are directly supported by the evidence presented in this paper, which may be followed by a paragraph of a more speculative character, clearly indicating that this reflects the authors' opinion.

I see a great value in confronting model simulations with independent observations to assess model skill and credibility, and in particular to identify model weaknesses and room for improvement. Therefore I think that the analysis presented here has potential to become a valuable addition to the scientific literature, after properly considering the reviewer comments and some additional points of my own, as explained above and below. Since this requires a major revision of the manuscript, I may have to send the revised paper out for another round of reviews.

Below, I listed some of my own thoughts about your responses to Reviewer #1 and #2, followed by a list of comments I had when re-reading your manuscript in the light of the reviewer comments. Please submit, along with the revised manuscript, point-by-point responses, linked to a list of changes in the revised manuscript. The latter can also be a tracked-changes version of the revised manuscript. Thank you for your contribution to HESS and for your willingness to produce the best possible outcome for the scientific community.

Re: REVIEWER #1

Specific Comment 1: I found a range of other statements of opinion that were not clearly separated from facts. I hope you will clear out those, too.

Specific Comment 2: I appreciate the authors' willingness to conduct additional analysis and look forward to seeing the results. I would propose to also add a discussion in the paper of the expectation that the data should scatter around the Budyko curve and present the data points used in the present study within the Budyko graphs to support this claim. This may also reveal any bias in precipitation or net radiation forcing, as dots outside of the Budyko envelope indicate violation of the mass or energy balance, as long as changes in storage are excluded (therefore long-term averages!).

P5L10: Please add the explanation to the paper and also explain what the different performance metrics mean in a practical sense. See also my comment below, wrt. P6L30.

P6L10: I did not realise that sigma values were calculated based on a gridded data set, rather than the catchment data. I do not understand your rationale for this. Why use an observation-based model as reference here, if you claim in the introduction that the models were evaluated using observations from 966 catchments as reference? Please modify your analysis for consistency or add a compelling explanation to the text. The explanation given in your response seems to undermine your original rationale for using the catchment data.

Tables 4 & 5: I do appreciate the detailed information, but I also agree that it is hard to read and interpret. Instead of the apparently continous colour scale, I would suggest three colours consistent with what you consider "unsatisfactory", "satisfactory" and "good" performance, or something along these lines. If this is more illustrative, you could also consider moving the table into the SI and provide bar charts instead, with a bar for each model and the different performance measures on the horizontal axis. Just an idea, not sure if it would help.

Re: REVIEWER #2

General comment: Please add a discussion of the reviewer's concerns about what is considered "poor" or "satisfactory" performance to the paper and justify your own criteria up front, before discussion the results. Please also make very clear what are the main insights to be gained from your analysis, also in the context of your 2016 WRR paper.

Specific comments:

1) I think the reviewer made a very good suggestion here, and I do not see a reason to stick to the eartH2Observe collection, if additional insights could be gained by adding one more model realisation. However, if this addition would not add new insights or change the conclusions, you may as well just discuss this issue in the paper and not "contaminate" your analysis with an additional simulation data set.

2) Please include a discussion of the reviewer's points in the context of your previous paper in the current manuscript. Please also refer to my comment below wrt. P17L22-, and add a table as suggested by the reviewer, or point to the exact table in Dutra et al. (2015), if such a table exists. I have not found it. You may put this table in the SI, if you feel that it would disrupt the flow of the paper, but please discuss it in the manuscript. Your response about the scope of the models wrt. representing reality at scales <5000 km2 and implications for the verifiability of the models is also worth including in the discussion, in my opinion.

3) Please add NSE values as suggested and a discussion of their meaning to the manuscript. The current blunt dismissal of NSE in the manuscript is not very helpful.

P6L14: If I am not mistaken, the reviewer is worried that the mean difference may obscure differences in timing, meaning that a value of 0 may be considered a very good result, whereas in reality over-estimation in one period is compensated by under-estimation in another. If adding Stdev is not justified because of deviations from normality, please at least discuss the meaning of the D values and explain how they should be interpreted in the context of other metrics. You could also think of providing some combined index that combines different metrics to produce a model-data correspondence between 0 and 1.

P17L34: Does this mean that the procedure was not sufficiently documented? Please explain in the text.


EDITOR's COMMENTS:

In addition to the reviewers' points, I would like to see clear statements about the expected model skills and the associated performance metrics and what would be considered satisfactory model performance. Just to grab an example, what do values of spatial correlation in the runoff coefficient between 0.3 and 0.67 mean and what values would be considered satisfactory? If the expected model skill is to predict trends in mean annual runoff, then neither of the models discussed in this paper appears to be useful, as they do not reproduce the trends. In this context, the mention of "studies assessing the hydrological impacts of climate change" in the abstract warrants a discussion of the suitability of these models for this purpose.

P2L17: Really all studies?

P2L24: In Table 1, most studies include the daily time scale, whereas in your study, you also included monthly runoff data in the evaluation. Could you clarify what you mean and give examples for the additional insights gained. This could then be moved to the discussion.

P2L28: What are the new insights gained by including more models?

P3L5: Need to discuss what is meant by "well". Reproduction of a time series?

P3L13: Really daily? On P4L25 you state that daily and monthly observed data was used, and on P6L2 you clarify that 325 catchments (i.e. 30%) had only monthly data.

P3L14: More reliabale conclusions than previous studies? In how far?

P15L9: Could you provide a clearer analysis illustrating the similarity in the produced trends? Fig. S1.8 contains the data, but it is hard to tell in how far the colours are similar between models for the same catchments.

P6L17: Do you mean spatial variability or variance? How was it calculated?

P6L30: A value of 0 for Pearson's r means no relationship, negative values imply a negative relationship. Clearly, one would not say that 0 is better than -0.2, so your statement here needs to be more specific. Please specify what values would be considered satisfactory.

P12L34: Was the performance satisfactory in other regions? How do you define satisfactory performance?

P17L4-7: Calibration does not compensate for lack of process understanding or other model deficiencies, it just reduces their effect for data similar to the calibration data. This is precisely what your results reflect. The more similar the calibration data and the metrics used in the calibration to your validation data, the better the perceived model performance. What can you tell about the degradation of the model performance under climate change? I would argue that a well-calibrated model is more likely to create a false sense of accuracy in prediction mode than an uncalibrated model.

P17L22-: Earlier in the manuscript, you refer to this section for details about the calibrations performed. The details given here are insufficient, as they do not contain any information about what kind of parameters were calibrated. Were these only parameters relating to physical catchment properties, or also related to e.g. vegetation properties, land cover etc.? What were the objective functions and how exactly do the objective functions and calibration data sets overlap with those used in the performance analysis presented here? Perhaps a table would be helpful here.

P18L29-: What evidence can be presented to claim that WFDEI precipitation is more biased than PRISM precipitation? In order to support your claim that bias in MAR is due to bias in WFDEI precipitation, you could use PRISM precipitation as input and see if the bias goes away. Otherwise, this seems like speculation.

P18L34: Please provide a quantitative analysis supporting "comparable bias pattern". At first eye shot, Panels a and b may look similar, but this is hardly a defendable analysis and the conclusion that the P bias propagates into MAR bias is hence not clearly supported by evidence.

P20L24: What is a "multi-parameterization ensemble"?

Conclusions:
4.: Clearly, if a model is calibrated to maximise the performance metrics ("performance metrics incorporated in the respective objective functions"), it will "perform" better than if it is not calibrated or calibrated to maximise some other metrics. Therefore, this line of argument is circular and the resulting recommendations (statements starting with "should be") potentially flawed. I consider both statements starting with "should be" as controversial and clearly the authors' opinion, not necessarily a result of the data presented. Please re-word or put your opinions in a separate paragraph, clearly identifiable as opinions.
5.: The statement in this form is not supported by the evidence presented. The evidence merely shows that the WFDEI P deviates from PRISM P, apparently in similar locations where simulated MAR also deviates from the observed. See my comments above, with respect to P18.

Dear authors,

I agree with the reviewers that the manuscript has improved substantially, but I think that some of the remaining points are not just minor, as they could potentially change some of the conclusions.

In Report #1, the reviewer doubts that the conclusion about the potential of multi-parameterization ensembles is backed by the analysis and asks for additional analysis to support the statements. The report further highlights a potential flaw in the evaluation of SWBM runoff results and the potential sensitivity of the conclusions to uncertainty in the daily climate data. If it is not feasible to remove these ambiguities, they should at least be clearly discussed.
In Report #2, the reviewer echoes my original concern about the normalization of the performance metrics using the GSCD data. In contrast to your assessment that a different choice of normalization would probably not have affected the results, the reviewer suggests that the normalization may affect the conclusions. I think this is too important to be left to speculations either way and I would like to ask you to provide evidence (in the manuscript) that the normalization does indeed not affect the results and conclusions. As pointed out by the reviewer, Table A5 is still very difficult to comprehend. I am not sure if a heat map or bar chart would help, but perhaps a reduction of the table to convey the main points needed to support the conclusions would be helpful, while the full table could be given in the SI. I also concur with the reviewer that the rationale for limiting the Budyko plots to the lower latitudes (<50 degr.) is not clear. Why not include all data? You should at least show the same plots including all data in the SI and briefly discuss any striking differences.

In addition to the reviewer comments, I still have a few comments of my own:

EC1 - The paper contains a number of value statements that are not clearly backed by the analysis, and some could even be misleading. In several occasions, you use the word "reliable", which implies some degree of trust and robustness. When referring to models, I would propose to replace "reliable models" by "accurate model simulations", as your assessments are drawn from correspondence between particular observations and model simulations, which strongly depend on the calibration procedure and data. When referring to conlusions, I really see no evidence to suggest that your conclusions are more reliable and generalizable than those presented in other papers, so I would remove this statement and instead be more specific about the additional insights gained.

EC2 - I also think that the value of calibration is over-stated and not discussed in a balanced way. As I alluded to before, calibration does not necessarily account for, nor compensate for inadequate understanding or missing information. All it evidently does is to reduce mismatch between model simulations and (calibration) data. Therefore, I would propose to replace the sentence on P17L12, "Calibration is a prerequisite for both conceptual and physically-based hydrological models to provide reliable runoff estimates, to account for (i) the impossibility of measuring all required model parameters at the model application scale, (ii) lack of process understanding..." by something like:
"Calibration is necessary whenever model parameters cannot be estimated a priori and is also used in both conceptual and physically-based hydrological models to reduce simulation-observation mismatch, commonly assumed to be caused by (i) lack of process understanding..."
Your result that calibration against one signature does not improve the model-observation mismatch in another signature does not fill me with confidence that the calibration makes the model any more "reliable", but you are of course free to express your own opinion, as long as you mark it as such.

EC3 - In any case, the reader is entitled to know what the calibration actually entailed, i.e. how much overlap between calibration and validation data there was for the different models and what process parameters were actually calibrated, be it conceptual or physically-based.

EC4 - This is crucial for the missing discussion of the reliability of the models or multi-model ensembles in assessing climate change impacts, which was also requested by Reviewer #2 in Report #1. For example, if one calibrates parameters relating to vegetation water use, one should expect that the calibration will become inadequate as vegetation properties change, e.g. due to change in rainfall or atmospheric CO2 concentrations. Please add a balanced discussion about the utility of this approach for climate change analysis or remove reference to climate change analysis and specify more clearly what the intended use of the models and multi-model ensembles is. I doubt that the intended use is to "mimic reality", where "reality" is expressed by a set of observations. One could use a purely statistical model for this, or, to be provocative, a set of polynomials of an arbitrary level of complexity.

I am sorry to ask for additional work at this stage, but given the potential impact of the concerns expressed in this round of review, I cannot accept the paper for publication in HESS in its current form. I hope that this last revision will result in a valuable paper.

Dear authors,

Thank you for your thoughtful responses and for addressing all the remaining points. I just have one more question regarding Fig. 3:
How come some models have no data for AI>2? Were they not all based on the same climate input data?
Please check and/or discuss the different ranges in AI and also please provide definitions of AI and RC in the figure caption.

Best regards,
Stan Schymanski