Do probabilistic forecasts lead to better decisions?

. The last decade has seen growing research in producing probabilistic hydro-meteorological forecasts and increasing their reliability. This followed the promise that, supplied with information about uncertainty, people would take better risk-based decisions. In recent years, therefore, research and operational developments have also started focusing attention on ways of communicating the probabilistic forecasts to decision-makers. Communicating probabilistic forecasts includes preparing tools and products for visualisation, but also requires understanding how decision-makers perceive and use uncertainty information in real time. At the EGU General Assembly 2012, we conducted a laboratory-style experiment in which several cases of ﬂood forecasts and a choice of actions to take were presented as part of a game to participants, who acted as decision-makers. Answers were collected and analysed. In this paper, we present the results of this exercise and discuss if we indeed make better decisions on the basis of probabilistic forecasts.


Introduction
There is a common concern in today's world about the vulnerability of water resources to fast-changing environments and the capacity of our societies to efficiently mitigate socioeconomic impacts of extreme events. On regional, national or international levels, efforts have been put into coupling meteorological and hydrologic prediction models for improved operational water management and a better anticipation of hydrologic extremes. Such forecasting and warning systems have been developed and applied to improve flood control and drought risk planning, as well as to optimise water management and regulation for different economic uses (domestic, industrial and agricultural water supply, hydroelectricity and thermal power plants, etc.).
The increasing use of outputs from numerical atmospheric models in hydrologic prediction systems, whether for climate change impact analyses or for short-to long-term streamflow forecasting, has been accompanied by an increasing interest of hydrologists to develop appropriate techniques for the quantification and propagation of uncertainties through the prediction chain (e.g. Krzysztofowicz, 2001;Pappenberger et al., 2005;Bogner and Pappenberger, 2011;Addor et al., 2011). To make the best possible decisions, forecasts with uncertainty estimates are essential. However, recognising that probabilistic hydrological predictions are necessary is not sufficient. In particular, probabilistic forecasts, predictive distributions or ensemble traces of future evolution of streamflows are not a guarantee that forecasts are going to be useful. High quality data, sophisticated models and colourful graphical representations may lose their value if probabilistic forecasts do not reach their users, if the provider is not trusted or if forecasts are not appropriately understood (Faulkner and Ball, 2007;McCarthy et al., 2007;Ramos et al., 2010;Frick and Hegg, 2011;Pappenberger et al., 2013).
Efforts must be put into providing guidance on how to interpret model outputs and how to use probabilistic predictions to support decisions. This implies communicating forecast verification information and building knowledge on the weaknesses and strengths of probabilistic forecasts. Practice is essential, but unfortunately not always easily acquired. In a recent interview-based research, Demeritt el al. (2010) showed that only 3 out of 24 flood forecasting centres in Europe have their own fully operational hydrological ensemble/probabilistic prediction system running, even though they have access to probabilistic meteorological predictions. These three forecasting centres may have learned valuable lessons, which can certainly be useful to other centres in their own implementations. Knowledge transfer is important for the community, but the question remains how to do it. Operational systems must often be adapted to local contexts and site-specific needs, and thus every new implementation is usually accompanied by site-specific challenges. This leads to a lack of practice on probabilistic-based systems and naturally leaves forecasters and decision-makers uncertain when an extreme event is predicted: how reliable is the system? What probability level is crucial to my forecast situation? How to act accordingly? On which scenario should I base my decision? Is the risk of making a wrong decision acceptable?
Unfamiliarity with probabilistic forecast products is a reality, even among well-trained operational forecasters and experienced managers. Although many are convinced that forecast uncertainty information is useful and must be provided, few claim that it does improve the entire decision-making process. "Do probabilistic forecasts really lead to better decisions?" This is the general question we address in this paper. The experiment described in the next section was set up to get hydrologists, forecasters, scientists and managers involved in a practical example and to provide some quantitative assessment of the benefits of uncertainty information in decision-making in the case of flood forecasting. Participants were prompted to make decisions when forecasts were provided with and without uncertainty information. Results of the experiment are presented in Sect. 3, while Sect. 4 is dedicated to discussion and conclusions.

Background
In this paper, we present a very simple case of decisionmaking under uncertainty for a hypothetical flood control problem. It can be viewed as a laboratory-based study, which is traditionally employed in experimental economics (Kagel and Roth, 1995). A controlled experiment (in our case, a simple game) is used to investigate decisions made by individuals when faced with forecasts that are displayed with and without uncertainty information. Roulston et al. (2006) demonstrate the value of such techniques to analyse risk exposure and decision-making for a case on the use of probabilistic information in temperature forecasts. With the aim to investigate psychological factors that influence decisions, Joslyn et al. (2011) also set up laboratory experiments where the most likely scenarios of weather forecasts were presented without or with the lower and/or upper bounds of the predictive interval.
In our experiment, the focus is on hydrological forecasting and flood protection. To implement our experiment, we integrated the first basic parts of the analysis of a decision problem, as identified by Wilks (1997) and Stewart (1997), and adapted them to our case study: 1. identification of the user's goals; 2. definition of the actions available to the decision-maker; 3. identification of the relevant information that should be available for the decision-making, including a decision rule based on a flood threshold level and a payoff function; 4. identification of the possible future unknown events that may occur and the probabilities/uncertainty associated with them; 5. specification of consequences following each possible action-event pair.
The experiment was set up as a game, based on a hypothetical situation. It was adapted to be conducted with attendees of the session on probabilistic/ensemble forecasting at the European Geophysical Union General Assembly meeting 2012 (Session HS4.3/AS1.18/NH1.2 -Ensemble hydrometeorological forecasting for improved risk management: across scales and applications). We had 15 min to explain the rules and conduct the experiment with a group that we expected to be composed of people from different backgrounds and levels of experience with probabilistic forecasts (the audience included operational forecasters, decision-makers, research scientists as well as students). Such time constraints and diversity of participants had to be taken into account when setting the experiment, as presented hereafter. Figure 1 illustrates the decision problem as it was posed to participants. Participants were informed that their company had received 30 000 tokens for a flood protection contract, and that they had to manage a gate which was the inlet of a retention basin designed to protect the town of Bigrivers. The decision they had to take was whether to open the gate to the retention basin or not. It was explained to them that:

Definition of the decision problem
-if they open the gate, the retention basin is flooded and the farmers in this basin demand a compensation for flooding their land: the cost of opening the gate is 2000 tokens; -if they decide not to open the gate and a flood occurs on the river, the town is flooded and they have to pay a fine of 7000 tokens.
Participants were informed that they would be presented with several consecutive forecasts of the river water level and that flooding on the river would occur if the actual water level exceeded 3.9 m.

Different levels of uncertainty information provided
To address the impact of uncertainty information on decision-making, our experiment was played in two phases, in the following order: Game 1 was designed to provide participants with all available information (i.e. the expected value, the error band on the expected value and the estimated probability of flooding), while Game 2 was designed to present first only the expected value of the forecast and then also the error band on the expected value in the last forecast cases. Due to time constraints, each game was composed of only six rounds (i.e. six consecutive forecast cases), of which two were associated with the occurrence of a flood. The players in the room were not previously informed that there would be two games to play. Besides, when moving to Game 2, they were not warned about the differences on the available forecast information between the games. Our aim here was to create a "surprise effect" when facing the same game again but then without information on uncertainty in the forecasts.

Game 1
Forecasts were presented as an expected value (e.g. a model output) with an error, and a probability of flooding. Figure 2 (top panel) shows the screen for Round 4 in Game 1.
The decision problem was stated as a question: "do you want to open the gate to the retention basin?" Time was given to participants and, once a decision was made ("yes" or "no"), participants were informed if a flood event had occurred or not. According to the appropriate situation (i.e. the action-event pair of that round), they were told one of the following consequences: -if a flood occurred and they had decided to open the gate, they had protected the city and had only to pay the farmers; -if a flood occurred, but they had decided not to open the gate, the city was flooded and they had to pay the fine of 7000 tokens; -if it did not flood, and they had decided to open the gate, they had to pay the farmers; Example of the rounds as they were presented to participants for their decision-making: during Game 1 with uncertainty information (top panel); during Game 2 without uncertainty information (bottom panel). Only the last two rounds of Game 2 showed ± error bands in additional to the expected output value.
-if it did not flood, and they had decided not to open the gate, neither the farmers nor the city were flooded and they had nothing to pay.
Participants were invited to adjust their initial purse of 30 000 tokens accordingly. The next round was then presented and the exercise was repeated until the last round.

Game 2
After playing the six rounds of Game 1, participants were invited to verify the final amount of money in their purse and to continue to Game 2. The rules were re-explained (they were the same) and a new 6-round play started, with a new initial purse of 30 000 tokens. Game 2 started without information on the forecast uncertainty, and only in Rounds 5 and 6 were error bounds shown together with the forecast value. Figure 2 (bottom panel) shows the screen for Round 3 as presented to participants in Game 2. Game 2 was designed using the same forecasts and events used in Game 1, but without displaying the full information on uncertainties or probabilities, and with a randomised order of appearance. Table 1 shows the data used in the forecast cases and their position (round number) in Game 1 and Game 2. It also indicates the cases associated with the occurrence of a flood.

Evaluation
Two strategies were adopted for the evaluation of the experiment.
1. During the game, helpers in the auditorium counted the number of decisions made in favour to open the gate and, subsequently, the number of decisions for not opening it (participants were asked to raise hands). The resulting majority vote was entered in the game during the presentation. In this paper, this is noted as "ensemble result", as it reflects the performance of all participants as a whole.
2. In the beginning of the session, we distributed A5-size worksheets and participants were encouraged to keep note of their decisions and the results of each round (one side contained a form for Game 1 and the other side for Game 2). The worksheets were collected at the end of the game. The results of their evaluation are presented in the next section.
For the evaluation of the worksheets, we focused initially on the relative frequencies of "yes" and "no" decisions for each forecast case. We also investigated the changes in decisions from "yes" (open the gate) to "no" (do not open the gate), and vice-versa, between consecutive rounds. We restricted our evaluation to a first-order process: the only relevant information to explain the decision at one round is in the previous round.
For each round k ∈ [2, 6], the one-step transition probability from state i to state j is defined as the conditional probability: (1) In our case, two states are possible: = 1 (decision to open the gate) or = 0 (decision not to open the gate). For each round k, we thus have the following 2 × 2 transition matrix: Each transition probability matrix can be estimated by counting the number of times, among all participants, the decisions went from state i to state j , n ij , when moving from round k − 1 to round k: For the "initial state" at round k = 1, it is a vector with the probabilities of "yes" decisions (open the gate) and "no" decisions (do not open the gate) that can be estimated: In our analysis, we focused on the n ij number of participants that moved from state i to state j for each round transition.
In a second step, we evaluated, for each participant, the differences between the remaining amounts of monetary resources left after all rounds were played in Game 1 and Game 2. They represent a loss (negative differences) or a gain (positive differences) in final purses when the decisions were taken without uncertainty information: They are also shown here as a percentage of the initial purse: Finally, in order to investigate if there is a relationship between available monetary resources and decisions (e.g. are participants more risk-avoiding, i.e. more inclined to answer "yes" to open the gate, if they have less money available in their purse?), we evaluated the distribution of the amounts of money participants had in hand before making a decision in round k, for all k ∈ [1, 6], according to the type of decision they made ("yes" or "no") in that round.

Results
We collected a total of 101 sheets, of which only 3 were incomplete. Here we present thus the results obtained from 98 players. When pertinent, the "ensemble result", i.e. the majority vote obtained from counting the hands raised during the game, is also presented.

General impact of uncertainty information on attitude towards risk
Individuals may have different risk attitudes: they can be risk-averse (or risk-avoiding), risk-neutral, or risk-loving (or risk-seeking) persons. A first general look at the data aimed at learning if there were any participants (and if so, how many participants) who always decided for a "yes" or a "no" answer during the games played, hence deciding towards a more risk-avoiding or risk-seeking attitude, respectively. Table 2 shows, for each game, the number of participants who decided to open the gate to the retention basin at n = 0, 1, 2 . . . 6 times during the six rounds of each game. For instance, if a player never decided to open the gate during the six rounds (i.e. a risk-seeking participant), it will be counted along with "0 yes answers during the six rounds" in Table 2. It can be seen that 3 participants out of 98 always decided not to open the gate during all the rounds of Game 1. This did not happen in Game 2. In both games, 2 participants decided to always open the gate (i.e. had a risk-avoiding attitude). In all of these cases, the participants were different persons.
The results in Table 2 also show that 53 % of the participants decided more often not to open the gate during Game 1, while this percentage in Game 2 is of only 15 %. In Game 2, 46 % of the participants decided more often to open the gate (and thus protect the town), while the corresponding percentage in Game 1 was only 26 %. It seems thus that participants were more risk-seeking in Game 1 and, on the contrary, were more towards a risk-averse attitude during Game 2, when they had no or less information on forecast uncertainty.
However, one should not neglect a possible effect of learning. In the context of the present data, a less risk-averse attitude in Game 1 may have been prompted by the fact that floods during this game occurred only in the last rounds (Rounds 5 and 6). This may have given participants some confidence in keeping the gate closed and not spending their money. Also, it is important to remember that in Game 2 participants were faced with a flood event already in Round 1. This, in its turn, may have contributed to an increased riskavoiding attitude during Game 2. Participants may have been more cautious in their decisions and tempted to decide on opening the gate to protect the city from the risk of flooding. Figure 3 shows the relative frequencies of "yes" and "no" answers to the decision-question of opening or not opening the gate to the retention basin for each of the six cases played in both games (see Table 1 for the forecast data of each case).

Role played by forecast values and uncertainty information in decisions
Frequencies were computed over all 98 participants. Results are shown for Game 1 (forecasts with full uncertainty information) and Game 2 (forecasts without uncertainty information -Cases 1, 3, 4 and 5 -or with less uncertainty information -Cases 2 and 6). The same case is shown across each row in the figure to allow comparisons between Game 1 and Game 2. Here we analyse the results without taking into account the order of presentation of the cases during the games.
Only the influence of displaying forecast uncertainty information is considered. The optimal decisions (based on an a posteriori knowledge of the occurrence of a flood) would have been to answer "no" in the cases of no flood (Cases 1 to 4) and "yes" in the cases of flood (Cases 5 and 6). In Game 1, the majority of participants followed the optimal decision in four cases: Cases 1, 3, 4, and 6. In Game 2, the optimal decision was only taken by the majority in two cases: Cases 4 and 6 (we recall that in Game 2 Case 4 was presented with no uncertainty information and Case 6 was presented without information on the probability of flooding, but with error bounds around the expected value). It is interesting to note that these two cases of common optimal decisions in both games (Cases 4 and 6) are those that have the lowest and the highest expected forecast values (3.52 and 3.94 m, respectively, with 3.9 m being the level of flooding in the games). Perhaps being presented with expected forecast values far below or above the threshold defining the level of flooding, compelled participants to take the optimal decision, regardless of the uncertainty information provided.
In Game 1, we can also see that the majority of participants did not follow the optimal "no" decision only in Case 2 (out of the four cases 1 to 4). It is interesting to note that Case 1 and Case 2 are, however, fairly similar in terms of expected value: in both cases, it is close to the 3.9 m level of flooding (see Table 1). These cases mainly differ by the width of the error bounds and the probability of flooding,  Table 1). Cases 5 and 6 are associated with a flood occurrence. The order of presentation of the cases during the games was: Game 1 = [1, 2, 3, 4, 5, 6]; Game 2 = [5, 1, 4, 3, 6, 2,].
which are much more important in Case 2 (see Table 1). The fact that during Game 1, when these cases were shown one immediately after the other, the majority decided for the optimal "no" decision in Case 1 (i.e. decided towards a riskseeking attitude), but did not do the same in Case 2 may be an indication of the influence of the uncertainty information. It seems that the higher values of the quantified uncertainty displayed in Case 2 prompted the participants to change towards a more risk-averse attitude, thus deciding on opening the gate to avoid the risk of flooding the city: there is a shift from 6.1 to 78.6 % in the percentage of "yes" (risk-averse) answers when we move from Case 1 to Case 2 in Game 1 (Fig. 3, left panel).
The suggestion that uncertainty information plays a role in decisions and that its absence may compel people towards a more risk-averse attitude is corroborated when we look at the results of Case 1 and Case 2 in Game 2 (Fig. 3, right  panel). In Game 2, Case 1 is displayed without uncertainty information. We can see that decisions concerning this case change significantly from Game 1 to Game 2: the percentage of "yes" (risk-averse) answers goes from 6.1 % when uncertainty information is provided (Game 1) to 82.6 % when it is not (Game 2). In Game 2, Case 2 is displayed in Round 6, hence without information on probability of flooding but with information on estimated value and error bounds only. In this case, the percentage of "yes" (risk-averse) answers goes from 78.6 % when uncertainty information is provided (Game 1) to 85.7 % when it is not fully provided (Game 2).
Another interesting result concerning the role played by forecast uncertainty information is given by Cases 3 and 4. In Game 1 (Fig. 3, left panel), Case 4 displays a near parity of participants who would open (44.9 %) or not open the gate (56.1 %). Case 3, in its turn, displays a clear majority of "no" (risk-seeking) answers (92.6 %). In Game 2 (Fig. 3, right panel), where both cases were presented without uncertainty information, it is the other way around: Case 4 displays a clear majority for the "no" (risk-seeking) answers (89.8 %), while it is in Case 3 that the participants seem to be more divided in their decisions: 62.2 % voted for opening the gate, while 37.8 % voted for not opening it. Although the forecast value in Case 3 is much closer to the flooding threshold than in Case 4, error bounds and probability of flooding are lower, which may have prompted participants to take the risk and answer "no" to opening the gate in Game 1. In Game 2, without information on forecast uncertainty only the expected value can influence the decision. The fact that this forecast value was low in Case 4 may have thus induced the majority of "no" answers observed in Game 2.
It is also interesting to note that in Game 2, the percentage of "yes" answers in favour of opening the gate and protecting the city from the risk of flooding increases as the expected (forecast) value of the water level increases (from the highest to the lowest value, cases follow the order: 6, 2, 1, 3, 5, 4; see Table 1). Such a relationship is not found in the results of Game 1. In Game 1, cases with narrower uncertainty bounds and smaller probability of flooding (Cases 1, 3 and 5) seem to induce a more risk-seeking decision towards not opening the gate (and thus leave the city to the risk of flooding). For instance, if we compare the results for Cases 4 and 5 in Game 1, for which expected values are close, we can see that the "no" answer in Case 5 is adopted by a larger number of participants (75.5 %, Fig. 3, left panel). Even though Case 5 has a slightly higher expected value, the fact that it displays the lowest values of error and probability of flooding may explain that more participants answered "no" to open the gate. Contrary to Case 4, a flood did occur in Case 5. Participants who had to pay the fine of 7000 tokens for flooding the city must have been very disappointed here.

Impact of uncertainty information on diversity in decisions
In order to analyse the diversity of decisions among participants and between games, we studied all combinations of "yes" and "no" answers and the number of times they were repeated in our sample. This also allowed us to check if there was a tendency for participants to "copy" the answers from the "ensemble result" without marking in the worksheets their own answers. Figure 4 shows the number of participants having shared every combination of "yes" (marked by 1) and "no" (marked by 0) answers found in Game 1 (left) and Game 2 (right). We can see that 36 participants in Game 1 (32 in Game 2) had the same sequence of "yes" and "no" answers as the one showed by the "ensemble result" (i.e. sequence 0-1-0-0-0-1 in Game 1 and 0-1-0-1-1-1 in Game 2). However, we observed that only 11 participants "copied" the "ensemble results" in both games. It seems that participants were indeed marking their own answers in the worksheet and not just systematically following the group. A detailed analysis of Fig. 4 also shows that the number of combinations of "yes" and "no" answers to the individual questions is higher in Game 2: 20 different combinations were observed for this game, while Game 1 had only 14. Also, the number of participants showing identical sequences drops more quickly in Game 2. It seems that having all uncertainty information available (the case of Game 1) narrows the range of possible decisions. This may be a signal that providing uncertainty information together with the expected value can help in assembling individual decisions.

Impact of sequential decision-making on risk attitudes
In a decision-making game, the consequences of an observed "action-event" pair in one round may influence the decisions taken in the round immediately next. Figure 5 shows the number of times (n ij ) a decision went from one state to another during each game and for each one-step transition of rounds. Here again "yes" answers are marked by 1 and "no" answers by 0. The results must be discussed per round transition and interpreted with caution as the number of rounds is small in our dataset. Additionally, they have to be understood in the light of the sequence of cases as displayed during each game (see Table 1) and of the event observed (i.e. if flood occurred or not). In Game 1 (the sequence in which cases were presented was 1, 2, 3, 4, 5 and 6), we observe that: -in the first round, the majority of participants had said "no" to open the gate (92 out of 98 participants, i.e. 93.9 %, Case 1, Game 1, in Fig. 3), and that was the optimal decision as no flood occurred. In Round 2 (Case 2), they were presented with a similar expected value, but with larger uncertainty bounds and probability of flooding. The transition to a "yes" answer is clear: 71 participants out of the 92 participants that said "no" in Round 1, moved to a "yes" answer in Round 2 (n 01 in Fig. 5, Round 1 to 2, Game 1). Only 21 kept their "no" decision from Round 1 to Round 2 (n 00 in Fig. 5).
-The expected and uncertainty information values become lower in Round 3 (Case 3). In the transition from Round 2 to 3, the strongest signal is n 10 : 70 participants (out of 77 that had decided to open the gate in Round 2) moved their "yes" decision towards a more conservative "no" answer to open the gate when going from Round 2 to 3. -Participants that had decided "no" in Round 3 (91 out of 98 participants, Case 3, Game 1, Fig. 3) are basically divided in Round 4 into those moving towards a "yes" answer (n 01 = 36 participants, Fig. 5, Round 3 to 4, Game 1) and those that decide to keep the "no" answer (n 00 = 55 participants), with a stronger tendency to keep a more risk-seeking "no" answer.
-In Round 5, Case 5 displays a slightly strong expected value, although with narrower error bounds and lower probability of flood comparatively to the previous round (Case 4). Under this situation, moving to or keeping a "no" answer is the strongest signal for the transition from Round 4 to 5: 53 out of 55 participants kept their "no" decision (n 00 in Fig. 5), and almost half of the participants that had given a "yes" (risk-averse) answer in Round 4 (43 participants) moved to a "no" (riskseeking) answer in Round 5 (n 10 = 21 in Fig. 5).
-In Round 5, a flood occurred and participants that had said "no" to open the gate had to pay the fine of 7000 tokens for flooding the city. They were 74 (out of 98) participants. In the round that follows (Round 6), 70 of these participants moved their decision to a "yes" (more risk-averse) answer (n 01 in Fig. 5, for Round 5 to 6, Game 1). This can be due to the impact of the flood occurrence while they had decided not to protect the city, but also due to the fact that Case 6 at Round 6 displays the highest expected value (even slightly higher than the level of flooding), error bounds and probability of flooding.
Considering Game 2 (the sequence in which cases were presented was 5, 1, 4, 3, 6, and 2): -Flood occurred in the first round and the majority of participants had said "no" (risk-seeking attitude) to open the gate (87.8 %, Case 5, Game 2, in Fig. 3). Thus, already in the beginning of the game, they had lost 7000 tokens from their initial purse. When we look at the transitional behaviour from Round 1 to Round 2, we can see that there is a strong signal of moving towards a "yes" (more risk-averse) answer in the second round: 72 participants (out of the 86 that had decided "no" in the first round) moved to a "yes" decision in Round 2 (n 01 in Fig. 5, Round 1 to 2, Game 2). Also, the majority of the 12 participants that had decided to open the gate in Round 1 kept their decision in Round 2 (n 11 = 9 in Fig. 5). In this case, decisions seem to have been influenced by the previous "action-event" pair.
-In Round 3, the expected value was much lower (3.52 m, Case 4, Table 1) comparatively to the previous round (Round 2 in Game 2, Case 1, Table 1). We observe a strong transition towards a "no" decision: 73 out of 81 participants that had decided "yes" in Round 2 move to a "no" decision in Round 3 (n 10 in Fig. 5, Round 2 to 3, Game 2). In Round 2, flood had not occurred and the high number of participants that had decided to open the gate had to cope with a non-optimal decision and to subtract 2000 tokens from their purse. This may have encouraged 90 % of them to decide towards a "no" answer (risk-seeking attitude) in Round 3 (if no Ranked differences in final purses (as percentage of the initial purse, % P 2−1 ) for the 98 participants and the "ensemble result". Differences < 0 mean that more money was lost when decisions were made without uncertainty information than with it, and vice-versa.
flood occurred they had a chance of keeping their purse without losses).
-The transition from Round 2 to 3 was the only moment in Game 2 where the number of participants going from a "yes" decision to a "no" decision (n 10 ) is significantly higher than the other transitional situations (Fig. 5). In general, participants in Game 2 more often move towards opening the gate (n 01 ), or choose to maintain a "yes" (risk-averse) answer to the decision problem (n 11 ). In Game 1, the n 01 transition is also important, but together with the n 00 transition, i.e. the choice to maintain a "no" (risk-seeking) answer to the decision problem. These remarks corroborate the observed general pattern of a more risk-avoiding attitude in Game 2 comparatively to Game 1 (Table 2). Figure 6 shows the distribution of differences in final purses (% P 2−1 ) for all participants and for the "ensemble result". On average, participants ended Game 2 with 3000 tokens less than the final amount they had after playing Game 1. Only 14 participants ended Game 2 with more money in their purse (positive differences in Fig. 6). 12 participants ended Game 2 with the same amount of money as they did in Game 1, while the majority of participants (72 out of 98) lost money when making decisions without uncertainty information (negative differences in Fig. 6). The majority that lost more money in Game 2, comparatively to their final purse in Game 1, had an average loss equivalent to approximately 15 % of the initial purse, with a maximum loss of up to 47 % (Fig. 6). The minority that ended Game 2 with more money had an average gain equivalent to approximately 9 % of the initial purse, with a maximum gain of 23 % of the initial purse (Fig. 6). The fact that the majority of participants had a worse economic performance when making their decisions without uncertainty information is also illustrated in Fig. 7 distribution of observed final purses for both games is represented. We can clearly see the shift to lower remaining amounts when moving from Game 1 to Game 2. We note that a participant that would know the results before playing, thus having all the "optimal decisions" (i.e. the ones that cost less money) would have ended both games with 26 000 tokens. A participant that, on the contrary, always made the wrong decisions in both games (i.e. the ones that would cost more money in all six rounds), would have ended both games with 8000 tokens. From Fig. 7, it can be seen that there are no occurrences of participants having followed these extreme  scenarios, always making optimal decisions or always making more costly decisions in both games. Also, participants making only "optimal decisions" in Game 1 but, inversely, only making "more costly decisions" in Game 2 would end up with a negative difference in final purses of 18 000 tokens (or, vice versa, would end up with a positive difference in final purses of the same amount). This represents a difference in final purses (% P 2−1 ) equal to 60 % of the initial purse. As can be seen from Fig. 6, these combinations of extreme decisions cases were not observed in our experiment either. The maximum (minimum) difference in final purses here observed was 7000 (−14 000) tokens.

Impact of decisions on economic performance and vice versa
To investigate the assumption that people's decisions depend on the economic resources they have left (i.e. if they become more risk-seeking when they still have a large amount of money in their purses, or, vice versa, if they become more risk-avoiding when they have less money remaining), we represent in Fig. 8 the conditioned distributions of available monetary resources and decisions for both games. Values of the purse participants had at the moment of making a decision are analysed separately, conditioned if the decision made was a "yes" answer to open the gate (i.e. a riskavoiding attitude) (Fig. 8, left panels) or a "no" answer to open the gate (i.e. a risk-seeking attitude) (Fig. 8, right panels). All rounds are considered to obtain the conditioned samples. Figure 8 shows that, in general, for both games, participants had a more risk-avoiding attitude ("yes" answers to the decision problem) when they had less money in their purses. On the contrary, when risk-seeking attitudes ("no" answers) were taken, the values of remaining money in their purses were higher. Finally, we note that this signal is more clearly observed in the results from Game 1 (Fig. 8, top panels), where all forecast uncertainty information was available to the decision-makers.

Discussion and conclusions
In this paper, we presented a simple experiment conducted during a conference, which aimed at making attendees play the role of decision-makers in flood forecasting. The experiment was presented as a flood control game, where participants had to make decisions based on forecasts that were presented with and without uncertainty information. The aim of this exercise was not only to investigate the benefits of probabilistic forecasts in the practice of decision-making, but also to provoke a thought process in the hydrologic community on how to integrate the decision-making process on the design of probabilistic forecast products and as part of daily routines of forecast systems. This issue needs to be investigated to avoid leaving room for a sceptical view of the benefits of probabilistic forecasts to end-users. By engaging people in thinking about the need of forecast uncertainty information to make better decisions, new ideas on how to produce, present and interpret probabilistic forecasts can emerge and be used for training purposes.
A similar question to the one asked in this study was investigated by Roulston and Kaplan (2009), who focused on the understanding of uncertainty using a 5-day temperature forecast. They show that uncertainty information leads to a better decision, independently of the type of academic background of the participants. Their study was followed up by Marimo et al. (2012) who confirmed the previous results, but also noted a learning effect as the experiment progressed. The importance of training with decision support products for probabilistic flood forecasts was pointed out by Ramos et al. (2007). From the results of a workshop conducted together with European operational forecasters on the use of ensemble-based forecasts for flood warning situations, Ramos et al. (2010) also draw attention to the way uncertainty information can be missed when suddenly it is made unavailable in a decision-making exercise. Comparing decisions made with and without uncertainty information is also the basis of a group of experiments presented by Joslyn and Leclerc (2012). Their results suggest that forecasts provided with reliable uncertainty estimates lead to better decisions and can improve trust in the forecast information source.
The experiment presented here is a simple one, but already yielded some interesting clues to the process of decisionmaking with probabilistic flood forecasts, as follows.
-There is evidence that decisions are based on a combination of what is displayed by the expected (forecast) value and what is given by the uncertainty information.
-The expected (forecast) value plays an important role in the decision: forecasts close to the critical flood level, although displaying large uncertainties, induced a majority of conservative risk-avoiding attitudes (here, it means a decision to open the gate and protect the city from an eventual flood event).
-However, uncertainty information also plays a role in the decision-making process: given similar expected values, decisions may differ according to the uncertainty information displayed. The tendency observed was: the higher the uncertainty information is, the more risk-avoiding the decisions are.
-The presence of uncertainty information seems to result in more optimal decisions and more coherent answers among individual decision-makers. There is evidence that decisions from individuals tend to converge when uncertainty information is provided, as it leads to a lower variety in the sequence of decisions. Uncertainty information leads thus not to a larger diversity in decisions, but, on the contrary, to a smaller one.
-In the absence of uncertainty information, decisionmakers are compelled towards a more risk-averse attitude. It was also observed that in this situation (i.e. when forecast are provided without uncertainty information), the percentage of risk-avoiding decisions among participants increased as the expected forecast value got closer to the critical level of flooding.
-When considering penalties to each "action-event" pair, more money was lost by a large majority of decisionmakers when they had to make decisions without uncertainty information. The absence of uncertainty information led to a worse economic performance in decisionmaking. Besides, the highest money loss was considerably more important than the higher gain registered.
-Despite the limitations of the present data, there is evidence that decisions are influenced by the previous "action-event" pair. A previous "action-event" pair may influence decisions coming immediately next, especially if the result of this previous "action-event" pair is not favourable to the decision-maker (i.e. if at the end of the action, less money would have been lost if the other alternative action was taken).
-When decision-makers have less money in hand, they more often decide towards a risk-avoiding attitude. Risk-seeking attitudes were more often taken when the values of remaining money in their purses were higher.
A generalisation of the results obtained from our experiment should be considered with caution. The conduct of the game during a conference presentation prompted for an experimental set-up that is not fully controlled. Several limitations to our game can be pointed out: e.g. the reduced number of cases played during the games, the time constraints (15 min to carry out the whole experiment) that limited the assessment of learning effects from previous decisions and from one game to another, the absence of training rounds to make sure that participants understood correctly the game and its rules, the limited sample size (98 participants), the fact that sampling was selective and predominately consisted of research scientists and professionals in hydrological forecasting and related fields, etc. Additionally, the experiment presented in this paper represents an attempt to examine the question: "Do probabilistic forecasts lead to better decisions?" The way we phrased our question is fairly simplistic and can be challenged on multiple accounts. For example, critical questions are: what does one mean by "probabilistic", how are these probabilities generated, how trustworthy are they, are they a synonym for uncertainties (see discussion in Koutsoyiannis, 2010 and Montanari, 2010)? In this paper we tried to avoid the discussion by displaying a probabilistic forecast as an expression of uncertainty around a mean (expected) forecast value and a probability of flood occurrence. This already excludes certain decision-makers or decision frameworks (e.g. Dale et al., 2012, present a decision-making framework which is based on the worst case scenario and freeboard concept).
One can further question what actually constitutes a "better decision" and how penalties can be assigned to "bad decisions". As noted by Wilks (1997), the "existence of a 'best' decision implies preferences among at least some of the possible consequences: if a decision maker doesn't care what eventually happens, the decision question is not a meaningful one". Traditionally, the quest for better decision is often broken down, either explicitly or implicitly, to the economic value of a utility or service: better decisions are those that provide the most economic benefit (higher gain or lower loss) for a target use (e.g. tradeoffs in allocating water resources) or the highest possible security to individuals (e.g. flood protection). In our exercise, a better decision is simply the one that allows a reduced (or no) loss in virtual money.
In our exercise, the decision-making process was presented on a single level, meaning that the only option given was to open or not the gate to the retention basin. Other levels of decision-making could have included checking the functioning of the gate or giving the farmers a warning to reduce potential damage. Zhu et al. (2002) pointed out that "for economic decision making it is imperative to use forecasts that provide multiple decision levels". Single-value or deterministic forecasts cannot fit the criteria, even if they are "perfect" (unbiased and accurate) forecasts. Our experiment focused on measuring the differences in final purses when decisions were made with and without forecast uncertainty information. Starting with 30 000 tokens, and considering the rules specifying the costs of their decisions, participants were allowed to play without economic pressures. It would, however, be interesting to investigate to what extent monetary gains and losses can affect the decision-making process in probabilistic streamflow forecasting. The use of probabilistic forecasts to achieve an increased potential economic value has been investigated in many disciplines such as meteorology (Zhu et al., 2002;Buizza, 2008), irrigation (Cai et al., 2011) or wind energy (Tina and Brunetto, 2010;Tsikalakis et al., 2009). In hydrology, recent studies have proposed methods based on cost/loss ratio analysis (McCollor and Stull, 2008;Muluye, 2011;Verkade and Werner, 2011) or stochastic optimisation models (Boucher et al., 2012). Based on simple optimal decision-making and different target applications (continuous flow forecasts, flood forecasts and warnings, inflows to hydroelectric production reservoirs), they all basically show that probabilistic or ensemble-based forecasts have the potential to generate higher benefits than climatology or deterministic forecasts.
Studies on forecast value, in general, will often consider decisions that are optimal or determined by the user's behaviour, as well as users that may or may not use information in an optimal manner (Stewart, 1997). As pointed out by Roulston et al. (2006), laboratory-based studies can provide complementary information to existing approaches and help in connecting acquired knowledge. The whole decisionmaking process is however a complex one. It is intrinsically related to how valuable information that can help decisionmakers to reduce forecast uncertainty and avoid the more costly errors is used (Joslyn et al., 2011). Most current decision-making situations taking place at operational hydrological forecasting centres are far more complex than what laboratory-based studies can replicate.
As mentioned earlier, we are conscious of the limitations of our experimental setting. Above all, the game presented here suffers from the lack of real consequences for wrong decisions. Decisions may be taken quite differently within the pressured environment of a real forecasting centre. However, a game setting, whatever it is, will hardly, and maybe even never, be able to replicate reality, and provide statistically robust results that are needed for a quantitative assessment of the role of uncertainty in decision-making. The experiment presented in this paper represents a simple attempt to understand the decision-making processes in one particular setting and to draw the community's attention to the need (and advantages) of promoting training and education on the use of probabilistic forecasts.
It is a crucial issue to interactively consider the development of training on decision-making under uncertainty in order to increase effectiveness in the use of probabilistic predictions to support decisions. Enhanced training may comprise learning tools for probabilistic predictions, training workshops or games, as the one presented here, designed to allow modellers and decision-makers to practice how to take actions (or which actions to take) based on uncertainty information. The development of understanding and practice can be based on cases involving specific hydrologic needs and situations, but also on more general pertinent cases, even if they are outside the hydrologic domain. Currently, knowledge and experiences still need to be put together to increase practice and enable training for a confident use of probabilistic outputs in decision-making. The community certainly needs to pursue in addressing remaining problems of uncertainty modelling, but this needs to be performed together with studies also addressing how probabilistic forecasts can be understood and used for improved water risk management. Innovative training activities based on a collection of case-studies and games will allow sharing problems and joining efforts to find solutions to an efficient use of probabilistic hydrologic predictions in decision-making.
In our opinion, a collection of training exercises, including topical games, made available through a web-based platform for exchange of e-learning tools can be a powerful tool to empower decision-makers with new capabilities and extended flexibility to handle different situations encountered in hazards prediction and mitigation. Training and games may be useful for both modellers and decision-makers. Training can help modellers to channel their efforts to improve the prediction chain. Its usefulness is also evident for decision-makers. It offers a unique opportunity to practice how to take actions (or which actions to take) based on uncertain information, and to optimise the use of probabilistic approaches in a range of applications. It can also help both actors to better understand the interactions between model outputs and forecasters or decision-makers. How much training can improve the use and the value of a forecast is an open question for the scientific and the operational communities. The value of a training program may not only rely on its quality, but also on how it integrates existing knowledge and capacities in operational forecasting centres.

Resources and other available products
All material (including power point presentation and a longer 10-round version of the game) is available upon request, and can be freely used by any interested group for teaching or training. The authors would also like to mention some available products: for instance, the exercises in weather forecasts created by Kahl and Horwitz, available at https: //pantherfile.uwm.edu/kahl/www/WebQuests/, or the "Wind Energy Game", which is part of the training on Ensemble Prediction System of the Meteorological Service of Canada (available at http://collaboration.cmc.ec.gc.ca/cmc/ ensemble/Formation-Training/, in English and French).