Optimising seasonal streamflow forecast lead time for operational decision making in Australia

Statistical seasonal forecasts of 3-month streamflow totals are released in Australia by the Bureau of Meteorology and updated on a monthly basis. The forecasts are often released in the second week of the forecast period, due to the onerous forecast production process. The current service relies on models built using data for complete calendar months, meaning the forecast production process cannot begin until the first day of the forecast period. Somehow, the bureau needs to transition to a service that provides forecasts before the beginning of the forecast period; timelier forecast release will become critical as sub-seasonal (monthly) forecasts are developed. Increasing the forecast lead time to one month ahead is not considered a viable option for Australian catchments that typically lack any predictability associated with snowmelt. The bureau’s forecasts are built around Bayesian joint probability models that have antecedent streamflow, rainfall and climate indices as predictors. In this study, we adapt the modelling approach so that forecasts have any number of days of lead time. Daily streamflow and sea surface temperatures are used to develop predictors based on 28-day sliding windows. Forecasts are produced for 23 forecast locations with 0–14and 21-day lead time. The forecasts are assessed in terms of continuous ranked probability score (CRPS) skill score and reliability metrics. CRPS skill scores, on average, reduce monotonically with increase in days of lead time, although both positive and negative differences are observed. Considering only skilful forecast locations, CRPS skill scores at 7-day lead time are reduced on average by 4 percentage points, with differences largely contained within+5 to−15 percentage points. A flexible forecasting system that allows for any number of days of lead time could benefit Australian seasonal streamflow forecast users by allowing more time for forecasts to be disseminated, comprehended and made use of prior to the commencement of a forecast season. The system would allow for forecasts to be updated if necessary.

once per month. However, the forecasts are frequently not released until the second week of the forecast season because the forecast production process is overly time-consuming.
The first step in the forecast process is to gather predictor data and perform quality control. Once the predictor data are assured to be of good quality, forecasts and forecast products are generated. The forecasts subsequently need to be checked for modelling errors and inconsistences. Once all the forecast products are ready, key messages and other communication 5 products are developed. Once the communication strategy is in place, the forecasts are released. The Bureau now issues forecasts for over 200 locations and so, understandably, the forecast production process is a time-consuming one.
Currently, the statistical forecasting models rely on predictor data observed up to the day prior to the first day of the forecast period. For example, a forecast for the austral spring (September-October-November) requires observed data up to and including the 31st of August. The forecasts effectively have no lead time and so the forecast lead time can be considered to 10 be 0-months or 0-days. Suppose predictor data were available immediately on the first day of the forecast period; then forecasts could be generated on the first day of the forecast period. Even in this case, the forecast release would occur several days into the forecast season, after forecast products and communication messages were created. In reality, by the time the forecasts are released, the forecasts typically have a lag time of at least 7 days.
The lag time in forecast release diminishes the value of the forecasts. Ideally, the forecasts would be in the hands of decision 15 makers well ahead of the forecast season. Furthermore, the lag time in forecast release is likely to become more prominent in the short term because the Bureau plans to release sub-seasonal (i.e. monthly) streamflow forecasts by 2017. It is therefore important to investigate ways to issue the forecasts earlier and to quantitatively analyse how forecast accuracy and reliability are affected.
The Bureau's current forecasting system is built around the Bayesian joint probability (BJP) modelling approach (Wang and 20 Robertson, 2011;Wang et al., 2009). BJP models make use of two types of predictors: predictors representing initial catchment conditions and predictors representing the climate state. The predictors are selected following a rigorous predictor selection process . Initial catchment condition predictors, which act as a soil-moisture proxy, include streamflow totals over the preceding 1, 2 or 3 months. Predictors representing the climate conditions are monthly climate indices lagged up to 3 months (Schepen et al., 2012;Kirono et al., 2010) and include indices representing the El Niño 25 Southern Oscillation, variability in the Indian Ocean and variability in the southern polar circulation.
Predictors in the BJP modelling approach vary by season and by catchment. Therefore in operational forecast production it is necessary to prepare a wide range of predictor data from a variety of data sources. Since the Water Act (2007) was introduced in Australia, the Bureau of Meteorology is authorised to collect streamflow data from gauge owners. Streamflow data is therefore easily obtained for most gauges; however, the data comes in a multitude of formats and needs to be 30 processed by data managers prior to being made available to forecasters. Usually, daily streamflow data for each of the forecast locations is available with a few days lag. Climate data is sourced from the Bureau of Meteorology and the United States National Oceanic and Atmospheric Administration (NOAA). NOAA ERSST (Huang et al., 2015) monthly sea surface temperature (SST) grids are used for calculation of climate indices such as Nino3.4 and the IOD mode index. The ERSST Hydrol. Earth Syst. Sci. Discuss., doi:10.5194/hess-2016-199, 2016 Manuscript under review for journal Hydrol. Earth Syst. Sci. Published: 9 May 2016 c Author(s) 2016. CC-BY 3.0 License. data set for the previous month is normally available by the 4th day of the month. Other indices, such as the Antarctic Oscillation Index (AAO) (Mo, 2000) representing the Southern Annular Mode can take up to a week to become available.
The Bureau of Meteorology updates Southern Oscillation Index values relatively swiftly, normally within a couple of days.
As just described, up to 7 days delay in the forecast production process can be attributed to delays in acquiring predictor data. Much of the problem lies in the reliance on data for complete calendar months. An immediate and obvious resolution to 5 the problem is to produce forecasts with 1-month lead time. N-month lead time forecasts are certainly feasible in a statistical framework and are operationally useful in many parts of the world where snow-melt is a major source of seasonal streamflow predictability, e.g. the western United States (e.g. Pagano et al., 2004) and the Three Gorges system in China (Xu et al., 2007) . However, increasing the forecast lead-time for Australian catchments beyond one month is undesirable because the primary source of skill is initial catchment conditions and catchments have limited memory or short response 10 times.
An alternative approach to get forecasts to users in good time is to make use of daily streamflow and climate data to generate predictors and forecasts with N-days lead time. Although it could be presumed that skill will tend to be reduced as lead time increases, it is not known to what degree skill will be impacted. An optimal N-days lead time forecast is the most skilful forecast that can be released prior to the beginning of the forecast period, given consideration for forecast reliability, forecast 15 preparation and communication time, and allowing enough time for users to ingest streamflow forecasts into their models ahead of the forecast season.
In this study, we develop BJP models to produce N-days lead time forecasts for 23 catchments across Australia and seek to demonstrate that it is possible to release operational forecasts ahead of the commencement of the forecast target season. We investigate the availability of daily climate data and establish necessary modifications to the predictor choices and length of 20 record used to establish the models. Cross-validation forecasting experiments are conducted to evaluate the quality of forecasts based on the N-days lead time approach. The skilfulness of forecasts are compared for 0-14 and 21 days lead time.
The results give an estimate of how forecast skill changes with increasing forecast lead time. Subsequently, the optimal Ndays lead time forecast is determined, based on considerations of forecast skill, data availability and forecast preparation time. 25 The remainder of the paper is organised as follows. Section 2 describes the study forecast catchments, streamflow and climate data. Section 3 details the study methods, including information about the BJP modelling approach and forecast verification. Section 4 presents the results. Section 5 discusses the results. Section 6 concludes the paper.

Catchments
Twenty-three forecast locations are selected for this study including forecast locations in all states: Queensland, New South Wales, Victoria, Tasmania, South Australia, Western Australia; plus the Northern Territory. Flows at the forecast locations are a mixture of perennial, ephemeral and intermittent flows. Table 1 summarises information about the forecast locations, 5 including long catchment name, short ID, state (in the jurisdictional sense), upstream catchment area and approximate centroid latitude and longitude. Furthermore, the forecast locations are plotted on a map of Australia in Figure 1..

Streamflow data
Daily streamflow data is sourced from the Bureau of Meteorology for all 23 forecast locations. The Bureau sources the data from the respective data owners and performs quality control. Daily streamflow data records often contain missing values, 10 for example, due to the failure of gauging equipment. The Bureau partially infills missing data records. Records for all forecast locations are infilled by linear interpolation if the number of missing days is equal to or less than 3 days. For some forecast locations, records are infilled by linear regression with nearby forecast locations if the number of missing days is equal to or less than 14 days.
In this study, the daily streamflow data are aggregated to 28 day totals for use as predictors and three-month totals for use as 15 predictands. Many of the data records still contain missing data after the infilling process, however the proportion of missing data is never greater than 10%.

Climate Data
The Bureau's current BJP seasonal streamflow forecasting models rely on predictors including surface and subsurface ocean temperatures, the Southern Oscillation Index (SOI) and the Antarctic Oscillation Index (AAO). The predictors are identified 20 through a rigorous predictor selection process . In this study, we adopt the climate predictors used by the Bureau in their operational models with some changes. The biggest change is that we restrict the set of climate predictors to sea surface temperature (SST) predictors. The subsurface ocean temperatures and SOI are not updated on a daily basis and it is therefore not possible to obtain up-to-date predictor values for earlier release forecasts. Because the SOI and subsurface temperatures reflect ENSO, we substitute an ENSO SST index wherever the climate predictor in the Bureau's 25 operational model is subsurface ocean temperatures or SOI. SST climate predictors representing ENSO and the Indian Ocean state are likely to be stable across a period of several weeks and are known to have strong lagged relationships with Australian rainfall up to three months ahead (Schepen et al., 2012). Therefore, the predictors are likely to remain valid predictors as forecast lead time is increased by a few weeks. On the other hand, it is less certain that the AAO, representing the Southern Annular Mode, will remain a valid climate predictor as lead time is increased since it has a weak lagging 30 Hydrol. Earth Syst. Sci. Discuss., doi:10.5194/hess-2016-199, 2016 Manuscript under review for journal Hydrol. Earth Syst. Sci. Published: 9 May 2016 c Author(s) 2016. CC-BY 3.0 License. relationship with Australian seasonal rainfall (Schepen et al., 2012). In cases where AAO is a climate predictor in the Bureau's operational model, no climate predictor is used in this study.
Daily SST data is obtained from the NOAA 1/4° daily Optimum Interpolation Sea Surface Temperature (daily OISST) (Reynolds et al., 2007). Full years of data are available from 1982 onwards. The daily OISST is constructed by combining data from satellites, ships, buoys, probes and ocean-dwelling robots. Interpolation fills in missing gaps and ensures a 5 complete historical record with no missing data. Compared to monthly ERSST records, which go back to 1854, the daily OISST record is relatively short but it is necessary for generating timelier forecasts. To obtain SST climate predictors we calculate 28-day average sea surface temperatures relative to 1982-2010 climatology. Area averaging is used to obtain the set of climate predictors: Nino3.4, Nino3, Nino4, the El Nino Modoki Index (Ashok et al., 2007), the Indian Ocean Dipole (Saji et al., 1999)

BJP forecasting models
Forecasting models are set up using the Bayesian joint probability (BJP) modelling approach (Wang and Robertson 2011;15 Wang et al. 2009). Separate forecasting models are established for each forecast location (23 locations), season (12 threemonth seasons from the start of each month) and for 0-14 and 21 days lead time (16 lead times).
The BJP forecasting models make use of two types of predictors: predictors representing initial catchment conditions and predictors representing the climate state. For all models, the initial catchment condition predictor is fixed to the previous 28 days total flow volume. This predictor will not be optimum for all forecast locations; however, it is a pragmatic choice that is 20 likely to represent the overall catchment wetness reasonably well. Additionally, it gives a consistent model set up without needing to undertake initial catchment condition predictor selection. SST climate predictors are adapted from the Bureau's current operational forecasting models as described in section 2.3. The same climate predictors are applied at all lead times to ensure a consistent model set up.
The full mathematical formulation of the BJP modelling approach is presented in Wang et al. (2009) and Wang and 25 Robertson (2011). Here, we note some key features of the BJP modelling approach.
The BJP models are able to effectively handle missing and non-concurrent records. The BJP models are based upon the multivariate normal distribution after allowing for data transformation using either the log-sinh  or Yeo-Johnson (Yeo and Johnson 2000) transformations. If a set of variables follow a multivariate normal distribution then a subset of those variables also follows a multivariate normal distribution. Thus the many instances of missing data in streamflow 30 records, as described in section 2.2, are easily handled. Several of the forecast locations experience ephemeral and Hydrol. Earth Syst. Sci. Discuss., doi:10.5194/hess-2016-199, 2016 Manuscript under review for journal Hydrol. Earth Syst. Sci. Published: 9 May 2016 c Author(s) 2016. CC-BY 3.0 License. intermittent flows, which can result in a probability mass for zero flows. This problem is handled in the BJP modelling approach by treating zero flows as censored data, meaning that the observations of zero flow are treated as being of unknown precise value equal to or below zero. Uncertainty in the model parameters due to the short data records is handled by inferring parameters using Markov Chain Monte Carlo methods (MCMC). Probabilistic (ensemble) forecasts are produced using conditional multivariate normal distribution equations. 5 Some aspects of the BJP implementation used in this study vary compared with those published in Wang et al. (2009) and Wang and Robertson (2011). We use fixed transformation parameters together with simplified parameter reparameterisations for more straightforward numerical inference. The changes reflect our experience in achieving more robust and efficient BJP modelling.

Verification 10
In this study, we assess the performance of forecasts for the period from JFM 1982 to DJF 2011-12. A separate BJP model is established for each season, forecast location, and lead time. Across different years, the model parameter inference and forecast process is cross-validated using leave-five-years-out cross-validation. In other words, for each historical forecast event to be tested, the data points for the year to be forecast plus the subsequent four years are left out. The leave-five-yearsout procedure is designed to account for strong persistence in streamflows, potentially over many years. 15 Forecasts from the BJP modelling approach are probabilistic. The continuous ranked probability score (CRPS; Matheson and Winkler, 1976) is used to assess full forecast probability distributions, therefore involving forecast ensemble spread as well as forecast accuracy. For convenience we define CRPS as the average CRPS across forecast time periods t ( 1, 2,..., ) tT  otherwise. To assess forecast value, the CRPS score is converted to a skill score, to measure the relative improvement of the CRPS over CRPSref, the CRPS for leave-five-years-out climatology reference forecasts: The CRPS skill score is positively oriented (whereas CRPS is negatively oriented). A maximum score of 100 is indicative of 25 perfect forecasts. A score of 0 indicates no overall improvement compared to the reference forecast. A negative score indicates poor quality forecasts in the sense that a naïve climatology forecast is more skilful.
Reliability refers to the statistical agreement of forecast probabilities with observed relative frequencies of events, which can be checked using probability integral transforms (PITs). At each lead time, the PIT values are calculated from the BJPgenerated ensemble forecasts for every forecast event, forecast location and season, and pooled in the analysis. The PIT represents the non-exceedance probability of observed streamflow obtained from the CDF of ensemble forecast. If the ensemble spread is appropriate and the forecasts are free of bias then observations will be contained within the forecast 5 ensemble spread. Reliable forecasts are evidenced by PIT values that follow a uniform distribution between 0 and 1.

Reliability
We first assess the reliability of the forecasts for the different lead times. Histograms of PIT values are constructed for each lead time from 0-14 days (Figure 3). For each lead time, the PIT histograms follow a similar distribution: approximately 10 uniform but not perfectly so. There are a couple of reasons why the PIT histograms are not perfectly uniform. First, it is very difficult to model the shape of the forecast distributions perfectly, particularly under cross-validation. Second, a prior distribution over the data transformation parameters is influential in preventing very skewed distributions when there is limited data or many instances of zero flows. Thus, the model attempts to reliably reflect uncertainty, but cannot tightly fit the shape of the data in all cases. 15 On-the-whole, the PIT values suggest that the forecasts capture the range of observations sufficiently well and there is no strong evidence of bias. Therefore, the spread of the ensembles is deemed to be appropriate. However, note that the shape of the PIT histograms suggests that an alternative distribution, for example a fat-tailed distribution, may be able to fit the tails of high-flow events better. The PIT histograms exhibit a similar shape for all lead times, meaning that the forecasts are similarly reliable across all lead times. 20

Accuracy Skill
We now assess the accuracy skill of the forecasts for the different lead times. To assess forecast accuracy we calculate CRPS for each forecast location, season and lead time. It is possible to decompose the CRPS into components reflecting accuracy and reliability. Results in the previous section demonstrate that the forecasts are similarly reliable across all times. Therefore differences in the CRPS will mainly reflect differences in forecast accuracy. CRPS scores are in the original units of the data 25 and therefore it is difficult to compare across forecast locations and seasons. CRPS skill scores, as defined in Equation (2), however, measure the relative reduction in error relative to a reference forecast, and therefore it is easier to compare relative skill levels across forecast locations, seasons and lead times. Hereafter, we consider only CRPS skill scores with modelled climatological distributions as reference forecasts.
CRPS skill scores for forecasts with 0-days lead time are plotted in Figure 4. Forecast locations form rows and target seasons 30 form columns. The CRPS skill scores vary highly between forecast locations and seasons, due to variations in catchment Hydrol. Earth Syst. Sci. Discuss., doi:10.5194/hess-2016-199, 2016 Manuscript under review for journal Hydrol. Earth Syst. Sci. Published: 9 May 2016 c Author(s) 2016. CC-BY 3.0 License. memory and climate predictability. The sources of predictability are not the main concern here though. Here, the aim is to analyse the changes in forecasting skill with increasing lead time. As an example, forecasts with 7-days lead time are plotted in Figure 5. By visual comparison, skill is overall reduced, however there are individual cases where skill increases.
Instances of positive CRPS skill score differences are conceivable given the small sample size (30 years) and associated uncertainty in the skill score. The patterns of skill across forecast locations and seasons are consistent at 0-and 7-days lead 5 time. Overall, increasing the forecast lead time to 7 days does not significantly decrease forecasting skill. Figure 6 illustrates for lead times of 0-, 7-and 14-days, the proportion of cases (catchments and seasons) where CRPS skill score thresholds are exceeded. CRPS skill scores between -5 and 5 are considered unskilful, below -5 is considered negative skill, and above 5 is considered skilful. The proportion of cases where certain CRPS skill score thresholds are exceeded decreases as the lead time increases. A CRPS skill score of -5 is exceeded in approximately 95%, 94% and 92% of cases for 10 0-, 7-and 14-days lead time respectively, indicating only a small proportion of cases exhibit skill worse than climatology.
The small difference in exceedance probabilities with increasing lead time is consistent with the knowledge that the BJP modelling approach should produce forecasts approximating climatology in the absence of any real forecasting skill. A CRPS skill score of 5 is exceeded in approximately 66%, 59% and 57% of cases for 0-, 7-and 14-days lead time respectively. The larger difference in exceedance probabilities suggests that the number of days lead time is important for 15 skilful forecast locations and catchments.
To evaluate the change in forecast skill for each 1 day increase in lead time, we calculate the difference between CRPS skill scores for 1-14 and 21-day lead times with CRPS skill scores at lead time 0. For each lead time, the skill score differences From Figure 4 and 5, it is evident that many forecast locations and seasons have low skill for reasons that may include aridity, poor catchment memory, extreme events and data problems. We now partition the catchment and seasons into two groups based on CRPS skill scores at 0-days lead time. If the CRPS skill score for a catchment and season at 0-days lead time is > 5, the case is assigned to the skilful group. Otherwise, the case is assigned to the unskilful group. Figure 8 plots the 30 change in forecast skill for each 1 day increase in lead time, considering only skilful cases. The reduction in skill is more marked for skilful cases compared with all cases. Figure 9 plots the change in forecast skill for each 1 day increase in lead time, considering only unskilful cases. The average reduction in CRPS skill scores is near zero, with variation mainly within ±5 pp. The results in Figure 8 (considering only skilful cases) are arguably the most important for analysing reductions in Hydrol. Earth Syst. Sci. Discuss., doi:10.5194/hess-2016-199, 2016 Manuscript under review for journal Hydrol. Earth Syst. Sci. Published: 9 May 2016 c Author(s) 2016. CC-BY 3.0 License. skill because the Bureau issues climatology forecasts when hindcast skill is poor. From Figure 8, at 7-days lead time the mean reduction in CRPS skill scores is about 4 pp, whereas at 14-days lead time it is about 6 pp. At 21-days the reduction in CRPS skill scores averages 7.5 pp with few cases associated with skill improvements.

Discussion
The results demonstrate that forecasts are similarly reliable for all lead times from 0-14 days. Hence, from the perspective of 5 reliability, any forecast lead time is similarly optimal. Forecasts released with 7-or 14-days lead time will be similarly reliable to forecasts released with 0 days lead time. Therefore, considerations for optimal forecast lead time do not need to be based on reliability.
The results demonstrate that mean skill, averaged across all forecast locations and seasons, decreases monotonically for lead times from 0-14 days and the trend continues to 21-days lead time. However, the reduction in skill ought to be distinguished 10 for skilful catchments, since the skill for difficult to model catchments with poor skill remains largely unchanged and the Bureau's forecasts are forced to a climatological distribution when skill is very low or negative. From the perspective of accuracy skill, the monotonicity of decreasing skill scores with lead time suggests that the optimum forecast lead time is the shortest lead time that allows for other factors including the number of days required for forecast preparation, dissemination, comprehension and application. 15 As discussed in the introduction, the Bureau currently issues forecasts at least 7 days into the forecast season. Thus 7 days can be regarded as the current time needed for forecast preparation and dissemination. Much of the delay in preparing the current operational forecasts is attributable to delays in obtaining various climate data sets and quality streamflow data. The approach taken in this study of relying solely on daily SST data for climate predictors significantly reduces the burden in preparing climate indices. Preliminary daily OISST data sets are available within 1 day, although they are subject to revision 20 for up to 14 days. Thus the necessary climate indices can theoretically be ready for inclusion in forecasting models within 1-2 days. As discussed in the introduction, quality-controlled streamflow data can take up to 3 days to enter the Bureau's forecasting database. It is therefore expected that the minimum predictor data preparation time is 3 days.
Forecast and communication strategy production is a process that takes 1-2 days. Thus for the Bureau to consider releasing forecasts prior to the beginning of the target season, it would be a safe choice to prepare forecasts with 7-day lead time. 25 Compared with forecasts with 0-days lead time, the mean reduction in CRPS skill scores in skilful cases is approximately 4 pp (Figure 8), which is likely to be tolerated by forecast users in exchange for earlier forecast release. However, it is to be reasonably expected that CRPS skill scores will reduce by up to 10-15 pp in some instances (Figure 8). The significant reduction in skill scores at 21-days lead time (Figure 8) highlights the importance of short-lead time forecasts for Australian catchments and confirms that simply switching to a 1-month ahead forecast system is undesirable. 30 A better, more flexible operational forecasting system for Australia could be built upon a flexible N-days lead time forecasting strategy. An N-days lead time forecasting system allows for multiple forecast runs prior to forecast release. To Hydrol. Earth Syst. Sci. Discuss., doi:10.5194/hess-2016-199, 2016 Manuscript under review for journal Hydrol. Earth Syst. Sci. Published: 9 May 2016 c Author(s) 2016. CC-BY 3.0 License. reduce forecast preparation time, while striving for optimal skill, a communication strategy for the forecasts can be developed based on a preliminary forecast run with, for example, 7-days lead time. The final forecast release could subsequently be based on a shorter lead time, e.g. 4-days. Furthermore, it sometimes happens that unexpected heavy rains fall within the period of forecast generation and forecast release. If significant events occur that change the hydrological outlooks dramatically, having the option for N-days lead time forecasts allows for reissue of forecasts. 5 The discussion thus far has considered only the forecast preparation and dissemination time. As identified previously, optimal forecast lead time for operational decision making depends on other factors including the time needed for comprehension and application. That is, forecast users need time to understand the forecasts and the likely impact on their operations. The process of understanding may include further sophisticated modelling using streamflow forecasts as inputs.
For water managers, lead times of a few days may be sufficient to assimilate the forecast information. For other operators, 10 longer lead-time forecasts may be preferred. In fact, the optimal forecast lead time for operational decision making across a range of industries is likely to vary. It therefore remains a research questions whether water forecasting services need to evolve to cater for the needs of different (sophisticated) water forecast users.

Conclusion
Currently the Bureau of Meteorology releases seasonal streamflow forecasts approximately 7 days into the forecast target 15 season. In this study we develop seasonal streamflow forecasting models with 0-14 and 21 days lead time to demonstrate that it is possible to release skilful operational forecasts ahead of the commencement of the forecast period. Forecasts were produced for 23 of the Bureau of Meteorology's seasonal streamflow forecast locations, using the Bayesian joint probability modelling approach. The forecasting models were constructed similarly to the Bureau's official models, using climate predictors and initial catchment condition predictors. Climate predictors were adapted to be based on SST predictors so that 20 daily SST data sets could be adopted.
The skill and reliability of the 0-14 day lead time forecasts were assessed. Reliability was found to be similar for all forecast lead times. Average skill reduces monotonically for each 1 day that forecast lead time increases. For forecasts with 7-days lead time, the mean reduction in CRPS skill scores is small, approximately 4 percentage points, although skill score differences within a range of +5 to -15 are possible. For forecasts with 14-days lead time, the mean reduction in CRPS skill 25 scores is approximately 6 percentage points. The reductions in skill are very likely to be tolerated by forecast users in exchange for forecasts released ahead of the commencement of the forecast target season. Particularly as the Bureau moves towards a monthly forecasting service, timelier forecast release is going to become critical.