Combining satellite observations to develop a global soil moisture product for near-real-time applications

The soil moisture dataset that is generated via the Climate Change Initiative (CCI) of the European Space Agency (ESA) (ESA CCI SM) is a popular research product. It is composed of observations from 10 different satellites and aims to exploit the individual strengths of active (radar) and passive (radiometer) sensors, thereby providing surface soil moisture estimates at a spatial resolution of 0.25. However, the annual updating cycle limits the use of the ESA CCI SM dataset for operational applications. Therefore, this study proposes an adaptation of the ESA CCI product for daily global updates via satellite-derived near-realtime (NRT) soil moisture observations. In order to extend the ESA CCI SM dataset from 1978 to present we use NRT observations from the Advanced Scatterometer on-board the two MetOp satellites and the Advanced Microwave Scanning Radiometer 2 on-board GCOM-W. Since these NRT observations do not incorporate the latest algorithmic updates, parameter databases and intercalibration efforts, by nature they offer a lower quality than reprocessed offline datasets. In addition to adaptations of the ESA CCI SM processing chain for NRT datasets, the quality of the NRT datasets is a main source of uncertainty. Our findings indicate that, despite issues in arid regions, the new CCI NRT dataset shows a good correlation with ESA CCI SM. The average global correlation coefficient between CCI NRT and ESA CCI SM (Pearson’s R) is 0.80. An initial validation with 40 in situ observations in France, Spain, Senegal and Kenya yields an average R of 0.58 and 0.49 for ESA CCI SM and CCI NRT, respectively. In summary, the CCI NRT product is nearly as accurate as the existing ESA CCI SM product and, therefore, of significant value for operational applications such as drought and flood forecasting, agricultural index insurance or weather forecasting.


Introduction
Soil moisture, the water in the soil's pore space, is one of very few environmental variables that directly link atmospheric processes to land surface conditions (Legates et al., 2010;Taylor et al., 2012).It is a decisive or even limiting factor in many processes related to agriculture, climate change, energy fluxes, hydrology and hydro-climatic extreme events (Brocca et al., 2010;Greve et al., 2014;Jung et al., 2010;Legates et al., 2010;Qiu et al., 2014;Seneviratne et al., 2010;Sheffield and Wood, 2008;Taylor et al., 2012;Trenberth et al., 2014).Along with temperature and precipitation, soil moisture is ranked a top priority variable in all societal benefit areas listed by the Group on Earth Observations (agriculture, biodiversity, climate, disasters, ecosystems, energy, health, water and weather) (Group on Earth Observations, 2012).Also, aid organizations or developers of financial instruments (e.g.weather index insurance), whose potential regions of interest may encompass whole sub-continents, are gradually discovering the importance of soil moisture for assessments of drought-related food insecurity.
Published by Copernicus Publications on behalf of the European Geosciences Union.
Traditional measurements of soil moisture relied on direct in situ methods, such as gravimetric samples or time domain reflectometry (Dorigo et al., 2011;Wagner et al., 2007).In situ observations are to date the most accurate localized measurements of soil moisture, but only models or satellites are able to provide spatially consistent information on a global scale.However, datasets derived from space-borne microwave sensors are not yet able to capture variability at the scale of metres at sub-daily intervals.Hence, the concept of temporal stability (Brocca et al., 2009;Vachaud et al., 1985), which describes a quasi-linear relationship between soil moisture variations over time on different spatial scales, allows using coarse information acquired via satellites to understand local to regional phenomena.
Satellite instruments capable of retrieving information about soil moisture have been available since the late 1970s.However, despite the existence of several individual spaceborne soil moisture products, a harmonized long-term dataset was missing until the Water Cycle Multi-mission Observation Strategy (WACMOS) project and the Climate Change Initiative (CCI) of the European Space Agency (ESA) released the first multi-sensor soil moisture product (Liu et al., 2011a(Liu et al., , 2012;;Wagner et al., 2012).This ESA CCI soil moisture dataset (ESA CCI SM) relies on the merging of different active (radar) and passive (radiometer) microwave instrument observations into a single consistent product (Dorigo et al., 2015) based on uncertainty information of the individual soil moisture products (Liu et al., 2011a;Dorigo et al., 2010).The latest official release of the ESA CCI SM product (CCI SM v02.2) covers a period from 1978 to 2014.Product updates that extend the temporal coverage are performed every year by incorporating new observations from radars and radiometers.
Since its release in 2012, the ESA CCI SM dataset has been used in a wide variety of studies (Dorigo and de Jeu, 2016).Yuan et al. (2015), for instance, analysed the performance of ESA CCI SM to detect short-term (monthly to seasonal) droughts in China with respect to in situ observations and two soil moisture reanalysis datasets, namely the Global Land Data Assimilation System (GLDAS) (Rodell et al., 2004) and ERA Interim (Dee et al., 2011).ESA CCI SM captured less than 60 % of drought months at the scale of in situ stations.However, comparable to the reanalysis products, it performed well with regard to the detection of inter-annual variations of short-term drought on river basin scale, particularly in sparsely vegetated areas.Nicolai-Shaw et al. (2015) confirm these findings over North America by comparing ESA CCI SM with reanalysis products of the European Centre for Medium Range Weather Forecasting (ECMWF) and in situ observations.Regarding the spatial representativeness, ESA CCI SM showed a higher agreement with the in situ observations than with the reanalysis data.With respect to the absolute values, however, the agreement between ESA CCI SM and the reanalysis data were higher.McNally et al. (2015) showed the superiority of the Water Requirement Satisfaction Index in Senegal and Niger when fed with ESA CCI SM instead of a water-balance model output.Finally, ESA CCI SM was also used to identify global trends in soil moisture with regard to vegetation (Barichivich et al., 2014;Dorigo et al., 2012;Muñoz et al., 2014) and to improve the understanding of the land-atmosphere coupling (Hirschi et al., 2014).
However, decision makers in various applications and domains (e.g.weather prediction, drought and flood monitoring, index-based agricultural insurance) need more timely soil moisture product updates at daily or sometimes even subdaily intervals.In case of weather prediction, for instance, satellite-derived soil moisture is usually assimilated via a nudging scheme or an ensemble Kalman filter approach at sub-daily (e.g.6-hourly) increments (Drusch, 2007;Drusch et al., 2009;Scipal et al., 2008).In case of drought monitoring, satellite-derived soil moisture can be used to fill the gap between satellite-based estimates of rainfall and vegetation vigour (Enenkel et al., 2014).However, the current ESA CCI SM product does not fulfil this requirement with regard to updates at appropriate time steps.Bridging this gap requires daily product updates of the ESA CCI SM dataset by seamlessly integrating near-real-time (NRT) soil moisture observations.Therefore, we use observations from two space-based sensors.One of these sensors is a radar, the Advanced Scatterometer (ASCAT) on-board the Meteorological Operational satellites MetOp-A and MetOp-B; the other is one a radiometer, the Advanced Microwave Scanning Radiometer (AMSR2) on-board the Global Change Observation Mission for Water (GCOM-W1) satellite.NRT means that both the observations from ASCAT and AMSR2 are available within 2-3 h after the satellite overpass.The resulting dataset is called CCI NRT.
This study has two complementary objectives.The first objective is to describe how the current ESA CCI processing chain is adapted to generate a CCI NRT soil moisture product by discussing issues related to the resampling of time series (ASCAT offline) and orbit format data (ASCAT NRT) to a quarter degree grid, missing surface state flags for snow-covered or frozen soils in ASCAT NRT or differences in the masking of radio frequency interference (RFI) in case of AMSR2 (Sect.3.1).The second objective is to investigate how well the CCI NRT dataset compares to ESA CCI SM on a global scale (Sect.4).In addition to the adaptations of the processing chain, we highlight that the difference in the backscatter and calibration levels of the NRT input datasets (compared to the offline datasets) naturally leads to differences in soil moisture estimates.Particularly in the case of AMSR2, issues related to its calibration resulted in different product versions, which we try to clarify in Sect.2.3.1.The initial sensor calibration of AMSR2 was recently improved after gathering a sufficiently large overlapping dataset with its predecessor AMSR-E through a dedicated "slow rotation" mode.This dataset is used to generate the ESA CCI SM dataset.However, the AMSR2 NRT dataset does not apply Hydrol.Earth Syst. Sci., 20, 4191-4208, 2016 www.hydrol-earth-syst-sci.net/20/4191/2016/ this calibration, potentially affecting the level of brightness temperature.We try to quantify the errors via an initial validation of the CCI NRT and the ESA CCI SM dataset with respect to 40 in situ stations in France, Senegal, Spain and Kenya.

Datasets used
Depending on the sensor and retrieval approach, space-based soil moisture retrievals show distinct variations in performance on a global scale (e.g.Crow et al., 2010;Dorigo et al., 2010).In combination with the TU Wien change detection algorithm, C-band radars (e.g.ASCAT), for instance, are better suited to retrieve soil moisture over moderate vegetation cover than radiometers (Al-Yaari et al., 2014;Dorigo et al., 2010;Gruhier et al., 2010;Rüdiger et al., 2009).Simultaneously, radars are facing challenges in arid regions that are often characterized by sandy soils (Wagner et al., 2003(Wagner et al., , 2007) ) due to volume scattering of the microwave beam.The following section describes the general characteristics of the reprocessed ESA CCI SM product, as well as the operational products from ASCAT and AMSR2 that are used to generate the extension of the ESA CCI SM dataset via daily updates.

ESA CCI surface soil moisture
The ESA CCI soil moisture product was generated in accordance with the World Meteorological Organization's 2008 report, "Future Climate Change Research and Observation".The report highlights the importance of collecting, harmonizing and validating soil moisture observations from different sources to extend the temporal and spatial coverage, to improve data quality (also for further data assimilation), to support the understanding of feedback mechanisms and the prediction of extreme events.The ESA CCI SM dataset incorporates the measurements of 10 satellites (Fig. 1).It is available at daily time steps and on a 0.25 • × 0.25 • latitude/longitude global array of grid points (i.e. a global 0.25 • grid).The quality flags, which are distributed in combination with the dataset, provide information about the sensor and observation frequency that was used for each soil moisture retrieval, the moment of the measurement, ascending or descending orbit and snow/frozen soil probability.According to Liu et al. (2011bLiu et al. ( , 2012)), soil porosity values derived from 1300 global samples are used in the algorithm developed by the VU University Amsterdam and the National Aeronautics and Space Administration (NASA) to generate soil moisture data from passive sensors via the Land Parameter Retrieval Model (LPRM) (Holmes et al., 2009;Owe et al., 2008).The same porosity values are also applied in GLDAS, which is used as a reference dataset to rescale soil moisture estimates from all active and passive sensors shown in Fig. 1 via cumulative distribution function (CDF) matching (Liu et al., 2009;Reichle and Koster, 2004).

Active microwave measurements from the ASCAT
The ASCAT sensors on-board MetOp-A/B are real aperture radar sensors.Their soil moisture retrieval is based on the backscatter of microwaves that are sensitive to the dielectric properties of the water molecule, resulting in a quasilinear increase relationship between increasing water content and microwave backscatter.ASCAT operates in C band (5.255 GHz), scanning two 550 km swaths with three antennas on each side.Consequently, every location is scanned from three different angles, enabling corrections for vegetation cover by analysing measurement differences at different angles.This principle is exploited by the TU Wien Water Retrieval Package (WARP), a change detection algorithm that results in surface soil moisture observations in relative units (%).These observations are scaled between the historically lowest and highest values, corresponding to a completely dry surface and soil saturation (Bartalis et al., 2005;Wagner et al., 1999Wagner et al., , 2013)).
WARP is optimized to estimate model parameters from multi-year backscatter time series on a discrete global grid (DGG).These parameters help to understand the characteristics of scattering effects on a global scale, which are affected by surface roughness and vary with land cover.However, there are large differences between soil moisture derived from ASCAT via the offline WARP processing chain and its operational version WARP NRT, which result in different backscatter levels.While the offline WARP processor generates soil moisture on a discrete global grid, the WARP NRT product is distributed from EUMETSAT (European Organisation for the Exploitation of Meteorological Satellites) in orbit geometry.It is available 135 min after the overpass of the two ASCAT sensors on board the MetOp-A and MetOp-B satellites.An advantage of WARP NRT is the high robustness and speed of the processing chain (less than a minute for one ASCAT orbit).However, updates related to algorithmic improvements and updates in the calibration of the backscatter measurement usually take a lot of time (Wagner et al., 2013).Several parameters, most importantly a dynamic mask for snow-covered/frozen soils, are not available in NRT.As a result, the quality of NRT soil moisture data lags behind the quality of reprocessed, offline datasets.
Validations of the NRT soil moisture product disseminated via EUMETCAST (Albergel et al., 2012) yielded an average root mean squared difference (RMSD) of 0.08 m 3 m −3 for more than 200 in situ stations around the globe.While the global average of all correlations was r = 0.50, the best correlation (r = 0.80) was achieved for an in situ network in Australia (OZNET).In general, the correlations were higher during winter months.

Passive microwave measurements from the AMSR2 radiometer
Passive soil moisture retrievals are based on the dielectric contrast between dry and wet soil that leads to changes in emissivity from 0.96 for dry soils to below 0.60 for wet soils (Njoku and Li, 1999;Schmugge and Jackson, 1994).
Being very similar to its predecessor AMSR-E, AMSR2 onboard the GCOM-W1 satellite measures brightness temperature at six different bands with vertical and horizontal polarizations at each frequency.In addition, the vertically polarized Ka-band (36.5 GHz) observations are used to simultaneously estimate land surface temperature (Holmes et al., 2009).In contrast to ASCAT, the AMSR sensors have a fixed observation angle at 55 degrees, resulting in a conically shaped footprint and a swath width of 1445 km.Both radiometer observations in the ESA CCI SM dataset and its NRT equivalent only use nighttime measurements (Liu et al., 2011), because a smaller temperature gradient between the soil and vegetation facilitates higher quality soil moisture retrievals (de Jeu et al., 2009).The LPRM transforms information about the dielectric constant into volumetric soil moisture by applying an empirical dielectric mixing model (Wang and Schmugge, 1980).Similar to ASCAT, reliable measurements over frozen or snow-covered soils are not possible due to the immovability of the water molecules.Several studies compared the performance of soil moisture products from the AMSR sensors and ASCAT (Brocca et al., 2011;Dorigo et al., 2010;Gruber et al., 2016), leading to overall comparable and complementary performance.An intercomparison over 17 European sites (Brocca et al., 2011), for

Issues related to the intercalibration of AMSR-E and AMSR2
The consistency of brightness temperature observations from AMSR-E to AMSR2, hence also soil moisture retrievals, is challenging due to the lack of an operational overlapping period between both sensors.AMSR-E was shut down in October 2011 while the AMSR2 soil moisture dataset started with July 2012.As a result, the first version of AMSR2 data was not perfectly intercalibrated with AMSR-E.In December 2012, AMSR-E was switched on again in a special slow rotation mode to get simultaneous observations of the sensors.Afterwards, the overlapping dataset between the operational AMSR2 and slow rotation AMSR-E was sufficiently large to re-calibrate AMSR2 and align those measurements based on this overlapping period (http://global.jaxa.jp/press/2015/12/20151207_amsr-e.html).Before JAXA corrected for these subtle differences, a preliminary solution was developed by Parinussa et al. (2015).This preliminary product was used to generate the ESA CCI dataset.As a consequence, the AMSR2 soil moisture product that was used to create the ESA CCI SM dataset is a different version than the current operational product that we use to develop the CCI NRT product, but both products are generally comparable (Parinussa et al., 2015).Just like its predecessor AMSR-E, AMSR2 needs to cope with RFI which is capable of jeopardizing whole satellite missions (Oliva et al., 2012).Currently, the RFI masking is based on a decisiontree that selects the passive microwave observations in the lowest available frequency that is free of RFI for each individual grid point (Fig. A7).AMSR2 offers an important advantage through additional observations at 7.3 GHz, which is a frequency that significantly improves the detection of RFI.However, in most cases the 6.9 GHz channel can be used.

In situ networks
All in situ measurements used for this study were obtained via the International Soil Moisture Network (Dorigo et al., 2011(Dorigo et al., , 2013)).The single probes/networks (Fig. 2) were selected to cover regions in which either the active, passive and merged component of the CCI NRT dataset (explained in Sect.3) are used.
Accordingly, we extracted measurements from the SMOS-MANIA network (Albergel et al., 2008) in the south of France to validate the active component of the daily ESA CCI surface soil moisture updates, from the REMEDHUS network (Sanchez et al., 2012) in the west of Spain to validate the merged active-passive component, from the Dahra network in Senegal and the Cosmos network in Kenya to validate the passive component.The SMOSMANIA (Albergel et al., 2008) and Dahra networks are equipped with the same type of probes (ThetaProbe ML2X), while the REMEDHUS network that covers the Duero Basin relies on Stevens Hy-draProbes.The Cosmos station in Kenya relies on a cosmicray probe.All in situ observations were filtered for stations that measure the soil moisture content up to a depth of 5 cm (respectively, 10 cm for the Cosmos station) to ensure the comparability with the satellite-derived surface soil moisture datasets.

Methods
The following section is divided into two parts.Section 3.1 concentrates on the adaptation of the ESA CCI SM processing chain for daily updates.Section 3.2 explains the corresponding validation of the new dataset on a global scale.

Integrating NRT ASCAT and AMRS2 into the ESA CCI SM dataset
The integration of NRT ASCAT and AMSR2 observations into the ESA CCI SM builds on the procedures used to gen-erate the standard ESA CCI SM products (Liu et al., 2011a(Liu et al., , 2012;;Wagner et al., 2012).Figure 3 illustrates the main processing steps for the integration of NRT soil moisture observation in a flow chart.The most recent official ESA CCI SM product covers the years 1978-2014.The CCI NRT dataset extends this temporal coverage to the present with an overlap for 2013-2014.
The aim is to keep the processing chain of the NRT datasets as similar as possible to the ESA CCI SM processing chain.However, several adaptations are unavoidable with regard to the resampling and the masking of snowcovered/frozen soils.In contrast to the offline soil moisture observations from ASCAT that were resampled to a quarter degree as time series to generate the ESA CCI, the NRT AS-CAT data from EUMETSAT have to be resampled from orbit geometry.Also, the masking of snow-covered/frozen soils needed to be adapted.While a surface state flag for snowcovered/frozen soils is available for the ASCAT observations in the ESA CCI dataset, the NRT ASCAT product from EU-METSAT is based on an older algorithm that is incapable of generating a surface state flag.As a consequence, we apply a mask based on a daily climatology (probability) for snowcovered/frozen soils.In addition to the snow flag, a second mask is applied to the ASCAT data based on vegetation optical depth (VOD).VOD is a dimensionless variable linked to the vegetation water content and above ground biomass (Liu et al., 2015).VOD has previously been used as an additional indicator for long-term vegetation dynamics (Liu et al., 2011b) and is retrieved simultaneously to soil moisture through the LPRM.Retrievals with VOD values > 0.8 (dense vegetation) are masked.The AMSR2 data are masked for soil skin temperature below freezing (Holmes et al., 2009), RFI and VOD.After the spatial resampling via a regular hamming window to a 0.25 • grid, we apply the temporal resampling to 00:00 UTC reference time via nearest neighbour search to both datasets.While we use both ascending and descending orbits in case of ASCAT, we only use the descending (nighttime) observations from AMSR2 (de Jeu et al., 2009;Lei et al., 2015).
Before the active and the passive datasets can be merged, it is vital to allow for different observation frequencies, observation principles and retrieval techniques.Consequently, we rescale both datasets to a reference soil moisture dataset (GLDAS-Noah) via piecewise CDF matching (Liu et al., 2011a;Reichle et al., 2004).The rescaling is carried out for each grid point individually.Also, values outside the range of the CDF curves can be rescaled by using the linear CDF equation of the closest value.The uncertainty (noise) of the rescaled soil moisture dataset is estimated by multiplying the ratio of the rescaled and the non-rescaled soil moisture value with the original noise.Due to the unavailability of the GLDAS dataset in NRT, we apply the scaling functions that were used to generate the original ESA CCI SM dataset.This way it is possible to preserve the datasets' original, relative dynamics, while adjusting them to the same range and distri- bution.Once this step is completed, the active and the passive datasets can be merged.
Figure 4 illustrates the coverage of active, passive and merged data on a global scale.The passive LPRM soil moisture product is used in regions with low vegetation density (VOD < 0.24), whereas the TU Wien ASCAT product is applied in regions with moderate to high vegetation density (VOD 0.60).So-called transition zones between dry and humid climates are characterized by VOD values between 0.24 and 0.60.In these regions the active and the passive products agree well (R > 0.65).Therefore, both products can be merged (green areas in Fig. 4).

Performance metrics and validation
According to Wagner et al. (2013), the validation of satellite data via in situ observations can be critical due to different issues, such as the high spatio-temporal variability of soil moisture (Western et al., 2002) or a lack of adequate reference datasets (Crow et al., 2012).There are no reference data that represent exactly the same physical quantity as the satellite observation.Acknowledging these limitations, this study concentrates on the following comparative assessments: Equation ( 1) shows that the bias E is expressed as the difference between the time series' f and reference r, corresponding to the mean values of CCI NRT and ESA CCI SM/in situ observations, respectively.
As the name suggests, the unbiased RMSD considers the overall bias related to the quadratic difference in observations (Taylor, 2001).Consequently, the unbiased RMSD E in Eq. ( 2) relates the individual bias for each time series to the corresponding observation values, whereas f n and r n again correspond to observations of ESA CCI SM and CCI NRT. (2)

Results
The Pearson correlation coefficient (R) yields an average correlation of 0.80 for ESA CCI SM and CCI NRT on a global scale (Fig. 5).Regions in which the NRT dataset does not correspond well with the offline dataset include parts of north Africa and the Sahara, the west coast of the United States and several large mountain ranges (e.g. the Andes in South America).Tropical forests are masked, because they are impenetrable to radars at the applied frequencies and block the soil moisture emission for radiometers.
Since the good agreement of the ESA CCI SM and the CCI NRT dataset is only meaningful if it represents actual surface soil moisture conditions on the ground, we calculate the performance metrics for both datasets related to daily in situ observations (Table 1).The average Pearson correlation coefficient for all in situ stations is 0.58 (ESA CCI SM) and 0.49 (CCI NRT), respectively.The statistical scores for the SMOSMANIA and the REMEDHUS network are comparable to the findings of Albergel et al. (2012) or Dorigo et al. (2015).The bias and the unbiased RMSD are slightly higher for CCI NRT.
The validation results for the corresponding anomalies, which were calculated based on a moving average with a window size of 35 days, are in line with the findings of Albergel et al. (2013).Table 2 lists the Pearson correlation coefficient, which is on average lower for the anomalies than for their normal time series and also lower for CCI NRT than for ESA CCI SM.Again, both the bias and the unbiased RMSD are higher for CCI NRT.
The Pearson (R) and Spearman (S) correlation coefficients between ESA CCI SM and CCI NRT over the locations of the in situ stations confirm the global picture with an average R of 0.80 and an S of 0.82.The best correlation is observed over the location of the Urgons station in the SMOSMANIA network, which is located in a cultivated area in the south of France.The corresponding Fig. 6 shows an R of 0.93 and a S of 0.96.However, in the same network we also observe the worst agreement (R = 0.62, S = 65) at a station named Savenes (not shown).
Global maps of the absolute differences between both datasets for 2013 (Fig. B8) and the four seasons (Figs.B9-B12) show a systematic positive bias in CCI NRT of up to 0.30 m 3 m −3 in regions like east Africa or Pakistan.This effect is stronger in spring and summer than in autumn and winter.In the central United States, large parts of Australia and southern Africa the bias overestimation is smaller.Since the overestimation mainly appears in regions where the AMSR2 dataset is used (Fig. 4) and to understand the bias of soil moisture over Europe during winter 2013, we also analyse the absolute difference between the offline and the NRT ASCAT and AMSR2 datasets (Figs.C13 and C14).Compared to the offline product, AMSR2 NRT tends to overestimate on a global scale, mainly in parts of the Horn of Africa, the Arabic peninsula, parts of Australia, South America and southern Africa.The strong overestimation in the Horn of Africa is also clearly visible in the CCI NRT dataset.On the contrary, ASCAT NRT tends to underestimate, mainly over Europe with the strongest signal over the winter season, parts of the western United States as well as areas north and east of the Black Sea.In summary, our validation results indicate that, with some exceptions, the new CCI NRT dataset performs well on a global scale in comparison to its offline counterpart.

Discussion and conclusions
The global daily update of the ESA CCI SM surface soil moisture dataset is motivated by an increasing interest in soil moisture products that offer long (> 30 years) reference periods for a wide range of applications.The need for improved and more timely soil moisture representations in agricultural drought monitoring is one of the strongest motivations (An-  derson et al., 2012;Bolten and Crow, 2012;Enenkel et al., 2014;Hirschi et al., 2014).Hence, this study concentrated on three main topics.First, we analyse the challenges related to the adaptation of the ESA CCI SM processing chain for NRT soil moisture observations from ASCAT and AMSR2.Just like in the case of ESA CCI SM, the CCI NRT merging scheme considers each sensor's individual strengths and limitations.ASCAT, for instance, performs better than AMSR2 at higher vegetation densities, while one strength of AMSR2 is the retrieval over semi-arid and arid regions (Liu et al., 2011a).The challenges are mainly related to the resampling of the NRT data to a common quarter degree grid and a quality flag for snow-covered/frozen soils, which does not exist for the NRT ASCAT dataset.Second, we identify the impact of NRT soil moisture algorithms and intercalibration issues of AMSR-E/AMSR2 on the final merged CCI NRT product.Third, we perform an initial validation on a global scale as well as based on in situ soil moisture observations that were selected based on their reliability, temporal coverage and ability to reflect the individual components (active/passive/combined) of the CCI NRT dataset.Finally, we also examine the agreement of the ESA CCI SM/CCI NRT/in situ anomalies and the absolute differences between ESA CCI SM and CCI NRT on a global scale.
Our main findings are as follows: -There is a high agreement between the CCI NRT dataset and the ESA CCI SM dataset on a global scale for the entire year 2013 (average R = 0.80).This finding also indicates a good performance of NRT soil moisture observations from ASCAT and AMSR2 and therefore the operational readiness of the CCI NRT algorithm.Low correlations are, for instance, observed in areas that permanently show low levels of soil moisture, e.g. the arid zones of northern Africa.The error sources in the CCI NRT product are likely due to the predominant use of AMSR2 in the merged dataset for these regions: calibration differences exist between the AMSR2 dataset used in ESA CCI SM and the latest AMSR2 NRT dataset used in CCI NRT, causing differences between the two merged products.Also, the challenging issue on aligning the brightness temperatures of both AMSR sensors was only recently solved through a slow rotation mode of AMSR-E that was dedicated to intercalibration (Sect.2.3.1.).
-The validation with in situ observations in Spain, France, Senegal and Kenya yields less accurate results for the CCI NRT dataset than for ESA CCI SM.The average Pearson correlation coefficient (R) for all in situ stations is 0.49 (0.58 for ESA CCI SM).The unbiased RMSD for CCI NRT is 0.008 m 3 m −3 (0.004 m 3 m −3 for ESA CCI SM).We observe hardly any difference in the overall bias (0.05 m 3 m −3 for both datasets).
Hydrol.Earth Syst.Sci., 20, 4191-4208, 2016 www.hydrol-earth-syst-sci.net/20/4191/2016/  -The performance metrics for the corresponding anomalies result in an average correlation coefficient (Pearson) of 0.44 for ESA CCI SM and 0.38 for CCI NRT, respectively.Also, with regard to absolute difference the general agreement between CCI NRT and ESA CCI SM is satisfying.A comparison of both datasets for 2013 reveals a bias of CCI NRT over Europe during winter 2013 (Fig. C13) and a bias over several dry areas, e.g. over parts of Africa and Australia (Fig. C14; Appendix), which is likely related to intercalibration issues between AMSR2 and its predecessor AMSR-E (Okuyama and Imaoka, 2015;Parinussa et al., 2015).
We expect that, apart from solving the AMSR2 intercalibration issues and a dynamic snow map for ASCAT, which should improve the performance during winter, two improvements in the processing chain could lead to considerable improvements in data quality.First, there are differences in the temporal coverage of the MetOp-A ASCAT data used to derive soil moisture model parameters for the offline ASCAT (2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014) and ASCAT NRT (2007NRT ( -2012) ) products.The offline and the NRT ASCAT product used in this study differ in their absolute calibration level affecting the soil moisture values.Despite the good correlation between both products it is likely that their consistency can be improved by reprocessing the rescaling parameters in the CCI NRT processing chain, which are currently based on parameters that had been developed for the offline ASCAT product.Second, the currently static RFI map for AMSR2 could be replaced by a dynamic map that is based on the average RFI values for the previous 6 months via a moving average.In a recent study (de Nijs et al., 2015), an improved algorithm to detect RFI at the global scale for 6.9 and 7.3 GHz AMSR2 observations was proposed, but remains to be tested for the specific implementation in the CCI NRT product.This is the first method that takes the additional 7.3 GHz channel into account, which was specifically added to the AMSR-E sensor constellation and proved to mitigate issues related to RFI.Despite these issues, the development of an operational processing chain that allows daily soil moisture updates is particularly promising with regard to applications that aim at the confirmation of satellite-based rainfall estimates (Brocca et al., 2013) or at closing the gap between rainfall estimates and the response of vegetation (Enenkel et al., 2014).In this regard, the integration of the latest generation of soil moisture sensors, such as Sentinel-1 of the ESA and the European Commission (EC) or NASA's SMAP (Soil Moisture Active/Passive), whose L-band radiometer is still active after the failure of the radar, could lead to further improvements.These new sensors are able to retrieve soil moisture at a far higher resolution than ASCAT or AMSR2: in case of Sentinel-1, around 1 km for operational products and below 100 m for research products.Of course the higher spatial resolution has a drawback, which is a decrease in temporal resolution.While ASCAT on MetOp-A alone covers more than 80 % of the globe every day, the two Sentinel-1 satellites will take 6-12 days to scan the total global land mass in the default interferometric wide swath (IWS) mode (World Meteorological Organization, 2013).Despite the differences in spatial resolution, it is possible to increase the temporal resolution of the CCI NRT dataset to fit various applications.
In the face of the latest generation of space-based soil moisture sensors, it seems to be the most promising approach to exploit each sensor's individual strength to generate the most accurate and complete soil moisture dataset.However, developing a user-friendly dataset means more than data access.As a consequence, software packages such as Python Open Earth Observation Tools (Mistelbauer et al., 2014) are necessary to enable automated updates, the visualization of images/time series/anomalies and the analysis of critical soil moisture thresholds.A pre-operational CCI NRT dataset will soon be available via the remote sensing research group of TU Wien (http://rs.geo.tuwien.ac.at/).The global dataset will be provided in NetCDF file format.Updates are planned for every 10th, 20th and last day of every month, resulting in a quasi-decadal (10 daily) dataset.

Data availability
For this study we used the official ESA CCI soil moisture product (CCI SM v02.2), which is available at http://www.esa-soilmoisture-cci.org/node/145.The ASCAT NRT dataset is available via EUMETSAT at http://www.eumetsat.int/website/home/Data/Products/Land/index.html.The AMSR2 NRT dataset is distributed from NASA and the Japan Aerospace Exploration Agency (JAXA).It is available at NASA's Global Change Master Directory at http://gcmd.gsfc.nasa.gov/r/d/[GCMD]GES_DISC_LPRM_AMSR2_SOILM2_V001.
In situ observations were accessed via the the International Soil Moisture Network (http://ismn.geo.tuwien.ac.at/).

Figure 1 .
Figure 1.Satellites and sensors used for generating the offline ESA CCI SM dataset and the daily continuation via ASCAT and AMSR2.Dotted lines indicate inactive missions, yellow arrows represent passive measurements, green arrows represent active measurements.The ESA CCI SM dataset only includes SSM/I data until 2007.

Figure 3 .
Figure 3. Schematic flowchart illustrating the methodology for extending the ESA CCI SM dataset via NRT observations from ASCAT and AMSR2.The GLDAS-Noah dataset is used as a scaling reference.

-
calculating the Pearson's correlation coefficient (R) between ESA CCI SM and CCI NRT for an overlapping year (2013) on a global scale; calculating the absolute differences in volumetric soil moisture between both datasets for the entire year 2013 (including individual calculations for all seasons) on a global scale; individual validation for ESA CCI SM and CCI NRT for 2013 over 40 in situ soil moisture stations in France, Kenya, Senegal and Spain.For each in situ observation a nearest neighbour search selects the closest grid point in the satellite-derived datasets.The performance metrics include -Pearson correlation (R), indicating a linear relationship between two variables, -Spearman correlation (S), a non-parametric test that does not rely on any assumption about the distribution of the data, the absolute bias in m 3 m −3 , and unbiased root mean squared difference (ubRMSD) in m 3 m −3 .

Figure 4 .
Figure 4. Global blending map illustrating where active sensors (red), passive sensors (yellow) and the average of both (green) is used to generate the ESA CCI SM product (modified from Liu et al., 2012).

Figure 5 .
Figure 5. Global correlation (Pearson's R) for ESA CCI SM and CCI NRT for 2013 (no negative correlations were observed).The white triangles illustrate the location of the in situ stations/networks.

Figure 6 .
Figure 6.Illustration of ESA CCI SM and CCI NRT over the Urgons station of the SMOSMANIA network (R = 0.93; S = 0.96).

Figure B2 .
Figure B2.Absolute differences in soil moisture (ESA CCI SM minus CCI NRT) for winter 2013.

Figure B3 .
Figure B3.Absolute differences in soil moisture (ESA CCI SM minus CCI NRT) for spring 2013.

Table 1 .
Statistical scores for ESA CCI SM/CCI NRT and in situ stations/networks (maximum depth 0.1 m) in Spain, France, Kenya and Senegal for 2013 (for the REMEDHUS and SMOSMANIA networks the table includes the bias range from minimum to maximum).

Table 2 .
Statistical scores for ESA CCI SM/CCI NRT anomalies and in situ stations/networks (maximum depth 0.1 m) in Spain, France, Kenya and Senegal for 2013 (for the REMEDHUS and SMOSMANIA networks the table includes the bias range from minimum to maximum).