IMSC2016 renard vidal
A performance weighting procedure for GCMs based on explicit probabilistic models and accounting for observation uncertainty Benjamin Renard1 & Jean-Philippe Vidal1 1 IRSTEA,
UR HHLY
9 June 2015
www.irstea.fr
1 / 16
Outline
1. Research questions 2. Methodology and example case study 3. Full-scale application 4. Conclusions 5. Open questions
2 / 16
Research questions
How well do GCMs simulate specific climate features? How to formalize this performance given the uncertainty in the observed climate features?
3 / 16
Methodology Ingredients Defining application-relevant climate features and probabilistic model Weather variable(s) series at location(s) (Leith and Chandler, 2010) z 1:T Statistics from the weather (mean, variance, trend, etc.) Explicit probabilistic model Y) z 1:T ∼ D (Y
4 / 16
Methodology Ingredients Defining application-relevant climate features and probabilistic model Weather variable(s) series at location(s) (Leith and Chandler, 2010) z 1:T Statistics from the weather (mean, variance, trend, etc.) Explicit probabilistic model Y) z 1:T ∼ D (Y Defining observation uncertainty Observed climate = true climate + observation errors Errors centred with known covariance matrix (Rougier, 2007) (0) (0) z 1:T ∼ D Y˜ + with ∼ N (00, Σ )
4 / 16
Example case study Winter minimum SLP in Reykjav´ıc 1950-2004 Renalysis data 56 20CR members (Compo et al., 2011) Observed weather: member #0
NCEP/NCAR (Kalnay et al., 1996)
GCM data 6 CMIP5 GCMs, some of them with multiple runs (Taylor et al., 2012)
ERA20C (Poli et al., 2016)
5 / 16
Example case study Winter minimum SLP in Reykjav´ıc 1950-2004 Renalysis data
GCM data 6 CMIP5 GCMs, some of them with multiple runs (Taylor et al., 2012)
56 20CR members (Compo et al., 2011) Observed weather: member #0
NCEP/NCAR (Kalnay et al., 1996) ERA20C (Poli et al., 2016) OBS
20CR
NCEP
ERA20C
CMCC
FGOALS
HadGEM2−CC
IPSL
MIROC5
MPI
1000 ●
●
SLP (hPa)
●
980
● ●
960
● ● ● ●
●
Observations
Specifying observation uncertainty
Verifying observation uncertainty
GCM
5 / 16
Example case study Probabilistic model
Hypotheses Probabilistic model: SLP ∼ GEV (µ, σ, ξ) Error covariance matrix derived from members 1-55 of 20CR
6 / 16
Example case study Probabilistic model
Hypotheses Probabilistic model: SLP ∼ GEV (µ, σ, ξ) Error covariance matrix derived from members 1-55 of 20CR 0.5
● OBS ●
900
●
0.4
ERA20C CMCC ● ● ● ● ● ● ● ● ● ● ● ●
FGOALS
●
HadGEM2−CC
Shape ξ
Scale σ
NCEP
800
● OBS
20CR
NCEP ERA20C
●
● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●
0.3
CMCC FGOALS
0.2
HadGEM2−CC
IPSL
700
MIROC5
IPSL 0.1
MIROC5
MPI 96000
96500
97000
Location µ
97500
20CR
MPI 700
800
Scale σ
900
6 / 16
Methodology Defining GCM correctness
H (k) hypothesis : GCM k simulates the true climate Weather from GCM k is a realisation from the distribution that generated the true weather (0) (k) z 1:T ∼ D Y˜
7 / 16
Methodology Defining GCM correctness
H (k) hypothesis : GCM k simulates the true climate Weather from GCM k is a realisation from the distribution that generated the true weather (0) (k) z 1:T ∼ D Y˜
H (0) hypothesis: No GCM simulates the true climate (0) ∀k, Y (k) 6= Y˜
7 / 16
Methodology Deriving hypotheses weights Binding together observed and simulated weather series z = z (0) , z (1) , . . . , z (N) Binding together observed and simulated parameters (0) Y = Y˜ , , Y (1) , . . . , Y (N) Computing maximum likelihood of weather data under H (k) Y |zz ) Lˆ(k) (Y Computing associated Information Criteria (AIC, BIC, ..) BIC (k) = −2 · log Lˆ(k) + 2 · p (k) Deriving weights associated to each hypothesis (Buckland et al., 1997) wk =
(k) exp − BIC2 N P (i) exp − BIC2
i=0
8 / 16
Example case study Results
With 20CR observational uncertainty 1.00
0.75 H0 (NONE)
weight
H1 (CMCC) H2 (FGOALS) 0.50
H3 (HadGEM2−CC) H4 (IPSL) H5 (MIROC5)
0.25
H6 (MPI)
0.00
9 / 16
Example case study Results
With 20CR observational uncertainty
With 20CR observational uncertainty inflated
1.00
1.00
0.75
0.75
H0 (NONE) H0 (NONE)
H2 (FGOALS) 0.50
H3 (HadGEM2−CC)
H1 (CMCC)
weight
weight
H1 (CMCC)
H2 (FGOALS) 0.50
H3 (HadGEM2−CC) H4 (IPSL)
H4 (IPSL) H5 (MIROC5) 0.25
H5 (MIROC5) 0.25
H6 (MPI)
H6 (MPI)
0.00 0.00
0
25
50
75
100
InflationFactor
9 / 16
Full-scale application Objectives
Generic objective for impact models Assess GCM performance on simulating: Predictors for Statistical Downscaling Methods (SDMs) Over specific subregions (see e.g. Radanovics et al., 2013; Caillouet et al., 2016, for local calibration of SDMs). Objective within the EU FP7 COMPLEX project Propagating GCM weights through the impact modelling chain to derive a weighted multimodel distribution of changes in renewable energy potential (hydro, wind, solar) for different European subregions defined by Fran¸cois et al. (2016)
10 / 16
Full-scale application Data and probabilistic model
Data Daily SLP fields 1950-2004 over 12 North-Atlantic domains (1+55) members of 20CR + 23 CMIP5 GCMs First 3 PCA components of observed fields + interannual mean field, projection of all fields into this basis ⇒ 4-dimensional daily weather variable Probabilistic model (for each of the 4 components) Gaussian distribution Seasonality in the mean and standard deviation 1-day autocorrelation
11 / 16
Full-scale application Results GCM ACCESS
bcc−csm
CMCC−CMS
CMCC−CM
BNU−ESM CSIRO
GISS−p1
GISS−p2
GISS−p3
MIROC5
MPI
GFDL
HadGEM2−CC
HadGEM2−ES
CCSM4
IPSL−LR
IPSL−MR
NorESM1
CanESM2
FGOALS
CNRM
MRI−p1
MRI−p2
None
France 1f00
Weight
0f75 0f50 0f25 0f00 0
25
50
75
100
Observationruncertaintyrinflationrfactor
Longitudinal gradient in GCM performance Strong impact of observation uncertainty 12 / 16
Conclusions
Properties of the GCM performance weighting procedure Based on explicit probabilistic model describing the features that ones want GCMs to reproduce (mean state, variance, inter-variable correlation, autocorrelation, trend, etc.) Uses all available runs from individual GCMs Accounts for observation uncertainty Considers the H (0) hypothesis that no GCM reproduces the true climate features
13 / 16
Open questions Observation uncertainty How to reliably quantify the observation uncertainty from reanalyses? Inflating 20CR uncertainty is at best a placeholder Inter-reanalysis uncertainty?
14 / 16
Open questions Observation uncertainty How to reliably quantify the observation uncertainty from reanalyses? Inflating 20CR uncertainty is at best a placeholder Inter-reanalysis uncertainty? When no GCM is able to simulate the true climate... What to do when H (0) weight is close to 1?
Winter SLP in Reykjav´ıc and Lisbon (NAO poles)
14 / 16
References Buckland, S. T., Burnham, K. P., and Augustin, N. H. (1997). Model selection: An integral part of inference. Biometrics, 53(2):603–618. Caillouet, L., Vidal, J.-P., Sauquet, E., and Graff, B. (2016). Probabilistic precipitation and temperature downscaling of the Twentieth Century Reanalysis over France. Climate of the Past, 12(3):635–662. Compo, G. P., Whitaker, J. S., Sardeshmukh, P. D., Matsui, N., Allan, R. J., Yin, X., Gleason, B. E., Vose, R. S., Rutledge, G., Bessemoulin, P., Br¨ onnimann, S., Brunet, M., Crouthamel, R. I., Grant, A. N., Groisman, P. Y., Jones, P. D., Kruk, M. C., Kruger, A. C., Marshall, G. J., Maugeri, M., Mok, H. Y., Nordli, O., Ross, T. F., Trigo, R. M., Wang, X. L., Woodruff, S. D., and Worley, S. J. (2011). The Twentieth Century Reanalysis Project. Quarterly Journal of the Royal Meteorological Society, 137(654):1–28. Fran¸cois, B., Hingray, B., Raynaud, D., Borga, M., and Creutin, J.-D. (2016). Increasing climate-related-energy penetration by integrating run-of-the river hydropower to wind/solar mix. Renewable Energy, 87, Part1:686–696. Kalnay, E., Kanamitsu, M., Kistler, R., W., C., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., Zhu, Y., Chelliah, M., Ebisuzaki, W., Higgins, W., Janoviak, J., Mo, K. C., Ropelewski, C., Wang, J., Leetmaa, A., Reynolds, R., Jenne, R., and Joseph, D. (1996). The NCEP/NCAR 40-year Reanalysis Project. Bulletin of the American Meteorological Society, 77(3):437–471. Leith, N. A. and Chandler, R. E. (2010). A framework for interpreting climate model outputs. Journal of the Royal Statistical Society: Series C (Applied Statistics), 59(2):279296. Poli, P., Hersbach, H., Dee, D. P., Berrisford, P., Simmons, A. J., Vitart, F., Laloyaux, P., Tan, D. G. H., Peubey, C., Th´ epaut, J.-N., Tr´ emolet, Y., H´ olm, E. V., Bonavita, M., Isaksen, L., and Fisher, M. (2016). ERA-20C: An atmospheric reanalysis of the twentieth century. Journal of Climate, 29(11):4083–4097. Radanovics, S., Vidal, J.-P., Sauquet, E., Ben Daoud, A., and Bontron, G. (2013). Optimising predictor domains for spatially coherent precipitation downscaling. Hydrology and Earth System Sciences, 17(10):4189–4208. Rougier, J. C. (2007). Probabilistic inference for future climate using an ensemble of climate model evaluations. Climatic Change, 81(3-4):247–264. Taylor, K. E., Stouffer, R. J., and Meehl, G. A. (2012). An overview of CMIP5 and the experiment design. Bulletin of the American Meteorological Society, 93(4):485–498. 15 / 16
Thank you for your attention
Contacts Benjamin Renard
Jean-Philippe Vidal
[email protected]
[email protected]
www.irstea.fr/en/renard
www.irstea.fr/en/vidal
16 / 16
UR HHLY
9 June 2015
www.irstea.fr
1 / 16
Outline
1. Research questions 2. Methodology and example case study 3. Full-scale application 4. Conclusions 5. Open questions
2 / 16
Research questions
How well do GCMs simulate specific climate features? How to formalize this performance given the uncertainty in the observed climate features?
3 / 16
Methodology Ingredients Defining application-relevant climate features and probabilistic model Weather variable(s) series at location(s) (Leith and Chandler, 2010) z 1:T Statistics from the weather (mean, variance, trend, etc.) Explicit probabilistic model Y) z 1:T ∼ D (Y
4 / 16
Methodology Ingredients Defining application-relevant climate features and probabilistic model Weather variable(s) series at location(s) (Leith and Chandler, 2010) z 1:T Statistics from the weather (mean, variance, trend, etc.) Explicit probabilistic model Y) z 1:T ∼ D (Y Defining observation uncertainty Observed climate = true climate + observation errors Errors centred with known covariance matrix (Rougier, 2007) (0) (0) z 1:T ∼ D Y˜ + with ∼ N (00, Σ )
4 / 16
Example case study Winter minimum SLP in Reykjav´ıc 1950-2004 Renalysis data 56 20CR members (Compo et al., 2011) Observed weather: member #0
NCEP/NCAR (Kalnay et al., 1996)
GCM data 6 CMIP5 GCMs, some of them with multiple runs (Taylor et al., 2012)
ERA20C (Poli et al., 2016)
5 / 16
Example case study Winter minimum SLP in Reykjav´ıc 1950-2004 Renalysis data
GCM data 6 CMIP5 GCMs, some of them with multiple runs (Taylor et al., 2012)
56 20CR members (Compo et al., 2011) Observed weather: member #0
NCEP/NCAR (Kalnay et al., 1996) ERA20C (Poli et al., 2016) OBS
20CR
NCEP
ERA20C
CMCC
FGOALS
HadGEM2−CC
IPSL
MIROC5
MPI
1000 ●
●
SLP (hPa)
●
980
● ●
960
● ● ● ●
●
Observations
Specifying observation uncertainty
Verifying observation uncertainty
GCM
5 / 16
Example case study Probabilistic model
Hypotheses Probabilistic model: SLP ∼ GEV (µ, σ, ξ) Error covariance matrix derived from members 1-55 of 20CR
6 / 16
Example case study Probabilistic model
Hypotheses Probabilistic model: SLP ∼ GEV (µ, σ, ξ) Error covariance matrix derived from members 1-55 of 20CR 0.5
● OBS ●
900
●
0.4
ERA20C CMCC ● ● ● ● ● ● ● ● ● ● ● ●
FGOALS
●
HadGEM2−CC
Shape ξ
Scale σ
NCEP
800
● OBS
20CR
NCEP ERA20C
●
● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●
0.3
CMCC FGOALS
0.2
HadGEM2−CC
IPSL
700
MIROC5
IPSL 0.1
MIROC5
MPI 96000
96500
97000
Location µ
97500
20CR
MPI 700
800
Scale σ
900
6 / 16
Methodology Defining GCM correctness
H (k) hypothesis : GCM k simulates the true climate Weather from GCM k is a realisation from the distribution that generated the true weather (0) (k) z 1:T ∼ D Y˜
7 / 16
Methodology Defining GCM correctness
H (k) hypothesis : GCM k simulates the true climate Weather from GCM k is a realisation from the distribution that generated the true weather (0) (k) z 1:T ∼ D Y˜
H (0) hypothesis: No GCM simulates the true climate (0) ∀k, Y (k) 6= Y˜
7 / 16
Methodology Deriving hypotheses weights Binding together observed and simulated weather series z = z (0) , z (1) , . . . , z (N) Binding together observed and simulated parameters (0) Y = Y˜ , , Y (1) , . . . , Y (N) Computing maximum likelihood of weather data under H (k) Y |zz ) Lˆ(k) (Y Computing associated Information Criteria (AIC, BIC, ..) BIC (k) = −2 · log Lˆ(k) + 2 · p (k) Deriving weights associated to each hypothesis (Buckland et al., 1997) wk =
(k) exp − BIC2 N P (i) exp − BIC2
i=0
8 / 16
Example case study Results
With 20CR observational uncertainty 1.00
0.75 H0 (NONE)
weight
H1 (CMCC) H2 (FGOALS) 0.50
H3 (HadGEM2−CC) H4 (IPSL) H5 (MIROC5)
0.25
H6 (MPI)
0.00
9 / 16
Example case study Results
With 20CR observational uncertainty
With 20CR observational uncertainty inflated
1.00
1.00
0.75
0.75
H0 (NONE) H0 (NONE)
H2 (FGOALS) 0.50
H3 (HadGEM2−CC)
H1 (CMCC)
weight
weight
H1 (CMCC)
H2 (FGOALS) 0.50
H3 (HadGEM2−CC) H4 (IPSL)
H4 (IPSL) H5 (MIROC5) 0.25
H5 (MIROC5) 0.25
H6 (MPI)
H6 (MPI)
0.00 0.00
0
25
50
75
100
InflationFactor
9 / 16
Full-scale application Objectives
Generic objective for impact models Assess GCM performance on simulating: Predictors for Statistical Downscaling Methods (SDMs) Over specific subregions (see e.g. Radanovics et al., 2013; Caillouet et al., 2016, for local calibration of SDMs). Objective within the EU FP7 COMPLEX project Propagating GCM weights through the impact modelling chain to derive a weighted multimodel distribution of changes in renewable energy potential (hydro, wind, solar) for different European subregions defined by Fran¸cois et al. (2016)
10 / 16
Full-scale application Data and probabilistic model
Data Daily SLP fields 1950-2004 over 12 North-Atlantic domains (1+55) members of 20CR + 23 CMIP5 GCMs First 3 PCA components of observed fields + interannual mean field, projection of all fields into this basis ⇒ 4-dimensional daily weather variable Probabilistic model (for each of the 4 components) Gaussian distribution Seasonality in the mean and standard deviation 1-day autocorrelation
11 / 16
Full-scale application Results GCM ACCESS
bcc−csm
CMCC−CMS
CMCC−CM
BNU−ESM CSIRO
GISS−p1
GISS−p2
GISS−p3
MIROC5
MPI
GFDL
HadGEM2−CC
HadGEM2−ES
CCSM4
IPSL−LR
IPSL−MR
NorESM1
CanESM2
FGOALS
CNRM
MRI−p1
MRI−p2
None
France 1f00
Weight
0f75 0f50 0f25 0f00 0
25
50
75
100
Observationruncertaintyrinflationrfactor
Longitudinal gradient in GCM performance Strong impact of observation uncertainty 12 / 16
Conclusions
Properties of the GCM performance weighting procedure Based on explicit probabilistic model describing the features that ones want GCMs to reproduce (mean state, variance, inter-variable correlation, autocorrelation, trend, etc.) Uses all available runs from individual GCMs Accounts for observation uncertainty Considers the H (0) hypothesis that no GCM reproduces the true climate features
13 / 16
Open questions Observation uncertainty How to reliably quantify the observation uncertainty from reanalyses? Inflating 20CR uncertainty is at best a placeholder Inter-reanalysis uncertainty?
14 / 16
Open questions Observation uncertainty How to reliably quantify the observation uncertainty from reanalyses? Inflating 20CR uncertainty is at best a placeholder Inter-reanalysis uncertainty? When no GCM is able to simulate the true climate... What to do when H (0) weight is close to 1?
Winter SLP in Reykjav´ıc and Lisbon (NAO poles)
14 / 16
References Buckland, S. T., Burnham, K. P., and Augustin, N. H. (1997). Model selection: An integral part of inference. Biometrics, 53(2):603–618. Caillouet, L., Vidal, J.-P., Sauquet, E., and Graff, B. (2016). Probabilistic precipitation and temperature downscaling of the Twentieth Century Reanalysis over France. Climate of the Past, 12(3):635–662. Compo, G. P., Whitaker, J. S., Sardeshmukh, P. D., Matsui, N., Allan, R. J., Yin, X., Gleason, B. E., Vose, R. S., Rutledge, G., Bessemoulin, P., Br¨ onnimann, S., Brunet, M., Crouthamel, R. I., Grant, A. N., Groisman, P. Y., Jones, P. D., Kruk, M. C., Kruger, A. C., Marshall, G. J., Maugeri, M., Mok, H. Y., Nordli, O., Ross, T. F., Trigo, R. M., Wang, X. L., Woodruff, S. D., and Worley, S. J. (2011). The Twentieth Century Reanalysis Project. Quarterly Journal of the Royal Meteorological Society, 137(654):1–28. Fran¸cois, B., Hingray, B., Raynaud, D., Borga, M., and Creutin, J.-D. (2016). Increasing climate-related-energy penetration by integrating run-of-the river hydropower to wind/solar mix. Renewable Energy, 87, Part1:686–696. Kalnay, E., Kanamitsu, M., Kistler, R., W., C., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., Zhu, Y., Chelliah, M., Ebisuzaki, W., Higgins, W., Janoviak, J., Mo, K. C., Ropelewski, C., Wang, J., Leetmaa, A., Reynolds, R., Jenne, R., and Joseph, D. (1996). The NCEP/NCAR 40-year Reanalysis Project. Bulletin of the American Meteorological Society, 77(3):437–471. Leith, N. A. and Chandler, R. E. (2010). A framework for interpreting climate model outputs. Journal of the Royal Statistical Society: Series C (Applied Statistics), 59(2):279296. Poli, P., Hersbach, H., Dee, D. P., Berrisford, P., Simmons, A. J., Vitart, F., Laloyaux, P., Tan, D. G. H., Peubey, C., Th´ epaut, J.-N., Tr´ emolet, Y., H´ olm, E. V., Bonavita, M., Isaksen, L., and Fisher, M. (2016). ERA-20C: An atmospheric reanalysis of the twentieth century. Journal of Climate, 29(11):4083–4097. Radanovics, S., Vidal, J.-P., Sauquet, E., Ben Daoud, A., and Bontron, G. (2013). Optimising predictor domains for spatially coherent precipitation downscaling. Hydrology and Earth System Sciences, 17(10):4189–4208. Rougier, J. C. (2007). Probabilistic inference for future climate using an ensemble of climate model evaluations. Climatic Change, 81(3-4):247–264. Taylor, K. E., Stouffer, R. J., and Meehl, G. A. (2012). An overview of CMIP5 and the experiment design. Bulletin of the American Meteorological Society, 93(4):485–498. 15 / 16
Thank you for your attention
Contacts Benjamin Renard
Jean-Philippe Vidal
[email protected]
[email protected]
www.irstea.fr/en/renard
www.irstea.fr/en/vidal
16 / 16
