IMSC Canmore Pretis
Testing Climate Models for Time-Varying Forecast Accuracy Using Indicator Saturation Felix Pretis Climate Econometrics & Department of Economics, University of Oxford Jun. 6th 2016 | IMSC Canmore
1/18
Climate Econometrics
0.0 −0.2 −0.4 −0.6
Temp., C
0.2
0.4
0.6
Global Mean Surface Temp. Anomaly
1950
1960
1970
1980
1990
2000
2010
2/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Forecast Error
3/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Shifts in Forecast Error
4/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Uncertainty in Magnitude of Shifts
5/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Uncertainty in Timing of Shifts
6/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
−0.2
0.0
0.2
0.4
1950
0.1 −0.1
No Forecast bias
Positive Forecast bias
Negative Forecast bias
−0.3
Forecast Bias, C
0.3
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Testing for Forecast Bias
1950
1960
1970
1980
1990
2000
2010
7/18
Forecasts from Competing Models
Climate Econometrics
0.0 −0.2 −0.4 −0.6
Temp., C
0.2
0.4
0.6
Time-Varying Diebold and Mariano (1995)
1950
1960
1970
1980
1990
2000
2010
8/18
Climate Econometrics
Testing Relative Forecast Performance
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.3
−0.1
0.1
0.3
1950
Improved Forecast Performance
−0.1
0.1
Improved Forecast Performance
Identical Forecast Performance
Identical Forecast Performance
−0.3
Forecast Error Loss Differential Forecast Error Loss Differential
−0.6
Temp., C
0.6
Forecast Err. Loss Differential: |Tt − TbRed,t | − |Tt − TbBlue,t |
1950
1960
1970
1980
1990
2000
2010
9/18
Climate Econometrics
Detecting Shifts using SIS
Model forecast error et using step-indicator saturation step-functions: 1 0 1 1 S1 = 1{t≥j} , j = 1, ..., T =⇒ D = 1 1 1 1
(SIS), full set of 0 0 0 1
0 0 1 1
2
1
0 0
1
2
3
4
5
General: et =
6
T X
7
αj 1{t≥j} j=1 K X
Specific: eˆt =
8
9
10
11
+ t
α ˆ j 1{t≥j}
j=1
10/18
Climate Econometrics
Illustrating ‘split-half’ SIS for a single location shift
Add half indicators and select ones significant at pα e.g. 1%. Indicators included initially 1.0
Indicators retained
Selected model: actual and fitted 15
Block 1
1.0
actual
fitted
10
0.5
0.5
5 0
0
50
100
0
50
100
0
50
100
11/18
Climate Econometrics
Illustrating ‘split-half’ SIS for a single location shift
Drop, add other half indicators and again select at 1%. Indicators included initially
Block 1
1.0
Indicators retained
Selected model: actual and fitted 15
1.0
actual
fitted
10 0.5
0.5
5 0
0
50
100
0
50
100
1.0
Block 2
1.0
0 15
50
100
50
100
10 0.5
0.5
5 0
0
50
100
0
50
100
0
11/18
Climate Econometrics
Illustrating ‘split-half’ SIS for a single location shift
Combine retained indicators and re-select at 1%. Indicators included initially 1.0
Indicators retained
Selected model: actual and fitted 15
1.0
actual
fitted
Block 1
10
0.5
0.5
5 0
0
50
100
0
50
100
1.0
1.0
0 15
50
100
0
50
100
0
50
100
Block 2
10 0.5
0.5
5 0
0
50
0
100
50
100 15
1.0
1.0
Final
10 0.5
0.5
5 0
0
50
100
0
50
100
Matching theory: initially retains last step indicator closest to mean shift, then finds correct shift, so eliminates redundant indicator. Just one step indicator needed. Under H0 : pα T
11/18
Climate Econometrics
Break Magnitude Uncertainty
eˆt =
K X
α ˆ j 1{t≥j}
0.2 0.0
^=α ^1 µ ^=α ^1 + α ^2 µ
−0.2
Temp. Model Forecast Error, C
0.4
j=1
^=α ^1 + α ^2 + α ^3 µ 1950
1960
1970
1980
1990
2000
2010
12/18
Climate Econometrics
Break Magnitude Uncertainty
eˆt =
K X
α ˆ j 1{t≥j}
0.2 0.0
^=α ^1 µ ^=α ^1 + α ^2 µ
−0.2
Temp. Model Forecast Error, C
0.4
j=1
^=α ^1 + α ^2 + α ^3 µ 1950
1960
1970
1980
1990
2000
2010
13/18
Climate Econometrics
0.8 0.6 0.4 0.2 0.0
Break Date Probability
0.2 0.0 −0.2
Temp. Model Forecast Error, C
0.4
1.0
Break Date Uncertainty, ˆ ∼ N(0, σ 2 )
1950
1960
1970
1980
1990
2000
2010
14/18
Climate Econometrics
0.8 0.6 0.4 0.2 0.0
Break Date Probability
0.2 0.0 −0.2
Temp. Model Forecast Error, C
0.4
1.0
Break Date Uncertainty, ˆ ∼ N(0, σ 2 )
1950
1960
1970
1980
1990
2000
2010
15/18
Climate Econometrics
Implementation
R-package ‘gets’ (Pretis, Reade & Sucarrat, 2016): isat: indicator saturation to detect shifts. isat(time series, t.pval=0.01, sis=TRUE)
isattest: testing on the coefficient path fitted
-0.3 0.0
0.3
y
1950
1960
1970
1980
1990
2000
2010
1970
1980
1990
2000
2010
1970
1980
1990
2000
2010
-2
0
2
standardised residuals
1950
1960
-0.4
-0.1
y: Coefficient Path
1950
1960
16/18
Conclusion
Climate Econometrics
Features Outliers jointly with step-shifts No min. break length – no maximum number of shifts
17/18
Climate Econometrics
Conclusion
Features Outliers jointly with step-shifts No min. break length – no maximum number of shifts Further Possibilities: Dynamic models of forecast error (autocorrelation) Autoregressive: isat(..., ar=1:2)
Add covariates (variables affecting forecasts) Covariates: isat(..., mxreg=Covariate)
17/18
Climate Econometrics
Conclusion
Features Outliers jointly with step-shifts No min. break length – no maximum number of shifts Further Possibilities: Dynamic models of forecast error (autocorrelation) Autoregressive: isat(..., ar=1:2)
Add covariates (variables affecting forecasts) Covariates: isat(..., mxreg=Covariate)
Beyond forecast evaluation: Homogenization (measurement corrections) Model Robustness Test ... try it out: ‘gets’ (in R)
17/18
Call for Papers – Conference: Econometric Models of Climate Change October 27-28th, CREATES, Aarhus, Denmark Submit: climateeconometrics.org/conference
18/18
References 1 2
3 4
5
6
7
8
Climate Econometrics
Diebold, F. X., and R. S. Mariano (1995) “Comparing Predictive Accuracy”, Journal of Business and Economic Statistics, 13, 3, 253263. Castle, J.L, Doornik, J.A, Hendry, D.F, and Pretis, F. (2015a) “Detecting Location Shifts during Model Selection using Step-Indicator Saturation” Econometrics, 3, 240-264. Ericsson, N.R. (2015) “How biased are US government forecasts of the federal debt?”, International Journal of Forecasting (in press) Hendry, D.F. and Pretis, F. (2016) “Quantifying the Uncertainty around Break Dates in Models using Indicator Saturation”. Oxford Department of Economics Working paper. Mincer, J., and V. Zarnowitz (1969) “The Evaluation of Economic Forecasts” in J. Mincer (ed.) Economic Forecasts and Expectations: Analyes of Forecasting Behavior and Performance, National Bureau of Economic Research, New York, 346. Pretis, F. Mann, M.L. and Kaufmann, R.K. (2015) “Testing competing models of the temperature hiatus: assessing the effects of conditioning variables and temporal uncertainties through sample-wide break detection”, Climatic Change, doi:10.1007/s10584-015-1391-5 Pretis, F. Reade, J. and Sucarrat, G. (2016) “General-to-Specific (GETS) Modelling and Indicator Saturation with the R Package gets”. Oxford Department of Economics Discussion Paper 794. Pretis, F. (2016) “Classifying Time-Varying Predictive Accuracy in Climate Econometrics Using Bias-Corrected Indicator Saturation”. Oxford Department of Economics Working paper. Under review.
19/18
1/18
Climate Econometrics
0.0 −0.2 −0.4 −0.6
Temp., C
0.2
0.4
0.6
Global Mean Surface Temp. Anomaly
1950
1960
1970
1980
1990
2000
2010
2/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Forecast Error
3/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Shifts in Forecast Error
4/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Uncertainty in Magnitude of Shifts
5/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.0
0.2
0.4
1950
−0.2
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Uncertainty in Timing of Shifts
6/18
Climate Econometrics
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
−0.2
0.0
0.2
0.4
1950
0.1 −0.1
No Forecast bias
Positive Forecast bias
Negative Forecast bias
−0.3
Forecast Bias, C
0.3
Temp. Model Forecast Error, C
−0.6
Temp., C
0.6
Testing for Forecast Bias
1950
1960
1970
1980
1990
2000
2010
7/18
Forecasts from Competing Models
Climate Econometrics
0.0 −0.2 −0.4 −0.6
Temp., C
0.2
0.4
0.6
Time-Varying Diebold and Mariano (1995)
1950
1960
1970
1980
1990
2000
2010
8/18
Climate Econometrics
Testing Relative Forecast Performance
0.2 −0.2
1960
1970
1980
1990
2000
2010
1950
1960
1970
1980
1990
2000
2010
0.3
−0.1
0.1
0.3
1950
Improved Forecast Performance
−0.1
0.1
Improved Forecast Performance
Identical Forecast Performance
Identical Forecast Performance
−0.3
Forecast Error Loss Differential Forecast Error Loss Differential
−0.6
Temp., C
0.6
Forecast Err. Loss Differential: |Tt − TbRed,t | − |Tt − TbBlue,t |
1950
1960
1970
1980
1990
2000
2010
9/18
Climate Econometrics
Detecting Shifts using SIS
Model forecast error et using step-indicator saturation step-functions: 1 0 1 1 S1 = 1{t≥j} , j = 1, ..., T =⇒ D = 1 1 1 1
(SIS), full set of 0 0 0 1
0 0 1 1
2
1
0 0
1
2
3
4
5
General: et =
6
T X
7
αj 1{t≥j} j=1 K X
Specific: eˆt =
8
9
10
11
+ t
α ˆ j 1{t≥j}
j=1
10/18
Climate Econometrics
Illustrating ‘split-half’ SIS for a single location shift
Add half indicators and select ones significant at pα e.g. 1%. Indicators included initially 1.0
Indicators retained
Selected model: actual and fitted 15
Block 1
1.0
actual
fitted
10
0.5
0.5
5 0
0
50
100
0
50
100
0
50
100
11/18
Climate Econometrics
Illustrating ‘split-half’ SIS for a single location shift
Drop, add other half indicators and again select at 1%. Indicators included initially
Block 1
1.0
Indicators retained
Selected model: actual and fitted 15
1.0
actual
fitted
10 0.5
0.5
5 0
0
50
100
0
50
100
1.0
Block 2
1.0
0 15
50
100
50
100
10 0.5
0.5
5 0
0
50
100
0
50
100
0
11/18
Climate Econometrics
Illustrating ‘split-half’ SIS for a single location shift
Combine retained indicators and re-select at 1%. Indicators included initially 1.0
Indicators retained
Selected model: actual and fitted 15
1.0
actual
fitted
Block 1
10
0.5
0.5
5 0
0
50
100
0
50
100
1.0
1.0
0 15
50
100
0
50
100
0
50
100
Block 2
10 0.5
0.5
5 0
0
50
0
100
50
100 15
1.0
1.0
Final
10 0.5
0.5
5 0
0
50
100
0
50
100
Matching theory: initially retains last step indicator closest to mean shift, then finds correct shift, so eliminates redundant indicator. Just one step indicator needed. Under H0 : pα T
11/18
Climate Econometrics
Break Magnitude Uncertainty
eˆt =
K X
α ˆ j 1{t≥j}
0.2 0.0
^=α ^1 µ ^=α ^1 + α ^2 µ
−0.2
Temp. Model Forecast Error, C
0.4
j=1
^=α ^1 + α ^2 + α ^3 µ 1950
1960
1970
1980
1990
2000
2010
12/18
Climate Econometrics
Break Magnitude Uncertainty
eˆt =
K X
α ˆ j 1{t≥j}
0.2 0.0
^=α ^1 µ ^=α ^1 + α ^2 µ
−0.2
Temp. Model Forecast Error, C
0.4
j=1
^=α ^1 + α ^2 + α ^3 µ 1950
1960
1970
1980
1990
2000
2010
13/18
Climate Econometrics
0.8 0.6 0.4 0.2 0.0
Break Date Probability
0.2 0.0 −0.2
Temp. Model Forecast Error, C
0.4
1.0
Break Date Uncertainty, ˆ ∼ N(0, σ 2 )
1950
1960
1970
1980
1990
2000
2010
14/18
Climate Econometrics
0.8 0.6 0.4 0.2 0.0
Break Date Probability
0.2 0.0 −0.2
Temp. Model Forecast Error, C
0.4
1.0
Break Date Uncertainty, ˆ ∼ N(0, σ 2 )
1950
1960
1970
1980
1990
2000
2010
15/18
Climate Econometrics
Implementation
R-package ‘gets’ (Pretis, Reade & Sucarrat, 2016): isat: indicator saturation to detect shifts. isat(time series, t.pval=0.01, sis=TRUE)
isattest: testing on the coefficient path fitted
-0.3 0.0
0.3
y
1950
1960
1970
1980
1990
2000
2010
1970
1980
1990
2000
2010
1970
1980
1990
2000
2010
-2
0
2
standardised residuals
1950
1960
-0.4
-0.1
y: Coefficient Path
1950
1960
16/18
Conclusion
Climate Econometrics
Features Outliers jointly with step-shifts No min. break length – no maximum number of shifts
17/18
Climate Econometrics
Conclusion
Features Outliers jointly with step-shifts No min. break length – no maximum number of shifts Further Possibilities: Dynamic models of forecast error (autocorrelation) Autoregressive: isat(..., ar=1:2)
Add covariates (variables affecting forecasts) Covariates: isat(..., mxreg=Covariate)
17/18
Climate Econometrics
Conclusion
Features Outliers jointly with step-shifts No min. break length – no maximum number of shifts Further Possibilities: Dynamic models of forecast error (autocorrelation) Autoregressive: isat(..., ar=1:2)
Add covariates (variables affecting forecasts) Covariates: isat(..., mxreg=Covariate)
Beyond forecast evaluation: Homogenization (measurement corrections) Model Robustness Test ... try it out: ‘gets’ (in R)
17/18
Call for Papers – Conference: Econometric Models of Climate Change October 27-28th, CREATES, Aarhus, Denmark Submit: climateeconometrics.org/conference
18/18
References 1 2
3 4
5
6
7
8
Climate Econometrics
Diebold, F. X., and R. S. Mariano (1995) “Comparing Predictive Accuracy”, Journal of Business and Economic Statistics, 13, 3, 253263. Castle, J.L, Doornik, J.A, Hendry, D.F, and Pretis, F. (2015a) “Detecting Location Shifts during Model Selection using Step-Indicator Saturation” Econometrics, 3, 240-264. Ericsson, N.R. (2015) “How biased are US government forecasts of the federal debt?”, International Journal of Forecasting (in press) Hendry, D.F. and Pretis, F. (2016) “Quantifying the Uncertainty around Break Dates in Models using Indicator Saturation”. Oxford Department of Economics Working paper. Mincer, J., and V. Zarnowitz (1969) “The Evaluation of Economic Forecasts” in J. Mincer (ed.) Economic Forecasts and Expectations: Analyes of Forecasting Behavior and Performance, National Bureau of Economic Research, New York, 346. Pretis, F. Mann, M.L. and Kaufmann, R.K. (2015) “Testing competing models of the temperature hiatus: assessing the effects of conditioning variables and temporal uncertainties through sample-wide break detection”, Climatic Change, doi:10.1007/s10584-015-1391-5 Pretis, F. Reade, J. and Sucarrat, G. (2016) “General-to-Specific (GETS) Modelling and Indicator Saturation with the R Package gets”. Oxford Department of Economics Discussion Paper 794. Pretis, F. (2016) “Classifying Time-Varying Predictive Accuracy in Climate Econometrics Using Bias-Corrected Indicator Saturation”. Oxford Department of Economics Working paper. Under review.
19/18
