A new approach for detecting shifts in forecast accuracy

TY - JOUR

T1 - A new approach for detecting shifts in forecast accuracy

AU - Chiu, Ching Wai (Jeremy)

AU - Hayes, Simon

AU - Kapetanios, George

AU - Theodoridis, Konstantinos

PY - 2019/10/1

Y1 - 2019/10/1

N2 - Forecasts play a critical role at inflation-targeting central banks, such as the Bank of England. Breaks in the forecast performance of a model can potentially incur important policy costs. However, commonly-used statistical procedures implicitly place a lot of weight on type I errors (or false positives), which results in a relatively low power of the tests to identify forecast breakdowns in small samples. We develop a procedure which aims to capture the policy cost of missing a break. We use data-based rules to find the test size that optimally trades off the costs associated with false positives with those that can result from a break going undetected for too long. In so doing, we also explicitly study forecast errors as a multivariate system. The covariance between forecast errors for different series, although often overlooked in the forecasting literature, not only enables us to consider testing in a multivariate setting, but also increases the test power. As a result, we can tailor our choice of the critical values for each series not only to the in-sample properties of each series, but also to the way in which the series of forecast errors covary.

AB - Forecasts play a critical role at inflation-targeting central banks, such as the Bank of England. Breaks in the forecast performance of a model can potentially incur important policy costs. However, commonly-used statistical procedures implicitly place a lot of weight on type I errors (or false positives), which results in a relatively low power of the tests to identify forecast breakdowns in small samples. We develop a procedure which aims to capture the policy cost of missing a break. We use data-based rules to find the test size that optimally trades off the costs associated with false positives with those that can result from a break going undetected for too long. In so doing, we also explicitly study forecast errors as a multivariate system. The covariance between forecast errors for different series, although often overlooked in the forecasting literature, not only enables us to consider testing in a multivariate setting, but also increases the test power. As a result, we can tailor our choice of the critical values for each series not only to the in-sample properties of each series, but also to the way in which the series of forecast errors covary.

KW - Central banking

KW - Forecast breaks

KW - Hypothesis testing with small sample sizes

KW - Optimal test sizes

KW - Statistical decision making

U2 - 10.1016/j.ijforecast.2019.01.008

DO - 10.1016/j.ijforecast.2019.01.008

M3 - Article

AN - SCOPUS:85066063425

SN - 0169-2070

VL - 35

SP - 1596

EP - 1612

JO - INTERNATIONAL JOURNAL OF FORECASTING

JF - INTERNATIONAL JOURNAL OF FORECASTING

IS - 4

ER -