Track 12 · Time Series and Forecasting · Lesson 6 of 11
Measuring forecast error
Calculate and interpret ME, MAE, MSE, RMSE, MAPE and MASE on held-out data, and choose a measure that fits the decision and the data.
Two analysts each claim their forecast is “90% accurate”. One computed MAPE on the data the model was fitted to; the other computed something else on next year's sales. Without agreeing how error is measured — and on which data — accuracy claims cannot be compared.
A forecast error is the actual value minus the forecast, , measured on periods the forecast did not use (a holdout or test period). A positive error means the forecast was too low. Errors from fitting the model (residuals) are almost always smaller than genuine forecast errors and say little about future accuracy.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Compute ME, MAE, MSE, RMSE, MAPE and MASE from actuals and forecasts
- Explain why errors must be measured on data the forecast did not use
- Explain why MAPE fails at or near zero and penalises over-forecasts more
- Interpret MASE relative to its in-sample naive yardstick
- Choose an error measure that fits the data and the cost of errors