Forecast accuracy: MAE, RMSE, MAPE and MASE

Summaries of forecast errors on data the model did not see: MAE and RMSE in the series' units, MAPE in percent, and MASE scaled by the in-sample naive method.

\text{MAE} = \frac{1}{n}\sum |e_t|, \quad \text{RMSE} = \sqrt{\frac{1}{n}\sum e_t^2}, \quad \text{MAPE} = \frac{100}{n}\sum \frac{|e_t|}{|y_t|}, \quad e_t = y_t - \hat y_t

Scale-dependent and percentage errors

$y_t, \hat y_t$: actual value and forecast for test period $t$
$n$: number of forecast periods evaluated

\text{MASE} = \frac{\text{MAE}}{\frac{1}{T-m}\sum_{t=m+1}^{T} |y_t - y_{t-m}|}

Mean absolute scaled error

$T$: length of the TRAINING series
$m$: seasonal period ( $m = 1$ : scale by the naive method; $m > 1$ : by the seasonal naive method)

When to use

Comparing forecasting methods on a holdout set or with rolling-origin (time-series) cross-validation, never on the data used to fit.
MAE: typical error in units, robust and easy to explain. RMSE: penalises large errors more; targets the mean. MAPE: unit-free, but only for strictly positive data far from 0. MASE: unit-free and safe for comparisons across series.

Assumptions

Errors are computed out of sample, with no look-ahead (training data strictly before the forecast origin).
MAPE requires every actual value to be non-zero (the engine refuses zeros) and is unstable for values near 0.

How to interpret

MASE < 1: on average more accurate than the one-step naive forecast was in-sample; MASE > 1: worse.
RMSE ≥ MAE always; a big gap means a few large errors.
MAPE treats over- and under-forecasts asymmetrically and can reward forecasts that are biased low.

Common mistakes

Reporting in-sample (fitted) errors as forecast accuracy.
Averaging MAE across series with different scales.
Using MAPE for intermittent demand or series that cross zero.

Example

Training series 40, 44, 43, 47, 46; test actuals 50, 54, 49; forecasts 52, 51, 50. Errors −2, 3, −1: MAE = 2, RMSE = 2.16, MAPE = 3.87%. Naive in-sample scale = mean(4, 1, 4, 1) = 2.5, so MASE = 2/2.5 = 0.8.