Track 12 · Time Series and Forecasting · Lesson 7 of 11
Holdouts and time-series cross-validation
Evaluate forecasting methods honestly with a final holdout and with rolling-origin cross-validation, and recognise look-ahead leakage.
A model that “explains” past sales with 3% error can still miss next quarter by 20%. Fitting is not forecasting. The only convincing evidence that a method forecasts well is how it performs on periods it had not seen when the forecast was made.
The holdout (train/test split) for time series. Keep the most recent periods aside as a test set. Fit and tune everything — method, parameters, transformations — on the earlier training set only, forecast steps from the end of the training set, and score those forecasts on the test set.
The split must respect time. A random split lets the model learn from months that come after the ones it is tested on, so it interpolates instead of forecasting and looks better than it will be in use.
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Split a series by time and keep the test period untouched until the end
- Run and count rolling-origin folds without look-ahead
- Explain why in-sample error and random k-fold cross-validation mislead
- Recognise leakage in preprocessing and model selection