Track 9 · Correlation and Regression · Lesson 15 of 15
Overfitting and validation
Recognise overfitting, estimate how a model will perform on new data with a seeded train/test split or k-fold cross-validation, and choose between models on out-of-sample error.
A marketing analyst presents a sales model with R² = 0.97 and proposes to use it for next quarter's budget. Every added term raised R², so the model looks better and better on the data it was fitted to. The question that matters is different: how well will it predict weeks it has never seen?
A model learns two things from its training data: the signal (the systematic relationship) and the noise (the random quirks of this particular sample). Adding predictors, squares and interactions always reduces the training error, because each new term gives least squares more freedom to chase the noise. A model that fits the noise is overfitted: it looks excellent in-sample and predicts worse out of sample. A model too simple to capture the signal is underfitted: it does badly on both.
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Explain why training error and R² always improve as a model grows
- Split data with a fixed seed and compute out-of-sample RMSE
- Use k-fold cross-validation to estimate prediction error
- Diagnose overfitting and underfitting from training and test errors
- Choose and justify a model by its performance on held-out data