Track 16 · Advanced Foundations · Lesson 6 of 8
Evaluating classifiers
Evaluate a binary classifier with a confusion matrix and the right metrics, explain why accuracy misleads under class imbalance, read ROC curves and AUC, choose a threshold from the costs of errors, and avoid leakage.
A lender's default model is “87% accurate” — and so is a model that approves everyone, because 87% of borrowers repay anyway. Judging a classifier means asking which mistakes it makes, how often, and what each mistake costs.
A classifier assigns each case to one of two classes, usually by computing a score (such as a predicted probability from logistic regression) and predicting “positive” when the score is at or above a threshold. The positive class is the one we are trying to detect — default, disease, fraud — whether or not it is good news. Track 9 (Correlation and Regression) showed how to fit such a model and check its AUC on held-out data; this lesson is about judging and tuning it.
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Build and read a confusion matrix and compute accuracy, precision, recall, specificity and F1
- Compare accuracy with the majority-class baseline when classes are imbalanced
- Interpret ROC curves and AUC as ranking quality
- Choose a threshold from the costs of false negatives and false positives
- Recognise leakage in predictors and in the evaluation process