Read these in order. Each one assumes the ones before it. About 40 minutes in total; stop after any step.
- Train, validation and test split: the three groups of data and the question each answers.
- Overfitting: why a perfect training score can mean a worse model.
- Bias-variance trade-off: where a model’s error comes from, and why the best model sits in the middle.
- Cross-validation: a steadier estimate when one validation split is too noisy.
- Data leakage: the ways information sneaks into training and makes offline scores lie.
Then go to the source: Leakage in data mining (Kaufman et al., 2012).
Then test yourself: the ML section of Practice, and the lesson #3 · Overfitting and the bias-variance trade-off.