Overfitting

When a model memorizes its training examples instead of learning the pattern behind them, so it aces the practice questions and flunks anything new.

A 70s flat-color illustration: a huge brass key bristling with dozens of jagged teeth, laid across a row of ordinary doors it was never going to open.

Overfitting happens when a model fits its training data too well: it learns the specific examples, quirks and noise included, instead of the general pattern they share. The result looks great during training and falls apart on anything it hasn’t seen.

The standard way to catch it is to hold back some data the model never trains on and keep testing on it. That score is the validation loss . While training and validation scores improve together, the model is learning. When training keeps improving but validation starts getting worse, it has started memorizing, and it’s time to stop (or go back to an earlier checkpoint ).

It’s a constant risk in fine-tuning , where the data is small and the model is huge: a model with billions of parameters has room to memorize a few thousand examples word for word. It also shows up in benchmarks , when a model has effectively seen the test questions during training.