Bagging and Random Forests

The previous lesson confronted a single decision tree's high variance directly by pruning it — trading away flexibility for stability. This note takes the opposite tack: keep the trees large and unpruned (low bias, high variance), and cancel out the variance by averaging many of them. This is the idea behind bagging and its refinement, the random forest — the single most widely deployed model family in tabular alpha research, and the direct analogue of every earlier lesson's bias–variance trade-off, now resolved by ensembling rather than by shrinkage or pruning.