Overview

Topic 3 relaxed linearity in controlled, smooth ways — splines, local regression, GAMs. This topic goes further: it abandons a fixed functional form altogether. $k$-nearest neighbours predicts from raw proximity in feature space with no fitted parameters at all; decision trees partition feature space with a sequence of if/then splits; and ensembles (bagging, random forests, boosting) combine many trees to fix a single tree's chief weakness — high variance — producing the workhorse models of tabular prediction, including the tree ensembles that dominate quant tabular alpha research.