Overview

The perceptron found a separating line between two classes; it never asked whether that line was the best one. This topic answers that question rigorously: the support vector machine (SVM) finds the separating hyperplane with maximum margin — the largest possible gap to the nearest training points — and, via Lagrange duality, expresses the solution purely in terms of dot products between training points. That single algebraic fact is the hinge on which the kernel trick turns: replace the dot product with a more general similarity function, and a linear classifier becomes a highly non-linear one without ever explicitly constructing the (possibly infinite-dimensional) feature space it implicitly operates in.