Neural Networks: Architecture and Non-Linear Models

Kernel methods achieve non-linearity by choosing a fixed similarity function $K$ up front and letting the algorithm work implicitly in the feature space it induces. This topic's route is different: rather than fixing a feature map, neural networks learn one, by stacking multiple layers of simple, adjustable non-linear units and fitting every layer's parameters jointly to the data. This note introduces the architecture — from the perceptron's single layer to the multilayer perceptron (MLP) — and states, at a conceptual level, why stacking non-linear layers is what makes deep networks strictly more expressive than any single linear or kernel model.