Why nonlinear activations matter
Question
What happens if every layer of a feed-forward neural network uses only an affine transformation and no nonlinear activation?
Short answer
The composition remains an affine transformation: multiplying the weight matrices and combining the bias terms gives a single equivalent layer. Extra depth does not create nonlinear decision boundaries in that setup, though parameterisation can change optimisation behaviour. Nonlinear activations allow richer functions. Constraints, bottlenecks and training procedures can still affect which affine map is learned.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Neural Networks: Architecture and Non-Linear Models
- Training Deep Networks: Initialization, Batch Normalization, Dropout, and Adam
- Optimisation Problems
Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free