All-zero hidden-layer initialisation
Question
Why is assigning identical zero weights to every hidden unit generally a poor neural-network initialisation?
Short answer
Symmetric units can receive identical gradients and remain identical, preventing the layer from learning distinct representations. Depending on activations and architecture, zero derivatives can also block learning. Use an appropriate asymmetric variance-scaled initialisation with a recorded RNG seed. Zero biases are often acceptable; the symmetry problem concerns equivalent hidden units, not a universal ban on every zero parameter.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Neural Networks: Architecture and Non-Linear Models
- Training Deep Networks: Initialization, Batch Normalization, Dropout, and Adam
- Optimisation Problems
Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free