Skip to content

All-zero hidden-layer initialisation

Question

Why is assigning identical zero weights to every hidden unit generally a poor neural-network initialisation?

Short answer

Symmetric units can receive identical gradients and remain identical, preventing the layer from learning distinct representations. Depending on activations and architecture, zero derivatives can also block learning. Use an appropriate asymmetric variance-scaled initialisation with a recorded RNG seed. Zero biases are often acceptable; the symmetry problem concerns equivalent hidden units, not a universal ban on every zero parameter.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question

Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.

Read it free