Training and evaluation modes for normalisation
Question
Why can a model with batch normalisation give different predictions for the same observation under training and evaluation modes?
Short answer
Training typically uses current minibatch statistics and updates running estimates; evaluation typically uses the stored running statistics. Predictions in training mode can therefore depend on the other rows in the batch. In a financial backtest, mixing future rows into training-mode prediction batches can also violate the intended information set. Freeze the model and use the intended evaluation behaviour, verifying batch-size sensitivity and stored-state provenance.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Neural Networks: Architecture and Non-Linear Models
- Training Deep Networks: Initialization, Batch Normalization, Dropout, and Adam
- Optimisation Problems
Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free