Dropout when making predictions
Question
Why must ordinary dropout behaviour differ between training and deterministic evaluation?
Short answer
Training dropout randomly removes activations to regularise the fit. Under inverted dropout, retained activations are rescaled during training so evaluation can use the full network without random dropping. Set evaluation mode before deterministic scoring. Deliberately retaining dropout for approximate uncertainty is a separate protocol requiring repeated draws and validation; it should not happen accidentally during a backtest.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Neural Networks: Architecture and Non-Linear Models
- Training Deep Networks: Initialization, Batch Normalization, Dropout, and Adam
- Optimisation Problems
Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free