Out-of-bag error for time series
Question
Why is a random forest's ordinary out-of-bag score not automatically an estimate of future forecasting performance?
Short answer
Bootstrap omission of a row does not prevent temporally adjacent, overlapping or contemporaneous observations from entering that tree's training set. The procedure can therefore exploit information unavailable in a forward forecast. Use chronological outer evaluation matching feature and label availability. Out-of-bag estimates remain useful diagnostics under suitable sampling assumptions, but they answer a different question from a strict future-period test.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solution