Skip to content

Out-of-bag error for time series

Question

Why is a random forest's ordinary out-of-bag score not automatically an estimate of future forecasting performance?

Short answer

Bootstrap omission of a row does not prevent temporally adjacent, overlapping or contemporaneous observations from entering that tree's training set. The procedure can therefore exploit information unavailable in a forward forecast. Use chronological outer evaluation matching feature and label availability. Out-of-bag estimates remain useful diagnostics under suitable sampling assumptions, but they answer a different question from a strict future-period test.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question