Random forests and correlated trees
Question
Why does reducing correlation among trees help a random forest, even if individual trees become slightly weaker?
Short answer
The variance of an average depends on both individual variances and pairwise covariances. Averaging highly correlated errors yields limited variance reduction. Feature subsampling and bootstrap variation can reduce correlation, potentially improving the ensemble despite weaker components. The benefit is empirical and does not remove bias or distribution shift. Tune feature sampling and leaf controls using the intended evaluation procedure.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Decision Trees: Splitting Criteria and Pruning
- Boosting: AdaBoost and Gradient Boosting
- Covariance and Correlation
Start with a free lesson: Expectation, Variance and Covariance is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free