Skip to content

Distribution shift after a market event

Question

A deployed model fails after a market-structure change. How would you distinguish covariate shift, target shift and a changed predictive relationship?

Short answer

Audit pipeline integrity first. Then compare feature distributions, outcome distributions and conditional performance using labels as they mature. Feature-distribution change alone does not identify whether the conditional relationship changed. Evaluate calibration and residuals by relevant groups and periods, allowing for sampling uncertainty. Choose monitoring and retraining rules on development evidence; automatic retraining can amplify data errors or transient noise.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question