Two models on the same dates
Question
How would you assess whether one forecasting model improves on another when both predict the same return observations?
Short answer
Compare paired loss differences on common evaluation observations rather than treating model errors as independent samples. Examine their mean, economic magnitude and serial dependence; use an appropriate dependence-aware uncertainty estimate. Account for overlapping horizons and whether either model was selected using the same sample. Also inspect performance by period and asset group so a single mean does not hide concentrated failures.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Finance-aware validation
- Cross-Validation and Information Criteria
- Sample Spaces, Events, and the Probability Axioms
Start with a free lesson: Probability Fundamentals is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free