Model Validation Quant
Independently checks the models a bank or fund relies on, before they cost money.
The job
A model validation quant independently checks the models a bank or fund uses for pricing, risk limits and capital before the firm relies on them, and again whenever they change. The job is to find what is wrong before it costs money: reproduce outputs from scratch, test assumptions, backtest forecasts, review proposed changes, and write findings the model owner must act on. Things go wrong when a model passes its headline test but fails in the tail, and when a change quietly uses future data.
- Supports
- Model risk teams, and the desks whose models they review
- Stages
- 5 to learn, then proof of work
- Two routes
- Free resources, or SelfMadeQuant programmes and projects
The roadmap
The same stages either way. The free route uses university courses, lectures, books and papers that are free online. The SelfMadeQuant route maps each stage to our programmes and ends in projects on real market data.
Stage 01 · Foundations
Statistics and hypothesis testing
The tests a validator uses to accept or reject a model.
- Likelihood ratio tests
- Power and size
- Goodness of fit
- MIT 18.650 Statistics for Applications (opens in a new tab)
Philippe Rigollet / MIT OpenCourseWare
Maximum likelihood, likelihood ratio tests and goodness-of-fit tests with proofs.
- STAT 415 Introduction to Mathematical Statistics (opens in a new tab)
Penn State Eberly College of Science
Free lessons on likelihood ratio tests, power and estimation.
- NIST/SEMATECH e-Handbook: Quantitative Techniques (opens in a new tab)
NIST
Reference pages for chi-square, Kolmogorov-Smirnov and Anderson-Darling goodness-of-fit tests.
Stage 02 · Core
Risk models and their backtests
VaR, expected shortfall and the backtests regulators expect.
- VaR and expected shortfall
- GARCH
- Kupiec and Christoffersen tests
- Basic Concepts and Techniques of Risk Management (opens in a new tab)
Martin Haugh / Columbia IEOR E4602
VaR, expected shortfall and their estimation, with the main pitfalls.
- A Review of Backtesting and Backtesting Procedures (opens in a new tab)
Sean Campbell / Federal Reserve Board
Kupiec, Christoffersen and related VaR backtests explained and compared.
- Supervisory framework for the use of backtesting with the internal models approach (opens in a new tab)
Basel Committee on Banking Supervision
The traffic-light rules supervisors apply to VaR exceptions.
- Practical Volatility and Correlation Modeling for Financial Market Risk Management (opens in a new tab)
Andersen, Bollerslev, Christoffersen and Diebold
GARCH and realised volatility inside VaR models, written for risk managers.
Stage 03 · Core
Validation design and leakage
How look-ahead gets in, and how a challenger model catches it.
- Out-of-sample design
- Look-ahead
- Benchmark and challenger models
- Forecasting: Principles and Practice, time series cross-validation (opens in a new tab)
Rob Hyndman and George Athanasopoulos
Rolling-origin evaluation that keeps every test point out of sample.
- Comparing Predictive Accuracy (opens in a new tab)
Francis Diebold and Roberto Mariano / NBER
The standard test for whether a challenger model forecasts better than the incumbent.
- Leakage and the Reproducibility Crisis in ML-based Science (opens in a new tab)
Sayash Kapoor and Arvind Narayanan / arXiv
A taxonomy of data leakage to check for when reviewing someone else's model.
- A Forecast Comparison of Volatility Models: Does Anything Beat a GARCH(1,1)? (opens in a new tab)
Peter Reinhard Hansen and Asger Lunde
A model to copy for benchmarking many challengers against a simple incumbent.
Stage 04 · Specialist
Model risk governance
Documentation, change control and findings that get acted on.
- Fed model risk guidance (SR 26-2)
- Change control
- Writing findings
- SR 26-2: Revised Guidance on Model Risk Management (opens in a new tab)
Board of Governors of the Federal Reserve System
The current US supervisory guidance, which supersedes SR 11-7, on validation and model governance.
- SS1/23 Model risk management principles for banks (opens in a new tab)
Prudential Regulation Authority, Bank of England
The UK principles on model inventory, tiering, validation and documentation.
- The Turing Way (opens in a new tab)
The Turing Way Community / Alan Turing Institute
Practical guidance on reproducible code, environments and documentation.
Stage 05 · Specialist
The models you will validate
Enough of each model family to challenge it.
- Rate models
- Option pricing models
- Counterparty exposure
- Credit Modeling and Credit Derivatives (Columbia IEOR E4706) (opens in a new tab)
Martin Haugh
Default intensities, survival curves and CDS pricing, the inputs to counterparty models.
- Model Risk (opens in a new tab)
Martin Haugh / Columbia IEOR E4602
Worked examples of how derivatives pricing models fail, from a validator's viewpoint.
- The standardised approach for measuring counterparty credit risk exposures (opens in a new tab)
Basel Committee on Banking Supervision
How exposure at default is built from replacement cost and potential future exposure.
- Financial Engineering and Risk Management Specialization (opens in a new tab)
Columbia University / Coursera
Free-to-audit lectures on fixed income, term structure models and derivatives pricing.
Stage 06 · Proof of work
Challenge a RiskMetrics-style VaR model
Rebuild a RiskMetrics EWMA 99% one-day VaR on US equity and EUR/USD returns, then backtest it with Kupiec and Christoffersen tests over calm and stressed periods. Challenge it with GARCH-t and filtered historical simulation and answer whether the champion model would pass validation.
Stage 01 · Foundations
Statistics and hypothesis testing
The tests a validator uses to accept or reject a model.
- Likelihood ratio tests
- Power and size
- Goodness of fit
- Probability, Statistics and Financial Data
- Likelihood Ratio Tests, Wilks' Theorem, and Goodness of Fit
- Neyman-Pearson Theory and Uniformly Most Powerful Tests
Stage 02 · Core
Risk models and their backtests
VaR, expected shortfall and the backtests regulators expect.
- VaR and expected shortfall
- GARCH
- Kupiec and Christoffersen tests
- Quantitative Risk Management
- Loss Distributions, VaR, and Coherent Risk Measures
- Empirical Properties of Financial Data and GARCH Volatility Models
Stage 03 · Core
Validation design and leakage
How look-ahead gets in, and how a challenger model catches it.
- Out-of-sample design
- Look-ahead
- Benchmark and challenger models
- Quantitative Research, Data and Backtesting
- Labels and Validation Design
- Financial Machine Learning
- Finance-aware training and validation
Stage 04 · Specialist
Model risk governance
Documentation, change control and findings that get acted on.
- Fed model risk guidance (SR 26-2)
- Change control
- Writing findings
- Professional Quant Practice
- Reproducible model development
- Governance and validation
- Communication, ethics and defence
Stage 05 · Specialist
The models you will validate
Enough of each model family to challenge it.
- Rate models
- Option pricing models
- Counterparty exposure
- Fixed Income and Interest-Rate Models
- Short-Rate Models
- Derivatives Pricing and Computational Finance
- Foundations of Computational Option Pricing
- Counterparty Credit Risk and CVA
- CVA Computation and Validation
Stage 06 · Proof of work
Build the projects
Each project is built on real market data and reports what the data said, including when the idea failed.
- VRVolatility forecasting and risk-limit calibration (the model under review)A next-day volatility forecast and exposure-limit rule on 2.45m checksum-verified BTC minute bars.
- IVIndependent model validation and change controlReproduced VR to 10⁻¹¹; the 99% VaR was well calibrated, but breach-day losses were 18% worse than forecast expected shortfall.
Practise the interview
Questions this role gets asked, each with a hint and a worked solution.
- Two researchers run the same model code and get different backtest results. What do you compare before blaming randomness?
- A model’s ridge penalty was chosen to minimise error on next quarter’s test set, and that same error is reported. What is wrong?
- You want a confidence interval for a statistic on daily returns. Why does an ordinary bootstrap mislead?
Start the SelfMadeQuant route free: a free account opens one complete topic from each primer and the interview question bank. No card needed.
Start free