Skip to content
Probability, Statistics and Financial Data / Linear Regression and Analysis of Variance

Overview — Linear Regression and Analysis of Variance

Every topic in this unit has been building toward a single practical question a quant desk asks constantly: does this variable explain that one, and how sure can we be? Point estimation gave the estimators, interval estimation gave the uncertainty bands, and Neyman-Pearson theory through the generalized likelihood ratio test gave the general machinery for comparing nested models. Linear regression and analysis of variance is where all three converge on the single most heavily used applied tool in quantitative finance: fitting a linear model to data and testing whether it earns its keep.
The topic runs in one direction, building a single unified picture rather than a list of unrelated techniques. It starts with the simplest possible case — one predictor, one response — and derives the ordinary least squares (OLS) estimator two ways: by calculus (minimising a sum of squares) and by geometry (orthogonal projection onto a subspace), the second route inherited directly from the QR/least-squares machinery of linear algebra. Every later result — the exact sampling distributions of the coefficients, the FF-test, R2R^2, the matrix generalisation to many predictors, and analysis of variance itself — turns out to be the same right-angle geometry, read in progressively more general settings. ANOVA in particular is revealed not as a separate technique but as regression with a categorical predictor recoded into indicator columns: one-way ANOVA compares group means, two-way ANOVA adds a second grouping factor and the question of whether the two factors interact, and the topic closes by turning the diagnostic lens on every model built along the way — checking, rather than assuming, that the machinery's assumptions actually hold.

By the end of this topic you should be able to:

  • derive the OLS estimators for simple linear regression both by calculus and by orthogonal projection, and state the residual identities that follow from each view;
  • construct exact confidence intervals and hypothesis tests for the slope and intercept, and distinguish a confidence interval for the mean response from a prediction interval for a new observation;
  • decompose total variation into explained and unexplained sums of squares, construct the corresponding FF-test, and define R2R^2 and adjusted R2R^2;
  • generalise every result to multiple predictors via the matrix formulation β^=(X⊤X)−1X⊤y\hat{\boldsymbol\beta}=(X^\top X)^{-1}X^\top\mathbf y, construct global and partial FF-tests, and diagnose multicollinearity;
  • construct and interpret one-way and two-way analysis of variance, including the Tukey and Bonferroni post-hoc procedures and the test for a factor interaction;
  • check a fitted model's assumptions using residual plots, Cook's distance, the Durbin-Watson statistic, and a QQ-plot of residuals.

In this lesson

  1. Contents
  2. The big picture

Before you start

The full lesson is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Probability, Statistics and Financial Data syllabus.