Skip to content
Linear Algebra, Optimisation and Numerical Methods / QR Factorisation and Least Squares

Derivation — The Normal Equations from Minimising ‖Ax − b‖

Method 1 of the previous note stated the normal equations, A⊤Ax=A⊤bA^\top A\mathbf{x}=A^\top\mathbf{b}, without deriving them. This sub-note gives that derivation two ways — directly from the orthogonality characterisation of Proposition 1 in the projections note, and via ordinary calculus — so a reader sees both why the equations are geometrically forced and how they'd be found by minimising directly.
Learning objectives. By the end of this note you should be able to:
  • derive the normal equations from the orthogonality of the least-squares residual;
  • derive the normal equations by differentiating the squared-error objective;
  • explain why the two derivations must agree;
  • verify the normal equations against a computed least-squares solution.

In this lesson

  1. Derivation 1: from orthogonality
  2. Derivation 2: from calculus

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Linear Algebra, Optimisation and Numerical Methods syllabus.