Skip to content
Probability, Statistics and Financial Data / Point Estimation: MOM, MLE, and Optimality Theory

The EM Algorithm

Every MLE construction in the previous lesson assumed the log-likelihood could be written down and differentiated directly. That assumption fails the moment a model has a latent component the data never reveal — a regime-switching return model where the market's volatility state is never labelled, a credit portfolio where a defaulted loan's true recovery driver is unobserved, or the missing-observation problem where a sensor simply failed to record a reading. In each case the log-likelihood a desk actually wants to maximise involves an intractable sum or integral over the unseen part, and the score equation of the last lesson offers no way in.
The EM (Expectation-Maximization) algorithm sidesteps this by trading one hard maximisation for an iterated sequence of easy ones. This note states the algorithm precisely, derives its two steps from a single likelihood identity, proves the headline guarantee — every EM iteration can only increase the observed-data likelihood, never decrease it — and works the two-component Gaussian mixture in full, the canonical model for a regime-switching return series.
Learning objectives. By the end of this note you should be able to:
  • distinguish the complete-data and incomplete-data (observed-data) likelihoods and state the identity linking their logarithms;
  • define the E-step quantity Q(θ∣θ(t))=E[log⁡Lc(θ;X,Z)∣x,θ(t)]Q(\theta\mid\theta^{(t)}) = \mathbb E\big[\log L_c(\theta;X,Z)\mid x,\theta^{(t)}\big] and the M-step update θ(t+1)=arg⁡max⁡θQ(θ∣θ(t))\theta^{(t+1)}=\arg\max_\theta Q(\theta\mid\theta^{(t)});
  • prove that the observed-data log-likelihood is non-decreasing across EM iterations;
  • derive the E-step and M-step updates for a two-component Gaussian mixture model, including the responsibility weights.

In this lesson

  1. Complete data, incomplete data, and the likelihood identity
  2. The algorithm
  3. Monotonicity
  4. Worked example: a two-component Gaussian mixture

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Probability, Statistics and Financial Data syllabus.