Skip to content
Probability, Statistics and Financial Data / Point Estimation: MOM, MLE, and Optimality Theory

Maximum Likelihood Estimation: Score and Log-Likelihood

The previous lesson built estimators by matching a handful of moments — fast, but blind to everything else the sample says. Maximum likelihood estimation (MLE) asks the sharper question the likelihood function was built for: among every candidate θ\theta, which one makes this exact sample most probable? A desk calibrating a GARCH volatility model, a credit-scoring logistic regression, or a jump-intensity Poisson model is, underneath the software, solving exactly this maximisation — and it is the workhorse estimator the rest of this topic measures every alternative against: the Cramér-Rao bound sets a variance floor the MLE asymptotically achieves, and the asymptotic-properties lesson closing this topic proves it.

This note defines the score function and the log-likelihood, shows that the MLE is (generically) a root of the score equation, proves that the root is unique for an exponential family, states the invariance property that lets you plug an MLE straight into any function of θ\theta without re-deriving anything, and closes with the Newton-Raphson iteration that finds the root numerically when — as for the Gamma family met last lesson — no closed form exists.

Learning objectives. By the end of this note you should be able to:
  • write the likelihood and log-likelihood of an i.i.d. sample and derive the score function s(θ;x)=∂log⁡L/∂θs(\theta;x)=\partial\log L/\partial\theta;
  • solve the score equation s(θ;x)=0s(\theta;x)=0 for the MLE in the Bernoulli, Poisson, and Normal families, and verify each is a global maximum;
  • prove that the log-likelihood of a one-parameter exponential family is concave in the natural parameter, so the score equation has at most one root;
  • state and apply the invariance property of the MLE;
  • execute one step of Newton-Raphson on a log-likelihood and state the algorithm's cost and stopping rule.

In this lesson

  1. The likelihood, log-likelihood, and score
  2. Worked examples
  3. Uniqueness for exponential families
  4. The invariance property
  5. Computing the MLE when there is no closed form

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Probability, Statistics and Financial Data syllabus.