Skip to content
Probability, Statistics and Financial Data / Statistical Models and Data Reduction

Fisher Information and the Likelihood Principle

The first lesson of this topic observed, without making it precise, that a sharper likelihood curve says the data pin down θ\theta more tightly than a flatter one. This note supplies the precise measure: Fisher information, the expected curvature of the log-likelihood at θ\theta, quantifies exactly how much a dataset "knows" about θ\theta — and it is the object that will set the best-possible precision of any unbiased estimator via the Cramér-Rao bound in Topic 3. Along the way, this note derives the two equivalent formulas for Fisher information, shows that for an exponential family it reduces to a quantity already computed two lessons ago, distinguishes the observed information a numerical optimiser reports from the expected (Fisher) information a textbook formula gives, and closes the topic with the likelihood principle — the claim, foundational to how this entire topic has been built, that the likelihood function alone should determine every inference about θ\theta, regardless of how or why the data collection stopped.
Learning objectives. By the end of this note you should be able to:
  • define the score function and prove Eθ[score]=0\mathbb E_\theta[\text{score}]=0 under regularity, identifying exactly where the regularity condition is used;
  • define Fisher information I(θ)I(\theta) and derive the computational shortcut I(θ)=−Eθ ⁣[∂2log⁡p(X;θ)/∂θ2]I(\theta) = -\mathbb E_\theta\!\left[\partial^2\log p(X;\theta)/\partial\theta^2\right];
  • compute Fisher information directly for the Bernoulli family and for a general exponential family via the log-partition function;
  • distinguish observed information from expected (Fisher) information and state when each is used in practice;
  • state the (weak) likelihood principle and explain, via the stopping-rule example, why it is violated by naive pp-value-based inference.

In this lesson

  1. The score function
  2. Fisher information
  3. Fisher information in an exponential family
  4. Observed versus expected information
  5. The likelihood principle

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Probability, Statistics and Financial Data syllabus.