Exponential Families

A credit-scoring model that writes the log-odds of default as a linear function of covariates (logistic regression), a jump-arrival model that treats trade counts as Poisson with intensity $\lambda$, and a GARCH innovation distribution fitted as Normal — these look like unrelated modelling choices, but they share a single algebraic skeleton: each density can be written as $h(x)\exp(\eta(\theta)T(x) - A(\theta))$ for some function $T$ of the data and some reparametrisation $\eta$ of $\theta$. Recognising this structure is not a cosmetic exercise. It is what makes the likelihood of every distribution in this family collapse onto the same low-dimensional summary $T(x)$ — the seed of the sufficiency idea developed in the next two lessons — and it turns moment computations that would otherwise require an integral into a single differentiation.

This note defines the one-parameter exponential family in its canonical form, works the Bernoulli and Poisson cases in full, proves the moment-generating shortcut $\mathbb E[T(X)]=A'(\eta)$, $\text{Var}(T(X))=A''(\eta)$, extends the definition to the multiparameter case needed for the Normal, Gamma, and Beta families, and flags the one common non-example — the Uniform$(0,\theta)$ family — that trips up nearly everyone on first exposure.

Learning objectives. By the end of this note you should be able to: