Overview — Bayesian Estimation and Posterior Inference

Topics 2 and 3 built an entire theory of estimation around a single object, the likelihood $L(\theta;x)$, while treating $\theta$ as a fixed, unknown constant. Bayesian inference is the natural completion of that story: attach a prior $\pi(\theta)$ encoding what is believed about $\theta$ before the data arrive, combine it with the same likelihood via Bayes' theorem, and read off a posterior distribution $\pi(\theta\mid x)$ that is the mathematically correct fusion of prior belief and observed evidence. Every quantity built in Topic 3 — the score, the MLE, the Fisher information — survives unchanged inside this machinery; nothing here replaces frequentist estimation, it reframes the same likelihood as one ingredient in a different calculation.

The topic runs a tight, self-contained arc. It opens with Bayes' theorem for a parameter and the "posterior $\propto$ likelihood $\times$ prior" mechanics, then systematises the three workhorse conjugate pairs (Beta-Binomial, Gamma-Poisson, Normal-Normal) that make closed-form Bayesian updating possible without numerical integration, and derives the general exponential-family conjugacy result that explains why those three pairs behave identically. It then asks the decision-theoretic question every posterior distribution raises — which single number should be reported? — and answers it with loss functions, Bayes risk, and the credible interval. The topic's key result threads through every lesson: the posterior mean is a shrinkage of the MLE toward the prior mean, precision-weighted by how much the prior and the data each know — the Bayesian version of the bias-variance trade-off, and the mechanism behind the Black-Litterman portfolio model. The topic closes with the Bernstein-von Mises theorem, which proves that this entire apparatus — however different its philosophy — converges numerically to the same asymptotic answer frequentist estimation reaches on its own.