Multiple Testing and the False Discovery Rate

Exercise 4 of the previous note did the arithmetic on a small scale: test $20$ genuinely null signals at $\alpha=0.05$ each, and about $1$ will show "significance" purely by chance. A systematic factor-research pipeline does not test $20$ signals — it tests hundreds or thousands, screening a universe of candidate predictors, lag structures, and interaction terms for anything that crosses a significance bar. This note builds the standard fixes, in increasing order of statistical power. The Bonferroni correction takes the most conservative route: shrink each individual test's threshold so the probability of making even one false rejection across the whole batch stays at $\alpha$. The Holm-Bonferroni step-down procedure tightens this at no cost, rejecting at least as much as flat Bonferroni for the identical guarantee. The Benjamini-Hochberg procedure takes a different, less conservative route entirely: control the expected proportion of false rejections among the rejections made, the false discovery rate (FDR) — a standard that trades a small amount of guaranteed-error control for substantially more power when screening a large number of hypotheses, exactly the tradeoff a factor-zoo screen needs to be usable at all.

Learning objectives. By the end of this note you should be able to: