Composite Hypotheses, p-Values, and the Significance Trap

The previous note proved a UMP test generally does not exist once a null hypothesis is challenged from both sides — the overwhelmingly common shape of a real testing question ("is this strategy's Sharpe ratio different from zero," not just "above" it). This note does two things with that gap. First, it works through what a two-sided test looks like in practice, splitting the significance level across both tails. Second, and more importantly, it replaces the binary reject/accept decision with the continuous-valued $p$-value almost every applied paper actually reports — defines it precisely, proves the single most useful fact about it (its distribution under $H_0$ is exactly $\text{Uniform}(0,1)$), and then spends real space on the significance trap: the specific, well-documented ways a $p$-value gets misread, misused, and — in a research pipeline that tries many strategies before reporting one — silently inflated. This is the note that turns "the test rejected at $5\%$" from a slogan into something a quant desk should read with real care.

Learning objectives. By the end of this note you should be able to: