Self-Supervised Representation Learning and Foundation-Model Connections

Labelled financial datasets are small relative to the raw tapes, filings, and price histories firms store. Self-supervised learning constructs a supervisory signal from the data itself — predict a masked token, decide whether two augmented views match, reconstruct a corrupted input — so a model learns $\phi_\theta(x)$ on abundant unlabelled observations before a downstream quant task is even specified. This note explains that pretraining objective, how to evaluate representations with a linear probe, and how the same logic connects to the foundation-model pattern surveyed in the attention lesson.