Skip to content
Text as Data and NLP for Finance / LLM annotation and measurement validity

Measurement Validity

Agreement says the coders match each other. Validity says the resulting number matches the construct in the hypothesis. A 10-K tone feature is a compression of a filing into a score. The compression is valid when a reader can see what was kept, rebuild it, and watch it fail on items that were not used to invent it. This note is that deliverable. A valid score is still not, by itself, a tradable signal.
Learning objectives. By the end of this note you should be able to:
  • list the five principles of a good text measurement;
  • separate discovery of the construct from measurement of it, using a sample split;
  • read a confusion matrix into accuracy, precision, and recall;
  • write the scope conditions for a tone score, including where it is a weak instrument.

In this lesson

  1. Measurement as a designed compression
  2. Five principles
  3. Splits, gold labels, and the matrix
  4. The tone-feature dossier
  5. Cross-validation and drift

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Text as Data and NLP for Finance syllabus.