Skip to content
Text as Data and NLP for Finance / LLM annotation and measurement validity

Overview

Earlier topics chose a representation, a document, and a labelling method for disclosure tone. This topic asks whether the label is a measurement, and whether a measurement is a tradable feature. Agreement among coders, validity against the construct, and the point-in-time and multiple-testing checks are three different deliverables. Passing one does not pass the others.

By the end of this note you should be able to:

  • describe chance-corrected agreement in words, using Gwet’s numerical example, and flag a garbled equation for the PDF;
  • apply the five measurement principles and a held-out confusion matrix to a tone score;
  • state why a text score still has to face point-in-time availability and the research search before anyone trades it.

In this lesson

  1. Contents
  2. The big picture

The full lesson is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Text as Data and NLP for Finance syllabus.