Skip to content
Text as Data and NLP for Finance / LLM annotation and measurement validity

Inter-Rater Agreement

A model label and a human label can both be stable and still disagree. Percent agreement counts how often two coders assign the same tone class to a filing or a call turn. Chance agreement is the part of that match you would expect from the raters’ marginal rates alone. Cohen’s kappa removes that part. For the equity-event feature, kappa is the reliability of the codebook, not the correlation of the score with returns. The typeset formula in the extract is garbled. The prose definition and the worked numbers are usable; the displayed algebra has to be confirmed on the page.
Learning objectives. By the end of this note you should be able to:
  • compute percent agreement and describe Cohen’s chance agreement from the marginal proportions;
  • state kappa in words and reproduce Gwet’s numerical example of about 0.51;
  • explain the prevalence paradox when almost every item falls in one class;
  • say why a Landis–Koch label is not a validity certificate.

In this lesson

  1. Agreement beyond chance
  2. What “chance” is doing
  3. Where the coefficient sits in the pathway
  4. Weighted agreement, without a garbled formula

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Text as Data and NLP for Finance syllabus.