A fixed embedding gives
“liability” one vector, averaged across every context in the training corpus. In a 10-K the same string is usually an
accounting line item. A contextualised embedding gives the token a vector that depends on the words around it, so the
accounting use and the moral-burden use need not share a point. This note states what that model class is for, how a
researcher is supposed to use a pretrained model, and the compute constraint on training one. It does not state an
attention equation. The extract names the Transformer architecture and does not transcribe its update rules.
Learning objectives. By the end of this note you should be able to:
- distinguish a type-level embedding from a contextualised embedding;
- name BERT and GPT-3 as pretrained models in this class, and say what fine-tuning means;
- compare the compute cost in the cited case with the cost of training a static embedding;
- state the validation deliverable for a 10-K tone feature that uses one of these models.
In this lesson
- What changes when the vector depends on context
- Pretrained models and fine-tuning
- Cost is part of the implementation
- Validation deliverable
- A filing sentence the bag of words flattens
Before you start
The full lesson, with worked examples and exercise solutions, is for members.
Read the full lessonNew to SelfMadeQuant.com? See membership options or browse the Text as Data and NLP for Finance syllabus.