Overview

Prices, transactions, text, and event streams are ordered: the model must decide what earlier observations still matter now. This topic begins with recurrent state as a learned sequential memory, then introduces attention and Transformers as a direct-access alternative for long-range dependencies.

By the end of this note you should be able to:

• state a basic RNN update and relate unrolling to gradient propagation;
• explain why basic recurrent cells have limited effective memory;
• describe attention and the Transformer at survey level.

1. Contents
2. The big picture