Longer Contexts with Rotary Positions

Rotary position encoding turns each pair of coordinates by an angle: the position m times a frequency theta i. Each dial is one pair, from the fastest on the left to the slowest on the right.

A model trained on four thousand and ninety six positions has seen, on each dial, only the shaded range of angles. The fast dials turn many times and see every angle; the slow dials on the right never complete a turn.

The rest of the narration is in the book's Companion.