The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

Layer Normalization: Project, Then Rescale

Chapter 1 · Mathematics for AI and Machine Learning · Advanced · 1:43

An interactive live film: in R^3, layer normalization projects x onto the plane orthogonal to the all-ones vector and rescales it onto the circle of radius sqrt(d); drag x and bring LN(x) onto a target.

Narration

Layer normalization takes an activation vector x. With three entries we can see every step in space. The blue arrow is x equals one, two, three, and the dashed grey line is the direction of the all ones vector.

First subtract the mean. The mean times the all ones vector is the projection of x onto that line: the orange arrow. What is left, x minus mu times one, is orthogonal to it. This purple arrow lies in the purple plane, where the entries sum to zero.

The rest of the narration is in the book's Companion.

More from Chapter 1

Interactive1:30Span and Independence: When the Dimension JumpsChapter 1Preview1:31Gram–Schmidt, One Vector at a TimeChapter 1Preview1:42Projection onto a Line: the Closest PointChapter 1Preview1:42Norms as Cones: the Unit Ball Is a SliceChapter 1Preview1:36The Parallelogram Law: Which Norms Come from an Inner ProductChapter 1Preview1:31The Outer Product: All of Space onto One LineChapter 1Preview1:29Cauchy-Schwarz: a Shadow Is Never Longer than the StickChapter 1