The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

Backward Error Analysis: What Gradient Descent Follows

Chapter 20 · Mathematics for AI and Machine Learning · Advanced · 1:40

An interactive live film: on the book's narrow valley in 3D, gradient descent steps stay closer to the gradient flow of the modified loss f + (eta/4)|grad f|^2 than to the flow of f; change eta and drag the start while a chart measures both gaps.

Narration

Here is the long valley, with gradient descent as amber dots, step k at time k times eta.

The green curve is the gradient flow of f. It matches gradient descent only to first order in eta, and the dots drift off it.

The rest of the narration is in the book's Companion.

More from Chapter 20

Interactive1:45Flow Matching: Velocity Fields and the Continuity EquationChapter 20Preview1:28From a Residual Network to a Neural ODEChapter 20Preview1:27Gradient Flow: Gradient Descent With Infinitely Small StepsChapter 20Preview1:26Memory by Orthogonal Projection (HiPPO)Chapter 20Preview1:36Momentum Methods: Second-Order ODEChapter 20Preview1:34Probability Flow ODE: Deterministic Paths, Identical MarginalsChapter 20Preview1:46Rectified Flow: Straightening by RetrainingChapter 20