A Large Step Selects a Flatter Minimum
An interactive live film: on the valley of minima of f(u, v) = (uv - 1)^2 / 2, raise the step size and watch gradient descent bounce off the sharp minima and settle where the sharpness is below 2/alpha.
Narration
Here is the loss f of u and v, one half of u times v minus one, squared. Its bottom is a whole valley: every point of the purple curve, u times v equals one, is a minimum.
These minima differ only in their sharpness, u squared plus v squared: steep walls near the right end of the curve, gentle ones in the middle.
The rest of the narration is in the book's Companion.
More from Chapter 12
Interactive1:08Gradient Descent in an Ill-Conditioned ValleyChapter 12
Preview1:28The Descent Lemma: A Bowl That Bounds Every StepChapter 12
Preview1:33Heavy-Ball Momentum: Zig-Zags Cancel, the Valley Floor Adds UpChapter 12
Preview2:12Sophia: Newton Steps with a ClipChapter 12
Preview2:14Adam and AdamW: A Step Size for Every CoordinateChapter 12
Preview1:50Proximal Gradient and Soft-Thresholding: Exact ZerosChapter 12
Preview1:53KKT Conditions: the Saddle Point of the LagrangianChapter 12
Preview1:52Mirror Descent Stays Inside the SimplexChapter 12
Preview1:55The Polyak-Łojasiewicz ConditionChapter 12