Heavy-Ball Momentum: Zig-Zags Cancel, the Valley Floor Adds Up
An interactive live film: in a long valley, gradient descent zig-zags for 43 steps; the heavy ball keeps a velocity, so the zig-zag cancels and the floor direction builds speed, reaching the bottom in 17 with the book's β. Too much momentum circles, too little blows up.
Narration
A long valley: steep across, nearly flat along the floor. The grey path is gradient descent with its best step size. It zig-zags across the valley and needs forty three steps to reach the bottom.
The heavy ball keeps a velocity. Each step it keeps a fraction beta of the old velocity, adds minus alpha times the gradient, and moves by the result. That is the orange path, and the orange arrow is its velocity.
The rest of the narration is in the book's Companion.
More from Chapter 12
Interactive1:08Gradient Descent in an Ill-Conditioned ValleyChapter 12
Preview1:28The Descent Lemma: A Bowl That Bounds Every StepChapter 12
Preview1:43A Large Step Selects a Flatter MinimumChapter 12
Preview2:12Sophia: Newton Steps with a ClipChapter 12
Preview2:14Adam and AdamW: A Step Size for Every CoordinateChapter 12
Preview1:55Backtracking Line Search and the Armijo ConditionChapter 12
Preview1:50Proximal Gradient and Soft-Thresholding: Exact ZerosChapter 12
Preview1:53KKT Conditions: the Saddle Point of the LagrangianChapter 12
Preview1:52Mirror Descent Stays Inside the SimplexChapter 12
Preview1:55The Polyak-Łojasiewicz ConditionChapter 12