Momentum Methods: Second-Order ODE

Gradient descent and gradient flow follow the steepest slope. In an elongated valley, they crawl along the floor and waste steps oscillating against steep walls.

Momentum changes the game. Adding inertia turns first-order gradient flow into a second-order differential equation, like a heavy ball rolling in a potential.

The rest of the narration is in the book's Companion.