Gradient Descent in an Ill-Conditioned Valley

Here is the loss surface from this chapter, f equals one half x one squared plus five halves x two squared.

Its Hessian is the diagonal matrix with entries one and five, so the condition number kappa is five.

Gradient descent starts at the point minus four, minus two and a half, and takes its first step against the gradient.

With the best step size, one third, the steep coordinate flips its sign at every step, while the flat coordinate only creeps.

Both coordinates shrink by the same factor, two thirds per step, which is kappa minus one over kappa plus one.

Pause the film and drag the surface: the valley is long along x one and steep along x two, exactly what the eigenvalues say.