The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

Proximal Gradient and Soft-Thresholding: Exact Zeros

Chapter 12 · Mathematics for AI and Machine Learning · Advanced · 1:50

An interactive live film: the lasso surface has creases along the axes; ISTA alternates gradient steps with soft-thresholding, whose bands set coordinates exactly to zero. Find a lambda with one exact zero.

Narration

Here is a composite objective: a smooth bowl f, plus lambda times the l one norm. The surface is the sum. Its minimum is what the lasso returns.

The l one term has a kink wherever a coordinate is zero, so the surface has two creases, along the axes. There the gradient does not exist, and gradient descent has nothing to follow.

The rest of the narration is in the book's Companion.

More from Chapter 12

Interactive1:08Gradient Descent in an Ill-Conditioned ValleyChapter 12Preview1:28The Descent Lemma: A Bowl That Bounds Every StepChapter 12Preview1:33Heavy-Ball Momentum: Zig-Zags Cancel, the Valley Floor Adds UpChapter 12Preview1:43A Large Step Selects a Flatter MinimumChapter 12Preview2:12Sophia: Newton Steps with a ClipChapter 12Preview2:14Adam and AdamW: A Step Size for Every CoordinateChapter 12Preview1:55Backtracking Line Search and the Armijo ConditionChapter 12Preview1:53KKT Conditions: the Saddle Point of the LagrangianChapter 12Preview1:52Mirror Descent Stays Inside the SimplexChapter 12Preview1:55The Polyak-Łojasiewicz ConditionChapter 12