Forward or Reverse: Filling the Jacobian
An interactive live film: forward mode pushes a direction onto the surface (a column of J), reverse mode pulls an output back to the floor (a row); drag v yourself, then count the sweeps.
Narration
Here is a function from the plane into space: every point of the floor is lifted onto the surface above it. Its Jacobian, on the right, has three rows, one per output, and two columns, one per input.
Forward mode pushes an input direction through the function. Push the first axis direction, and it comes out as the teal arrow along the surface: that arrow is the first column of J. One J V P fills one column.
The rest of the narration is in the book's Companion.
More from Chapter 11
Interactive1:54Backpropagation, One VJP per LayerChapter 11
Preview1:27Partial Derivatives Are Two SlicesChapter 11
Preview1:34The Gradient Is the Steepest DirectionChapter 11
Preview1:42Jacobian Matrix: Linearizing a MapChapter 11
Preview1:43The Derivative of the Matrix InverseChapter 11
Preview1:32Taylor Expansion, One Term at a TimeChapter 11
Preview1:30Second-Order Taylor: Where the Quadratic Model Can Be TrustedChapter 11
Preview1:38A Hessian-Vector Product from Two GradientsChapter 11
Preview1:34The Gauss–Newton MatrixChapter 11
Preview1:44Hessian Matrix: Bowls, Saddles and Curvature DirectionsChapter 11
Preview1:37Why the Hessian Is SymmetricChapter 11
Preview1:26The Newton Step: Jumping to the Taylor Bowl's BottomChapter 11
Preview2:36Corners, Maxima and Steps in the Chain RuleChapter 11
Preview1:59GPTQ: Rounding with the HessianChapter 11
Preview2:12AWQ: Activation-Aware Weight QuantizationChapter 11
Preview1:41The Implicit Function TheoremChapter 11