AWQ: Activation-Aware Weight Quantization
An interactive live film: scaling a salient input channel before 4-bit rounding shrinks its error until it coarsens the group's grid; the layer error over the scale and the activation size is a valley, and the reader drags (s, a) to find its bottom.
Narration
A linear layer multiplies weights by activations. AWQ rounds the weights to four bits and keeps the activations in full precision. Here is one group of eight weights from one row, and the grey lines are the grid they must snap to. The largest weight in the group sets the spacing.
One input channel, in red, sees activations much larger than the others. Its rounding error is multiplied by that large activation, so this single weight dominates the output error.
The rest of the narration is in the book's Companion.
More from Chapter 11
Interactive1:54Backpropagation, One VJP per LayerChapter 11
Preview1:27Partial Derivatives Are Two SlicesChapter 11
Preview1:34The Gradient Is the Steepest DirectionChapter 11
Preview1:42Jacobian Matrix: Linearizing a MapChapter 11
Preview1:43The Derivative of the Matrix InverseChapter 11
Preview1:54Forward or Reverse: Filling the JacobianChapter 11
Preview1:32Taylor Expansion, One Term at a TimeChapter 11
Preview1:30Second-Order Taylor: Where the Quadratic Model Can Be TrustedChapter 11
Preview1:38A Hessian-Vector Product from Two GradientsChapter 11
Preview1:34The Gauss–Newton MatrixChapter 11
Preview1:44Hessian Matrix: Bowls, Saddles and Curvature DirectionsChapter 11
Preview1:37Why the Hessian Is SymmetricChapter 11
Preview1:26The Newton Step: Jumping to the Taylor Bowl's BottomChapter 11
Preview2:36Corners, Maxima and Steps in the Chain RuleChapter 11
Preview1:59GPTQ: Rounding with the HessianChapter 11
Preview1:41The Implicit Function TheoremChapter 11