Corners, Maxima and Steps in the Chain Rule

Backpropagation multiplies local derivatives, but some layers have corners. Here is the rectified linear unit. For positive x its slope is one, for negative x it is zero, and the tangent line follows the point.

At zero there is a corner, and no tangent. Every line through the corner with a slope between zero and one stays below the graph. These slopes form the subdifferential, the interval from zero to one. Frameworks pick one of them, usually zero.

The rest of the narration is in the book's Companion.