The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

Entropy, Cross-Entropy and KL: the Wasted Bits

Chapter 14 · Mathematics for AI and Machine Learning · Beginner · 1:57

An interactive live film: on the triangle of three-outcome distributions, the entropy dome, the cross-entropy bowl of a true p and the sheet at H(p); the gap is KL. Put the model q where half a bit is wasted.

Narration

With three outcomes, every probability distribution is a point of this triangle. The corners are the certain outcomes, and the center is the uniform distribution.

Above each point stands its entropy, in bits: the grey dome. It is zero at the corners, where nothing is uncertain, and highest at the center: the logarithm of three, in base two.

The rest of the narration is in the book's Companion.

More from Chapter 14

Interactive1:42KL Divergence: Two Orders, Two Different FitsChapter 14Preview1:38Entropic Optimal Transport and SinkhornChapter 14Preview1:58The ELBO: a Hill under the EvidenceChapter 14Preview1:45Wasserstein Distance: the Cheapest Way to Move the EarthChapter 14Preview2:04Fisher Information: the Curvature of KLChapter 14Preview1:53The InfoNCE BoundChapter 14