The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

Fisher Information: the Curvature of KL

Chapter 14 · Mathematics for AI and Machine Learning · Advanced · 2:04

An interactive live film: over the steps delta of a Gaussian's parameters, the exact KL bowl and its quadratic delta^T F delta / 2 agree near the center. Change the variance until the ellipses of equal KL are circles.

Narration

A Gaussian with mean mu and variance sigma squared is one point in the plane of its parameters. Move it by a small step delta, and you get a nearby Gaussian: the two curves in the panel.

The Fisher information is the expected square of the score, the derivative of the log density. For this Gaussian it is a diagonal matrix: one over sigma squared for the mean, and one over two sigma to the fourth for the variance.

The rest of the narration is in the book's Companion.

More from Chapter 14

Interactive1:42KL Divergence: Two Orders, Two Different FitsChapter 14Preview1:57Entropy, Cross-Entropy and KL: the Wasted BitsChapter 14Preview1:38Entropic Optimal Transport and SinkhornChapter 14Preview1:58The ELBO: a Hill under the EvidenceChapter 14Preview1:45Wasserstein Distance: the Cheapest Way to Move the EarthChapter 14Preview1:53The InfoNCE BoundChapter 14