The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

The InfoNCE Bound

Chapter 14 · Mathematics for AI and Machine Learning · Advanced · 1:53

An interactive live film: for a correlated Gaussian pair, the InfoNCE bound log N minus the loss rises with the number of samples N toward the true mutual information, and never above log N; choose N and the correlation rho.

Narration

Take a pair X and Y, jointly Gaussian with correlation rho, here nine tenths. Their mutual information is minus one half log of one minus rho squared, about zero point eight three nats.

InfoNCE draws one positive Y that goes with X, and N minus one negatives drawn on their own, and asks a critic to pick out the positive. We use the best critic, the log density ratio.

The rest of the narration is in the book's Companion.

More from Chapter 14

Interactive1:42KL Divergence: Two Orders, Two Different FitsChapter 14Preview1:57Entropy, Cross-Entropy and KL: the Wasted BitsChapter 14Preview1:38Entropic Optimal Transport and SinkhornChapter 14Preview1:58The ELBO: a Hill under the EvidenceChapter 14Preview1:45Wasserstein Distance: the Cheapest Way to Move the EarthChapter 14Preview2:04Fisher Information: the Curvature of KLChapter 14