KL Divergence: Two Orders, Two Different Fits

The true distribution p, in blue, has two peaks. We want to approximate it by a single Gaussian q, with a mean mu and a width sigma. Every choice of q is a point on this floor.

Forward KL, p against q, is an average under p. It is drawn as a height above every point of the floor. Its lowest point puts q in the middle, wide enough to cover both peaks.

Wherever p has mass and q has almost none, the log ratio blows up. So forward KL forces q to cover everything.

Reverse KL, q against p, averages under q instead. Its surface has two valleys, one under each peak, with a ridge between them.

Reverse KL only charges q where q itself has mass. A narrow q sitting on one peak pays only log two for ignoring the other one, far less than forward KL would charge. It is mode seeking.

Your turn. Drag the point on the floor, and switch between the two surfaces with the slider. Find a valley of reverse KL.

The same two distributions, in a different order, give a different answer. Maximum likelihood uses forward KL, and variational inference uses reverse KL.