Mixture of Experts: Routing Tokens to a Few Blocks

A mixture of experts layer has several expert blocks, here four, and a router in front of them. The dots on the floor are tokens; to draw them, they live in two dimensions.

The router gives every expert a score, a linear function of the token: h i equals w i dot x, where w i is a row of the router matrix. Above the floor, each score is a tilted plane, and the arrow on the floor is its direction w i.

The rest of the narration is in the book's Companion.