The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

Continuous-Time Markov Chains: Probability Flowing on a Triangle

Chapter 15 · Mathematics for AI and Machine Learning · Advanced · 1:34

An interactive live film: distributions of a three-state chain are points of a triangle, time runs upward; p(t) = exp(tQᵀ)p₀ climbs to the stationary distribution Qᵀπ = 0 from every start. Tune a rate to make π₀ = 0.4.

Narration

A chain with three states that can jump at any moment. A probability distribution over them is a point of this triangle on the floor, and time runs upward.

The rate matrix Q holds the jump rates: off the diagonal, how fast probability leaks from one state to another; on the diagonal, minus the total rate of leaving, so every row sums to zero.

The rest of the narration is in the book's Companion.

More from Chapter 15

Interactive1:41Value Iteration and Policy Iteration: Many Small Steps or a Few Big JumpsChapter 15Preview1:32The Discounted Return: Now or LaterChapter 15Preview1:39The Bellman Equation as a Linear SystemChapter 15Preview1:49Bellman Operator: A Contraction With One Fixed PointChapter 15Preview1:58The KL-Regularized Optimal PolicyChapter 15Preview1:51The Deadly Triad: When TD Learning ExplodesChapter 15