The interactive film is in the book's Companion

Play it, pause it, turn it and drag it with Mathematics for AI and Machine Learning.

Get the book

The Bellman Equation as a Linear System

Chapter 15 · Mathematics for AI and Machine Learning · Beginner · 1:39

An interactive live film: value bars over three states climb by iteration to the exact solution v = (I − γP)⁻¹r; as gamma approaches one the system becomes nearly singular. Choose gamma so that v(s₀) = 5.

Narration

Three states and a fixed policy. The arrows are the transition probabilities under that policy, and the thin pins are the expected rewards in each state.

The value of a state is its expected reward plus gamma times the expected value of the next state. Written for all states at once, v equals r plus gamma P v.

The rest of the narration is in the book's Companion.

More from Chapter 15

Interactive1:41Value Iteration and Policy Iteration: Many Small Steps or a Few Big JumpsChapter 15Preview1:32The Discounted Return: Now or LaterChapter 15Preview1:49Bellman Operator: A Contraction With One Fixed PointChapter 15Preview1:34Continuous-Time Markov Chains: Probability Flowing on a TriangleChapter 15Preview1:58The KL-Regularized Optimal PolicyChapter 15Preview1:51The Deadly Triad: When TD Learning ExplodesChapter 15