From 4d1950f4c8fcb488401b43d709b7f231e8c2bad6 Mon Sep 17 00:00:00 2001 From: Elijah Voigt Date: Tue, 24 Feb 2026 14:02:02 -0800 Subject: [PATCH] =?UTF-8?q?docs(edu):=20write=20markov=20=C2=A72=20states?= =?UTF-8?q?=20and=20transitions=20[738be2]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the stub in Section 2 with full prose covering: - State space definition (finite vs countably infinite) - Transitions as directed, probability-labelled edges - ASCII state-transition diagram for the weather model - Absorbing, transient, and recurrent state classification --- edu/markov.md | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/edu/markov.md b/edu/markov.md index 870169a..f9c6b37 100644 --- a/edu/markov.md +++ b/edu/markov.md @@ -66,9 +66,33 @@ Under the Markov assumption, these two rules are *all* you need. If today is Sun ### 2. States and Transitions -Every Markov chain consists of a finite (or countably infinite) set of **states** and a rule for how the system moves between them. This section formalizes the vocabulary: states, transitions, directed graphs, and the notion of a chain's **step** or **time**. By the end you will be able to draw a state-transition diagram for a simple real-world system. +Every Markov chain consists of a set of **states** and a rule for how the system moves between them. The set of all possible states is called the **state space**, commonly written *S*. In this course the state space is always discrete — it is either finite (e.g., {Sunny, Rainy}) or countably infinite (e.g., the non-negative integers {0, 1, 2, …}, as in a random walk on the number line). Discrete state spaces are by far the most common setting for introductory Markov chain theory and for the kinds of simulations you will build in Rust. -> 🚧 This section is a stub — see nbd ticket `738be2` +A **transition** is a single step from one state to another. At each discrete time step the chain occupies some state *i* ∈ *S*, then moves to state *j* ∈ *S* with probability *P(i → j)*. Two things matter: (1) transitions are directed — going from *i* to *j* and going from *j* to *i* are distinct transitions with potentially different probabilities; and (2) every transition has an associated probability, and the probabilities of all transitions out of a given state must sum to 1. A transition from *i* back to *i* — a self-loop — is perfectly valid and simply means the chain stays put with some nonzero probability. + +**State-transition diagrams** make these rules visual. Draw one node for each state and one labelled arrow for each possible transition; the label is the transition probability. For the two-state weather model from Section 1 the diagram looks like this: + +``` + 0.8 0.6 + ╭─────────╮ ╭─────────╮ + │ │ 0.2 │ │ + ▼ │ ─────────────► │ ▼ + ╔═══════╗ ───┘ └─── ╔═══════╗ + ║ SUNNY ║ ║ RAINY ║ + ╚═══════╝ ───┐ ┌─── ╚═══════╝ + │ 0.4 │ + └──────────────────┘ +``` + +The self-loops (Sunny → Sunny with 0.8, Rainy → Rainy with 0.6) show that the chain can stay in the same state. The cross-arrows capture the transitions between states. Every row of outgoing arrows sums to 1: from Sunny, 0.8 + 0.2 = 1; from Rainy, 0.4 + 0.6 = 1. + +States in a Markov chain differ in how they relate to the chain's long-run behaviour. Three categories matter most: + +- An **absorbing state** is one you can never leave: once the chain enters it, it stays there forever. A self-loop with probability 1 is the defining feature. A simple example is a gambler who reaches £0 in a gambling game with no credit — they are stuck. +- A **transient state** is one you are not guaranteed to return to. If you leave a transient state there is some positive probability of never coming back. In many models early states are transient — the chain passes through them and moves on. +- A **recurrent state** (also called a **persistent state**) is one you are guaranteed to return to eventually, with probability 1. In the weather model both Sunny and Rainy are recurrent: no matter which state you are in, you will eventually visit both states again and again. + +The distinction between transient and recurrent states determines the chain's long-run behaviour. A chain that only visits transient states will eventually leave them forever; a chain trapped in recurrent states will cycle among them indefinitely. Understanding this classification is the foundation for studying stationary distributions, which Section 9 covers in detail. ---