The random walk generalises in exactly one direction here: the rule for the next step is now allowed to depend on the current state, not just be a fixed i.i.d. increment. This single change turns a narrow object (a walk on the integers) into the single most widely used discrete stochastic model in applied probability — weather, credit ratings, market regimes, board games, population genetics, and web search all reduce, at some level of abstraction, to a Markov chain. This topic builds the complete theory of finite-state discrete-time chains, following Durrett's Chapter 1 section by section: from the bare definition through classification, stationary distributions, detailed balance, convergence, and exit analysis, closing with a worked-examples capstone that puts every tool to use on credit migration, PageRank, and Markov chain Monte Carlo.
By the end of this topic you should be able to:
- define a Markov chain, verify the Markov property, and compute multistep probabilities via the Chapman-Kolmogorov equation;
- classify every state of a finite chain as transient or recurrent, and compute its period;
- compute a stationary distribution — directly, via detailed balance, or via the doubly-stochastic shortcut — and interpret it as a reciprocal mean return time;
- state and use the convergence theorem and the ergodic theorem, and explain the coupling argument behind the former;
- solve exit-distribution and exit-time problems for a general absorbing chain.
In this lesson
- Contents
- The big picture
The full lesson is for members.
Read the full lessonNew to SelfMadeQuant.com? See membership options or explore the curriculum.