Continuous-State MDPs and Function Approximation

Value iteration requires enumerating every state and updating $V(s)$ for each one — a hard requirement that breaks the moment a state is a real-valued vector, such as a car's $(x, y, \theta, \dot x, \dot y, \dot \theta)$ or a portfolio's continuous vector of holdings. This note covers the two standard ways to bring value iteration to continuous state spaces: discretization, which is simple but scales catastrophically with dimension, and fitted value iteration, which fits $V$ directly as a parametrised function using the same supervised-regression machinery from Topic 1.