The Markov Property
A state is Markov when it captures all history relevant to the future, so the past adds nothing once the present is known.
Memoryless states
A stochastic process has the Markov property when the distribution of the next state depends only on the current state and action, not on the full history that led there. Formally P(S_{t+1} | S_t, A_t, S_{t-1}, A_{t-1}, ...) = P(S_{t+1} | S_t, A_t). The present screens off the past.
Why it matters
The Markov property is what makes reinforcement learning tractable. If the next state and reward depend only on the current state and action, then a policy that maps the current state to an action loses nothing by ignoring history. Value functions and the Bellman equation rely on this so that a single state variable carries all the information needed to plan.
Designing Markov states
Whether a representation is Markov depends on what you include in the state. A single camera frame of a moving ball is not Markov because velocity is missing; stacking several frames restores it. Good state design folds enough history into the current state to make it approximately Markov.
When it fails
- Partial observability: sensors hide relevant variables (a POMDP).
- Non-stationary dynamics: the environment's rules change over time.
- Hidden context: an unseen mode or regime alters transitions.
When states are not Markov, agents use memory — recurrent networks or explicit belief states — to summarize history. A physical plant controlled from limited sensors, such as a simulated fusion device with only a few diagnostic channels, is naturally partially observable, so a controller may need to integrate readings over time rather than react to a single instant.