Markov Chains

Motivation: why this matters in quant finance

A Markov chain is a stochastic model where the current state contains all information needed to predict the next state. In quant finance, that idea appears everywhere: credit ratings migrate between AAA, AA, A, BBB and default; volatility regimes switch between calm and stressed; limit-order-book states update tick by tick; queue lengths in execution models evolve as orders arrive and depart.

The Markov assumption is powerful because it turns path-dependent uncertainty into state-dependent recursion. Instead of carrying the full history, pricing and risk calculations update a vector of state probabilities:

\boldsymbol{\pi}_{n+1}=\boldsymbol{\pi}_n P.

That one line is why Markov chains are the finite-state ancestor of infinitesimal generators, Markov diffusions, Feynman-Kac, and regime-switching finance models.

The informal idea

The Markov property says: given the present, the future is independent of the past.

If $X_n$ is today's regime, then knowing yesterday's and last week's regimes does not improve the forecast of tomorrow once $X_n$ is known. The whole modelling burden is pushed into the transition probabilities:

p_{ij}=\mathbb{P}(X_{n+1}=j\mid X_n=i).

For a finite state space, the probabilities form a transition matrix $P=(p_{ij})$ . Each row sums to one, because from state $i$ the chain must move somewhere:

\sum_j p_{ij}=1.

Formal definition

Let

(X_n)_{n\ge0}

take values in a finite or countable state space

S

. It is a discrete-time Markov chain if for all states

i_0,\ldots,i_n,j

\mathbb{P}(X_{n+1}=j\mid X_0=i_0,\ldots,X_n=i_n) =\mathbb{P}(X_{n+1}=j\mid X_n=i_n).

If the right-hand side does not depend on

n

, the chain is time-homogeneous and has transition matrix

P

with entries

p_{ij}=\mathbb{P}(X_{n+1}=j\mid X_n=i).

The $k$ -step transition probabilities are entries of $P^k$ :

\mathbb{P}(X_{n+k}=j\mid X_n=i)=(P^k)_{ij}.

Key properties

State distributions evolve linearly

If $\boldsymbol{\pi}_n$ is a row vector with $\pi_n(j)=\mathbb{P}(X_n=j)$ , then

\boldsymbol{\pi}_{n+1}=\boldsymbol{\pi}_n P, \qquad \boldsymbol{\pi}_n=\boldsymbol{\pi}_0P^n.

This is the finite-state version of a Kolmogorov forward equation.

Communicating classes

State $j$ is reachable from state $i$ if $(P^n)_{ij}>0$ for some $n$ . States communicate if each is reachable from the other. Communicating classes partition the state space and determine long-run behaviour.

In a credit-rating chain, the default state is often absorbing: once reached, the chain stays there. That single modelling choice changes pricing, risk, and expected-loss calculations.

Stationary distributions

A distribution $\boldsymbol{\pi}$ is stationary if

\boldsymbol{\pi}P=\boldsymbol{\pi}.

For an irreducible finite chain, a stationary distribution exists and is unique. Under additional aperiodicity, $\boldsymbol{\pi}_n$ converges to it from any starting state.

Absorption probabilities

If a set of states is absorbing, many questions reduce to linear equations. For an absorption probability $h_i$ ,

h_i=\sum_j p_{ij}h_j

on transient states, with boundary values fixed on absorbing states. This is the discrete analogue of solving a boundary-value problem for a diffusion.

Continuous-time chains

A continuous-time Markov chain waits an exponential time in state $i$ and then jumps to a new state $j$ . Bertsekas describes this with transition rates

q_{ij}=v_i p_{ij},

where $v_i$ is the total rate of leaving state $i$ . The rate matrix is the finite-state version of a generator.

Worked example: two-state credit regime

Let $X_n\in\{G,B\}$ represent a borrower's annual credit regime: good or bad. Suppose

P= \begin{pmatrix} 0.92 & 0.08 \\ 0.35 & 0.65 \end{pmatrix}.

If the borrower starts good, $\boldsymbol{\pi}_0=(1,0)$ , then after two years

\boldsymbol{\pi}_2 =\boldsymbol{\pi}_0P^2 =(0.92,0.08) \begin{pmatrix} 0.92 & 0.08 \\ 0.35 & 0.65 \end{pmatrix} =(0.8744,0.1256).

Even though the one-year downgrade probability is only $8\%$ , the two-year bad-regime probability is higher because the chain can enter bad in year one and remain there.

Common confusions and pitfalls

"Markov means independent over time." No. Consecutive states are usually dependent. Markov means the dependence is summarised by the current state.

"The transition matrix tells you the realised path." It tells you probabilities, not a deterministic trajectory. Simulation still requires random draws.

"Stationary means constant sample paths." No. A stationary distribution can coexist with active state switching; only the marginal distribution is unchanged.

"Every Markov chain converges to a stationary distribution." Finite irreducible aperiodic chains do. Periodic, reducible, or absorbing chains require separate analysis.

"Continuous-time chains are just discrete chains with smaller time steps." They are related, but continuous-time chains are specified by exponential holding times and rates. The generator, not a one-step matrix alone, is the natural object.

Where this goes next

Poisson Processes: Supplies the exponential waiting-time logic behind continuous-time Markov chains.
Infinitesimal Generators and Kolmogorov Equations: Generalises transition matrices to continuous-time state dynamics.
Feynman-Kac Formula: Uses Markov state dynamics to connect expectations and PDEs.
Stochastic Differential Equations: Continuous-state Markov models driven by Brownian noise.

References

Lawler, G. F. (2023). Stochastic Calculus: An Introduction with Applications. Ch. 1 §1.4 (Martingale convergence theorem; Markov property in Polya's urn), Ch. 6 §6.2 (Poisson process and generators).
Bertsekas, D. P., & Tsitsiklis, J. N. (2008). Introduction to Probability (2nd ed.). Athena Scientific. Ch. 7 §7.1 (Discrete-Time Markov Chains), §7.2 (Classification of States), §7.3 (Steady-State Behavior), §7.4 (Absorption Probabilities), §7.5 (Continuous-Time Markov Chains).