Solution: The LLN Fails — Running Mean of i.i.d. Cauchy

Exercise: The LLN Fails: Running Mean of i.i.d. Cauchy

Part 1

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
N = 1_000_000
X = rng.standard_cauchy(N)
running_mean = np.cumsum(X) / np.arange(1, N + 1)

for n in [100, 1_000, 10_000, 100_000, 1_000_000]:
    print(f"n={n:8d}:  running_mean = {running_mean[n-1]:+.4f}")
# n=     100:  running_mean = +1.3847
# n=    1000:  running_mean = +0.1362
# n=   10000:  running_mean = -0.2134
# n=  100000:  running_mean = +0.8905
# n= 1000000:  running_mean = -0.0041

The running mean does not settle — it wanders. A single large draw at any

n

can shift the cumulative sum by an arbitrary amount, and because the Cauchy has power-law tails with exponent

1

, very large draws arrive too often to be averaged out.

Part 2

for seed in [0, 1, 2]:
    rng = np.random.default_rng(seed)
    X = rng.standard_cauchy(N)
    running_mean = np.cumsum(X) / np.arange(1, N + 1)
    plt.plot(running_mean, label=f"seed={seed}", alpha=0.8)
plt.xscale('log'); plt.ylabel('running mean'); plt.xlabel('n'); plt.legend()
# The three curves diverge wildly and never agree on a limit.

Three seeds → three trajectories that wander independently and do not converge to any common limit — or any limit at all.

Part 3

The LLN requires the

X_i

to have a finite mean

\mu

(for the strong form) or additionally finite variance (for the Chebyshev-style weak form). The Cauchy distribution has neither:

\int x f(x)\,dx

does not converge absolutely, so no mean exists. With no target to converge to, "convergence" is not a well-defined question. A remarkable consequence: the sample mean of

n

i.i.d. standard Cauchy variates is itself standard Cauchy — so the distribution of

\bar X_n

is the same for every

n

; averaging does nothing.

Part 4

rng = np.random.default_rng(0)
X = rng.standard_cauchy(N)
running_median = np.array([np.median(X[:n]) for n in [100, 1_000, 10_000, 100_000, 1_000_000]])
for n, m in zip([100, 1_000, 10_000, 100_000, 1_000_000], running_median):
    print(f"n={n:8d}:  running_median = {m:+.4f}")
# n=     100:  running_median = -0.0872
# n=    1000:  running_median = -0.0172
# n=   10000:  running_median = -0.0094
# n=  100000:  running_median = +0.0032
# n= 1000000:  running_median = -0.0007

The running median converges to the population median of the standard Cauchy, which is $0$ by symmetry.

Why the median works. The median depends only on the ranks of the observations, not their magnitudes. A single extreme observation affects the rank sorting of a huge sample by exactly one position, which changes the median by the width of one gap between neighbouring order statistics — a quantity that shrinks with

n

(for any distribution with a continuous density at the median). The sample mean, by contrast, is a linear functional: one extreme observation shifts it by

O(X_{\max}/n)

, and Cauchy extremes grow like

n

, so the shift is

O(1)

— unbounded in total. The Glivenko-Cantelli theorem (uniform convergence of the empirical CDF) gives rigorous convergence of quantile estimators for any distribution.

Takeaways

The LLN is a one-way implication: finite mean ⇒ convergence. Without finite mean there is nothing to converge to.
The Cauchy is the canonical counterexample in every first course on probability. Keep it as a standing reminder to check moment conditions.
Median-based estimators are robust to LLN failure. This is why robust statistics (median, MAD, trimmed means) exist — they maintain consistency under heavy tails where the ordinary mean fails.
Heavy-tailed real financial data. Real returns have finite variance (the sample always does), but their population kurtosis can be infinite and the approach to the mean is slow. Robust estimators are quietly routine in risk management: VaR from historical simulation, trimmed returns in factor models, median regression for alpha spread.