Exercise: Fisher Information and the Cramér-Rao Bound for Exponential Rate

Prerequisites: Maximum Likelihood Estimation

Problem

Let $X_1, \ldots, X_n$ be i.i.d. $\text{Exp}(\lambda)$: $p(x; \lambda) = \lambda e^{-\lambda x}$ for $x \ge 0$.

1. Compute the log-likelihood $\ell(\lambda)$ and derive the MLE $\hat\lambda = 1/\bar x$.

2. Compute the Fisher information per observation: $I(\lambda) = -\mathbb{E}\!\left[\frac{\partial^2 \log p(X_1; \lambda)}{\partial \lambda^2}\right].$ (Hint: $\partial^2/\partial\lambda^2\,(\log\lambda - \lambda x) = -1/\lambda^2$.)

3. The Cramér-Rao lower bound for unbiased estimators of $\lambda$ is $\text{Var}(\hat\lambda) \ge 1/(n I(\lambda)) = \lambda^2/n$. Does $\hat\lambda = 1/\bar X$ saturate this bound? (Note: $\hat\lambda$ is biased.)

4. Asymptotic normality simulation. For $\lambda = 2$ and $n = 100$, generate $m = 10{,}000$ sample sets and compute $\hat\lambda$ for each. Compare the empirical distribution of $\sqrt n(\hat\lambda - \lambda)$ to $\mathcal{N}(0, \lambda^2)$. Plot a histogram overlay or report mean and variance.

Hint

For part 2: $\log p(x; \lambda) = \log\lambda - \lambda x$. First derivative: $1/\lambda - x$. Second derivative: $-1/\lambda^2$. Fisher information: $-\mathbb{E}[-1/\lambda^2] = 1/\lambda^2$.