Exercise: MLE for a Normal with Unknown Mean and Variance

Prerequisites: Maximum Likelihood Estimation

Problem

Let $X_1, \ldots, X_n$ be i.i.d. $\mathcal{N}(\mu, \sigma^2)$.

1. Write out the joint log-likelihood $\ell(\mu, \sigma^2)$.

2. Set partial derivatives $\partial\ell/\partial\mu$ and $\partial\ell/\partial\sigma^2$ to zero and solve for the MLEs $\hat\mu$ and $\hat\sigma^2$.

3. Compute $\mathbb{E}[\hat\sigma^2]$ and show that the MLE for variance is biased: $\mathbb{E}[\hat\sigma^2] = (n-1)\sigma^2/n$.

4. State the bias-corrected (Bessel) version $\hat\sigma^2_{\text{Bessel}} = \frac{1}{n-1}\sum(x_i - \bar x)^2$, which is unbiased.

5. Numerical simulation. For true $\mu = 0, \sigma^2 = 1$, generate $m = 10{,}000$ sample sets of size $n = 10$. Compute both $\hat\sigma^2$ (MLE) and $\hat\sigma^2_{\text{Bessel}}$ for each. Report the average of each across the $m$ experiments and verify they match $(n-1)/n = 0.9$ and $1$ respectively.

Hint

For part 3: use $\text{Var}(\bar X) = \sigma^2/n$, so $\mathbb{E}[(\bar X - \mu)^2] = \sigma^2/n$. Then: