Exercise: Regression as Projection — Computing the Hat Matrix

Prerequisites: Linear Regression Derivation

Problem

Consider the matrix $X \in \mathbb{R}^{n \times p}$ with full column rank ($p \le n$) and the hat matrix $H = X(X^\top X)^{-1}X^\top$.

1. Prove that $H$ is symmetric: $H^\top = H$.

2. Prove that $H$ is idempotent: $H^2 = H$.

3. Show $\text{tr}(H) = p$. (Hint: use the cyclic property of trace: $\text{tr}(ABC) = \text{tr}(BCA) = \text{tr}(CAB)$.)

4. Interpret: the residual vector $e = (I - H)y$ is orthogonal to the column space of $X$. Verify directly that $X^\top e = 0$ for the fitted $\hat\beta$.

5. Numerical example. Construct $X = \begin{pmatrix}1 & 0 \\ 1 & 1 \\ 1 & 2 \\ 1 & 3\end{pmatrix}$ and $y = \begin{pmatrix}1 \\ 2 \\ 2 \\ 3\end{pmatrix}$. Compute $H$, $\hat y = Hy$, and the residuals $e = y - \hat y$. Verify that $\text{tr}(H) = 2 = p$.

Hint

For trace: $\text{tr}(H) = \text{tr}(X(X^\top X)^{-1}X^\top) = \text{tr}((X^\top X)^{-1}X^\top X) = \text{tr}(I_p) = p$.