Exercise: Deriving the Normal Equations by Calculus

Prerequisites: Linear Regression Derivation

Problem

We have data $(x_1, y_1), \ldots, (x_n, y_n)$ with $x_i, y_i \in \mathbb{R}$ (simple 1-d regression with intercept). The model is $y_i = \alpha + \beta x_i + \epsilon_i$.

1. Write the sum of squared residuals $L(\alpha, \beta) = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2$ and take partial derivatives with respect to $\alpha$ and $\beta$. Set both to zero.

2. Solve the resulting two equations to derive the classical closed forms $\hat\beta = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2}, \quad \hat\alpha = \bar y - \hat\beta \bar x.$

3. Verify that $\hat\beta$ equals the sample covariance of $(x, y)$ divided by the sample variance of $x$: $\hat\beta = \widehat{\text{Cov}}(x, y)/\widehat{\text{Var}}(x)$.

4. Numerical sanity check. Generate $n = 100$ points with $x_i \sim \mathcal{N}(0, 1)$, $y_i = 2 + 3 x_i + \epsilon_i$, $\epsilon_i \sim \mathcal{N}(0, 1)$. Use numpy.polyfit or your closed-form formulas to estimate $\hat\alpha, \hat\beta$. Both should be close to $(2, 3)$.

Hint

For part 1: $\partial L/\partial \alpha = -2\sum (y_i - \alpha - \beta x_i) = 0$ gives $\sum y_i = n\alpha + \beta\sum x_i$. Similarly for $\beta$.