Solution: Deriving the Normal Equations by Calculus

Exercise: Deriving the Normal Equations by Calculus

Part 1

$L(\alpha, \beta) = \sum (y_i - \alpha - \beta x_i)^2$ .

$\partial L/\partial\alpha = -2\sum(y_i - \alpha - \beta x_i) = 0$ , giving

\sum y_i = n\alpha + \beta\sum x_i. \tag{A}

$\partial L/\partial\beta = -2\sum x_i(y_i - \alpha - \beta x_i) = 0$ , giving

\sum x_i y_i = \alpha\sum x_i + \beta\sum x_i^2. \tag{B}

Part 2

From (A): $\hat\alpha = \bar y - \hat\beta \bar x$ .

Substitute into (B):

\sum x_i y_i = (\bar y - \hat\beta\bar x)\sum x_i + \hat\beta\sum x_i^2 = \bar y\sum x_i - \hat\beta\bar x\sum x_i + \hat\beta\sum x_i^2.

Using $\sum x_i = n\bar x$ :

\sum x_i y_i = n\bar x\bar y - n\hat\beta\bar x^2 + \hat\beta\sum x_i^2.

Rearrange:

\hat\beta(\sum x_i^2 - n\bar x^2) = \sum x_i y_i - n\bar x\bar y.

\hat\beta = \frac{\sum x_i y_i - n\bar x\bar y}{\sum x_i^2 - n\bar x^2} = \frac{\sum (x_i - \bar x)(y_i - \bar y)}{\sum (x_i - \bar x)^2}. \quad \checkmark

(The last equality uses the algebraic identities $\sum x_iy_i - n\bar x\bar y = \sum(x_i - \bar x)(y_i - \bar y)$ and $\sum x_i^2 - n\bar x^2 = \sum(x_i - \bar x)^2$ .)

Part 3

Sample covariance $\widehat{\text{Cov}}(x, y) = \tfrac{1}{n-1}\sum(x_i - \bar x)(y_i - \bar y)$ and sample variance $\widehat{\text{Var}}(x) = \tfrac{1}{n-1}\sum(x_i - \bar x)^2$ . Their ratio:

\frac{\widehat{\text{Cov}}(x, y)}{\widehat{\text{Var}}(x)} = \frac{\sum(x_i - \bar x)(y_i - \bar y)}{\sum(x_i - \bar x)^2} = \hat\beta. \quad \checkmark

The Bessel-correction factor $(n - 1)$ cancels.

Part 4 — Numerical sanity check

import numpy as np
rng = np.random.default_rng(0)
n = 100
x = rng.standard_normal(n)
eps = rng.standard_normal(n)
y = 2 + 3 * x + eps

# closed-form
x_bar, y_bar = x.mean(), y.mean()
beta_hat = np.sum((x - x_bar)*(y - y_bar)) / np.sum((x - x_bar)**2)
alpha_hat = y_bar - beta_hat * x_bar
print(f"alpha={alpha_hat:.3f}, beta={beta_hat:.3f}")
# alpha=2.020, beta=3.094

Close to the true $(2, 3)$ ; deviations are consistent with $\text{SE}(\hat\beta) = \sigma/\sqrt{\sum(x_i - \bar x)^2} \approx 1/\sqrt{100} = 0.1$ .

Takeaways

Normal equations emerge from setting partial derivatives of the squared-loss to zero. No calculus tricks; direct application of first-order conditions.
Closed form in simple (1-d) regression: $\hat\beta$ is a covariance-variance ratio. This is the "rise over run" intuition for slope, made rigorous.
Sample covariance and variance forms are the numerator and denominator. Bessel's correction $(n - 1)$ cancels in the ratio, so using "sum" or "sum divided by $n - 1$ " gives the same $\hat\beta$ .
Standard error decreases as $1/\sqrt{\sum(x_i - \bar x)^2}$ . More data and more variation in the predictor both improve precision.