Solution: Why Bias Correction Changes Early Steps

Exercise: Why Bias Correction Changes Early Steps

Solution

$m_1=0.1(2)=0.2$. Then $m_2=0.9(0.2)+0.1(2)=0.38$. Both are far below the constant gradient $2$ because the moving average started at zero. Dividing by $1-0.9^t$ gives $2$ at both $t=1$ and $t=2$ for a constant gradient.

Takeaways

• Zero initialisation creates downward bias.
• Bias correction is most important early.
• The correction fades as $t$ grows.