Exercise: Why Bias Correction Changes Early Steps

Prerequisites: Adam Optimizer

Problem

Assume a constant scalar gradient $g_t=2$. Explain why the uncorrected first moment $m_t$ underestimates the true gradient early in training and compute $m_1$ and $m_2$ for $\beta_1=0.9$.

Hint