Solution: Compute the First Adam Update by Hand

Exercise: Compute the First Adam Update by Hand

Solution

$m_1=0.1(4)=0.4$ and $v_1=0.001(16)=0.016$. Bias correction gives $\hat m_1=0.4/0.1=4$ and $\hat v_1=0.016/0.001=16$. The step is $-0.1\cdot4/\sqrt{16}=-0.1$.

Takeaways

• Bias correction exactly removes the initial zero bias at $t=1$.
• The first Adam step has sign equal to the gradient sign.
• The scale is controlled by $\alpha$ when $\epsilon=0$.