Residual Connections
Formal definition. A residual connection adds a transformation \(F(x)\) to its input: \(y=x+F(x)\), with both operands of matching shape [d] at each token. He et al. (2016) formulate residual learning as learning \(F\) relative to an identity shortcut. For this bare addition, the Jacobian is \(I+J_F\), giving gradients a direct additive path alongside the transformed path. This does not ensure that the learned residual is small or that gradients cannot vanish or explode. Projections can align dimensions when the shortcut and transformation otherwise differ. The shortcut and transformed contributions are added element by element, so their dimensions must be compatible for this operation.
Why it matters here. Residual additions wrap each attention and feed-forward sublayer in the original Transformer. Its post-normalized form is \(\operatorname{LN}(x+\operatorname{Sublayer}(x))\), so the Jacobian also includes the LayerNorm Jacobian; it is not simply \(I+J_F\). In pre-normalization, \(x+\operatorname{Sublayer}(\operatorname{LN}(x))\) instead retains an unnormalized identity path across the complete block. The ebook's characterization of the residual as necessarily a small correction is an intuition, not a mathematical constraint.
Appears in: Week 2: Transformer Decoding, Position, Normalization, and Training; Week 2: Transformer Decoder Exam-Focused Notes