Week 1 in plain language: from one squeezed summary to attention and the Transformer encoder

eli12week-01

Explain-like-I'm-12 version of the full Week 1: From Sequence Models to Transformer Encoders. Compact revision lives in Week 1: Attention and Encoder Exam Guide, and the real past-paper questions with worked answers live in Week 1: LLM Exam Questions and Worked Answers. Everything hangs off the LLM Knowledge Map.

Note

The big idea of Week 1. An old-style translator reads a whole sentence, squeezes it into one fixed-size summary, and then writes the translation from that summary alone. That is like asking a friend to retell a long story after reading only a single sticky note. Attention fixes this: instead of one sticky note, the writer may look back at every word of the source and decide, at each step, which words matter right now. The Transformer pushes this to the limit. Every word looks at every other word at once (self-attention), several "readers" (heads) look in parallel, a small per-word network (the FFN) digests the result, and we stack that block into an encoder. The week ends by counting exactly how many learnable numbers one encoder layer holds.

The method throughout is Concept, Code, Output, Interpretation: say the idea, run it, read the result.

1. The squeezed summary (seq2seq bottleneck)

A sequence-to-sequence (seq2seq) model turns one sequence into another, for example an English sentence into a French one. The early version used an encoder RNN that reads the source and a decoder RNN that writes the output. The only thing passed between them was one fixed-width vector, the context \(c\).

Analogy: the encoder reads a whole book and hands the decoder a single sticky note. Whatever fits on the note is all the decoder ever knows.

Two problems follow (see Sequence-to-Sequence Bottleneck):

Bottleneck: the one vector \(c\) must carry everything the decoder will need for every output word. Long or detail-rich inputs strain it. (Notes: this is a strain, not a law that long sentences always translate worse.)
Slow chain: an RNN's state depends on the previous state, so words are processed one after another, and information from far-apart words has to travel through many steps (about \(T\) steps for \(T\) words). A Transformer connects any two positions in one layer, a path length of \(O(1)\).

Tiny check: a 3-word and a 30-word sentence both end up as the same size of \(c\). That is the bottleneck in one sentence.

Warning

Mistakes people make

"Seq2seq means exactly two RNNs and one vector." No. That is one early design. Attention-based RNN models and Transformer encoder-decoders are also seq2seq.
"Long sentences always get worse." The notes say degradation is not inevitable at a particular length; the fixed channel is simply a stress point.

Where the exam asks this

2025 T1 Quiz 1, Q107: which models struggle to encode context for long sentences. The derived answer is the plain RNN (option A).
2026 T1 Quiz 2, Q9: RNN vs Transformer comparison (path length \(O(1)\) vs \(O(T)\), parallel training, Q/K/V). Derived answer: A, C and D.

Answers here and below are derived study aids, not official keys.

2. Looking back instead of remembering (Bahdanau attention)

Idea: at each output word, the decoder asks "which source words matter now?", scores every encoder state, turns the scores into percentages, and takes a weighted average of the encoder states. That average is a fresh context \(c_t\) for that step.

\[e_{ti}=v_a^\top\tanh(W_a s_{t-1}+U_a h_i),\qquad \alpha_{ti}=\frac{e^{e_{ti}}}{\sum_j e^{e_{tj}}},\qquad c_t=\sum_i\alpha_{ti}h_i .\]

\(e_{ti}\) is a score: how relevant source word \(i\) is at decoder step \(t\) ("additive" because \(s_{t-1}\) and \(h_i\) are added inside the \(\tanh\)).
\(\alpha_{ti}\) are the weights from softmax: positive, summing to 1, like percentages of attention.
The weights form an alignment table of shape [T_target, T_source], and every row sums to 1.

Classroom analogy: the decoder is a teacher asking four students (encoder states) a question; each raises a hand with a different height (score). Softmax turns hand heights into shares of the class's attention.

Hand-checkable example. Two source words with states \(h_1=[1,0]\) and \(h_2=[0,1]\), and (made-up) scores \(e=[2,0]\):

\[\alpha_1=\frac{e^2}{e^2+1}\approx 0.881,\quad \alpha_2\approx 0.119,\quad c_t=0.881\,[1,0]+0.119\,[0,1]=[0.881,\;0.119].\]

The context leans toward word 1, as expected.

Executed output (05_additive_attention, four encoder states of size 3):

alignment scores=[ 0.699  0.539 -0.081 -0.522]
alpha=[0.384 0.327 0.176 0.113]; sum=1.000000
context=[ 0.351 -0.993 -0.053]

The highest score (0.699) gets the biggest weight (0.384), the weights add to 1, and the context is their weighted mix of the four encoder states.

Warning

Mistakes people make

Weights are over encoder states (source positions), not over all decoder states.
A weight is a learned soft alignment, not a guaranteed explanation of why the model decided something.
A diagonal pattern is not required: when word order changes between languages, the highlight moves off the diagonal.

Where the exam asks this

2026 T1 Quiz 1, Q13: compute one additive score. Derived steps: with \(h_1=(1,0)\) the sum is \((2,2)\) and \(v=(1,1)\), so the score is \(2\tanh 2\approx 1.93\).
2026 T1 Quiz 1, Q14: softmax the scores (derived answer 0.35).
2026 T1 Quiz 1, Q15: build the context vector (derived answer 1.36).
2026 T2 Quiz 1, Q2: true statements about traditional attention (derived: A and C).

Full details: Bahdanau Attention.

3. Self-attention and the three roles Q, K, V

Now drop the RNN. Let every word in the sentence look at every other word of the same sentence. Each word is turned into three vectors by three learned matrices:

Role	Made by	Plain meaning	Library analogy
Query \(Q=XW^Q\)	learned matrix \(W^Q\)	what this word is looking for	the question you type in the search box
Key \(K=XW^K\)	learned matrix \(W^K\)	what this word offers to be matched on	the label on a book's spine
Value \(V=XW^V\)	learned matrix \(W^V\)	the content handed over if matched	the book's contents

Score = query matched against every key; weights = softmax of the scores; output = weighted sum of values. Shapes for one head, with X of shape [B, T, d_model]: \(Q\) and \(K\) are [B, T, d_k], \(V\) is [B, T, d_v], the weight matrix \(A\) is [B, T, T], and the output \(Z=AV\) is [B, T, d_v].

Hand-checkable example (2 tokens, 2-dim). Let \(X=\begin{bmatrix}1&0\\0&1\end{bmatrix}\) and, to keep it simple, take \(W^Q=W^K=W^V=I\) (identity). Then \(Q=K=V=X\), and \(QK^\top=I\). Token 1 matches itself with score 1 and token 2 with score 0. Dividing by \(\sqrt{d_k}=\sqrt2\) (next section) gives \([0.707,0]\). Softmax:

\[\alpha=\left[\tfrac{e^{0.707}}{e^{0.707}+1},\ \tfrac{1}{e^{0.707}+1}\right]\approx[0.670,\ 0.330],\qquad z_1=0.670\,[1,0]+0.330\,[0,1]=[0.670,\ 0.330].\]

Token 1's new vector is mostly itself, plus a bit of token 2. By symmetry token 2 gets \([0.330, 0.670]\). Each word is now a contextual representation.

Warning

Mistakes people make

The weighted sum is over values, not keys. (Notes example: \(q=1\), \(k_1=k_2=0\), \(v_1=[1,0]\), \(v_2=[0,2]\) gives weights \([\tfrac12,\tfrac12]\) and \(z=[0.5,1]\).)
Softmax runs over the keys for each query, not over queries.
\(W^Q\), \(W^K\), \(W^V\) are three separately learned matrices, not one copy. (My identity trick above is a teaching shortcut.)
Attention exists in the encoder too, not only the decoder.
Without positional information, self-attention cannot tell word order by itself.

Where the exam asks this

2026 T1 Quiz 1, Q2: shape of \(Q=XW_Q\). Derived answer: [T, d_k].
2024 T1 End Term AN, Q153: compute a word's representation and sum its elements (derived answer 1.676).
2026 T2 End Term AN, Q18: compute \(W_Q x_m\) (derived answer: option C, \([5,3,-1,-1]^\top\)).
2025 T3 End Term FN, Q18: how many scalar scores for 10 sentences of length 100, one head (derived: 100,000).

Full details: Self-Attention.

4. Why divide by \(\sqrt{d_k}\) (scaled dot-product)

The full formula is

\[\text{Attention}(Q,K,V)=\operatorname{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)V .\]

The problem: a dot product adds up \(d_k\) products. If the numbers in \(q\) and \(k\) are independent with average 0 and variance 1, the sum has variance \(d_k\), so the scores spread out as vectors get longer. Big spread makes softmax nearly one-hot (one weight near 1, the rest near 0), and then the gradients are close to zero, so learning stalls.

The fix: divide by \(\sqrt{d_k}\), which brings the variance back to about 1.

Executed evidence (02_dot_product_variance, 20,000 random pairs per row):

d_k=  4 Var(q.k)=    4.04 target=  4 scaled_var=1.011
d_k= 16 Var(q.k)=   15.87 target= 16 scaled_var=0.992
d_k= 64 Var(q.k)=   63.83 target= 64 scaled_var=0.997
d_k=256 Var(q.k)=  253.99 target=256 scaled_var=0.992

Raw variance tracks \(d_k\); after scaling it stays near 1.

Hand-checkable example. Take \(d_k=64\) and (invented) raw scores \([8,0]\). Without scaling, softmax gives \([0.9997,\ 0.0003]\): practically one-hot. With scaling, \(8/\sqrt{64}=1\), so the scores become \([1,0]\) and softmax gives \([0.731,\ 0.269]\): still a clear favourite, but not a winner-takes-all.

Executed check (01_attention_from_scratch): a from-scratch NumPy version matches PyTorch's own attention.

attention shape=(1, 2, 3, 4); row-sum range=1.000000..1.000000
max |NumPy - torch SDPA|=0.00000011

Every softmax row sums to 1, and the difference to PyTorch is only rounding noise.

Warning

Mistakes people make

Scaling does not make every row uniform. It shrinks the size of the scores but keeps their order, so unequal scores still give unequal weights.
Scaling is not what makes rows sum to 1; softmax does that.
Scaling does not guarantee no saturation. The variance argument assumes random, independent numbers, and trained projections need not obey that.
Scaling does not change any shape and adds no parameters.

Where the exam asks this

2025 T3 Quiz 1, Q173 and 2026 T1 Quiz 1, Q3: the reason for dividing by \(\sqrt{d_k}\). Derived answers: avoid softmax saturation and vanishing gradients (D and C respectively).
2025 T2 Quiz 1, Q136: purpose of softmax (derived: normalise scores to a probability distribution).
2025 T2 Quiz 1, Q146: weighted sum of values. Derived: \(0.1[1,0]+0.2[0,1]+0.5[1,1]+0.2[2,1]=[1.0,0.9]\), so the first element is 1.00.

Full details: Scaled Dot-Product Attention.

5. Several readers at once (multi-head attention and \(W^O\))

Idea: one attention pattern can only focus one way. Give the layer several heads; each has its own \(W^Q,W^K,W^V\), so each can look for a different relationship (for example, one links "it" to "animal", another links "it" to "tired"). Then glue the heads' outputs side by side (concatenate) and mix them with one more learned matrix, \(W^O\):

\[\operatorname{MHA}(X)=\operatorname{Concat}(H_1,\dots,H_h)\,W^O .\]

Original base sizes: \(d_{\rm model}=512\), \(h=8\), so each head is \(d_k=d_v=64\). Eight heads of width 64 concatenate to [B, T, 512], and \(W^O\) is [512, 512]. It mixes the head features and restores the width the residual connection needs.

Executed shape check (03_multihead_shape_check, a smaller model with \(d_{\rm model}=16\), 4 heads):

input [B,T,d_model]=[2,5,16]
heads [B,h,T,d_k]=[2,4,5,4]; concat=[2, 5, 16]
max |manual - torch.nn.MultiheadAttention|=0.00000009

Four heads of width 4 concatenate back to width 16, and the hand-built version agrees with PyTorch's module.

Hand-checkable \(W^O\) example (from the notes). Two heads give one token the outputs \([1,2]\) and \([3,4]\). Concatenate: \([1,2,3,4]\). If \(W^O\) has columns \([1,0,1,0]^\top\) and \([0,1,0,1]^\top\), the result is \([1+3,\ 2+4]=[4,6]\). So \(W^O\) takes [1, 4] back to [1, 2], a learned mixture of all head outputs.

Warning

Mistakes people make

More heads do not automatically mean more weights. With head width \(d/h\), each projection still totals \(h\cdot d\cdot(d/h)=d^2\). (Count what the question includes: \(W^O\), biases.)
Heads can specialise, but distinct, readable roles are not guaranteed.
With row-vector inputs, \(W^O\) multiplies the concatenation on the right.

Where the exam asks this

2026 T1 Quiz 2, Q2: width of the concatenated output for \(d_{\rm model}=512\), \(h=8\). Derived: [T, 512].
2026 T1 Quiz 1, Q9: motivation for multi-head attention. Derived: A, B and D (C is false: it does not cut parameters).
2025 T3 Quiz 1, Q191: parameters in \(W_O\). Derived: \(768^2=589{,}824\).
2025 T3 Quiz 1, Q183: statements about multiple heads (see the bank for the derived answer).

Full details: Multi-Head Attention.

6. The encoder layer: attention, then a per-word brain (FFN)

One encoder layer has two sub-blocks, each wrapped in a "keep the original and add the new part" residual step followed by LayerNorm:

\[U=\operatorname{LN}_1\big(X+\operatorname{MHA}(X)\big),\qquad Y=\operatorname{LN}_2\big(U+\operatorname{FFN}(U)\big).\]

Attention is where words talk to each other (information moves between positions).
The FFN is where each word thinks alone: \(\operatorname{FFN}(x)=\operatorname{ReLU}(xW_1+b_1)W_2+b_2\) with \(W_1\) of shape [d_model, d_ff] and \(W_2\) of shape [d_ff, d_model]. The same weights are applied to every position independently.
The ReLU in the middle is what makes two stacked affine maps more expressive than one.
The residual additions need matching widths, so the layer keeps shape [B, T, d_model]. Stack six layers (each with its own weights) and the shape is still [B, T, 512]: one vector per token.

Classroom analogy: attention is group discussion; the FFN is each student quietly working on their own notebook afterwards.

Hand-checkable example. Input [T, d_model] = [50, 512] through 6 encoder layers gives [50, 512]: length and width are preserved at every layer.

Warning

Mistakes people make

"The FFN looks at neighbouring tokens." No; it works on each position separately. Mixing across tokens comes from attention.
"Identical layers share weights." They share the architecture; each layer has separately learned parameters.
The standard encoder has unmasked self-attention and an FFN; cross-attention and masked self-attention belong to the decoder (Week 2).
The original paper applies LayerNorm after the residual addition (post-norm).

Where the exam asks this

2026 T1 Quiz 1, Q4: output shape of the 6th layer for \(T=50\), \(d_{\rm model}=512\). Derived: [50, 512].
2026 T1 Quiz 2, Q8: components of a standard encoder layer. Derived: A and B.
2026 T1 End Term FN, Q4: output volume for 5 sentences padded to 20 tokens with \(d_{\rm model}=64\). Derived: \(5\cdot20\cdot64=6400\).
2025 T2 Quiz 1, Q135: how Transformers process tokens (derived: all in parallel).

Full details: Position-Wise Feed-Forward Network.

7. Counting the learnable numbers (parameters)

A parameter is a number the model learns. Count only what the question asks for. With \(d=d_{\rm model}\) and \(f=d_{\rm ff}\):

Piece	Weights	Extra if biases / norms are counted
Q, K, V projections (all heads together)	\(3d^2\)
Output projection \(W^O\)	\(d^2\)
FFN (\(W_1\), \(W_2\))	\(2df\)	\(+f+d\) for FFN biases
Attention biases		\(+4d\)
Two affine LayerNorms		\(+4d\)

Weights only: \(4d^2+2df\). Fully counted (as PyTorch does): \(4d^2+2df+f+9d\).

Worked example with \(d=512\), \(f=2048\) (executed in 04_encoder_parameter_count, and each line below recomputed by hand arithmetic):

MHA: \(4\cdot512^2+4\cdot512=1{,}048{,}576+2{,}048=1{,}050{,}624\)
FFN: \(2\cdot512\cdot2048+2048+512=2{,}097{,}152+2{,}560=2{,}099{,}712\)
two LayerNorms: \(4\cdot512=2{,}048\)
total: \(1{,}050{,}624+2{,}099{,}712+2{,}048=3{,}152{,}384\)

MHA formula=1,050,624; FFN formula=2,099,712; two LayerNorms=2,048
encoder-layer formula=3,152,384; torch count=3,152,384; equal=True
Course shorthand: about 3.15M with FFN biases; original paper's base layer count is approximate.

The formula and PyTorch's own module count agree. Six independently parameterised layers hold \(6\times3{,}152{,}384=18{,}914{,}304\) (excluding embeddings, positions and task heads). The FFN holds about twice as many numbers as attention: the big part of a layer is the "per-word brain".

Small counts to do in your head:

One head's \(W_Q\) with \(d_{\rm model}=512\), \(d_k=64\): \(512\cdot64=32{,}768\).
Length does not matter for the layer's weights: a longer sentence changes activations and compute, not shared projection weights (unless a learned position table is counted).
Score entries for a dense layer: \(B\cdot h\cdot T^2\). With \(B=4\), \(h=2\), \(T=12\) that is \(4\cdot2\cdot144=1{,}152\). Doubling \(T\) quadruples the score entries.

Warning

Mistakes people make

"Increasing the number of heads adds weights." Not at fixed \(d_{\rm model}\) (see section 5).
Forgetting to say whether biases, LayerNorm, \(W^O\), embeddings and positions are included. Some bank questions are marked ambiguous for this reason.
The ebook's "1M + 2M = 3M" is a mnemonic, not an exact count; the notes also flag the slide value 32,178 as a typo for 32,768.
Attention is not linear in length: there are \(T\) queries each looking at \(T\) keys.

Where the exam asks this

2025 T3 Quiz 1, Q188: parameters in \(W_Q\) for one head. Derived: 32,768.
2026 T1 End Term FN, Q2: Q, K, V and \(W^O\) parameters for \(d_{\rm model}=256\), 4 heads. Derived: \(196{,}608+65{,}536=262{,}144\).
2026 T1 End Term FN, Q3: T5-Base FFN, \(2\cdot768\cdot3072\). Derived: 4,718,592.
2025 T3 Quiz 1, Q186: count of attention scores for \(B=4\), \(T=12\), \(h=2\). Derived: 1,152.

Full details: Encoder Layer Parameter Count.

Check yourself

Why is one fixed context vector a problem? Answer: it is the only source-dependent channel the decoder has, so it must carry everything needed for every output word; attention instead builds a new weighted context for each step.
Scores \([2,0]\) with encoder states \([1,0]\) and \([0,1]\): what is the context? Answer: weights \([0.881,0.119]\), so \(c=[0.881,0.119]\).
Why \(\sqrt{d_k}\)? Answer: dot-product variance grows like \(d_k\); dividing by \(\sqrt{d_k}\) keeps it near 1 (the run above shows 1.011, 0.992, 0.997, 0.992), which limits softmax saturation and vanishing gradients.
Eight heads of width 64: what is the concatenation width and the shape of \(W^O\)? Answer: 512, and [512, 512].
Parameters in one encoder layer at \(d=512\), \(f=2048\) with biases and affine norms? Answer: 3,152,384 (MHA 1,050,624 + FFN 2,099,712 + norms 2,048).

Bridge to Week 2

Week 1 built the encoder: words look at each other, then each word is processed alone, stacked layer upon layer. But a Transformer needs more: a way to know word order (positions), a decoder that writes the output one word at a time without peeking ahead (masked attention), and cross-attention so the decoder can look at the encoder. That is where Week 2 in Plain Language: How a Transformer Decoder Writes, Keeps Its Place and Stays Stable picks up, with the longer notes in Week 2: Transformer Decoding, Position, Normalization, and Training.