Teacher Forcing
Formal definition. Teacher forcing conditions each training prediction on the observed target prefix rather than on tokens previously generated by the model. For source \(x\) and target \(y_{1:T}\), the conditional sequence likelihood factorizes as \(p_\theta(y_{1:T}\mid x)=\prod_{t=1}^T p_\theta(y_t\mid y_{<t},x)\). The decoder receives the target shifted right, typically beginning with a start token, and its output at position \(t\) is scored against \(y_t\). The autoregressive factorization is exact; the model's conditional probabilities are learned approximations. A causal attention mask prevents a training position from accessing later shifted inputs, so all target positions can be evaluated in parallel without revealing the answer.
Why it matters here. At inference the true target prefix is unavailable, so generated tokens are fed back sequentially. Errors can therefore change later contexts, producing a training–inference distribution mismatch often called exposure bias. Teacher forcing does not itself guarantee accurate free-running generation, and parallel training does not make autoregressive inference parallel across output positions.
Appears in: Week 2: Transformer Decoding, Position, Normalization, and Training; Week 2: Transformer Decoder Exam-Focused Notes