Amy, a graduate student, plans her reading-group talk around one question: why abandon recurrence? She frames sequence modeling as a trade between accuracy and practicality, and decides her narrative must explain not just what the new approach does, but why it changes the day-to-day reality of training and deploying models on long, real-world sequences.

She starts with the pain point: recurrent models compute states one step at a time, so a single example can’t be fully parallelized. Longer sequences mean higher latency, and bigger batches hit memory limits before they hit throughput. For Amy’s audience, this becomes the core motivation: sequential dependence turns into the dominant cost.

Amy contrasts “many hops” with “direct contact.” In self-attention, any token can interact with any other token within a layer, so the effective dependency path between distant positions is short. She emphasizes why that matters in practice: shorter paths generally help gradients travel, making long-range structure easier to learn than with deep stacks of stepwise updates.

She explains the architectural bet: if the model can process all positions simultaneously, training time and hardware utilization improve dramatically. Instead of designing around time steps, the model is designed around matrix operations that scale well on GPUs. Amy frames this as turning sequence modeling into a problem of batched linear algebra rather than serialized computation.

Amy introduces the Transformer as an encoder–decoder that keeps the familiar translation pipeline but swaps out recurrence for repeated blocks. Each block is built from self-attention plus a position-wise feed-forward network, wrapped with residual connections and normalization so depth stays stable. The result feels modular: stack layers, keep widths consistent, and train efficiently.

In the encoder, each layer first mixes information across the input sequence using self-attention, then refines features at each position using the same small MLP applied independently per token. Amy highlights the separation of roles: attention handles cross-token communication, while the feed-forward sublayer increases per-token representation power without needing extra sequence steps.

Amy then walks through the decoder: it mirrors the encoder but adds encoder–decoder attention so output tokens can consult the encoded input. The key guardrail is causal masking—each position can only attend to earlier outputs—so generation remains autoregressive. She describes it as letting the model “look back, not ahead,” while still using parallel computation during training.

Loading equations

Loading equations

Loading equations

Amy clarifies why splitting into heads doesn’t explode cost: if the model keeps total width fixed, each head operates on smaller vectors, so total computation stays comparable to a single large attention. The payoff is qualitative: different heads can specialize, reducing the pressure for one attention map to average over many relationships, which often improves modeling of syntax, alignment, and context.

Loading equations

Amy closes the loop with translation outcomes: the model achieves strong WMT14 BLEU improvements and does so with notably reduced training cost compared with prior approaches that relied heavily on sequential computation or expensive ensembles. She frames this as the practical win—better quality is important, but better quality that trains faster is what changes what labs can iterate on.

To finish, Amy highlights what the ablations teach: performance depends on choices like the number of heads, per-head dimensions, dropout, and how positional information is represented, but the throughline is consistent. Self-attention shortens dependency paths while removing per-step recurrence, enabling faster training and strong accuracy—her concluding argument for why this design becomes a new default.