Transformers Cheatsheet
Core concepts from 'Attention Is All You Need' — architecture, attention mechanisms, and generation
📚 Learning Resources
| Topic | Source |
|---|---|
Good for setting the context for the problem set-up. Doesn't go into scaled dot product. | DeepLearning.AI |
Excellent visual walkthrough of attention mechanism with matrix illustrations. | Jay Alammar |
Hands-on tutorial building GPT from the ground up with detailed explanations. | Andrej Karpathy |
Very readable and understandable reference implementation. | nanoGPT (Karpathy) |
The foundational paper introducing the Transformer architecture. | arXiv (Vaswani et al.) |
Line-by-line annotation of the original paper with working PyTorch code. | Harvard NLP |
Why Transformers Matter
Transformers were revolutionary because they let language models attend to all parts of a sequence in parallel, making it much easier to learn long-range relationships than with older recurrent models. That parallelism also made training vastly more scalable on modern hardware, which unlocked the huge data and model sizes behind today's strongest language models.
vs. prior NLP models (RNNs, LSTMs):
- Contextual understanding — each token learns which other tokens matter for interpreting it
- Parallelization — can be optimized on GPU/TPUs (no sequential dependency)
- Long-range dependencies — captures relationships across entire sequence
- Stable gradients — avoids vanishing/exploding gradient problems
Transformer Architecture Types
| Architecture | How it attends | Intuition | Best for | Examples |
|---|---|---|---|---|
| Encoder-only (e.g., BERT) | Bidirectional self-attention — each token attends to tokens on both left and right | Builds deep contextual understanding of the full input by looking at the whole sentence at once | Tasks where the goal is to interpret or label text rather than generate it | sentiment classification, NER, search/ranking, information extraction |
| Decoder-only (e.g., GPT) | Masked self-attention — each token can only attend to earlier tokens | Learns to generate text one token at a time, always predicting what comes next from prior context | Autoregressive generation and open-ended text production | chatbots, story writing, code completion, next-token prediction |
| Encoder–Decoder (e.g., T5) | Encoder uses bidirectional attention; decoder uses masked self-attention + cross-attention to encoder states | First reads and understands the input, then generates an output conditioned on that input | Input-to-output tasks where the model transforms one sequence into another | translation, summarization, question answering, paraphrasing |
Example: Encoder–Decoder Architecture
Encoder–Decoder architecture. From Vaswani et al., "Attention Is All You Need" (2017).
Pipeline: Input → Output
INPUT
OUTPUT
A decoder-only transformer produces a probability distribution over the vocabulary for the next token. The model must then use a decoding strategy to decide which token to output.
Other Architectural Design Choices
Attention Mechanism
The attention mechanism acts as a dynamic spotlight, allowing the model to rank the importance of every word in a sentence simultaneously, regardless of distance.
The implementation uses a retrieval metaphor — each token asks: "what information do I need?" and fetches it from the sequence:
Q, K, V are learned linear projections of token representations:
The projection matrices WQ, WK, WV are learned during model training via backpropagation — the model discovers which projections produce the most useful attention patterns.
Self-Attention vs Cross-Attention
Attention Score Computation
The output matrix O (one row per token) encodes all contextual information — each row is a weighted sum of value vectors, capturing what each token should "know" given the full context.
Why divide by √dk?
Dividing by √dk scales the variance of the attention scores back to 1:
This is critical because without scaling, high-dimensional dot products explode in magnitude, saturating softmax into a hard one-hot distribution — which effectively zeros out the gradients and freezes the model's ability to learn through backpropagation.
Matrix View (Q, K, V and Scaled Dot Product)
Illustration adapted from Jay Alammar's The Illustrated Transformer.
Multi-Head Attention
Multiple attention heads run in parallel, each learning different relationship patterns (syntax, coreference, long-range links). Outputs are concatenated and projected back down.
Masking
Key Code: Causal Self-Attention (nanoGPT)
Source: karpathy/nanoGPT `model.py`
def forward(self, x):
B, T, C = x.size() # batch size, sequence length, embedding dimensionality (n_embd)
# calculate query, key, values for all heads in batch and move head forward to be the batch dim
q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs)
q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs)
v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs)
# causal self-attention; Self-attend: (B, nh, T, hs) x (B, nh, hs, T) -> (B, nh, T, T)
if self.flash:
# efficient attention using Flash Attention CUDA kernels
y = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=self.dropout if self.training else 0, is_causal=True)
else:
# manual implementation of attention
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))
att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))
att = F.softmax(att, dim=-1)
att = self.attn_dropout(att)
y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs)
y = y.transpose(1, 2).contiguous().view(B, T, C) # re-assemble all head outputs side by side
# output projection
y = self.resid_dropout(self.c_proj(y))
return yMLP / Feed-Forward Network (FFN)
What it is
After each attention layer, a feed-forward network (FFN) is applied independently to each token's representation. It transforms the contextual information in the attention output matrix into the model's final probability distribution over all tokens in the vocabulary.
Why is it added?