Start here
Transformer Architecture in plain language
Self-attention lets a representation incorporate information from permitted positions in the input. The permitted positions depend on the architecture and masking: a causal language model cannot use later tokens when predicting the next token.
The original transformer used an encoder and decoder for translation. Later models can use encoder-only, decoder-only or encoder–decoder designs for different tasks.
How it works
Represent tokens and position
Convert input tokens into vectors and provide a way to account for their positions.
Combine context through attention
Attention weights determine how information from available positions contributes to a representation.
Transform and produce task output
Layers also apply other transformations. The output head and training objective determine the task.
Illustrative example · Fictional data
The same word in two fictional sentences
- Input
- “The bank approved the request” and “They sat beside the river bank.”
- Process
- Represent the word in the context of the surrounding sentence rather than assigning it one fixed interpretation for every occurrence.
- Expected output
- Context-sensitive representations that can support different interpretations in downstream tasks.
- How to check it
- This illustrates the purpose of contextual representation; it is not a measured attention map or proof that a particular model understands either sentence correctly.
Where it helps
- Language modelling and translation.
- Context-sensitive text representations.
- Architectures adapted for visual and other sequence-like inputs.
Encoder, decoder and encoder–decoder
- Encoder
- Builds input representations, often using context from both directions.
- Decoder-only
- Commonly generates sequentially with causal masking.
- Encoder–decoder
- Encodes a source and generates a corresponding output sequence.
Multi-head attention and feed-forward layers
Attention computes relationships between token representations through query, key and value projections. Multiple heads can learn different useful relationships; their outputs are combined. It is not safe to assume a particular head always corresponds to one human-readable concept such as grammar.
A transformer block also includes a position-wise feed-forward network, residual connections and normalisation. Stacking blocks transforms representations repeatedly. Positional information and the attention mask affect which relationships the model can use. Attention alone is not the whole architecture.
Training, generation and image patches
During causal language-model training, many positions can contribute to the loss in parallel while masking prevents access to future tokens. During ordinary autoregressive generation, the next token depends on the generated prefix, so output is produced sequentially. Parallel training does not mean the model generates an entire answer at once.
A Vision Transformer can represent an image as a sequence of patches rather than text tokens. Position information helps preserve spatial relationships. This demonstrates that attention can operate on non-text representations; it does not establish the unpublished architecture of every commercial multimodal model.
Related reading and learning pathways
Common pitfalls
- An attention weight is not automatically a faithful explanation of a prediction.
- Longer context capacity does not guarantee every detail is used correctly.
- Different transformer variants have different computational and input constraints.
Put the idea to work
Try it yourself
Create two sentences using the same ambiguous word. Identify which surrounding words resolve its meaning, then explain why this is a language exercise rather than a measured model explanation.
Common questions
Is a transformer the same as an LLM?
No. Transformer is an architecture; many LLMs use it, but transformer models can serve other tasks. BERT, for example, is a transformer used for understanding and classification rather than open-ended generation. Vision transformers apply the same architecture to images.
Does a transformer always see future tokens?
No. Attention masks can restrict access. Causal generation uses prior context rather than future tokens. Encoder models such as BERT can attend to the whole input at once, while decoder models such as GPT generate one token at a time and only attend to earlier tokens.
Your next step
Continue learning
Connected concepts
Sources and further reading
Examples on this page are teaching exercises using fictional information. They are not client case studies or measured outcomes.