GLOSSARY
Attention
The mechanism that lets a model decide which parts of the input matter for each output — the single idea that made modern AI possible.
Attention answers a question at every step: “of everything I've seen, what matters right now?” Reading the word “bank” in “river bank”, attention pulls in “river” and deprioritizes “finance”. Every token computes relevance scores against every other token, and the model becomes a weighted blend of what it attends to.
This is why transformers handle long-range meaning, why translation remembers a gender mentioned paragraphs ago, and why context windows are measured in tokens — attention's cost grows with every pair. Its variants (multi-head, grouped-query, flash attention) are the plumbing optimizations behind inference engines like vLLM.