An attention mechanism is a neural-network operation that allows a model to assign different levels of importance to different parts of its available input when computing a representation or prediction. Instead of treating every input element as equally relevant, attention calculates learned relationships between elements and uses those relationships to construct weighted combinations of information. In transformer models, attention is central to how tokens interact with one another. The mechanism is particularly useful for sequences because the meaning or interpretation of one element can depend strongly on other elements, including elements that occur far away. Attention therefore provides a flexible method for dynamically selecting and combining contextual information.
In transformer self-attention, each input representation is transformed into a query, a key, and a value. Queries represent what a position is seeking in the current computation, keys provide representations against which queries are compared, and values contain the information that is combined after those comparisons. A similarity calculation, commonly a scaled dot product, produces attention scores between queries and keys. These scores are normalised, usually using softmax, to form weights. The weighted values are then summed to produce the attention output. In simplified form, scaled dot-product attention is commonly represented as Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V, where dₖ is the key-vector dimensionality.
Self-attention means that the queries, keys, and values originate from the same sequence or representation set. This allows each token to incorporate information from other tokens in the same input. For example, in a sentence containing an ambiguous pronoun, useful contextual information may occur several words earlier. Self-attention can directly establish a relationship between the pronoun and relevant surrounding tokens rather than requiring information to pass sequentially through every intermediate position. The mechanism does not explicitly encode human grammatical rules; rather, its learned weights arise from optimisation during training and represent statistical relationships useful for the model's objectives.
Cross-attention is a related mechanism in which queries come from one representation sequence while keys and values come from another. It is common in encoder-decoder transformer architectures. During machine translation, for example, a decoder position can use cross-attention to access representations produced by the encoder for the source sentence. Cross-attention is also useful in multimodal systems, where one modality can attend to representations of another modality. The distinction between self-attention and cross-attention therefore concerns the source of the query, key, and value representations and the interaction being modelled.
Multi-head attention performs several attention calculations using different learned projections and then combines their outputs. The motivation is that a single attention operation may have limited representational flexibility, whereas multiple heads can process different relational patterns simultaneously. Each head has its own query, key, and value projections, although the heads operate within the same overall layer. The outputs are concatenated or otherwise combined and passed through another learned projection. Multi-head attention does not guarantee that each head corresponds neatly to one interpretable concept; learned representations can be distributed and different heads can have overlapping or specialised functions depending on the model.
Attention has important computational characteristics. Full self-attention constructs relationships among all relevant positions, so the number of pairwise interactions increases rapidly as sequence length grows. For a sequence of length n, the conventional attention matrix contains n² position-to-position scores. This contributes to the memory and computational cost of long-context processing. Various efficient-attention approaches reduce the number of interactions, approximate attention, restrict attention to local or selected positions, or optimise the implementation. These methods can improve scalability, but they can also alter which dependencies are directly accessible and may involve trade-offs between efficiency, capacity, and quality.
Attention should not be interpreted simply as a human-like spotlight indicating exactly what a model “is thinking about”. Attention weights are mathematically defined intermediate values, and their interpretation as explanations of model reasoning is an active area of research. Multiple internal mechanisms can contribute to an output, and attention weights alone may not fully explain causal influence. Nevertheless, attention is a genuine computational mechanism that determines how representations are combined. Its importance comes from its ability to construct context-dependent representations dynamically rather than applying the same fixed combination of surrounding information to every input.
Attention mechanisms predate transformers and were used in earlier sequence-to-sequence neural networks, particularly to address limitations of compressing an entire input sequence into a single fixed-size representation. Transformers made attention the principal mechanism for connecting sequence positions and removed the need for recurrence in the original architecture. Today, attention appears in language models, vision models, speech systems, multimodal architectures, retrieval systems, and other neural networks. Related concepts include self-attention, cross-attention, causal attention, bidirectional attention, masked attention, sparse attention, and linear or approximate attention, each describing different restrictions or computational formulations of the basic idea.
