A transformer model is a neural-network architecture designed to process sequences by using attention mechanisms to determine how different elements of the sequence relate to one another. Transformers were introduced in the 2017 research paper “Attention Is All You Need” by Vaswani and colleagues, which demonstrated that sequence modelling could be performed without the recurrent and convolutional structures that had dominated many earlier natural-language-processing systems.
A transformer typically represents input elements as vectors, adds information about their positions, processes them through repeated layers containing attention and feed-forward transformations, and produces contextual representations or predictions. The architecture became particularly important in natural language processing because it can model relationships between words or tokens that are far apart in a sequence while also allowing substantial parallelisation during training.
The core computational operation in a transformer is self-attention. Given representations for a sequence, the model creates query, key, and value vectors and uses relationships between queries and keys to determine how strongly each position should incorporate information from other positions. The resulting weighted combinations of value vectors produce contextualised representations. Multi-head attention repeats this process through multiple learned projections, allowing different heads to capture different relationships or patterns. For example, one attention head may become sensitive to syntactic relationships while another may capture longer-range associations, although the precise interpretation of individual heads is model-dependent and should not automatically be treated as a direct linguistic explanation.
A transformer layer normally contains more than attention. In the original architecture, attention is followed by a position-wise feed-forward network, with residual connections and layer normalisation used to support optimisation and information flow. Modern transformer implementations vary considerably, but commonly retain some combination of these fundamental components. The feed-forward portion applies learned nonlinear transformations independently to each position after attention has mixed information across positions. Residual connections allow information to pass through layers more directly, while normalisation helps maintain numerically stable representations during deep computation. Stacking many layers allows increasingly complex transformations to be learned.
Transformers can be organised according to how their attention and output mechanisms are used. Encoder-only architectures, such as BERT-style models, are commonly suited to understanding or representation tasks because they can process contextual information from both directions. Decoder-only architectures, such as GPT-style language models, generate tokens sequentially and are widely used for text generation. Encoder-decoder architectures, exemplified by the original Transformer and models such as T5, use an encoder to represent an input sequence and a decoder to generate an output sequence. These categories are architectural patterns rather than absolute limitations, and modern models can introduce specialised variations.
Position information is necessary because ordinary self-attention does not inherently encode the order in which tokens occur. Transformer systems therefore use positional information in some form. The original Transformer used sinusoidal positional encodings, while many later models use learned positional embeddings or other mechanisms such as rotary position embeddings. The treatment of position affects how a model represents sequence order and can influence its ability to generalise to sequence lengths or positional relationships not encountered during training. The distinction between token identity and positional information is important: knowing which tokens are present is not sufficient to know their order or structural relationship.
Transformers are computationally powerful because the attention operation can connect each token with many other tokens directly, rather than requiring information to propagate through a long chain of recurrent states. During training, many sequence positions can be processed in parallel, making transformer architectures well suited to modern accelerator hardware. However, standard full self-attention has computational and memory costs that grow approximately quadratically with sequence length for the attention matrix, which creates challenges for very long contexts. Numerous techniques, including efficient attention variants, sparse attention, grouped-query attention, sliding-window approaches, and other architectural optimisations, have been developed to address different efficiency and scaling requirements.
Although transformers became famous through language models, the architecture is not limited to text. Transformer-based systems are used for images, audio, video, multimodal data, biological sequences, code, and other domains. Vision transformers divide images into patches that can be treated as sequence elements, while multimodal architectures can combine representations from different modalities. Large language models are a prominent application because transformers can learn statistical relationships among tokens and generate or transform sequences. The same architectural principles can therefore support many sequence-like or relational modelling problems, even though implementation details differ substantially between systems.
A transformer model should not be confused with a complete AI application. The transformer is an architecture; a deployed model also requires learned parameters, training procedures, tokenisation or input representations, inference software, and often additional components such as retrieval systems, tools, safety controls, or application-specific processing. Pretraining teaches a transformer general statistical patterns from large datasets, while later stages such as supervised fine-tuning or preference optimisation can alter its behaviour for particular purposes. A transformer also does not literally store a conventional database of sentences. Its learned parameters encode statistical structure acquired during training, and its output is generated through computations over its current inputs, internal state, and learned parameters.
