A token in AI is a discrete unit of input or output used by a model to represent text or, in some systems, another type of data. In language models, a token is not necessarily equivalent to a word. Depending on the tokenisation method, a token can represent an entire word, part of a word, punctuation, whitespace-related information, a number, a symbol, or another fragment of text. Tokenisation converts raw text into a sequence of token identifiers, and a model processes those identifiers through learned representations. For example, a common subword tokenizer may represent an uncommon word as several pieces rather than requiring a separate vocabulary entry for every possible word.
Tokenisation is necessary because neural models do not normally consume arbitrary strings directly. A tokenizer defines a vocabulary and an algorithm for mapping text to vocabulary entries, which are usually represented by integer IDs. Subword tokenisation approaches such as Byte Pair Encoding, WordPiece, and SentencePiece-based methods have become common because they provide a practical compromise between vocabulary size and the ability to represent unfamiliar words. Character-level systems use much smaller units, while word-level tokenisation uses larger units. The exact token boundaries therefore depend on the tokenizer associated with a particular model, meaning that the same text can have different token counts under different models.
A token has an important relationship with an embedding. After tokenisation, token IDs are typically mapped through an embedding matrix into numerical vectors. The model then transforms these vectors through its neural-network layers. Consequently, the token ID itself is essentially a symbolic index rather than a semantic numerical representation. The embedding supplies a learned continuous representation used by the neural network. Positional information is generally incorporated separately or through an architectural mechanism so that the model can distinguish the order of tokens. The sequence processed by a transformer can therefore be understood as a series of token representations enriched with positional information and repeatedly transformed by the network.
Token counts matter operationally because language models have finite context limits and because many APIs or model interfaces price usage according to processed tokens. Input tokens consume part of the model's available context, while generated output tokens consume additional capacity. Tokenisation also affects computational cost because longer token sequences require more processing. A word split into several subword tokens occupies more sequence positions than a word represented by one token. Languages, writing systems, formatting, source code, punctuation, and unusual strings can therefore have substantially different tokenisation characteristics. Token count should not be assumed to equal word count.
Special tokens are vocabulary entries or token-like markers used for structural or control purposes. Examples can include beginning-of-sequence, end-of-sequence, padding, separator, classification, or instruction-related markers, depending on the model and tokenizer. Chat-oriented systems may use specialised representations for roles or message boundaries, although the exact mechanism is model-specific and not necessarily visible to the user. These markers can influence how the model interprets the structure of its input. Developers must therefore use the tokenizer and formatting conventions expected by the specific model rather than assuming that ordinary text alone determines the complete token sequence.
Tokens are also used outside ordinary natural-language text. Code is tokenised into units representing identifiers, operators, punctuation, literals, and other fragments. Multimodal models can represent images, audio, or other inputs through specialised tokens or token-like latent representations, although the internal mechanisms differ from ordinary text tokenisation. An image may be divided into patches or encoded into learned discrete or continuous representations before being processed alongside text. Consequently, “token” is best understood as a model-specific unit of representation or processing rather than a universal synonym for a word.
Tokenisation creates several practical consequences. Text containing rare names, URLs, long identifiers, source code, numerical strings, or unusual Unicode sequences may require many tokens. Whitespace and punctuation can sometimes affect token boundaries. Case can also matter because tokenizers may distinguish differently cased forms. Developers who need accurate token budgets should therefore use the tokenizer associated with the target model rather than estimating from characters or words alone. Tokenisation can also affect model behaviour because the granularity of the input representation influences which patterns are easy or difficult for the network to learn.
A common misconception is that a token always corresponds to approximately one word or a fixed fraction of a word. In reality, tokenisation is vocabulary- and algorithm-dependent. Another misconception is that tokens themselves contain the meaning of words. The token ID is merely an index; meaning emerges from learned representations and transformations in the model. Tokens also should not be confused with embeddings: a token is a discrete symbolic unit, whereas an embedding is a continuous numerical vector associated with a token or other object. Understanding this distinction is essential when reasoning about context windows, model inputs, inference costs, and how language models convert human-readable text into numerical computations.
