AI inference is the process of using a trained or otherwise parameterised artificial-intelligence model to produce an output from an input. During inference, the model's learned parameters are normally held fixed rather than being updated through training. An input might be an image, text prompt, audio recording, sensor reading, structured record or sequence of tokens, and the output could be a classification, numerical prediction, generated text, image, recommendation, embedding, detection result or other model-defined response.
In a neural network, inference generally involves applying learned weights and biases through a sequence of computational operations to calculate an output. The term therefore describes model execution for prediction or generation, in contrast to training, which adjusts model parameters based on data and an optimisation objective. In practical AI systems, inference also includes surrounding operations such as input preprocessing, tokenisation, model execution, decoding or post-processing, depending on the application.
Training and inference have different computational purposes. Training normally requires repeated forward passes, calculation of a loss against a learning objective, gradient computation through backpropagation and parameter updates. Inference generally requires a forward computation without updating the model's learned parameters. This makes inference typically less computationally demanding per example than training, although large generative models can make inference extremely expensive because each generated token may require substantial computation and because long contexts increase memory and processing requirements.
Training may occur periodically, while inference can happen millions or billions of times after a model has been deployed. This distinction has major infrastructure implications: training systems prioritise sustained high-throughput optimisation and large-scale dataset processing, while inference systems often prioritise latency, throughput, cost efficiency, availability, memory usage and predictable response times.
AI inference can be deterministic or probabilistic depending on the model and execution settings. A conventional classifier may return a probability distribution over classes and a selected prediction, while a generative language model produces probability distributions over possible next tokens and a decoding procedure chooses which tokens to generate. Sampling temperature, top-k or top-p methods and random seeds can affect generative outputs.
Some models can produce different outputs for identical inputs when stochastic sampling is enabled, while deterministic decoding can produce repeatable results under sufficiently controlled conditions. The model's output probabilities should not automatically be interpreted as calibrated real-world probabilities: a model can be highly confident while being wrong. Inference therefore produces model outputs, not guaranteed facts or truths. Downstream applications may require confidence thresholds, abstention mechanisms, validation rules, retrieval systems or human review depending on the consequences of errors.
Inference pipelines commonly include several stages. Raw input may first undergo validation and preprocessing, such as resizing an image, normalising numerical features, converting speech to an appropriate representation or tokenising text. The resulting representation is passed through the model. The model output may then require post-processing, such as converting class probabilities into labels, applying non-maximum suppression to object detections, decoding generated tokens into text, or transforming embeddings into a similarity search operation.
A production inference service may additionally perform authentication, rate limiting, batching, caching, logging, monitoring and resource allocation. Consequently, the latency experienced by a user is not necessarily identical to the neural network's raw computation time. Network transmission, preprocessing, queueing, model loading, database access and post-processing can all contribute to end-to-end inference latency.
Inference can run in many environments. Cloud inference servers can provide large models through APIs, while edge inference executes models on local devices such as phones, cameras, vehicles, industrial controllers or embedded computers. Edge inference can reduce network latency and avoid transferring raw data to a remote service, but hardware constraints may require smaller models, quantisation, pruning or specialised accelerators. Cloud systems can support larger models and centralised updates but introduce network dependency, operating costs and potentially additional privacy considerations.
Some systems use hybrid architectures in which a local model performs initial processing and a remote service handles more complex tasks. Hardware such as GPUs, TPUs, NPUs and other accelerators can substantially affect inference throughput and latency, especially for neural networks involving large matrix operations.
Inference performance is often measured using latency, throughput, resource utilisation and cost. Latency describes the time required to produce a result, while throughput describes how many requests or examples can be processed per unit of time. Batch inference processes many examples together and is common in data pipelines where immediate responses are unnecessary. Online or real-time inference processes requests as they arrive and usually has stricter latency requirements.
Large language-model inference introduces additional measures such as time to first token and token-generation throughput. Memory bandwidth, model size, numerical precision, cache behaviour and sequence length can all influence performance. Quantisation reduces numerical precision to decrease memory use and potentially accelerate computation, but excessive quantisation can reduce model quality. Efficient inference therefore requires balancing accuracy, latency, memory and cost rather than optimising one metric in isolation.
Inference can also expose security and reliability risks. An attacker may deliberately construct inputs that cause a model to misclassify an example, exploit weaknesses in an AI application, overload an inference service or attempt to extract information about the model. Generative systems can produce incorrect, fabricated or unsafe outputs even when the underlying model is functioning according to its learned distribution.
Monitoring can track input distributions, error rates, latency, resource consumption and output characteristics, while application-level controls can constrain what the model is permitted to do. For high-impact applications, inference may need human review, deterministic business rules or independent validation rather than allowing model outputs to directly trigger consequential actions. A model's inference-time behaviour should therefore be evaluated in the complete application context rather than treated as an isolated mathematical operation.
AI inference is ultimately the operational use of learned model parameters to calculate predictions or generate outputs. It is distinct from training, although modern systems can blur the boundary through online learning, continual learning or retrieval systems that modify context or models between uses. Retrieval-augmented generation, for example, may perform inference using both a trained language model and externally retrieved documents; the retrieval step is not itself model training.
Inference can be performed locally or remotely, synchronously or in batches, with deterministic or stochastic decoding, and on general-purpose processors or specialised accelerators. The quality and usefulness of inference depend on the trained model, input distribution, preprocessing, runtime configuration, decoding strategy and surrounding application controls. A reliable AI system therefore treats inference as an engineered production process rather than simply the act of “asking the model a question”.
