Natural Language Processing is the field of artificial intelligence, computer science and computational linguistics concerned with enabling computers to analyse, represent, understand, transform and generate human language. Human language includes written and spoken communication, and Natural Language Processing encompasses methods for handling both. The field includes tasks such as tokenisation, part-of-speech analysis, syntactic parsing, named-entity recognition, information extraction, sentiment analysis, question answering, machine translation, summarisation, speech-related language processing and text generation. Natural Language Processing combines linguistic knowledge with statistical modelling, machine learning and computational methods. It is broader than any particular architecture, so systems based on rules, statistical models, neural networks, Transformer Models or Large Language Models can all belong to Natural Language Processing.
Language presents distinctive computational challenges because meaning depends on syntax, context, ambiguity, world knowledge, discourse and social conventions. A word can have several meanings, and the intended meaning may depend on surrounding text. Pronouns can refer to entities mentioned earlier, while the interpretation of a sentence can depend on information outside the sentence itself. Natural Language Processing systems therefore need representations capable of capturing relationships at multiple levels, including characters or subword units, words, phrases, sentences and documents. Spoken language adds further complications such as pronunciation variation, accents, background noise, disfluencies and timing. These properties make language substantially more complex than treating text as a simple sequence of independent symbols.
The history of Natural Language Processing includes rule-based, statistical and neural approaches. Early systems frequently relied on manually written grammars, dictionaries, semantic rules and domain-specific knowledge. Statistical approaches subsequently used probabilities estimated from corpora, enabling systems to learn language patterns from data rather than requiring every rule to be explicitly encoded. Machine-learning methods improved tasks such as tagging, parsing, classification and translation.
Neural networks then introduced learned distributed representations, recurrent architectures improved sequence modelling, and Transformer Models enabled highly scalable contextual processing. Large Language Models have subsequently become prominent because broad pretraining can support many language tasks through a common model. Modern Natural Language Processing therefore incorporates techniques from several historical paradigms rather than replacing all earlier methods with one universal approach.
Tokenisation is a foundational Natural Language Processing operation because computational systems need a representation of linguistic input that can be processed by algorithms. Traditional tokenisation often separated text into words and punctuation, but modern neural systems frequently use subword tokenisation, allowing uncommon words to be decomposed into smaller units. Stemming reduces words to approximate roots, while lemmatisation attempts to map inflected forms to dictionary-like base forms. Part-of-speech tagging assigns grammatical categories such as noun, verb or adjective. Syntactic parsing represents grammatical relationships, while named-entity recognition identifies entities such as people, organisations, places or dates. These tasks can be performed independently or as components of larger models, and modern end-to-end systems may learn several capabilities jointly without explicitly exposing every intermediate representation.
Natural Language Processing supports a wide range of applications. Search systems can analyse queries and documents to improve retrieval. Machine translation converts content between languages. Summarisation produces shorter representations of longer documents. Information extraction converts unstructured text into structured records. Question-answering systems identify or generate responses to queries. Text classification can categorise messages, documents or customer requests. Speech technologies combine language processing with acoustic processing for transcription and spoken interaction. Sentiment and opinion analysis can identify linguistic patterns associated with expressed attitudes, although inferring actual beliefs or intentions from text is more difficult. Writing assistance, accessibility technologies, document processing and conversational interfaces are additional application areas.
Modern Natural Language Processing increasingly uses representation learning, where models learn numerical representations that encode useful linguistic relationships from large datasets. Contextual representations allow the representation of a word or token to vary according to its surrounding context, helping distinguish meanings that cannot be determined from the token alone. Transformer-based models can represent relationships across long portions of text and can be pretrained on broad corpora before being adapted to specific tasks.
Retrieval-augmented systems can combine language modelling with external documents so that responses can be grounded in a specified information source. Tool-integrated systems can also connect language understanding to deterministic computation, databases or software execution. These developments have expanded Natural Language Processing from narrowly defined linguistic pipelines towards more general language-oriented systems.
Evaluation in Natural Language Processing is complicated because language quality is multidimensional. A translation can preserve literal meaning while sounding unnatural, or sound fluent while introducing a factual error. A summarisation system can produce grammatically correct text that omits an important qualification. A question-answering system can produce a plausible answer that is unsupported by its source. Automated metrics can measure lexical overlap, similarity or task-specific properties, but they do not capture every aspect of meaning, factuality, usefulness or social appropriateness. Evaluation should therefore use task-relevant datasets, human assessment where appropriate and tests designed around realistic failure modes. Language technologies can also encode demographic and cultural biases present in training data, making dataset composition and fairness evaluation important considerations.
Natural Language Processing has limitations arising from the difference between linguistic form and underlying meaning. A system can detect statistical patterns in language without possessing the same contextual experience or grounding as a human. Ambiguity, sarcasm, irony, implicit references and specialised terminology can remain difficult. Models can also reproduce errors or biases from their data and may generate fluent but unsupported statements.
For sensitive applications, confidentiality and data protection are important because language inputs can contain personal, commercial or otherwise restricted information. Security concerns include malicious instructions embedded in retrieved documents or user content, especially when Natural Language Processing systems are connected to external tools. Robust deployment therefore requires appropriate validation, access controls and monitoring in addition to linguistic performance.
