AI safety is the field concerned with preventing artificial intelligence systems from causing unintended harm and reducing the likelihood and severity of failures, accidents and misuse. The term encompasses technical, operational and governance issues across a wide range of AI systems, from conventional machine-learning applications to highly capable generative and autonomous systems. Safety concerns can include incorrect decisions, unsafe physical actions, privacy violations, cybersecurity vulnerabilities, unreliable automation, harmful outputs, model misuse and failures of human oversight.
AI safety is broader than simply preventing malicious use because an AI system can cause harm through ordinary operation if it behaves incorrectly or is deployed in an unsuitable environment. The appropriate safety measures depend heavily on the system's capabilities, autonomy, domain, access to resources and potential consequences of failure.
AI safety includes several different classes of risk. Accidental or reliability failures occur when a system does not perform as intended, such as a perception system misclassifying an object or an automated agent executing the wrong workflow. Misuse involves people deliberately using an AI system to cause harm, evade controls or conduct abusive activities. Security risks involve attacks against models or their surrounding infrastructure, including prompt injection, data poisoning, model extraction and compromised tools.
Privacy risks arise when systems expose or infer sensitive information. Physical-safety risks are especially important for robotics, autonomous vehicles, industrial control and medical systems. At a broader level, safety research can also examine risks associated with highly capable systems that operate autonomously or interact with critical infrastructure. These categories overlap but require different mitigation strategies.
Reliability engineering is a major foundation of AI safety. Developers can test systems against representative inputs, edge cases, adversarial conditions and known failure modes. Red teaming involves deliberately attempting to expose weaknesses, while stress testing examines behaviour under unusual or degraded conditions. Monitoring can detect anomalous outputs or actions after deployment. Incident response processes allow organisations to investigate failures and prevent recurrence.
Evaluation should measure not only average performance but also tail risks and catastrophic or high-impact failure modes where appropriate. For safety-critical applications, independent validation, redundancy, fail-safe behaviour and formal methods may be appropriate. The required level of assurance should be proportional to the potential consequences of failure rather than simply to the sophistication of the underlying AI model.
AI safety also concerns security because AI systems often operate as components of larger software systems. A language model connected to a database, shell, browser or financial API can create risks that do not exist when the same model is restricted to producing text. Least-privilege access limits what an AI system can do if it behaves incorrectly or is compromised. Sandboxing can isolate potentially dangerous operations, while approval workflows can require human confirmation before consequential actions.
Input and output validation can constrain data passed between components. Audit logs provide records for investigating actions. Secrets should not be exposed to models unnecessarily, and credentials should be scoped to the minimum permissions required. These are conventional security principles applied to AI-enabled architectures, demonstrating that AI safety is not solely a model-training problem.
Human oversight is another major component. A human-in-the-loop system requires a person to approve or participate in decisions, while human-on-the-loop arrangements may allow automated operation with monitoring and intervention capabilities. Human oversight is not automatically effective: people can become over-reliant on automated recommendations, misunderstand model limitations or fail to notice rare errors. Interface design should therefore communicate uncertainty and provide sufficient information for meaningful review. Automation bias, alert fatigue and workload are relevant operational risks.
In some domains, the safest design may involve limiting automation rather than simply adding a human approval step after the AI has already made a difficult decision. Safety engineering consequently considers the complete human-machine system rather than treating the human as an infallible final checkpoint.
AI safety differs from AI alignment, although the concepts overlap. Alignment focuses on whether system behaviour corresponds with intended objectives, whereas safety includes preventing and managing harmful outcomes even when the intended objective itself is clear. A system might be aligned with a user's stated goal but still be unsafe if the goal is carried out through a dangerous mechanism.
Safety also differs from AI ethics: ethics asks broader normative questions about acceptable behaviour and social consequences, while safety focuses more directly on preventing or controlling harmful failures and risks. Responsible AI encompasses safety along with governance concerns such as accountability, fairness and transparency. These distinctions are useful because a complete AI governance programme normally requires multiple perspectives rather than treating every AI risk as one category.
Safety practices vary according to application. A low-risk writing assistant may primarily require privacy protections, abuse controls and accuracy safeguards, whereas an AI system controlling industrial equipment may require rigorous testing, redundancy, deterministic interlocks and fail-safe mechanisms. A medical AI system may need clinical validation and careful monitoring because errors can affect patient care. An enterprise agent with access to confidential files requires strong identity, access-control and audit mechanisms. Safety evaluation should therefore consider foreseeable misuse, operating environment, users, dependencies, failure consequences and recovery mechanisms. The model itself is only one component. Data pipelines, prompts, retrieval systems, APIs, software dependencies, user interfaces, deployment infrastructure and organisational procedures can all create or mitigate risk.
AI safety is therefore a multidisciplinary discipline combining machine learning, software engineering, cybersecurity, reliability engineering, human factors, risk management and governance. Its purpose is not to make AI systems incapable of useful action but to ensure that their capabilities operate within acceptable risk boundaries. Effective safety programmes identify hazards, estimate potential impact, test for failure modes, limit permissions, monitor operation, provide recovery mechanisms and update controls as systems and environments change.
As AI systems become more capable and autonomous, the distinction between a model and the surrounding system becomes increasingly important. A safe deployment is not established merely by demonstrating that a model performs well on a benchmark; it requires evidence that the complete system behaves acceptably under normal, unusual, adversarial and failure conditions relevant to its intended use.
