Hash Functions

Technology — James William Steven Parker

Hash Functions are mathematical functions that accept input of arbitrary or practically variable length and produce a fixed-length output known as a hash, digest or message digest. A cryptographic hash function is designed so that it is computationally difficult to reverse the digest to recover an appropriate original input, find a second input producing the same digest as a given input, or find any two distinct inputs producing the same digest, subject to the function's security definition.

Important properties include preimage resistance, second-preimage resistance and collision resistance. Cryptographic hashes are therefore fundamentally different from encryption: encryption is designed to be reversible by an authorised party with a key, whereas a cryptographic hash is normally designed as a one-way transformation without a decryption key.

Common cryptographic hash functions include SHA-256 and SHA-3, while older functions such as MD5 and SHA-1 have significant collision weaknesses and should not be selected for modern security-critical applications. SHA-256 produces a 256-bit digest, while SHA-3 is based on the Keccak sponge construction rather than the Merkle-Damgård architecture used by SHA-2. The output size influences generic collision resistance: for an ideal n-bit hash, a collision can theoretically be expected after roughly 2^(n/2) operations because of the birthday phenomenon, whereas generic preimage search requires roughly 2^n operations. These are theoretical generic bounds rather than direct predictions of practical attacks against a particular implementation.

Hash functions are widely used for integrity verification because a change to the input normally produces a substantially different digest. Software publishers can publish a known cryptographic hash of a file, allowing users to calculate the digest independently and compare it with the expected value. Hashes are also fundamental to digital signatures because signing an entire large message directly is usually unnecessary; systems can hash the message and sign the resulting digest using an appropriate signature scheme. Merkle trees use hashes recursively to represent large collections of data efficiently and allow proofs that individual records belong to a committed dataset. Version-control systems, content-addressable storage and distributed systems likewise use hashes to identify or link data structures.

Hashing is commonly used for passwords, but simply applying a fast general-purpose hash to a password is inadequate. Passwords have low entropy and attackers can test huge numbers of candidate passwords, so password storage should use a deliberately expensive password-hashing or password-based key-derivation algorithm such as Argon2id, scrypt or an appropriately configured bcrypt implementation. A unique salt should be generated for each password and stored with the resulting password verifier.

The salt prevents identical passwords from producing identical stored values and makes large precomputed lookup tables substantially less useful. Work factors can be tuned to make offline guessing more expensive. Hashing a password with SHA-256 alone does not provide these memory-hard or deliberately costly properties.

Hash functions are also used inside message authentication, but a plain hash is not automatically an authentication mechanism. HMAC combines a cryptographic hash with a secret key to provide message authentication under an established security model. Similarly, a hash of a downloaded file proves little about authenticity if an attacker can replace both the file and the published hash. Authenticity requires a trusted distribution channel, digital signature or other authenticated mechanism.

A hash therefore provides a compact representation with useful integrity properties, but the security meaning of that representation depends on how the digest is distributed, stored and authenticated. This distinction explains why a checksum obtained from the same untrusted source as the downloaded file may fail to protect against deliberate substitution.

Hash functions are deterministic: identical inputs produce identical outputs, and even a one-bit change to the input is normally expected to produce a substantially different digest because of the avalanche effect. They do not provide confidentiality because an attacker can calculate the hash of any known or guessed input. This makes them unsuitable for hiding small or predictable secrets such as short PINs without additional password-specific protection. Hashes are also not compression algorithms in the ordinary data-storage sense, even though their fixed output can be much shorter than the input. They are better understood as cryptographic fingerprints with formally defined resistance properties rather than reversible data transformations.

Hash functions play major roles in distributed and cryptographic protocols. Merkle trees allow systems to commit to large sets of transactions or records using a single root hash, while inclusion proofs allow a verifier to check membership without receiving the entire dataset. Blockchains use hashes extensively for transaction identifiers, block linking, Merkle roots and proof-of-work constructions, although the security of an entire blockchain does not arise from hashing alone. Hash-based constructions also appear in pseudorandom generators, key derivation, commitment schemes and signature algorithms. Their versatility comes from the ability to compress arbitrary structured information into a fixed-size value while making certain forms of deliberate manipulation computationally difficult.

Hash functions have limitations related to collisions, algorithm ageing and misuse. No fixed-length hash can be mathematically collision-free for arbitrary inputs because infinitely many possible inputs must map into a finite output space. Security therefore depends on making useful collisions computationally infeasible. Once a practical collision attack becomes available, protocols relying on collision resistance may need migration even if preimage resistance remains comparatively strong. The SHA-1 collision demonstrations illustrate this distinction: an algorithm can remain difficult to invert while no longer providing adequate collision security. Choosing modern standardised functions, adequate digest lengths and protocol constructions appropriate to the threat model is therefore essential.