Parallel Computing is the execution of multiple computational operations concurrently using multiple processing resources. The objective may be to reduce the time required to solve a problem, increase throughput, process larger workloads or make efficient use of available hardware. Parallelism can occur at several levels, including instruction-level, data-level, task-level and thread-level parallelism. Systems may use multiple CPU cores, vector units, graphics processors, specialised accelerators or collections of processors. Parallel Computing is especially effective when a problem can be divided into relatively independent portions that can execute simultaneously. It is therefore fundamental to scientific simulation, numerical analysis, image processing, machine learning, engineering, cryptography, rendering and many high-performance computing applications.
Parallel computation has a long history, but its importance increased as simply raising processor clock frequency became progressively more constrained by power dissipation and physical limitations. Vector and array-processing systems provided early large-scale parallel capabilities, followed by shared-memory multiprocessors and massively parallel systems. Graphics processors evolved from specialised graphics hardware into highly programmable parallel processors suitable for general-purpose workloads. Multicore CPUs made parallel execution commonplace in ordinary computers, while clusters and supercomputers combined large numbers of processors. Modern processor architectures exploit parallelism internally through pipelines, superscalar execution and vector instructions even when programmers write sequential-looking code. Consequently, parallelism exists at both hardware and software levels and is not restricted to specialised supercomputers.
A key distinction is between data parallelism and task parallelism. Data parallelism applies the same operation to many independent data elements, making it well suited to vector processors and GPUs. For example, applying a mathematical transformation independently to millions of image pixels can expose substantial data parallelism. Task parallelism divides a workload into different operations that can execute concurrently, such as processing independent stages or services.
Instruction-level parallelism allows hardware to overlap or execute independent machine instructions, while thread-level parallelism allows multiple software execution contexts to run concurrently. These forms can coexist. A program might use multiple CPU threads, each processing a portion of a dataset using vector instructions, while specialised accelerators perform additional parallel operations.
Shared-memory and distributed-memory parallel systems use different programming and communication models. In shared-memory systems, multiple processors or cores access a common logical address space, although caches and memory hierarchies introduce important performance and consistency considerations. Programming models such as threads, OpenMP and related abstractions can express shared-memory parallelism. Distributed-memory systems provide separate memory spaces for different processing nodes and require explicit communication, commonly through message-passing systems such as MPI. Hybrid systems combine both approaches, for example using multiple processes across nodes and multiple threads within each node. The choice affects programming complexity, communication overhead, scalability and the types of algorithms that can be implemented efficiently.
Parallel algorithms must address dependencies and synchronisation. If an operation depends on the result of another operation, the dependent work cannot always execute independently. Synchronisation mechanisms establish points or conditions at which execution must coordinate, but excessive synchronisation can reduce performance. Communication can also become a bottleneck when processors spend more time exchanging data than performing useful computation. Load balancing is important because if one processing resource receives substantially more work than others, the faster resources may remain idle. Algorithms therefore need to expose sufficient independent work while limiting dependencies, communication and coordination. Data locality is also significant because accessing local cache or memory is generally cheaper than repeatedly transferring data across slower memory or interconnect paths.
Amdahl's law illustrates a fundamental limitation of parallel speedup. If a fraction of a program is inherently sequential, increasing the number of parallel processors cannot accelerate that sequential fraction indefinitely. Gustafson's law provides another perspective by considering how larger problem sizes can make parallel computation more useful. Real performance is additionally affected by communication overhead, synchronisation, scheduling, memory bandwidth, contention, load imbalance and hardware utilisation. Consequently, doubling the number of processing units rarely guarantees a doubling of performance. Efficient parallel software requires algorithms and implementations designed around the characteristics of the target architecture, workload and data movement patterns. Measuring actual scaling behaviour is more reliable than assuming ideal linear speedup.
GPUs have become important parallel-computing platforms because their architectures contain large numbers of arithmetic execution resources designed for workloads with substantial regular parallelism. Programming frameworks and interfaces allow developers to launch large numbers of lightweight computational threads or work items. GPUs are highly effective for workloads such as matrix operations, graphics, scientific calculations and machine learning, but they are not universally faster than CPUs. Irregular control flow, small workloads, frequent synchronisation and data-transfer overhead can reduce GPU benefits. Specialised accelerators can similarly provide major improvements for targeted algorithms while being less flexible than general-purpose processors. Choosing an accelerator therefore requires understanding both the computational structure of the workload and the cost of moving data to and from the accelerator.
Parallel Computing differs from Distributed Computing primarily in emphasis. Parallel Computing concerns simultaneous execution and performance through multiple computational resources, whereas Distributed Computing concerns cooperation among autonomous networked machines and must explicitly address independent failures and communication uncertainty. A parallel program can run entirely within one multicore computer, while a distributed system may perform operations sequentially on separate machines. The two can also overlap, such as a distributed supercomputer in which each node performs parallel computation internally. Modern high-performance computing frequently combines CPU cores, GPUs, high-speed interconnects, distributed memory and parallel algorithms. Effective systems therefore require consideration of algorithmic decomposition, processor architecture, memory hierarchy, communication, synchronisation and workload scaling simultaneously.
