Statistical significance is a concept used in statistical hypothesis testing to describe whether observed data would be considered sufficiently incompatible with a specified null hypothesis according to a predefined statistical decision rule. A common approach uses a significance level, such as 0.05, and calculates a p-value representing the probability, under the assumptions of the null hypothesis and statistical model, of obtaining a result at least as extreme as the observed result.
If the p-value is below the chosen significance level, the result is described as statistically significant under that procedure. Statistical significance is therefore a property of an inference relative to a model, hypothesis, dataset, test statistic, and decision threshold; it is not an intrinsic property of a finding. The term does not mean that an effect is large, important, practically useful, or certainly real.
Statistical significance is often misunderstood because a p-value does not represent the probability that the null hypothesis is true. A p-value is calculated conditional on the null hypothesis and other model assumptions, not conditional on the hypothesis being true or false. Likewise, a p-value of 0.03 does not mean there is a 3% probability that the observed effect occurred by chance in the ordinary-language sense.
It means that, under the specified null model, results at least as extreme as the observed result have a probability of approximately 0.03 according to the test's assumptions and definition. Interpretation also depends on whether the statistical test was appropriate, whether assumptions were satisfied, whether the analysis was prespecified, and whether multiple comparisons or selective reporting affected the result.
Statistical significance is distinct from effect size. With a sufficiently large sample, a very small effect can produce a statistically significant result because larger samples generally provide more precise estimates and greater power to detect small departures from a null hypothesis. Conversely, a substantial effect can fail to reach conventional statistical significance when the sample is small, variability is high, measurements are imprecise, or the study has low statistical power.
Researchers therefore commonly report effect sizes alongside confidence intervals or other uncertainty measures. For example, an estimated difference of a fraction of a measurement unit might be statistically significant in a very large dataset while having little practical relevance, whereas a larger estimated difference in a small exploratory study may be important but imprecisely estimated.
The significance level, often denoted alpha, is selected before or independently of the observed result in conventional hypothesis testing. A threshold of 0.05 means that, under the relevant assumptions, the procedure is designed to reject the null hypothesis when the test result falls within a specified region having probability 0.05 under the null. This does not mean that 5% of significant findings are false.
The false-positive rate depends on the testing context, prior probabilities, number of hypotheses tested, analytical flexibility, study design, and other factors. If researchers test many hypotheses and report only those producing small p-values, the chance of obtaining at least one apparently significant result can become much larger than the nominal alpha level for an individual test. Multiple-testing procedures can adjust inference when many hypotheses are evaluated.
Statistical significance is also affected by statistical power. Power is the probability that a specified statistical test will reject the null hypothesis when a particular alternative condition is true. It depends on factors such as sample size, effect size, variability, significance threshold, and test design. Low-powered studies may generate unstable estimates and may fail to detect genuine effects, while repeated low-powered testing combined with selective publication can distort the apparent evidence base.
Increasing sample size generally improves precision and can increase power, but it does not automatically correct biased measurement, confounding, inappropriate modelling, or poor study design. Power calculations and sample-size planning are therefore useful components of study design, although they depend on assumptions about expected effects and variability.
Statistical significance also has a historical context. The modern use of significance testing developed through work by statisticians including Ronald Fisher and the Neyman–Pearson framework, which differ conceptually in important respects. Fisher emphasised p-values as measures of evidence against a null hypothesis, whereas Neyman–Pearson hypothesis testing formalised decision procedures involving null and alternative hypotheses, error rates, and long-run operating characteristics. Contemporary research often combines elements of these traditions, sometimes without clearly distinguishing their underlying interpretations. This history helps explain why the simple rule "p less than 0.05 equals important" is an inadequate representation of statistical inference. Different statistical frameworks, including Bayesian methods and estimation-focused approaches, can address scientific questions without relying on conventional statistical significance thresholds.
Common criticisms of statistical significance concern dichotomous thinking, threshold dependence, publication bias, p-hacking, HARKing, selective reporting, and overinterpretation of small p-values. Treating 0.049 and 0.051 as fundamentally different levels of evidence can obscure the continuous nature of statistical information. Researchers may also unintentionally or deliberately alter analyses, exclude observations, add covariates, try multiple outcomes, or repeatedly inspect data until a desired threshold is reached. Such practices can increase the rate of false-positive findings. Improved reporting therefore emphasises estimates, uncertainty intervals, study design, prespecified analyses, transparency about all tested outcomes, and substantive interpretation. Statistical significance can remain useful, but it should be treated as one component of evidence rather than a universal decision rule for scientific importance.
In practical interpretation, a statistically significant result indicates that the observed data are sufficiently inconsistent with the specified null model according to the chosen test and threshold. It does not establish causation, practical importance, replicability, or truth of the preferred explanation. A non-significant result likewise does not establish that there is no effect; it may indicate insufficient information to distinguish the null from plausible alternatives.
Proper interpretation requires considering the effect estimate, confidence interval, sample size, measurement quality, study design, assumptions, multiplicity, and substantive context. Statistical significance is therefore best understood as a technical property of a particular inferential procedure rather than as a general synonym for "important", "real", or "proven".
