P-Values
A p-value is the probability of data at least as extreme as observed, assuming the null hypothesis is true.
Precise definition
The p-value is P(test statistic at least as extreme as observed | H₀ true). A small p-value means the observed data would be unusual if the null hypothesis held, which is taken as evidence against the null.
What it is not
- It is not the probability that H₀ is true.
- It is not the probability the result was due to chance.
- It is not the probability of replicating the result.
- 1 − p is not the probability that H₁ is true.
Why the misreadings matter
The p-value is a statement about data given a hypothesis, not about a hypothesis given data. Inverting that direction is the same base-rate error as confusing P(A|B) with P(B|A). Turning a p-value into the probability a hypothesis is true requires a prior and Bayes theorem.
Multiple testing
Run twenty independent tests at α = 0.05 and you expect one false positive by chance alone. Testing many hypotheses without correction manufactures spurious significance. Corrections such as Bonferroni or false-discovery-rate control adjust the threshold to keep the overall error rate in check.
Good practice
A p-value is one summary, not a verdict. Report the effect size, a confidence interval, the sample size, and how many comparisons were made. A p-value just under 0.05 from a small study is weak evidence, easily produced by chance or by selective analysis — the pattern behind much irreproducible research.