Why do false positives appear when you run many statistical tests?

At a 0.05 significance threshold, each test of pure noise has a 5% chance of a false positive. Run 10,000 such tests and roughly 500 — about 5% — cross p < 0.05 by chance alone. Corrections like Bonferroni (α/m) and Benjamini-Hochberg control this multiplicity.

P-hacking · multiple comparisons

Torture the data, and it confesses.

There is no real effect here — every cell below is a fresh test on pure random noise. Slice it enough ways and "significance" appears anyway. About 1 in 20 tests will cross p<0.05 by chance alone.

● live Auto-generated headline

Run the study to manufacture a finding.

False positives

—

tests with p < 0.05

Expected by chance

—

= 0.05 × tests

False-positive rate

—

should hover near 5%

Every square = one test on noise n.s. "significant" corrected away

How many ways to slice it100 tests

More comparisons = more chances for noise to look real. This is the "garden of forking paths."

Sample size per test30

Two random groups compared with a two-sample t-test. No effect exists — bigger samples don't change the 5% false-alarm rate.

Apply a correction

No correctionthreshold α = 0.05

Run a study to see how many "findings" survive.

False-positive rate vs tests run0 tests

observed rate α = 0.05

How to use

Set how many tests and the sample size, then hit Run the study. Each square is one t-test on freshly generated noise — red squares "found" significance. Now switch on a correction and watch the spurious findings get clawed back to (almost always) zero.

What's actually happening

Every cell is random noise — two groups drawn from the same distribution, so the true effect is exactly zero. Under that null, p-values are uniform on [0,1], so each test has a flat 5% chance of landing below 0.05.

Run 20 tests → expect ~1 false positive. Run 100 → expect ~5. Run 10,000 → expect ~500. The "finding" isn't luck, it's arithmetic: E[false positives] = 0.05 × m.

Bonferroni & the FDR fix

Bonferroni divides the threshold by the number of tests: α' = 0.05 / m. With m = 20 you now need p < 0.0025. It controls the chance of any false positive (family-wise error) — strict, and most noise dies under it.

Benjamini-Hochberg controls the false discovery rate instead: sort the m p-values ascending, find the largest k with p(k) ≤ (k/m)·0.05, reject all below it. More powerful than Bonferroni when real effects exist — here, with pure noise, it also clears almost everything out.

Examples in the wild

Subgroup analyses in clinical trials. Multiple outcomes reported when only one reached significance. Multiple time windows for an economic effect. Multiple comparison groups.

The "garden of forking paths" (Gelman & Loken): one study with many analyst choices is effectively many studies. Preregistration commits you to one test before you see the data — p-hacking is "look at the data, try 20 things, report the one that worked."

Citations

Simmons, J.P., Nelson, L.D. & Simonsohn, U. (2011). "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant." Psychological Science 22(11): 1359–1366.
Ioannidis, J.P.A. (2005). "Why Most Published Research Findings Are False." PLoS Medicine 2(8): e124.
Gelman, A. & Loken, E. (2014). "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no 'fishing expedition' or 'p-hacking'." American Statistician 68(2).
Bonferroni, C.E. (1936). "Teoria statistica delle classi e calcolo delle probabilità." Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze 8: 3–62.
Benjamini, Y. & Hochberg, Y. (1995). "Controlling the false discovery rate." J. Royal Statistical Society B 57(1): 289–300.

Built by Joshua Chua · Fable rebuild · vanilla JS + Canvas