Multiple Hypothesis Testing

When to Use Multiple Testing Corrections#

Apply multiple testing corrections whenever your analysis involves more than one hypothesis test within a family of related tests — multiple outcomes, multiple subgroups, multiple specifications, or multiple time horizons addressing the same research question. The correction is needed regardless of whether the tests were pre-registered. Common settings include: multi-arm experiments, heterogeneity analyses, event study pre-trend tests, and specification curve analyses.

The Problem of Too Many Tests#

Imagine you run an experiment evaluating a tutoring program. You measure its effect on math scores, reading scores, attendance, behavior, self-confidence, parental involvement, peer relationships, and teacher ratings. That list is eight outcomes. Even if the program does absolutely nothing, with eight independent tests at the 5% level, you have roughly a 34% chance of finding at least one "significant" result.

If this probability does not alarm you, it should.

This inflation is the multiple testing problem, and it is everywhere in empirical research. Any time you test more than one hypothesis — multiple outcomes, multiple subgroups, multiple specifications, multiple time periods — the probability of finding at least one spurious "significant" result rises rapidly with the number of tests. Without correction, your paper's headline finding might be nothing more than a statistical accident.

Why Testing Many Hypotheses Inflates False Positives#

Under the null hypothesis of no effect, each individual test has a 5% probability of a false positive (a Type I error). For $m$ independent tests, the probability of at least one false positive — the familywise error rate (FWER) — is:

For $\alpha = 0.05$:

With 20 tests, you are more likely than not to find a false positive. With 100 tests, it is virtually certain.

Two Philosophies: FWER vs. FDR#

Before choosing a correction method, you need to decide what you are trying to control.

Familywise Error Rate (FWER)#

Control the probability of making even one false positive across all tests. FWER control is the strictest standard. You use FWER control when even a single false claim is unacceptable — clinical trials, policy decisions, confirmatory analyses.

False Discovery Rate (FDR)#

Control the expected proportion of false positives among all rejected hypotheses. If you reject 20 hypotheses and allow FDR = 0.05, you expect about 1 of those 20 to be a false discovery. FDR is less conservative and more powerful. You use FDR control when you are doing exploratory analysis, testing many outcomes, or when some false positives are tolerable.

The Methods#

1. Bonferroni Correction#

The simplest approach. Divide your significance threshold by the number of tests:

Equivalently, multiply each p-value by $m$ and compare to $\alpha$.

Pros: Dead simple. Valid under any dependence structure. Cons: Extremely conservative, especially with many correlated tests. If your 20 outcomes are all measuring similar things (and thus correlated), Bonferroni overcorrects badly.

2. Holm Step-Down Correction#

A strict improvement over Bonferroni that is always at least as powerful, yet still controls FWER.

Procedure:

1. Sort your $m$ p-values from smallest to largest: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$
2. Starting from the smallest, reject $H_{(k)}$ if $p_{(k)} \leq \frac{\alpha}{m - k + 1}$
3. Stop at the first non-rejection; do not reject any remaining hypotheses.

The threshold becomes less stringent as you work through the sorted list, giving you more power than Bonferroni.

3. Benjamini-Hochberg (BH) FDR Correction#

Controls the false discovery rate rather than the familywise error rate (Benjamini & Hochberg, 1995).

Procedure:

1. Sort p-values from smallest to largest: $p_{(1)} \leq \cdots \leq p_{(m)}$
2. Find the largest $k$ such that $p_{(k)} \leq \frac{k}{m} \cdot q$, where $q$ is the desired FDR level
3. Reject all hypotheses $H_{(1)}, \ldots, H_{(k)}$

The BH procedure is substantially more powerful than Bonferroni or Holm, but it controls a weaker error rate.

4. Anderson (2008) Sharpened q-Values#

Anderson (2008) adapted the BH procedure for the economics context, producing "sharpened" q-values that account for the share of true null hypotheses in the family of tests.

The key insight: the standard BH procedure assumes all nulls could be true. If you can estimate the share of true nulls ($\pi_0$), you can sharpen the threshold and gain power. Anderson provides a Stata do-file and algorithm for computing these sharpened q-values.

5. Romano-Wolf Stepdown#

The most widely recommended method for FWER control in applied economics. Romano and Wolf (2005) developed a stepdown procedure that uses resampling (bootstrap or randomization) to capture the joint dependence structure of the test statistics.

Why it matters: Unlike Bonferroni and Holm, Romano-Wolf accounts for the correlation between test statistics. If your outcomes are highly correlated (as is typical), the effective number of independent tests is much smaller than the nominal count, and Romano-Wolf exploits this to give you more power.

Procedure (simplified):

1. Compute all test statistics from the original data
2. Bootstrap (or permute) the data many times, recomputing all test statistics each time
3. Use the joint distribution of the bootstrapped statistics to compute adjusted p-values
4. Apply a stepdown algorithm that sequentially removes rejected hypotheses

How to Do It: Code#

1# Raw p-values from multiple tests
2p_values <- c(0.003, 0.012, 0.041, 0.052, 0.089, 0.210, 0.430)
3
4# Bonferroni
5p.adjust(p_values, method = "bonferroni")
6
7# Holm
8p.adjust(p_values, method = "holm")
9
10# Benjamini-Hochberg FDR
11p.adjust(p_values, method = "BH")
12
13# Compare all methods
14data.frame(
15raw = p_values,
16bonferroni = p.adjust(p_values, method = "bonferroni"),
17holm = p.adjust(p_values, method = "holm"),
18bh = p.adjust(p_values, method = "BH")
19)

Anderson Sharpened q-Values and Romano-Wolf Stepdown#

1# Requires: wildrwolf (install.packages("wildrwolf"))
2library(wildrwolf)
3
4# Romano-Wolf stepdown adjusted p-values
5# Fit models for each outcome
6models <- list(
7lm(outcome1 ~ treatment + x1 + x2, data = df),
8lm(outcome2 ~ treatment + x1 + x2, data = df),
9lm(outcome3 ~ treatment + x1 + x2, data = df),
10lm(outcome4 ~ treatment + x1 + x2, data = df)
11)
12
13# Romano-Wolf adjusted p-values via wild bootstrap
14rw_result <- rwolf(
15models = models,
16param = "treatment",
17B = 1000,
18seed = 12345
19)
20print(rw_result)

Interactive: Watching False Discoveries Accumulate#

Set the share of true effects to zero and run 20 tests. You will almost certainly find at least one "significant" result. Now turn on corrections and watch the false positives disappear.

Compare the performance of Bonferroni, Holm, and Benjamini-Hochberg corrections side by side. Adjust the number of tests, fraction of true nulls, and effect size to see how each method trades off power against false discovery control:

How to Report Multiple Testing Corrections#

A good reporting practice includes:

1. State how many tests you run and how you group them into families.
2. Report both unadjusted and adjusted p-values (or q-values) side by side.
3. Specify the method and why you chose it (FWER vs. FDR, accounting for correlation or not).
4. Define families clearly. Not every test in your paper needs to be in the same family. Group tests that address the same question or the same set of outcomes. Primary outcomes are one family; secondary outcomes may be another; subgroup analyses may be a third. When designing your study, a pre-analysis plan that specifies family groupings in advance adds credibility.

Example table format:

Common Mistakes#

Concept Check#