Randomization Inference

The Idea That Predates Everything#

Before regression. Before t-tests. Before the entire apparatus of parametric inference that fills your econometrics textbook. There was a simpler, more elegant idea.

In 1935, Ronald Fisher proposed the following procedure: if you want to know whether a treatment had an effect, shuffle the treatment labels, recompute your test statistic, and see how unusual your original result is compared to the shuffled versions. No assumptions about normality. No asymptotics. No functional form. Just the physical act of randomization, turned into an inference engine (Fisher, 1935).

This procedure is randomization inference, and it is experiencing a renaissance in applied economics. Young (2019) applied randomization tests to treatment effects estimated across 53 experimental papers from journals of the American Economic Association. He found 13–22% fewer significant individual treatment effects under randomization inference than authors had reported using conventional methods, and 33–49% fewer significant results in joint tests of the multiple treatment effects appearing together in published tables.

If that evidence does not make you take randomization inference seriously, nothing will.

Why It Matters#

Randomization inference provides exact p-values that are valid regardless of sample size or distributional assumptions, making it a key robustness tool for any study where standard inference is questionable.

When Conventional Asymptotics Fail#

Standard inference — computing a t-statistic and comparing it to a normal or t-distribution — relies on asymptotics. The central limit theorem guarantees that, with enough observations, the sampling distribution of your estimator is approximately normal. But "enough" is carrying substantial weight in that sentence.

Conventional p-values rely on asymptotic approximations that can fail badly with small samples, few clusters, or non-standard test statistics. When the approximations break down, reported significance levels are wrong — sometimes dramatically so. The situations where conventional inference goes badly wrong fall into four categories:

Small Samples#

With 30 or 50 observations, the central limit theorem (CLT) approximation may be poor, especially if the outcome distribution is skewed. Your t-statistic might not actually follow a t-distribution, and your p-value might be too small.

Few Clusters#

This limitation is the most consequential one for applied economists. If you have data on students in 20 schools, and treatment is assigned at the school level, you effectively have 20 observations for inference — regardless of how many students you have. With 20 clusters, cluster-robust standard errors can be severely biased downward, leading to over-rejection of the null (Cameron et al., 2008).

Non-Standard Test Statistics#

What if you want to test a null hypothesis about a quantile, a ratio, or an interaction? The asymptotic distribution of these statistics may not be normal, and deriving the correct distribution analytically may be difficult. Randomization inference handles any test statistic.

Discrete Outcomes#

With binary or count outcomes and small samples, the discreteness of the test statistic means the normal approximation is poor. Randomization inference naturally handles discrete distributions.

How It Works: Fisher's Sharp Null#

Step 1: State the Null Hypothesis#

The sharp null hypothesis is:

The sharp null says the treatment has zero effect on every single unit. Not on average — on every individual. The sharp null is stronger than the usual null of "no average effect," but it is what makes randomization inference work.

Under this null, every person's observed outcome is the same regardless of whether they were treated or not. That invariance means we know what would have happened under any treatment assignment — because nothing would have changed.

Step 2: Choose a Test Statistic#

You can use any test statistic you like. Common choices:

• Difference in means: $T = \bar{Y}_{\text{treated}} - \bar{Y}_{\text{control}}$
• t-statistic from a regression
• Kolmogorov-Smirnov statistic (for distributional differences)
• Rank-based statistics (Wilcoxon, etc.)
• Any function of the data

This flexibility is a major advantage. There is no requirement that the statistic have a known analytical distribution.

Step 3: Compute the Test Statistic on the Actual Data#

Run your analysis as you normally would. Record the test statistic: $T_{\text{obs}}$.

Step 4: Construct the Randomization Distribution#

Now, re-randomize. Under the sharp null, every unit's outcome is fixed. The only thing that changes is the assignment vector. So:

1. Generate a new treatment assignment consistent with your experimental design (e.g., if you assigned 50 of 100 to treatment, randomly choose a different 50)
2. Recompute $T$ using this new assignment but the same outcomes
3. Repeat many times (e.g., 10,000 permutations — or enumerate all $\binom{100}{50}$ assignments if feasible)

The resulting distribution of $T$ values is the randomization distribution — the distribution of your test statistic under the null, given the exact experimental design.

Step 5: Compute the p-Value#

The randomization p-value is the fraction of randomization distribution values that are at least as extreme as $T_{\text{obs}}$:

where $B$ is the number of permutations. (The +1 in numerator and denominator ensures the p-value is never exactly zero and includes the observed statistic in the reference set.)

Respecting the Design: Stratified and Cluster Randomization#

A critical requirement of randomization inference is that the permutations must mirror the actual experimental design. If you randomized within strata, you typically need to permute within strata. If you randomized at the cluster level, you typically need to permute clusters.

Stratified Randomization#

If treatment was assigned within blocks (e.g., 5 of 10 students treated in each classroom), permutations must shuffle treatment labels within each classroom. Cross-classroom permutations would not reflect the actual design and would invalidate the test.

Cluster Randomization#

If entire clusters (schools, villages, firms) are assigned to treatment, the unit of permutation is the cluster. You shuffle which clusters are treated, keeping all individuals within a cluster in the same treatment status. With 10 treated and 10 control clusters, you have $\binom{20}{10} = 184{,}756$ possible assignments — enough for exact enumeration if you are patient, or easily approximated with 10,000 random draws.

Randomization Inference vs. the Bootstrap#

Readers occasionally conflate randomization inference (RI) with the bootstrap. Both are resampling methods, but they answer different questions and rely on different assumptions.

The practical implication: in a small randomized experiment, RI delivers an exact p-value that does not rely on asymptotic approximations — the resampling enumerates (or samples from) the exact distribution of the test statistic under the design. The bootstrap, in contrast, approximates the sampling distribution of an estimator under the assumption that the empirical distribution converges to the population distribution. With small samples or heavy-tailed outcomes, the bootstrap approximation can be poor; RI sidesteps the approximation entirely.

A complementary observation: when the assignment mechanism is unknown — as in observational data with selection-on-observables — RI is generally not available, and the bootstrap (or analytic asymptotics) remains the default inference tool.

The Sharp Null vs. the Weak Null#

A subtlety that matters: randomization inference tests the sharp null ($Y_i(1) = Y_i(0)$ for all $i$), not the weak null ($E[Y_i(1) - Y_i(0)] = 0$). These nulls are different hypotheses.

The sharp null says the treatment has zero effect on every unit. The weak null says the average effect is zero, but allows the treatment to help some individuals and hurt others.

Randomization inference can reject the sharp null even when the average treatment effect (ATE) is exactly zero, if the treatment has heterogeneous effects. Suppose a program helps half the population by +2 and hurts the other half by -2. The ATE is zero, but the distribution of outcomes differs between treatment and control groups, and a test statistic sensitive to distributional differences (like the Kolmogorov-Smirnov statistic) could detect this difference.

This sensitivity to heterogeneity is a feature, not a bug — but be aware of what you are testing. Note that when testing many hypotheses via randomization inference (RI) — for example, across multiple outcomes or subgroups — multiple testing corrections still apply.

Interactive: Permuting Treatment and Seeing the Null Distribution#

When to Use Randomization Inference#

How to Do It: Code#

1# Requires: ri2
2library(ri2)
3
4# Declare the experimental design
5# Here: 100 units, 50 assigned to treatment, complete randomization
6declaration <- declare_ra(N = 100, m = 50)
7
8# Conduct randomization inference
9ri_result <- conduct_ri(
10formula = outcome ~ treatment,
11declaration = declaration,
12sharp_hypothesis = 0,  # test sharp null of zero effect
13data = df,
14sims = 5000  # number of permutations
15)
16
17# View the p-value
18summary(ri_result)
19
20# Plot the randomization distribution
21plot(ri_result)

Cluster-Level Permutation#

1# Requires: ri2
2library(ri2)
3
4# Cluster-randomized design: 20 clusters, 10 treated
5declaration <- declare_ra(
6N = nrow(df),
7clusters = df$cluster_id,
8m = 10  # number of treated clusters
9)
10
11ri_result <- conduct_ri(
12formula = outcome ~ treatment,
13declaration = declaration,
14sharp_hypothesis = 0,
15data = df,
16sims = 5000
17)
18
19summary(ri_result)

How to Report Randomization Inference#

A typical reporting pattern:

We supplement conventional inference with randomization inference (Fisher, 1935). Under the sharp null of no treatment effect for any unit, we permute the treatment assignment 5,000 times, respecting the original design (complete randomization within strata). The randomization-based p-value for the main treatment effect is 0.023, compared to the conventional p-value of 0.018. The close agreement between the two suggests that our results are not driven by distributional assumptions or finite-sample bias in standard errors.

When the two p-values diverge substantially, that divergence is informative — it typically means the asymptotic approximation is unreliable, and the randomization inference result is more reliable in such cases.

Common Mistakes#

Concept Check#