Experimental Design

Motivating Example: The Oregon Health Insurance Experiment

In 2008, Oregon expanded its Medicaid program but had far more applicants than slots. The state held a lottery — a literal random draw — to decide who would get the opportunity to enroll. The lottery created one of the most important experiments in health economics.

The researchers could compare lottery winners (who were offered insurance) to lottery losers. Because assignment was random, the two groups were identical in expectation on every dimension — income, health status, education, motivation, everything. Any difference in outcomes could be attributed to the insurance offer itself.

This balance is the power of experimental design. You do not need to measure and control for every confounder. Randomization handles it for you.

But here is the catch: not everyone who won the lottery actually enrolled in Medicaid. And this non-compliance creates a gap between what was randomly assigned (the offer) and what was actually received (the insurance). Understanding this gap is one of the central lessons of this page.

A. Overview: Why Experiments Are the Benchmark for Causal Inference

The average treatment effect (ATE) is defined as:

The fundamental problem is that we never observe both potential outcomes for the same unit. But random assignment solves the comparison problem — it eliminates selection bias in expectation, which is why causal inference requires careful research design. When treatment is randomly assigned:

In plain language: the average untreated outcome is the same for the treated group and the control group. The control group is a valid stand-in for the counterfactual. A simple difference in means recovers the ATE:

This estimator is exactly equivalent to running OLS with a single treatment dummy. The regression just gives you standard errors for free.

The Three Pillars of a Good Experiment

1. Random assignment — units are allocated to treatment and control by a mechanism the researcher controls.
2. No interference — one unit's treatment does not affect another unit's outcome (the Stable Unit Treatment Value Assumption, or SUTVA).
3. Excludability — the assignment mechanism affects outcomes only through the treatment itself, not through other channels.

Common Confusions

B. Identification: What Makes Randomization Work

The Mechanics of Randomization

Randomization creates identification through a simple but powerful mechanism: it makes treatment assignment statistically independent of potential outcomes.

This independence means there is no selection bias — a concept explored in depth on the selection bias foundations page. The people in the treatment group are, on average, identical to those in the control group in every way — observed and unobserved.

Intent-to-Treat (ITT)

When you compare outcomes by assignment (regardless of whether subjects actually took up the treatment), you get the ITT:

where $Z_i$ is the random assignment indicator. Under intact randomization and no differential attrition, the ITT is a valid causal effect. It answers: "What is the effect of being assigned to the treatment group?"

LATE for Non-Compliance

In the Oregon experiment, $Z_i$ was winning the lottery, but $D_i$ (actually enrolling in Medicaid) was a choice. Some winners did not enroll (never-takers), and in principle, some non-winners might have found other ways to enroll (always-takers).

Using the lottery as an instrument for actual enrollment, you can estimate the Local Average Treatment Effect (LATE):

This expression is the Wald ratio: reduced form ÷ first stage. The numerator ($\text{ITT}_Y$) is the reduced-form effect of the instrument $Z$ on the outcome $Y$; the denominator ($\text{ITT}_D$) is the first-stage effect of $Z$ on the treatment take-up $D$. The ratio gives you the causal effect of treatment for compliers — those whose treatment status was actually changed by the random assignment.

C. Visual Intuition

Think of randomization as a shuffling machine. You take your sample of people — with all their differences in motivation, ability, health, income — and you shuffle them into two groups completely at random. Each group ends up being a miniature copy of the other, on average.

The key visual: imagine a balance scale. Before randomization, the treatment group could be heavier on one side (more motivated people, higher income, whatever). After randomization, the scale is balanced — not perfectly for any single experiment, but in expectation across repeated randomizations.

This expectation is why balance tables matter. If your randomization worked, the treatment and control groups should look similar on all observed characteristics. A balance table lets you verify this expectation.

D. Mathematical Derivation

E. Implementation

1library(fixest)
2library(modelsummary)
3
4# ---- Balance table ----
5# Regress each covariate on treatment to verify randomization worked
6balance_vars <- c("age", "female", "income", "education")
7bal <- lapply(balance_vars, function(v) {
8feols(as.formula(paste(v, "~ treatment")), data = df)
9})
10# Display balance results; small coefficients = good randomization
11modelsummary(bal, stars = TRUE)
12
13# ---- ITT estimate ----
14# Simple regression of outcome on treatment assignment
15# vcov = "HC1" gives heteroskedasticity-robust standard errors
16itt <- feols(outcome ~ treatment, data = df, vcov = "HC1")
17summary(itt)
18
19# ---- LATE via IV (for non-compliance) ----
20# outcome ~ exogenous | fixed_effects | endogenous ~ instrument
21# Uses assignment as instrument for actual takeup
22late <- feols(outcome ~ 1 | 0 | takeup ~ assignment, data = df, vcov = "HC1")
23summary(late)

F. Diagnostics and Robustness Checks

Balance Checks

An essential diagnostic for any experiment. Compare pre-treatment covariates across treatment and control groups. Report:

• Group means and standard deviations
• Difference and its p-value (or standardized difference)
• An F-test for joint significance of all covariates predicting treatment

Attrition Checks

Attrition (people dropping out of the study) is only a problem if it is differential — if treatment causes people to leave the sample at different rates. Check:

1. Is the attrition rate similar across treatment and control?
2. Among non-attritors, is balance still maintained?
3. Consider Lee bounds for worst-case scenarios.

Compliance Checks

Report the first-stage compliance rate: what fraction of the assigned-to-treatment group actually received treatment? A first-stage below 100% means you need to decide between ITT and LATE.

Interpreting Results

• The ITT is a policy-relevant parameter: it tells you what happens when you roll out an intervention in practice, including non-compliance.
• The LATE tells you what the treatment does for people who actually take it up, but it only applies to compliers.
• If compliance is near 100%, ITT and LATE are essentially the same.
• It is recommended to report the ITT. Report the LATE as a complement, not a replacement.

G. What Can Go Wrong

H. Practice

I. Swap-In: When to Use Something Else

If randomization is infeasible (ethical constraints, cost, or lack of control), the closest alternatives are:

• Natural experiments — situations where nature or policy creates as-if random assignment. See IV / 2SLS and Regression Discontinuity.
• Matching — construct a comparison group that looks similar on observables.
• Difference-in-differences — exploit a policy change that affects some groups but not others.

For any of these approaches, sensitivity analysis is essential for assessing how robust your conclusions are to potential violations of identifying assumptions. The further you move from randomization, the more assumptions you need, and the less credible your causal claims become. But a well-designed quasi-experiment often beats a poorly executed RCT.

J. Reviewer Checklist