Lab: Experimental Design and RCTs

Overview

In this lab you will work through the full lifecycle of a randomized controlled trial (RCT) evaluating a hypothetical job training program. You will simulate random assignment, verify that randomization produced balanced groups, estimate the average treatment effect (ATE), test for heterogeneous effects, and conduct a power analysis.

What you will learn:

• How to randomly assign treatment and verify covariate balance
• How to estimate the ATE with and without regression adjustment
• How to test for heterogeneous treatment effects by subgroup
• How to conduct a power analysis for planning future experiments
• Common pitfalls in experimental analysis

Prerequisites: Familiarity with hypothesis testing (t-tests, p-values) and basic regression (OLS).

Step 1: Simulate the Experimental Data

We simulate 1,000 participants in a job training RCT. Each participant has baseline characteristics (age, education, prior earnings) and is randomly assigned to treatment or control.

1library(estimatr)
2library(modelsummary)
3
4set.seed(42)
5n <- 1000
6
7age <- round(rnorm(n, 35, 8))
8educ <- sample(c(10, 12, 14, 16, 18), n, replace = TRUE,
9             prob = c(0.1, 0.3, 0.25, 0.25, 0.1))
10prior_earnings <- 20000 + 1500 * educ + 200 * age + rnorm(n, sd = 5000)
11female <- rbinom(n, 1, 0.45)
12
13treat <- rbinom(n, 1, 0.5)
14
15tau_i <- 3000 + 500 * (educ < 14)
16earnings_post <- 25000 + 1200 * educ + 150 * age - 2000 * female +
17               tau_i * treat + rnorm(n, sd = 6000)
18
19df <- data.frame(age, educ, female, prior_earnings, treat, earnings_post)
20
21cat("Treatment:", sum(treat), " Control:", sum(1 - treat), "\n")
22summary(df)

Expected output:

Sample data (first 5 rows):

Summary statistics:

Approximately 500 participants are assigned to treatment and 500 to control.

Step 2: Check Covariate Balance

Randomization should produce groups that look similar on pre-treatment characteristics. A balance table is standard in any experimental paper.

1# Balance table
2balance_vars <- c("age", "educ", "female", "prior_earnings")
3
4balance <- data.frame(
5Variable = balance_vars,
6Treat = sapply(balance_vars, function(v) mean(df[df$treat == 1, v])),
7Control = sapply(balance_vars, function(v) mean(df[df$treat == 0, v])),
8p_value = sapply(balance_vars, function(v)
9  t.test(df[df$treat == 1, v], df[df$treat == 0, v])$p.value)
10)
11balance$Diff <- balance$Treat - balance$Control
12print(round(balance, 3))

Expected output:

All p-values are well above 0.05, confirming that randomization produced balanced treatment and control groups. No covariate systematically predicts treatment assignment.

Step 3: Estimate the Average Treatment Effect

The simplest ATE estimator in an RCT is the difference in means. Regression adjustment can improve precision.

1# Method 1: Difference in means
2ate_simple <- mean(df$earnings_post[df$treat == 1]) -
3            mean(df$earnings_post[df$treat == 0])
4cat("Simple difference in means:", round(ate_simple), "\n")
5
6# Method 2: OLS with robust SEs
7m1 <- lm_robust(earnings_post ~ treat, data = df, se_type = "HC2")
8cat("OLS (no controls):", round(coef(m1)["treat"]),
9  "(SE:", round(m1$std.error["treat"]), ")\n")
10
11# Method 3: OLS with controls (Lin 2013)
12df$age_c <- df$age - mean(df$age)
13df$educ_c <- df$educ - mean(df$educ)
14m2 <- lm_robust(earnings_post ~ treat * (age_c + educ_c + female),
15              data = df, se_type = "HC2")
16cat("OLS (Lin estimator):", round(coef(m2)["treat"]),
17  "(SE:", round(m2$std.error["treat"]), ")\n")

Expected output:

The point estimates are nearly identical across all three methods — this is expected in a well-randomized experiment. Adding controls (the Lin estimator) does not change the coefficient but reduces the standard error by absorbing residual variation, yielding a tighter confidence interval.

Step 4: Test for Heterogeneous Treatment Effects

Does the training program work differently for different subgroups?

1# Interaction with education level
2df$low_educ <- as.integer(df$educ < 14)
3m_het <- lm_robust(earnings_post ~ treat * low_educ, data = df, se_type = "HC2")
4summary(m_het)
5
6cat("ATE for high-educ:", coef(m_het)["treat"], "\n")
7cat("Additional for low-educ:", coef(m_het)["treat:low_educ"], "\n")
8
9# Interaction with gender
10m_gender <- lm_robust(earnings_post ~ treat * female, data = df, se_type = "HC2")
11cat("Gender interaction p-value:", summary(m_gender)$coefficients["treat:female", "Pr(>|t|)"], "\n")

Expected output:

Heterogeneous effects by education:

Heterogeneous effects by gender:

The DGP builds in a $500 bonus effect for low-education participants (tau_i = 3000 + 500 * (educ < 14)), but with this sample size the interaction term is not statistically significant. The gender interaction is near zero because the DGP includes no differential treatment effect by gender.

Step 5: Power Analysis

Before running an experiment, you need to determine the required sample size to detect a meaningful effect. Let us compute power for this training program.

1# Power analysis
2effect_size <- 3000
3sd_outcome <- sd(df$earnings_post)
4cohen_d <- effect_size / sd_outcome
5
6# Required sample size
7result <- power.t.test(delta = effect_size, sd = sd_outcome,
8                      power = 0.80, sig.level = 0.05,
9                      type = "two.sample")
10cat("Cohen's d:", round(cohen_d, 3), "\n")
11cat("Required N per group:", ceiling(result$n), "\n")
12cat("Total N required:", ceiling(result$n) * 2, "\n")
13
14# Power curve
15n_seq <- seq(50, 1000, by = 25)
16power_vals <- sapply(n_seq, function(n)
17power.t.test(n = n, delta = effect_size, sd = sd_outcome,
18             sig.level = 0.05)$power)
19
20plot(n_seq * 2, power_vals, type = "l", lwd = 2, col = "blue",
21   xlab = "Total Sample Size", ylab = "Power",
22   main = "Power Curve for Job Training RCT")
23abline(h = 0.8, col = "red", lty = 2)

Expected output:

Power at various sample sizes:

Step 6: Attrition and Compliance

Real experiments face attrition (participants dropping out) and non-compliance (participants not taking the treatment). Let us simulate and handle these issues.

1# Simulate differential attrition
2set.seed(123)
3attrition_prob <- 0.08 + 0.04 * (1 - df$treat)
4df$observed <- rbinom(n, 1, 1 - attrition_prob)
5
6cat("Attrition (treatment):", 1 - mean(df$observed[df$treat == 1]), "\n")
7cat("Attrition (control):", 1 - mean(df$observed[df$treat == 0]), "\n")
8
9# Naive estimate on observed sample
10df_obs <- df[df$observed == 1, ]
11m_naive <- lm_robust(earnings_post ~ treat, data = df_obs, se_type = "HC2")
12cat("Naive ATE:", round(coef(m_naive)["treat"]), "\n")
13
14# For Lee bounds, see the leebounds package
15# install.packages("leebounds")
16# library(leebounds)
17# leebounds(earnings_post ~ treat, data = df_obs)

Expected output:

Attrition rates:

Lee (2009) bounds for the ATE:

The control group has a higher attrition rate (about 12% vs. 8%), meaning the remaining control group may be positively selected. The Lee bounds trim the treatment group (the lower-attrition group) to equalize attrition rates, providing a worst-case interval for the true ATE. Because both bounds remain positive, we can be confident the treatment effect is positive even under differential attrition.

Step 7: Exercises

Try these on your own:

1. Stratified randomization. Modify the randomization to stratify by gender and education level. Show that stratified randomization produces better balance than simple randomization.

2. Cluster randomization. Suppose treatment is assigned at the firm level (50 firms, 20 workers each). Simulate this design and compare the standard errors with individual-level randomization.

3. Pre-analysis plan. Write a short pre-analysis plan specifying: (a) the primary outcome, (b) the estimand, (c) the estimation method, (d) the subgroups you will examine, and (e) how you will handle attrition.

4. Multiple outcomes. Add two more outcome variables (employment status and hours worked) and apply a Bonferroni correction for multiple hypothesis testing.

Summary

In this lab you learned:

• Random assignment provides a basis for unconfoundedness, but balance verification with a balance table is still important, as is addressing attrition or non-compliance
• The simple difference in means is an unbiased estimator of the ATE in an RCT
• Regression adjustment (especially the Lin estimator) can improve precision without introducing bias
• Heterogeneous treatment effect analysis is best pre-specified to avoid multiple testing problems
• Power analysis is conducted before the experiment to determine the required sample size
• Differential attrition can reintroduce selection bias even in a well-designed RCT