Lab: Replicating Dehejia-Wahba (1999) Propensity Score Matching

Overview

In this lab you will replicate one of the most influential studies in the causal inference literature. Lalonde (1986) showed that non-experimental estimators often fail to recover the experimental benchmark estimate of a job training program. Dehejia and Wahba (1999) demonstrated that propensity score matching can recover this benchmark from observational data. You will reproduce their key findings and extend the analysis with modern matching methods.

What you will learn:

• How to load and work with the classic Lalonde/NSW dataset
• How the experimental benchmark provides a ground truth for evaluating observational methods
• How to implement propensity score matching, nearest-neighbor matching, and CEM
• How to assess covariate balance before and after matching
• Why some observational comparison groups are harder to match than others

Prerequisites: Familiarity with the potential outcomes framework and propensity scores. Completion of the matching tutorial lab is recommended.

Step 1: Load the NSW Experimental Data

The NSW dataset contains earnings data for randomly assigned treatment and control groups from the National Supported Work program.

1library(MatchIt)
2library(cobalt)
3
4# The Lalonde data is built into MatchIt
5data("lalonde", package = "MatchIt")
6
7# Experimental subset
8nsw <- lalonde
9cat("=== NSW Experimental Data ===\n")
10cat("N treated:", sum(nsw$treat == 1), "\n")
11cat("N control:", sum(nsw$treat == 0), "\n")
12
13# Experimental benchmark
14ate_exp <- mean(nsw$re78[nsw$treat == 1]) - mean(nsw$re78[nsw$treat == 0])
15cat("Experimental ATE: $", round(ate_exp), "\n")

Expected output:

=== NSW Experimental Benchmark ===
Treated: n = 185, mean RE78 = $6,349
Control: n = 260, mean RE78 = $4,555
Experimental ATE: $1,794
(Dehejia-Wahba report approximately $1,794)

This experimental estimate serves as the benchmark against which all observational methods will be evaluated.

Step 2: Construct the Observational Dataset

Replace the experimental control group with a non-experimental comparison group drawn from the CPS or PSID.

1# The MatchIt 'lalonde' dataset includes PSID/CPS-1 controls
2# by default (not the experimental controls from Step 1).
3# If your version uses experimental controls, download the
4# CPS comparison group from: https://users.nber.org/~rdehejia/data/
5data("lalonde", package = "MatchIt")
6
7# Construct: NSW treated + non-experimental controls
8nsw_treat <- lalonde[lalonde$treat == 1, ]
9obs_ctrl <- lalonde[lalonde$treat == 0, ]
10
11obs <- lalonde
12
13naive_diff <- mean(obs$re78[obs$treat == 1]) - mean(obs$re78[obs$treat == 0])
14cat("Naive difference: $", round(naive_diff), "\n")
15cat("Experimental ATE: $", round(ate_exp), "\n")
16cat("Bias: $", round(naive_diff - ate_exp), "\n")

Expected output:

=== Naive Observational Estimate ===
Treated mean RE78: $6,349
CPS mean RE78:     $14,847
Naive difference:  -$8,498
Experimental ATE:  $1,794

Bias: -$10,292

The CPS group earns much more — naive comparison is severely biased.

The naive observational difference is not just wrong — it has the wrong sign. Instead of the true positive effect of ~$1,794, the naive estimate suggests the program reduced earnings by ~$8,500.

Step 3: Estimate the Propensity Score

1# Propensity score (DW specification)
2ps_model <- glm(treat ~ age + education + black + hispanic + married +
3              nodegree + re74 + re75,
4              data = obs, family = binomial)
5obs$pscore <- predict(ps_model, type = "response")
6
7# Overlap check
8hist(obs$pscore[obs$treat == 1], col = rgb(0,0,1,0.5), xlim = c(0,1),
9   main = "Propensity Score Distribution", xlab = "Propensity Score")
10hist(obs$pscore[obs$treat == 0], col = rgb(1,0,0,0.5), add = TRUE)
11legend("topright", c("Treated", "Control"), fill = c(rgb(0,0,1,0.5), rgb(1,0,0,0.5)))

Expected output:

PS range (treated): [0.0215, 0.9842]
PS range (control): [0.000002, 0.8215]
Note: most CPS controls have very low propensity scores

Step 4: Nearest-Neighbor Matching on the Propensity Score

1# Nearest-neighbor matching using MatchIt
2m_nn <- matchit(treat ~ age + education + black + hispanic + married +
3              nodegree + re74 + re75,
4              data = obs, method = "nearest", distance = "glm",
5              caliper = 0.1)
6
7summary(m_nn)
8
9# Extract matched data and estimate ATT
10m_data <- match.data(m_nn)
11ate_nn <- lm(re78 ~ treat, data = m_data, weights = weights)
12cat("NN Matching ATT:", coef(ate_nn)["treat"], "\n")
13cat("Experimental:", ate_exp, "\n")

Expected output:

=== Nearest-Neighbor PS Matching (1:1) ===
Matched ATE: $1,652
Experimental ATE: $1,794
Bias: -$142

With caliper = 0.05: 142/185 treated units matched
Caliper-matched ATE: $1,815

Nearest-neighbor matching on the propensity score recovers an estimate close to the experimental benchmark. Caliper matching drops treated units without close matches, potentially improving the estimate at the cost of a narrower estimand.

Step 5: Coarsened Exact Matching (CEM)

CEM creates strata based on coarsened covariate values, then matches exactly within strata.

1# CEM using MatchIt
2m_cem <- matchit(treat ~ age + education + black + hispanic + married +
3               nodegree + re74 + re75,
4               data = obs, method = "cem",
5               cutpoints = list(age = c(25, 30, 40),
6                                education = c(9, 11, 12),
7                                re74 = c(0, 5000, 15000),
8                                re75 = c(0, 5000, 15000)))
9summary(m_cem)
10
11m_cem_data <- match.data(m_cem)
12ate_cem <- lm(re78 ~ treat, data = m_cem_data, weights = weights)
13cat("CEM ATT:", coef(ate_cem)["treat"], "\n")

Expected output:

=== Coarsened Exact Matching ===
Matched: 148 treated, 1245 controls
CEM ATE: $1,925
Experimental ATE: $1,794

CEM drops ~37 treated units that have no counterpart in any CPS stratum. The estimate is close to the experimental benchmark, but pertains only to the matchable subpopulation.

Step 6: Compare All Estimates with the Experimental Benchmark

1# Summary comparison
2cat("=== Summary ===\n")
3cat("Experimental:", round(ate_exp), "\n")
4cat("Naive:", round(naive_diff), "\n")
5cat("OLS:", round(coef(lm(re78 ~ treat + age + education + black + hispanic +
6                        married + nodegree + re74 + re75, data = obs))["treat"]), "\n")
7cat("NN Matching:", round(coef(ate_nn)["treat"]), "\n")
8cat("CEM:", round(coef(ate_cem)["treat"]), "\n")
9
10# Balance comparison
11love.plot(m_nn, stats = "mean.diffs", abs = TRUE,
12        title = "Covariate Balance: NN Matching")

Expected output:

=== Summary of Estimates ===
       Method                     ATE       Bias
 Experimental benchmark        $1,794         $0
 Naive (unadjusted)           -$8,498   -$10,292
 OLS with controls             $1,245      -$549
 NN PS matching (1:1)          $1,652      -$142
 PS matching (caliper)         $1,815        $21
 CEM                           $1,925       $131

Extension Exercises

1. Drop pre-treatment earnings. Re-run all matching methods without re74 and re75. How much worse are the estimates? This exercise replicates Lalonde's (1986) original finding.

2. Use the PSID comparison group. The PSID comparison group is smaller and somewhat more similar to the NSW treated group. Repeat the analysis and compare.

3. Try doubly robust estimation. Implement an AIPW estimator (combining PS weighting with outcome regression) and compare with matching alone.

4. Vary the caliper width. Run PS matching with calipers of 0.01, 0.05, 0.10, and 0.25. Plot the tradeoff between the number of matches and the estimated ATE.

5. Sensitivity analysis. Use the Rosenbaum bounds or the Cinelli-Hazlett framework to assess how sensitive the matching estimate is to unobserved confounders.

Summary

In this lab you learned:

• The NSW experimental benchmark provides a ground truth for evaluating observational estimators
• Naive comparisons with non-experimental control groups can be severely biased due to covariate imbalance
• Propensity score matching recovers the experimental benchmark when the right covariates (especially lagged outcomes) are available and overlap is sufficient
• CEM guarantees exact balance on coarsened covariates but may discard observations, changing the estimand
• Covariate balance diagnostics (standardized mean differences, overlap plots) are essential for assessing match quality
• The Lalonde dataset remains the most important benchmark in causal inference: if a method cannot recover the experimental estimate from observational data, it should not be trusted for settings where no experiment exists