Lee Bounds for Attrition

When You Cannot Pin It Down

Sometimes you have to admit that the data cannot tell you exactly what the answer is — only that it lies within a range. This situation is the world of partial identification, and if you find it frustrating, you are in good company. But it turns out that knowing an effect is "between 0.05 and 0.25" is often far more useful — and far more honest — than reporting a precise but biased point estimate.

The most common setting where partial identification arises in applied economics is sample selection: the outcome you care about is only observed for a non-random subset of your sample. The classic example is wages. You want to estimate the effect of a job training program on wages, but wages are only observed for people who are employed. If the training program itself changes who is employed (as it almost certainly does), then comparing wages among the employed is contaminated by selection.

Sample selection is not a minor technical issue. It threatens the validity of any study where the treatment affects whether the outcome is observed.

Why It Matters

If you ignore sample selection and simply compare outcomes among observed units, your treatment effect estimate is biased — potentially severely. Lee bounds give you honest, assumption-lean bounds on the true effect, letting you report what the data can actually support rather than a precise but misleading point estimate. Reviewers increasingly expect attrition analysis in experimental work, and Lee bounds are the standard tool for it.

Why Point Identification Fails with Differential Attrition

Consider a randomized experiment evaluating a job training program. You randomly assign 1,000 people to training and 1,000 to control. After six months, you observe:

• Training group: 700 employed (70%), wages observed
• Control group: 600 employed (60%), wages observed

You want to estimate the effect of training on wages. But here is the problem: the 700 employed in the treatment group and the 600 employed in the control group are different populations. Training caused 100 more people to be employed. Those 100 "marginal" workers — people who would not have been employed without the training — are probably different from the "always-employed" workers (e.g., lower skill, lower potential wages).

When you compare average wages among the employed, you are comparing:

• Treatment: a mixture of always-employed workers and newly employed workers
• Control: only always-employed workers

The comparison is contaminated by the composition change. You cannot separate the effect of training on wages from the effect of training on who is observed. Randomization guarantees balance in the full sample, but it says nothing about the selected subsample.

Two Approaches: Heckman vs. Bounds

The Heckman Selection Model

The traditional solution to sample selection is the Heckman (1979) model.

The Heckman model requires:

1. A selection equation (a model of who is observed)
2. An exclusion restriction (a variable that affects selection but not the outcome)
3. Joint normality of the error terms

These conditions are strong requirements. The exclusion restriction is often hard to justify — what affects employment but not wages? Joint normality is a functional form assumption that may not hold. In practice, Heckman estimates can be fragile and sensitive to specification.

The Bounds Approach

The alternative, pioneered by Charles Manski and applied powerfully by David Lee, is to give up on point identification and instead bound the treatment effect using weaker assumptions.

The key insight: if you are willing to assume less, you learn less — but what you learn is more credible. A wide but honest bound beats a precise but questionable point estimate.

Lee (2009) Bounds: The Method

Lee (2009) developed a practical, widely used method for bounding treatment effects in the presence of sample selection.

The Monotonicity Assumption

Lee bounds require a single key assumption: monotonicity of selection. The treatment must affect selection in one direction only:

where $S_i(d)$ is the selection indicator (1 = observed, 0 = not observed) under treatment status $d$.

In the job training example: training can only increase (or leave unchanged) the probability of employment for every individual. No one who would have been employed without training becomes unemployed because of training.

The Trimming Procedure

Under monotonicity, the treatment group contains everyone the control group contains plus some extra individuals who were "brought in" by the treatment. To make the groups comparable, we need to remove those extra individuals. The question is: which ones?

We do not know. But we can construct the best and worst cases.

Step 1: Compute the selection rates:

• $p_1 = P(\text{observed} | \text{treated})$
• $p_0 = P(\text{observed} | \text{control})$

Under monotonicity with the treatment increasing selection: $p_1 > p_0$.

Step 2: Compute the trimming proportion:

This quantity is the fraction of the treatment group's observed sample that was "brought in" by the treatment.

Step 3: Trim the treatment group to make it comparable:

• Upper bound: Remove the bottom $q$ fraction of the treated group's outcome distribution. The remaining treated individuals have the highest outcomes. Comparing them to the control group gives an upper bound on the treatment effect for always-observed individuals.
• Lower bound: Remove the top $q$ fraction of the treated group's outcome distribution. Comparing the remaining (lowest) outcomes to the control group gives a lower bound.

A Worked Example

Return to our training program:

• Treatment: 700 of 1,000 employed (70%)
• Control: 600 of 1,000 employed (60%)

Step 1: $p_1 = 0.70$, $p_0 = 0.60$

Step 2: $q = (0.70 - 0.60) / 0.70 = 0.143$

So 14.3% of the treatment group's employed workers were "brought in" by the program.

Step 3: Trim the treatment group.

Suppose the average wage in the control group is $15/hour. The average wage in the full treatment group is $15.50/hour.

• Trim the bottom 14.3% of the treatment wage distribution. The remaining 85.7% have an average wage of $16.20. Upper bound = $16.20 - $15.00 = $1.20/hour.
• Trim the top 14.3% of the treatment wage distribution. The remaining 85.7% have an average wage of $14.80. Lower bound = $14.80 - $15.00 = -$0.20/hour.

The Lee bounds are [-$0.20, $1.20]. The training program's effect on wages (for always-employed workers) is somewhere in this range. Notice the bounds include zero — we cannot rule out that training has no wage effect, even though the naive comparison shows a positive difference.

Interactive: Attrition and Bounds

Try setting both observation rates to the same value. The bounds collapse because there is no differential selection. Now widen the gap and watch the bounds expand — this widening is the cost of not knowing who was "brought in" by the treatment.

Tightening the Bounds

Lee bounds can be wide, especially when differential attrition is large. Two main strategies help:

1. Condition on Pre-Treatment Covariates

If you have baseline covariates that predict the outcome, compute Lee bounds within covariate cells and then average. Within-cell outcome distributions are less dispersed, so the bounds within each cell are tighter. The overall bounds (a weighted average across cells) are tighter than the unconditional bounds.

This tightening works because trimming removes a fixed fraction of the outcome distribution. If the within-cell distribution has less spread, trimming removes less extreme values, producing tighter bounds.

2. Reduce Differential Attrition

The most effective strategy is prevention. Minimize differential attrition through:

• Intensive follow-up and tracking
• Administrative data linkage (which eliminates survey non-response)
• Incentive payments for survey completion
• Short, simple outcome measures that maximize response

Every percentage point of differential attrition widens the bounds. A 2-percentage-point differential is far more manageable than a 15-percentage-point differential. Accounting for expected attrition in your power analysis at the design stage helps ensure the resulting bounds remain informative.

Lee Bounds vs. Manski Worst-Case Bounds

It is important not to confuse Lee bounds with Manski's "worst-case" bounds, which use no assumptions at all beyond the support of the outcome:

Manski bounds are often uninformatively wide because they impute the worst possible outcomes for missing data. Lee bounds are typically much tighter because monotonicity restricts the set of possible imputations.

When to Use Lee Bounds

How to Do It: Code

1lee_bounds <- function(y, treatment, selection, alpha = 0.05, n_boot = 1000) {
2# Observation rates
3p1 <- mean(selection[treatment == 1])
4p0 <- mean(selection[treatment == 0])
5
6# Determine trimming direction
7if (p1 >= p0) {
8  q <- (p1 - p0) / p1
9  trim_group <- "treatment"
10} else {
11  q <- (p0 - p1) / p0
12  trim_group <- "control"
13}
14
15# Observed outcomes
16y1 <- y[treatment == 1 & selection == 1]
17y0 <- y[treatment == 0 & selection == 1]
18
19compute_bounds <- function(y1, y0, q) {
20  if (q < 1e-10) {
21    # No differential attrition
22    return(c(lower = mean(y1) - mean(y0), upper = mean(y1) - mean(y0)))
23  }
24  # Upper bound: trim bottom q of treatment group
25  cutoff_low <- quantile(y1, q)
26  upper <- mean(y1[y1 >= cutoff_low]) - mean(y0)
27
28  # Lower bound: trim top q of treatment group
29  cutoff_high <- quantile(y1, 1 - q)
30  lower <- mean(y1[y1 <= cutoff_high]) - mean(y0)
31
32  c(lower = lower, upper = upper)
33}
34
35# Point estimates
36bounds <- compute_bounds(y1, y0, q)
37
38# Bootstrap confidence intervals
39n1 <- sum(treatment == 1)
40n0 <- sum(treatment == 0)
41boot_bounds <- replicate(n_boot, {
42  idx1 <- sample(which(treatment == 1), n1, replace = TRUE)
43  idx0 <- sample(which(treatment == 0), n0, replace = TRUE)
44  b_y1 <- y[idx1[selection[idx1] == 1]]
45  b_y0 <- y[idx0[selection[idx0] == 1]]
46  b_p1 <- mean(selection[idx1])
47  b_p0 <- mean(selection[idx0])
48  b_q <- max(0, (b_p1 - b_p0) / b_p1)
49  compute_bounds(b_y1, b_y0, b_q)
50})
51
52ci_lower <- quantile(boot_bounds["lower", ], c(alpha/2, 1-alpha/2))
53ci_upper <- quantile(boot_bounds["upper", ], c(alpha/2, 1-alpha/2))
54
55list(
56  lower = bounds["lower"],
57  upper = bounds["upper"],
58  ci_lower = ci_lower,
59  ci_upper = ci_upper,
60  trimming_proportion = q,
61  p_treated = p1,
62  p_control = p0
63)
64}
65
66# Usage
67result <- lee_bounds(
68y = df$wage,
69treatment = df$treatment,
70selection = df$employed
71)
72
73cat(sprintf("Lee Bounds: [%.3f, %.3f]\n", result$lower, result$upper))
74cat(sprintf("95%% CI for lower bound: [%.3f, %.3f]\n",
75  result$ci_lower[1], result$ci_lower[2]))
76cat(sprintf("95%% CI for upper bound: [%.3f, %.3f]\n",
77  result$ci_upper[1], result$ci_upper[2]))

Stata: Manual Implementation

1* Step 1: Selection rates
2sum employed if treatment == 1
3local p1 = r(mean)
4sum employed if treatment == 0
5local p0 = r(mean)
6
7* Step 2: Trimming proportion
8local q = (`p1' - `p0') / `p1'
9di "Trimming proportion: " `q'
10
11* Step 3: Quantiles for trimming
12_pctile wage if treatment == 1 & employed == 1, p(`=`q'*100')
13local cutoff_lower = r(r1)
14_pctile wage if treatment == 1 & employed == 1, p(`=(1-`q')*100')
15local cutoff_upper = r(r1)
16
17* Upper bound: keep above q-th percentile
18sum wage if treatment == 1 & employed == 1 & wage >= `cutoff_lower'
19local mean_t_upper = r(mean)
20sum wage if treatment == 0 & employed == 1
21local mean_c = r(mean)
22di "Upper bound: " `mean_t_upper' - `mean_c'
23
24* Lower bound: keep below (1-q)-th percentile
25sum wage if treatment == 1 & employed == 1 & wage <= `cutoff_upper'
26local mean_t_lower = r(mean)
27di "Lower bound: " `mean_t_lower' - `mean_c'

How to Report Lee Bounds

A well-reported Lee bounds analysis includes:

1. The attrition or selection rates for treatment and control groups.
2. A test for differential attrition (is the difference statistically significant?).
3. The monotonicity assumption, stated explicitly and justified for your setting.
4. The bounds with confidence intervals (bootstrapped).
5. Comparison with the naive estimate (ignoring selection).
6. Whether covariates were used to tighten bounds.

Example write-up:

Employment rates are 70% in the treatment group and 60% in the control group (p < 0.001), indicating that the training program increased employment. Because wages are only observed for employed individuals, the naive comparison of treatment and control wages is contaminated by differential selection. Following Lee (2009), we compute bounds on the wage effect under the assumption that training weakly increases employment for all individuals (monotonicity). The trimming proportion is 14.3%. The Lee bounds for the treatment effect on hourly wages are [-$0.20, $1.20] (95% CI for the lower bound: [-$0.85, $0.45]; 95% CI for the upper bound: [$0.55, $1.85]). The bounds include zero, so we cannot reject that training has no effect on wages for always-employed workers. Conditioning on baseline covariates (age, gender, education) tightens the bounds to [$0.05, $0.95].

Common Mistakes

Concept Check