Sensitivity Analysis for Unobservables

The Problem You Cannot See

You have run your regression, included every control variable you can think of, and your treatment effect is statistically significant. Your advisor nods. A referee writes: "How robust is this result to omitted variable bias?"

Addressing this question is central to building a convincing empirical argument.

Every observational study — and even many quasi-experimental ones — faces the same uncomfortable question: what if there is an unobserved variable that, if you could control for it, would shrink your estimate to zero? You can never prove this is not the case. What you can do is formally characterize how strong such a confounder would have to be in order to eliminate your findings. If the answer is "implausibly strong," your result gains credibility. If the answer is "a modest confounder would do it," you should worry.

This formalization is what sensitivity analysis for unobservables does. It does not prove your result is causal. It tells you — and your readers — exactly how much unobserved confounding it would take to overturn it.

Why Does This Matter?

The traditional approach to omitted variable bias was informal: run a regression with few controls, then add more, and argue that if the coefficient does not move much, unobservables probably would not change things either. This logic was formalized by Altonji et al. (2005), who proposed comparing the coefficient with and without controls as a way to bound the bias from unobservables.

The intuition is clean: if adding observed controls barely moves the coefficient, then unobserved confounders — which, under certain assumptions, produce bias of similar magnitude — probably would not move it much either. But this argument had limitations. It was hard to make precise, and it ignored the role of R-squared movements.

Two frameworks have since emerged as the standard tools for formalizing this argument. Both are worth knowing.

Framework 1: Oster's Delta and Bias-Adjusted Estimates

Oster (2019) extended the Altonji, Elder, and Taber logic into a complete framework that accounts for both coefficient movements and R-squared movements when controls are added.

The Core Idea

Oster's approach asks: if selection on unobservables is proportional to selection on observables, how large would that proportionality factor (called delta, or $\delta$) have to be to drive the treatment effect to zero?

Here is the setup. You run two regressions:

1. Short regression (no controls): coefficient $\dot{\beta}$, R-squared $\dot{R}$
2. Long regression (with controls): coefficient $\tilde{\beta}$, R-squared $\tilde{R}$

If adding observables moves the coefficient from $\dot{\beta}$ to $\tilde{\beta}$ and moves R-squared from $\dot{R}$ to $\tilde{R}$, then the bias from unobservables depends on two things:

• Delta ($\delta$): the ratio of selection on unobservables to selection on observables. If $\delta = 1$, unobservables are equally important as observables for confounding. If $\delta = 2$, unobservables are twice as important.
• $R_{\max}$: the R-squared you would obtain if you could include all relevant variables (observed and unobserved). This bound caps how much explanatory power remains to be captured.

How to Interpret Delta

• $\delta > 1$: unobservables would need to be more important than observables to explain away the result. This magnitude is generally reassuring.
• $\delta < 1$: even modest unobserved confounding could overturn the result. This vulnerability is concerning.
• $\delta < 0$: the result would actually get stronger if you added more confounders. This sign reversal can happen and is very reassuring.

Framework 2: Cinelli & Hazlett's Partial R-Squared Approach

Cinelli and Hazlett (2020) propose a different — and in many ways more intuitive — framework based on partial R-squared values.

The Core Idea

Instead of asking "how proportional is selection on unobservables to selection on observables?", Cinelli and Hazlett ask: how much residual variance in the treatment and in the outcome would a confounder have to explain in order to change the conclusion?

They parameterize the confounder's strength using two quantities:

• $R^2_{Y \sim Z | X, D}$: partial R-squared of the confounder $Z$ with the outcome $Y$, after partialing out treatment $D$ and controls $X$
• $R^2_{D \sim Z | X}$: partial R-squared of the confounder $Z$ with the treatment $D$, after partialing out controls $X$

The key result is that the omitted variable bias from leaving out $Z$ is a function of these two partial R-squared values. This framework lets you ask: "How strong would a confounder have to be — in terms of its association with treatment and outcome — to change my conclusion?"

Robustness Values

The framework produces two key quantities:

• Robustness Value (RV): the minimum strength of confounding (in partial R-squared terms) that would reduce the estimate to zero. Formally, it is the value such that a confounder with $R^2_{Y \sim Z | X, D} = R^2_{D \sim Z | X}$ equal to that value would reduce the estimate to zero. A related quantity, $RV_{q,\alpha}$, gives the minimum confounding strength that would make the estimate statistically insignificant at level $\alpha$.
• Benchmarking: you compare the robustness value against the partial R-squared of observed covariates. If no observed covariate is as strong as the robustness value, an unobserved confounder would need to be stronger than anything you observe.

Contour Plots

The most distinctive feature of this framework is the contour plot: a two-dimensional graph with $R^2_{D \sim Z | X}$ on one axis and $R^2_{Y \sim Z | X, D}$ on the other, showing contour lines for different levels of bias. Benchmark covariates are plotted as points, giving readers an immediate visual sense of how strong a confounder would have to be.

When to Use Which Framework

In practice, many papers now report both. They complement rather than substitute for each other. For an alternative robustness approach that focuses on researcher degrees of freedom rather than unobservables, see specification curve analysis.

How to Do It: Code

Cinelli & Hazlett (sensemakr)

1library(sensemakr)
2
3# Fit the main regression
4model <- lm(outcome ~ treatment + x1 + x2 + x3, data = df)
5
6# Run sensitivity analysis
7sensitivity <- sensemakr(
8model = model,
9treatment = "treatment",
10benchmark_covariates = c("x1", "x2", "x3"),  # observed covariates for benchmarking
11kd = 1:3,  # multiples of each benchmark's strength to consider
12ky = 1:3
13)
14
15# Print summary
16summary(sensitivity)
17
18# Contour plot
19plot(sensitivity)
20
21# Extreme scenario plot
22ovb_contour_plot(sensitivity, sensitivity.of = "t-value")

Oster (2019) Coefficient Stability

Oster (2019) recommends $R_{\max} = 1.3 \tilde{R}$, capped at 1, where $\tilde{R}$ is the $R^2$ from the regression with all observed controls. The R2max / rmax() arguments take an absolute $R^2$ value, so you must compute it first.

1library(robomit)
2
3# Step 1: Estimate the fully controlled model to get R-tilde
4full_model <- lm(outcome ~ treatment + x1 + x2 + x3, data = df)
5R_tilde <- summary(full_model)$r.squared
6
7# Step 2: Apply Oster's heuristic with cap at 1
8R2max <- min(1.3 * R_tilde, 1)
9
10# Compute delta for beta = 0
11o_delta(
12y = "outcome",          # outcome variable
13x = "treatment",        # treatment variable
14con = "x1 + x2 + x3",  # controls (as a formula string)
15beta = 0,               # hypothesized true beta (0 = test for sign change)
16R2max = R2max,          # absolute R-max value (not a multiplier)
17type = "lm",
18data = df
19)
20
21# Bias-adjusted beta for delta = 1
22o_beta(
23y = "outcome",
24x = "treatment",
25con = "x1 + x2 + x3",
26delta = 1,
27R2max = R2max,          # same computed R-max
28type = "lm",
29data = df
30)

Interactive: Adjusting Confounding Strength

How to Report Sensitivity Analysis

A good sensitivity analysis section includes:

1. The controlled estimate and its standard error.
2. The robustness value (Cinelli & Hazlett) or delta (Oster) — stated clearly in words: "A confounder would need to explain X% of the residual variation in both treatment and outcome to reduce the estimate to zero."
3. Benchmarking against observed covariates: "The strongest observed covariate explains Y% of residual variation; a confounder would need to be Z times as strong."
4. A contour plot or table showing sensitivity across a range of assumptions.
5. Bias-adjusted estimates for key scenarios ($\delta = 1$, $R_{\max} = 1.3\tilde{R}$ for Oster; benchmark multiples for Cinelli & Hazlett).

Here is an example of how to write this up:

We assess robustness to omitted variable bias using the partial R-squared framework of Cinelli and Hazlett (2020). The robustness value for our main estimate is 0.12, meaning that an unobserved confounder would need to explain at least 12% of the residual variance in both treatment assignment and the outcome to reduce the point estimate to zero. For comparison, the strongest observed predictor in our model (household income) explains only 4% of the residual variance in treatment. A confounder would therefore need to be three times as strong as income — a scenario we consider implausible given the institutional setting.

Common Mistakes

Concept Check