Double/Debiased Machine Learning (DML)

Motivating Example: Estimating Price Elasticity of Demand#

You want to estimate the causal effect of price on demand for a product. You have observational data with hundreds of potential confounders: competitor prices, seasonality, weather, local demographics, marketing spend, and more.

Traditional regression forces you to specify a functional form: maybe you add log-transformed variables, interactions, and polynomials. But with hundreds of confounders, getting the functional form right is extremely difficult. Machine learning excels at flexibly fitting complex relationships — random forests, gradient boosting, and neural networks can capture nonlinearities and interactions that you would never think to specify.

But there is a catch. If you simply run a random forest to predict the outcome, extract the predicted values, and use them in a second-stage regression, your standard errors are wrong and your point estimate may be biased. ML models overfit, regularize in ways that bias coefficients, and do not distinguish between causal and predictive relationships.

Chernozhukov et al. (2018) addressed this problem with Double/Debiased Machine Learning (DML). The key ideas are:

1. Neyman orthogonality: Construct the causal estimating equation so that small errors in the ML-estimated nuisance functions do not bias the causal parameter.
2. Cross-fitting: Split the data to avoid overfitting bias — train ML models on one subset, predict on another.

The result: you can use any well-behaved ML method to control for confounders while still getting valid confidence intervals for the causal effect.

The Problem with Naive ML#

Suppose you want to estimate $\theta_0$ in:

where $g_0(X)$ is an unknown, potentially complex function of high-dimensional confounders $X$.

Naive approach: Estimate $\hat{g}(X)$ using ML, then regress $Y - \hat{g}(X)$ on $D$.

Problem: Even if $\hat{g}$ converges to $g_0$, ML methods typically converge at a rate slower than $\sqrt{n}$. This "regularization bias" contaminates the estimate of $\theta_0$, making standard errors invalid and the point estimate biased.

The DML Solution#

DML addresses this through two innovations:

1. Neyman orthogonality. Instead of directly partialing out confounders from $Y$, also partial out confounders from $D$ — that is, residualize both $Y$ and $D$ on $X$. This "double residualization" makes the estimating equation for $\theta_0$ insensitive to first-order errors in $\hat{g}$. The idea goes back to Robinson (1988) and Frisch-Waugh-Lovell, but DML generalizes it to the ML setting.

2. Cross-fitting. Split the sample into $K$ folds. For each fold, train the ML models on the other $K-1$ folds and predict on the held-out fold. This sample-splitting avoids the overfitting problem that arises when the same data are used for both ML estimation and causal inference.

Neyman Orthogonality (Plain Language)#

Imagine you are estimating a treatment effect, and your estimate depends on how well you model the outcome as a function of confounders. If a small error in your outcome model directly translates into a proportional error in your treatment effect estimate, you have a problem — because ML models inevitably have some error.

Neyman orthogonality means the treatment effect estimate is locally insensitive to errors in the nuisance models. Geometrically, the "gradient" of the causal parameter with respect to the nuisance function is zero at the true values. First-order errors in the nuisance function produce only second-order errors in the causal parameter.

This insensitivity property is why you partial out confounders from both the outcome and the treatment. The resulting estimating equation has the Neyman-orthogonal property.

Common Confusions#

"Can I use any ML method?" Almost. The ML learners must satisfy certain convergence rate conditions (roughly, faster than $n^{-1/4}$; more precisely, the product of the two nuisance estimation errors must be $o_p(n^{-1/2})$). Most standard methods (random forests, gradient boosting, lasso, neural networks) satisfy this condition. But consider avoiding methods that do not converge at all or that have extremely high variance.

"Does DML give me causal effects even with observational data?" DML gives you valid inference conditional on the unconfoundedness assumption, similar to matching methods. It does not solve the omitted variable problem. If there are unmeasured confounders, DML is biased just like OLS would be. DML's advantage is in handling observed confounders more flexibly.

"How is DML different from doubly robust estimation?" The doubly robust property is a building block of DML. DML adds cross-fitting (to handle ML overfitting) and the formal Neyman orthogonality framework (to handle regularization bias). You can think of DML as "doubly robust estimation done right when using ML."

"How many folds for cross-fitting?" 5 folds is a common default. Too few folds means the training set is small (reducing ML performance). Too many folds means each held-out set is small (increasing variance). The theoretical results are robust to the number of folds as long as $K \geq 2$.

The Partially Linear Model#

The simplest DML setup is the partially linear regression model:

where:

• $\theta_0$ is the causal parameter of interest
• $g_0(X) = E[Y - D\theta_0 | X]$ captures the confounding relationship between $X$ and $Y$ (nuisance)
• $m_0(X) = E[D | X]$ is the treatment confounding function (nuisance / propensity score)
• $U, V$ are residuals

Identifying Assumptions#

1. Conditional exogeneity (unconfoundedness): $E[U | X, D] = 0$ — conditional on the observed covariates $X$, the treatment $D$ is as-good-as randomly assigned. All confounders are observed and included in $X$.

2. Overlap (positivity): $0 < P(D = 1 \mid X = x) < 1$ for all $x$ in the support of $X$ — every unit has a positive probability of receiving treatment. Without overlap, the propensity score $m_0(X)$ would be degenerate and residualization would fail.

3. SUTVA: No interference between units — one unit's treatment does not affect another unit's outcome.

4. Sufficient ML convergence: The nuisance estimators $\hat{g}$ and $\hat{m}$ converge at a rate faster than $n^{-1/4}$, so that the product of their errors is $o_p(n^{-1/2})$.

The DML Estimator#

Step 1: Double residualization.

• Estimate $\hat{\ell}_0(X) \approx E[Y|X]$ and $\hat{m}_0(X) \approx E[D|X]$ using ML. Here $\ell_0(X) \equiv E[Y|X] = m_0(X)\theta_0 + g_0(X)$ is the reduced-form conditional expectation — we estimate this directly rather than $g_0(X)$ because $g_0$ depends on the unknown $\theta_0$.
• Compute residuals: $\tilde{Y}_i = Y_i - \hat{\ell}(X_i)$ and $\tilde{D}_i = D_i - \hat{m}(X_i)$

Step 2: Estimate $\theta_0$.

This estimator is just OLS of $\tilde{Y}$ on $\tilde{D}$ — the Frisch-Waugh-Lovell theorem applied to ML residuals.

Step 3: Cross-fitting. Do the above with sample splitting:

1. Split data into $K$ folds
2. For each fold $k$: train $\hat{\ell}^{(-k)}$ and $\hat{m}^{(-k)}$ on all data except fold $k$; predict $\tilde{Y}_i, \tilde{D}_i$ for observations in fold $k$
3. Pool all residuals and run the final regression

Formal Statement#

The DML procedure can be visualized in three steps:

1. Partial out X from Y: Remove the part of the outcome explained by confounders (the ML-predicted component). What remains ($\tilde{Y}$) is the variation in $Y$ not explained by $X$.

2. Partial out X from D: Remove the part of the treatment explained by confounders. What remains ($\tilde{D}$) is the variation in treatment not predicted by observables — the "residual" or "exogenous" variation.

3. Regress $\tilde{Y}$ on $\tilde{D}$: The slope of this regression is the causal effect $\hat{\theta}_0$.

1# Requires: DoubleML, mlr3, mlr3learners
2# DoubleML: R implementation of Chernozhukov et al. (2018) DML framework
3library(DoubleML)
4library(mlr3)          # mlr3: machine learning framework for R
5library(mlr3learners)  # mlr3learners: additional ML algorithms for mlr3
6
7# --- Step 1: Prepare data for DML ---
8# DoubleMLData specifies the causal structure: outcome, treatment, and confounders
9dml_data <- DoubleMLData$new(
10df,
11y_col = "outcome",     # outcome variable
12d_cols = "treatment",   # treatment variable
13x_cols = paste0("x", 1:50)  # high-dimensional confounders
14)
15
16# --- Step 2: Choose ML learners for nuisance estimation ---
17# ml_l: predicts E[Y|X] (outcome model) — removes confounding from outcome
18# ml_m: predicts E[D|X] (treatment model) — removes confounding from treatment
19ml_l <- lrn("regr.ranger", num.trees = 500)     # random forest for outcome
20ml_m <- lrn("classif.ranger", num.trees = 500, predict_type = "prob")  # random forest for propensity
21
22# --- Step 3: Fit DML with cross-fitting ---
23# DoubleMLPLR: partially linear model Y = theta*D + g(X) + epsilon
24# n_folds = 5: 5-fold cross-fitting prevents overfitting bias
25# Cross-fitting is essential — without it, ML overfitting biases theta
26dml_plr <- DoubleMLPLR$new(
27dml_data,
28ml_l = ml_l,
29ml_m = ml_m,
30n_folds = 5
31)
32dml_plr$fit()
33dml_plr$summary()
34# theta: debiased causal effect of treatment on outcome
35
36# --- Step 4: Confidence interval ---
37# Valid inference from Neyman orthogonality + cross-fitting
38dml_plr$confint(level = 0.95)

1. Check the quality of the first-stage ML models. Report cross-validated $R^2$ or MSE for both the outcome model ($\hat{g}$) and the treatment model ($\hat{m}$). If the ML models do not fit well, the residualization may not adequately remove confounding.

2. Compare ML learners. Run DML with different ML methods (random forest, gradient boosting, lasso) and check whether the causal estimate changes. Robustness to the ML learner choice increases credibility.

3. Check residual balance. After double residualization, the residualized treatment $\tilde{D}$ should be uncorrelated with $X$. Check this by regressing $\tilde{D}$ on $X$ — the $R^2$ should be near zero.

4. Sensitivity to the number of folds. Re-run with $K = 2, 5, 10$ folds. Results should be stable.

5. Compare with simple OLS. If DML and OLS give similar results, the confounding is well-captured by linear terms and DML's flexibility was not needed (but also did not hurt).

Interpreting Your Results#

DML and OLS agree: The confounding relationship is approximately linear. Both are valid. DML's standard errors may be somewhat wider, reflecting additional estimation variance from the cross-fitting procedure.

DML and OLS disagree: Nonlinear confounding matters. Report DML as your main result and discuss why the linear approximation fails.

DML results vary across ML learners: The nuisance functions may not be well-estimated. Consider using ensemble methods or SuperLearner to aggregate multiple ML methods.

Outcome model CV R²: 0.72 (RF), 0.75 (GBM), 0.58 (Lasso)
Treatment model CV R²: 0.35 (RF), 0.38 (GBM), 0.22 (Lasso)

• OLS with controls: When the number of controls is small, functional form is known, and there is no need for machine-learning flexibility.
• IV / 2SLS: When endogeneity cannot be addressed by conditioning on observables and a valid instrument is available — DML assumes conditional exogeneity.
• Matching: When a transparent matched-pair design is preferred over a regression-based approach, and the covariate space is moderate.
• Doubly robust estimation: When double robustness is desired but the parametric setting suffices — DR estimators share the same doubly-robust logic as DML but are typically applied without cross-fitting.