Double/Debiased Machine Learning (DML)

Motivating Example

You want to estimate the causal effect of price on demand for a product. You have observational data with hundreds of potential confounders: competitor prices, seasonality, weather, local demographics, marketing spend, and more.

Traditional regression forces you to specify a functional form: maybe you add log-transformed variables, interactions, and polynomials. But with hundreds of confounders, you cannot possibly get the functional form right. Machine learning excels at flexibly fitting complex relationships — random forests, gradient boosting, and neural networks can capture nonlinearities and interactions that you would never think to specify.

But there is a catch. If you simply run a random forest to predict the outcome, extract the predicted values, and use them in a second-stage regression, your standard errors are wrong and your point estimate may be biased. ML models overfit, regularize in ways that bias coefficients, and do not distinguish between causal and predictive relationships.

Chernozhukov et al. (2018) solved this problem with Double/Debiased Machine Learning (DML). The key ideas are:

1. Neyman orthogonality: Construct the causal estimating equation so that small errors in the ML-estimated nuisance functions do not bias the causal parameter.
2. Cross-fitting: Split the data to avoid overfitting bias — train ML models on one subset, predict on another.

The result: you can use any well-behaved ML method to control for confounders while still getting valid confidence intervals for the causal effect.

A. Overview

The Problem with Naive ML

Suppose you want to estimate $\theta_0$ in:

where $g_0(X)$ is an unknown, potentially complex function of high-dimensional confounders $X$.

Naive approach: Estimate $\hat{g}(X)$ using ML, then regress $Y - \hat{g}(X)$ on $D$.

Problem: Even if $\hat{g}$ converges to $g_0$, ML methods typically converge at a rate slower than $\sqrt{n}$. This "regularization bias" contaminates the estimate of $\theta_0$, making standard errors invalid and the point estimate biased.

The DML Solution

DML addresses this through two innovations:

1. Neyman orthogonality. Instead of directly partialing out confounders from $Y$, also partial out confounders from $D$ — that is, residualize both $Y$ and $D$ on $X$. This "double residualization" makes the estimating equation for $\theta_0$ insensitive to first-order errors in $\hat{g}$. The idea goes back to Robinson (1988) and Frisch-Waugh-Lovell, but DML generalizes it to the ML setting.

2. Cross-fitting. Split the sample into $K$ folds. For each fold, train the ML models on the other $K-1$ folds and predict on the held-out fold. This sample-splitting avoids the overfitting problem that arises when the same data are used for both ML estimation and causal inference.

Neyman Orthogonality (Plain Language)

Imagine you are estimating a treatment effect, and your estimate depends on how well you model the outcome as a function of confounders. If a small error in your outcome model directly translates into a proportional error in your treatment effect estimate, you have a problem — because ML models inevitably have some error.

Neyman orthogonality means the treatment effect estimate is locally insensitive to errors in the nuisance models. Geometrically, the "gradient" of the causal parameter with respect to the nuisance function is zero at the true values. First-order errors in the nuisance function produce only second-order errors in the causal parameter.

This insensitivity property is why you partial out confounders from both the outcome and the treatment. The resulting estimating equation has the Neyman-orthogonal property.

Common Confusions

"Can I use any ML method?" Almost. The ML learners must satisfy certain convergence rate conditions (roughly, faster than $n^{-1/4}$). Most standard methods (random forests, gradient boosting, lasso, neural networks) satisfy this condition. But consider avoiding methods that do not converge at all or that have extremely high variance.

"Does DML give me causal effects even with observational data?" DML gives you valid inference conditional on the unconfoundedness assumption, similar to matching methods. It does not solve the omitted variable problem. If there are unmeasured confounders, DML is biased just like OLS would be. DML's advantage is in handling observed confounders more flexibly.

"How is DML different from doubly robust estimation?" The doubly robust property is a building block of DML. DML adds cross-fitting (to handle ML overfitting) and the formal Neyman orthogonality framework (to handle regularization bias). You can think of DML as "doubly robust estimation done right when using ML."

"How many folds for cross-fitting?" 5 folds is a common default. Too few folds means the training set is small (reducing ML performance). Too many folds means each held-out set is small (increasing variance). The theoretical results are robust to the number of folds as long as $K \geq 2$.

B. Identification

The Partially Linear Model

The simplest DML setup is the partially linear regression model:

where:

• $\theta_0$ is the causal parameter of interest
• $g_0(X) = E[Y - D\theta_0 \mid X]$ captures the confounding relationship between $X$ and $Y$ (nuisance)
• $m_0(X) = E[D | X]$ is the treatment confounding function (nuisance / propensity score)
• $U, V$ are residuals

The DML Estimator

Step 1: Double residualization.

• Estimate $\hat{\ell}(X) \approx E[Y|X]$ and $\hat{m}(X) \approx E[D|X]$ using ML (note: in practice, one estimates $\ell_0(X) = E[Y|X]$ directly rather than $g_0(X)$, since $g_0$ depends on the unknown $\theta_0$)
• Compute residuals: $\tilde{Y}_i = Y_i - \hat{\ell}(X_i)$ and $\tilde{D}_i = D_i - \hat{m}(X_i)$

Step 2: Estimate $\theta_0$.

This estimator is just OLS of $\tilde{Y}$ on $\tilde{D}$ — the Frisch-Waugh-Lovell theorem applied to ML residuals.

Step 3: Cross-fitting. Do the above with sample splitting:

1. Split data into $K$ folds
2. For each fold $k$: train $\hat{\ell}^{(-k)}$ and $\hat{m}^{(-k)}$ on all data except fold $k$; predict $\tilde{Y}_i, \tilde{D}_i$ for observations in fold $k$
3. Pool all residuals and run the final regression

Formal Statement

C. Visual Intuition

The DML procedure can be visualized in three steps:

1. Partial out X from Y: Remove the part of the outcome explained by confounders (the ML-predicted component). What remains ($\tilde{Y}$) is the variation in $Y$ not explained by $X$.

2. Partial out X from D: Remove the part of the treatment explained by confounders. What remains ($\tilde{D}$) is the variation in treatment not predicted by observables — the "residual" or "exogenous" variation.

3. Regress $\tilde{Y}$ on $\tilde{D}$: The slope of this regression is the causal effect $\hat{\theta}_0$.

D. Mathematical Derivation

E. Implementation

1library(DoubleML)
2library(mlr3)
3library(mlr3learners)
4
5# Prepare data
6dml_data <- DoubleMLData$new(
7df,
8y_col = "outcome",
9d_cols = "treatment",
10x_cols = paste0("x", 1:50)
11)
12
13# Choose ML learners
14ml_l <- lrn("regr.ranger", num.trees = 500)   # outcome model
15ml_m <- lrn("classif.ranger", num.trees = 500) # treatment model
16
17# Fit DML (partially linear model)
18dml_plr <- DoubleMLPLR$new(
19dml_data,
20ml_l = ml_l,
21ml_m = ml_m,
22n_folds = 5
23)
24dml_plr$fit()
25dml_plr$summary()
26
27# Confidence interval
28dml_plr$confint(level = 0.95)

F. Diagnostics

1. Check the quality of the first-stage ML models. Report cross-validated $R^2$ or MSE for both the outcome model ($\hat{g}$) and the treatment model ($\hat{m}$). If the ML models do not fit well, the residualization may not adequately remove confounding.

2. Compare ML learners. Run DML with different ML methods (random forest, gradient boosting, lasso) and check whether the causal estimate changes. Robustness to the ML learner choice increases credibility.

3. Check residual balance. After double residualization, the residualized treatment $\tilde{D}$ should be uncorrelated with $X$. Check this by regressing $\tilde{D}$ on $X$ — the $R^2$ should be near zero.

4. Sensitivity to the number of folds. Re-run with $K = 2, 5, 10$ folds. Results should be stable.

5. Compare with simple OLS. If DML and OLS give similar results, the confounding is well-captured by linear terms and DML's flexibility was not needed (but also did not hurt).

Interpreting Your Results

DML and OLS agree: The confounding relationship is approximately linear. Both are valid. DML's standard errors may be somewhat wider, reflecting additional estimation variance from the cross-fitting procedure.

DML and OLS disagree: Nonlinear confounding matters. Report DML as your main result and discuss why the linear approximation fails.

DML results vary across ML learners: The nuisance functions may not be well-estimated. Consider using ensemble methods or SuperLearner to aggregate multiple ML methods.

G. What Can Go Wrong

H. Practice

Outcome model CV R²: 0.72 (RF), 0.75 (GBM), 0.58 (Lasso)
Treatment model CV R²: 0.35 (RF), 0.38 (GBM), 0.22 (Lasso)

I. Swap-In: When to Use Something Else

• OLS with controls: When the number of controls is small, functional form is known, and there is no need for machine-learning flexibility.
• IV / 2SLS: When endogeneity cannot be addressed by conditioning on observables and a valid instrument is available — DML assumes conditional exogeneity.
• Matching: When a transparent matched-pair design is preferred over a regression-based approach, and the covariate space is moderate.
• Doubly robust estimation: When double robustness is desired but the parametric setting suffices — DR estimators share the same doubly-robust logic as DML but are typically applied without cross-fitting.

J. Reviewer Checklist