Specification Curve Analysis

When to Use Specification Curve Analysis#

Use specification curve analysis whenever your analysis involves choices that could plausibly affect your results — different outcome measures, control variable sets, sample restrictions, functional forms, or standard error specifications. It is especially valuable for observational studies where there is no single "correct" specification, and for pre-registered studies where you want to demonstrate that your headline result is not an artifact of one particular set of choices.

Why It Matters#

When a paper reports a single specification, readers have no way of knowing whether that combination of choices is the one that happened to produce the most favorable result. Specification curve analysis makes this transparent by showing all defensible specifications simultaneously, letting readers judge whether the finding is robust or fragile. It is increasingly expected in top journals as a complement to pre-registration and sensitivity analysis.

The Problem of Researcher Degrees of Freedom#

Every empirical analysis involves dozens of choices. Which sample do you use? Which control variables do you include? How do you define the outcome? Do you winsorize outliers at the 1st or 5th percentile? Do you use OLS or a nonlinear model? Do you cluster standard errors at the state level or the county level?

Each of these choices is defensible. But different choices lead to different results. And when a paper reports a single specification — the one the authors settled on after weeks of analysis — the reader has no way of knowing whether that particular combination of choices is the one that happened to produce the most favorable result.

This selectivity is the problem of researcher degrees of freedom. It is distinct from outright fraud or even intentional p-hacking. It arises because the space of reasonable analytic choices is large, and reporting only one point in that space is inherently selective.

Specification curve analysis is the systematic method for exploring the entire space of defensible choices and presenting the results transparently (Simonsohn et al., 2020).

Two Frameworks, One Idea#

Specification Curve Analysis (SCA)#

Simonsohn et al. (2020) introduced SCA as a method for assessing the robustness of a finding across all reasonable specifications. The key insight: rather than presenting one "preferred" specification with a few robustness checks in an appendix, run every defensible specification and show the full distribution of results.

Multiverse Analysis#

Steegen et al. (2016) proposed multiverse analysis, which emphasizes the data processing choices that precede statistical analysis — how variables are constructed, how missing data are handled, how the sample is defined.

How It Differs from Sensitivity Analysis for Unobservables#

It is important to distinguish specification curve analysis from the sensitivity analysis covered in the sensitivity analysis practice page. They address different questions:

A result can be robust to specification choices but fragile to unobserved confounding (or vice versa). Ideally, both dimensions are assessed.

Building a Specification Curve: Step by Step#

Step 1: Define the Analytic Universe#

List all defensible choices along each dimension. "Defensible" means a reasonable researcher could justify this choice on methodological grounds — not that you personally prefer it.

If you have 3 outcome definitions, 3 sample definitions, 3 control sets, 2 functional forms, 4 FE structures, and 2 SE choices, that combination is $3 \times 3 \times 3 \times 2 \times 4 \times 2 = 432$ specifications.

Step 2: Run All Specifications#

This step is computationally intensive but straightforward. Loop over all combinations, estimate each model, and store the coefficient of interest, its standard error, and the p-value. Keep track of which choices produced each result.

Step 3: Plot the Specification Curve#

The canonical specification curve figure has two panels:

Top panel: Point estimates (and optionally confidence intervals) sorted from smallest to largest. A horizontal line at zero shows where effects flip sign.

Bottom panel: A grid showing which analytic choices were active for each specification. Each row corresponds to one dimension (outcome, controls, sample, etc.), and dots or highlighting indicates the active choice. This layout lets readers see which choices drive the variation in results.

The bottom panel is often more informative than the top. It reveals patterns like: "the estimate is always large when we use log earnings and always small when we use levels" — which tells you something important about the nature of the result.

Step 4: Joint Inference#

Simonsohn et al. (2020) propose a permutation-based joint test. Under the null of no effect:

1. Randomly permute the treatment variable
2. Re-run all specifications on the permuted data
3. Compute a summary statistic (e.g., the median estimate across specs, or the share of specs with positive estimates)
4. Repeat many times to build a null distribution
5. Compare the observed summary statistic to this distribution

This joint test asks: "Is the overall pattern of results across all specifications more extreme than what you would expect by chance alone?"

Interactive: Exploring the Specification Space#

How to Do It: Code#

1# --- Option 1: Using the specr package (>= 1.0 API) ---
2# specr automates specification curve construction and plotting
3library(specr)
4
5# --- Step 1: Prepare the data ---
6# Create a subgroup variable for subsetting specifications
7df$age_group <- ifelse(df$age >= 25 & df$age <= 55, "prime_age", "other")
8
9# --- Step 2: Define the analytic universe ---
10# setup() enumerates all combinations of outcomes, controls, and subsets
11specs <- setup(
12data = df,
13y = c("earnings", "log_earnings"),       # two outcome definitions
14x = "treatment",                          # treatment variable (fixed)
15model = c("lm"),                          # model type(s)
16controls = c(
17  "age + gender",                         # minimal controls
18  "age + gender + education",             # moderate controls
19  "age + gender + education + baseline_earnings"  # full controls
20),
21subsets = list(age_group = c("prime_age"))  # optional subsample
22)
23
24# --- Step 3: Run all specifications and inspect results ---
25# specr() estimates each combination and collects coefficients + SEs
26results <- specr(specs)
27summary(results)  # shows distribution of estimates across specs
28
29# --- Step 4: Plot the specification curve ---
30# patchwork arranges the estimate panel and choice-indicator panel
31library(patchwork)
32plot(results)  # top = sorted estimates, bottom = active choices
33
34# ============================================================
35# --- Option 2: Manual loop + joint permutation test ---
36# Use this when you need full control or specr does not fit your design
37library(broom); library(dplyr); library(purrr)
38
39# --- Step 1: Define each dimension of the analytic universe ---
40# Outcomes: different ways to measure the dependent variable
41outcomes <- c("earnings", "log_earnings", "employed")
42
43# Control sets: from no controls to a full covariate set
44control_sets <- list(
45none = character(0),
46demo = c("age", "gender"),
47full = c("age", "gender", "education", "baseline_earnings")
48)
49
50# Sample definitions: full sample vs. prime-age restriction
51samples <- list(
52full = function(d) d,
53prime = function(d) filter(d, age >= 25, age <= 55)
54)
55
56# --- Step 2: Run every combination and extract treatment coefficient ---
57run_specs <- function(data) {
58# expand.grid creates one row per specification
59specs <- expand.grid(
60  outcome = outcomes,
61  controls = names(control_sets),
62  sample = names(samples),
63  stringsAsFactors = FALSE
64)
65# Iterate over specifications and collect tidy regression output
66map_dfr(seq_len(nrow(specs)), function(i) {
67  s <- specs[i, ]
68  d <- samples[[s$sample]](data)
69  covs <- control_sets[[s$controls]]
70  # Build the regression formula dynamically
71  fml <- reformulate(c("treatment", covs), response = s$outcome)
72  fit <- lm(fml, data = d)
73  # Extract only the treatment row from the coefficient table
74  tidy(fit) %>%
75    filter(term == "treatment") %>%
76    mutate(outcome = s$outcome, controls = s$controls, sample = s$sample)
77})
78}
79
80# Run all specifications on the observed data
81observed <- run_specs(df)
82
83# --- Step 3: Joint permutation test (Simonsohn et al. 2020) ---
84# Randomly permute treatment to build a null distribution
85# of the median estimate across all specifications
86set.seed(42)
87null_medians <- replicate(1000, {
88df_perm <- df
89df_perm$treatment <- sample(df_perm$treatment)  # break treatment assignment
90perm_results <- run_specs(df_perm)
91median(perm_results$estimate)  # summary statistic under the null
92})
93# p-value: fraction of null medians as extreme as the observed median
94p_joint <- mean(abs(null_medians) >= abs(median(observed$estimate)))

How to Report a Specification Curve#

A well-reported specification curve section includes:

1. A clear description of the analytic universe. What choices did you vary, and why is each option defensible?
2. The total number of specifications.
3. The specification curve figure with both the estimate panel and the choice indicator panel.
4. Summary statistics. What fraction of specifications produce a positive estimate? What fraction are significant? What is the median estimate and its interquartile range?
5. Joint inference. The p-value from the permutation test.
6. Discussion of what drives variation. Which choices matter most?

Example:

Figure 3 displays the specification curve across 432 specifications varying the outcome definition (3 options), sample restrictions (3 options), control variables (3 options), fixed effects (4 options), and standard error clustering (2 options). Across all specifications, 94% produce positive estimates and 78% are statistically significant at the 5% level. The median estimate is 0.12 (IQR: 0.08–0.18). The joint permutation test, based on 1,000 permutations, rejects the null of no effect (p < 0.001). The primary source of variation is the outcome definition: specifications using log earnings yield systematically larger effects than those using levels.

Common Mistakes#

Concept Check#