Specification Curve Analysis

When to Use Specification Curve Analysis

Use specification curve analysis whenever your analysis involves choices that could plausibly affect your results — different outcome measures, control variable sets, sample restrictions, functional forms, or standard error specifications. It is especially valuable for observational studies where there is no single "correct" specification, and for pre-registered studies where you want to demonstrate that your headline result is not an artifact of one particular set of choices.

Why It Matters

When a paper reports a single specification, readers have no way of knowing whether that combination of choices is the one that happened to produce the most favorable result. Specification curve analysis makes this transparent by showing all defensible specifications simultaneously, letting readers judge whether the finding is robust or fragile. It is increasingly expected in top journals as a complement to pre-registration and sensitivity analysis.

The Problem of Researcher Degrees of Freedom

Every empirical analysis involves dozens of choices. Which sample do you use? Which control variables do you include? How do you define the outcome? Do you winsorize outliers at the 1st or 5th percentile? Do you use OLS or a nonlinear model? Do you cluster standard errors at the state level or the county level?

Each of these choices is defensible. But different choices lead to different results. And when a paper reports a single specification — the one the authors settled on after weeks of analysis — the reader has no way of knowing whether that particular combination of choices is the one that happened to produce the most favorable result.

This selectivity is the problem of researcher degrees of freedom. It is distinct from outright fraud or even intentional p-hacking. It arises because the space of reasonable analytic choices is large, and reporting only one point in that space is inherently selective.

Specification curve analysis is the systematic method for exploring the entire space of defensible choices and presenting the results transparently.

Two Frameworks, One Idea

Specification Curve Analysis (SCA)

Simonsohn et al. (2020) introduced SCA as a method for assessing the robustness of a finding across all reasonable specifications. The key insight: rather than presenting one "preferred" specification with a few robustness checks in an appendix, run every defensible specification and show the full distribution of results.

Multiverse Analysis

Steegen et al. (2016) proposed multiverse analysis, which emphasizes the data processing choices that precede statistical analysis — how variables are constructed, how missing data are handled, how the sample is defined.

How It Differs from Sensitivity Analysis for Unobservables

It is important to distinguish specification curve analysis from the sensitivity analysis covered in the sensitivity analysis practice page. They address different questions:

A result can be robust to specification choices but fragile to unobserved confounding (or vice versa). Ideally, both dimensions are assessed.

Building a Specification Curve: Step by Step

Step 1: Define the Analytic Universe

List all defensible choices along each dimension. "Defensible" means a reasonable researcher could justify this choice on methodological grounds — not that you personally prefer it.

If you have 3 outcome definitions, 3 sample definitions, 3 control sets, 2 functional forms, 4 FE structures, and 2 SE choices, that combination is $3 \times 3 \times 3 \times 2 \times 4 \times 2 = 432$ specifications.

Step 2: Run All Specifications

This step is computationally intensive but straightforward. Loop over all combinations, estimate each model, and store the coefficient of interest, its standard error, and the p-value. Keep track of which choices produced each result.

Step 3: Plot the Specification Curve

The canonical specification curve figure has two panels:

Top panel: Point estimates (and optionally confidence intervals) sorted from smallest to largest. A horizontal line at zero shows where effects flip sign.

Bottom panel: A grid showing which analytic choices were active for each specification. Each row corresponds to one dimension (outcome, controls, sample, etc.), and dots or highlighting indicates the active choice. This layout lets readers see which choices drive the variation in results.

The bottom panel is often more informative than the top. It reveals patterns like: "the estimate is always large when we use log earnings and always small when we use levels" — which tells you something important about the nature of the result.

Step 4: Joint Inference

Simonsohn et al. (2020) propose a permutation-based joint test. Under the null of no effect:

1. Randomly permute the treatment variable
2. Re-run all specifications on the permuted data
3. Compute a summary statistic (e.g., the median estimate across specs, or the share of specs with positive estimates)
4. Repeat many times to build a null distribution
5. Compare the observed summary statistic to this distribution

This joint test asks: "Is the overall pattern of results across all specifications more extreme than what you would expect by chance alone?"

Interactive: Exploring the Specification Space

Set the true effect to zero and watch how many specifications still appear significant due to chance. This exercise illustrates why joint inference is essential.

How to Do It: Code

1library(specr)
2
3# Create a subsetting variable for the specification space
4df$age_group <- ifelse(df$age >= 25 & df$age <= 55, "prime_age", "other")
5
6# Define the specification space (specr >= 1.0 API)
7specs <- setup(
8data = df,
9y = c("earnings", "log_earnings"),        # outcome options
10x = "treatment",                            # treatment (fixed)
11model = c("lm"),                            # model type
12controls = c(                               # control sets
13  "age + gender",
14  "age + gender + education",
15  "age + gender + education + baseline_earnings"
16),
17subsets = list(age_group = c("prime_age"))  # named list: variable = levels
18)
19
20# Run all specifications
21results <- specr(specs)
22
23# Summary statistics
24summary(results)
25
26# The canonical specification curve plot (estimates + choice indicators)
27library(patchwork)
28plot(results)

How to Report a Specification Curve

A well-reported specification curve section includes:

1. A clear description of the analytic universe. What choices did you vary, and why is each option defensible?
2. The total number of specifications.
3. The specification curve figure with both the estimate panel and the choice indicator panel.
4. Summary statistics. What fraction of specifications produce a positive estimate? What fraction are significant? What is the median estimate and its interquartile range?
5. Joint inference. The p-value from the permutation test.
6. Discussion of what drives variation. Which choices matter most?

Example:

Figure 3 displays the specification curve across 432 specifications varying the outcome definition (3 options), sample restrictions (3 options), control variables (3 options), fixed effects (4 options), and standard error clustering (2 options). Across all specifications, 94% produce positive estimates and 78% are statistically significant at the 5% level. The median estimate is 0.12 (IQR: 0.08–0.18). The joint permutation test, based on 1,000 permutations, rejects the null of no effect (p < 0.001). The primary source of variation is the outcome definition: specifications using log earnings yield systematically larger effects than those using levels.

Common Mistakes

Concept Check