Difference-in-differences (diff-in-diffs) experiments

Attempt to replicate Callaway and Sant’Anna (2021).

Get libraries loaded:

library(tidyverse)
library(did)
library(sandwich)
library(lmtest)

The example dataset supplied with {did}:

dat <- mpdta  # In case I wanted to change anything...
head(dat)

Estimate ATT(g, t) for each group g and time t:

did_mod <- att_gt(
  yname = "lemp",
  gname = "first.treat",
  control_group = "notyettreated",
  idname = "countyreal",
  tname = "year",
  data = dat,
  est_method = "reg",
  base_period = "varying"
)

did_mod

Call:
att_gt(yname = "lemp", tname = "year", idname = "countyreal", 
    gname = "first.treat", data = dat, control_group = "notyettreated", 
    est_method = "reg", base_period = "varying")

Reference: Callaway, Brantly and Pedro H.C. Sant'Anna.  "Difference-in-Differences with Multiple Time Periods." Journal of Econometrics, Vol. 225, No. 2, pp. 200-230, 2021. <https://doi.org/10.1016/j.jeconom.2020.12.001>, <https://arxiv.org/abs/1803.09015> 

Group-Time Average Treatment Effects:

---
Signif. codes: `*' confidence band does not cover 0

P-value for pre-test of parallel trends assumption:  0.16814
Control Group:  Not Yet Treated,  Anticipation Periods:  0
Estimation Method:  Outcome Regression

Now my attempt, initially focussed on the estimates. Standard errors can come later!

What made intuitive sense:

• We’re sweeping along times and groups (the year when a unit is first treated), building the canonical 2×2 datasets and estimating the usual TWFE model as we zoom along.

Here’s what initially tripped me up:

• att_gt treats group = 0 as never treated. A better choice might have been any number greater than the maximum time (e.g., year) in the dataset (including Inf).
• As we sweep along and build the 2×2 datasets, we consider a unit “treated” if it’s ever treated and in the second time point, i.e., so we can check for baseline parallel trends.
• If we are still in the untreated phase, the comparison time point is one before the current time point. If we are in the treated phase, then the comparison time point is the one just before when the unit first received treatment.
• Also a bit of silliness with how I setup the fixed effect model: I tried to include a main effect of the treatment group; however, that’s perfectly correlated with the unit fixed effects, obvs.

ATT_group_time <- function(the_group, the_year) {
  # The first time point we choose as we go depends on whether
  # the unit is treated in this "canonical" 2×2 dataset
  comparision_year <- ifelse(the_year < the_group,
                             the_year - 1,
                             the_group - 1)
  
  this_dat <- dat |>
    mutate(treat = as.numeric(first.treat == the_group),
           time  = as.numeric(year == the_year)) |>
    filter(treat |                # ever treated (current group)
           first.treat == 0 |     # never treated (weird convention of {did})
           year < first.treat) |> # not yet treated (other groups)
    filter(year %in% c(comparision_year, the_year))
  
  mod <- lm(lemp ~ 0 + factor(countyreal) + time + time:treat,
            data = this_dat)
  summary(mod) |>
    tidy() |>
    filter(term == "time:treat") |>
    mutate(group = the_group, .before = "term") |>
    mutate(year = the_year, .before = "term") |>
    mutate(sub_n = nrow(this_dat)) |>
    rename(my_est = estimate) |>
    select(-term)
}

Now zap the whole dataset and compare with {did}’s estimates.

res <- map2(did_mod$group, did_mod$t, ATT_group_time) |>
  bind_rows() |>
  mutate(did_est = did_mod$att, .after = "my_est") |>
  mutate(correct = abs(my_est - did_est) < 1e-6, .after = "did_est")
stopifnot(all(res$correct))
res |> select(group, year, my_est, did_est, correct, sub_n)

Hurrah - it worked! I’m going to worry about the standard errors another day.

Pooling the ATT(g,t)s?

Here are {did}’s estimates, pooled by group:

did_agg_group <- aggte(did_mod, type = "group")
did_agg_group

Call:
aggte(MP = did_mod, type = "group")

Reference: Callaway, Brantly and Pedro H.C. Sant'Anna.  "Difference-in-Differences with Multiple Time Periods." Journal of Econometrics, Vol. 225, No. 2, pp. 200-230, 2021. <https://doi.org/10.1016/j.jeconom.2020.12.001>, <https://arxiv.org/abs/1803.09015> 


Overall summary of ATT's based on group/cohort aggregation:  

Group Effects:

---
Signif. codes: `*' confidence band does not cover 0

Control Group:  Not Yet Treated,  Anticipation Periods:  0
Estimation Method:  Outcome Regression

We can replicate the estimated by just averaging the ATT(g,t) estimates across the t’s:

pool_estimates_naive <- function(ests) {
  data.frame(
    my_pooled_est = mean(ests),
    n_t = length(ests)
  )
}

agg_res <- res |>
  filter(year >= group) |>
  group_by(group) |>
  summarise(pool_estimates_naive(my_est)) |>
  mutate(did_agg_es = did_agg_group$att.egt, .after = "my_pooled_est") |>
  mutate(correct = abs(my_pooled_est - did_agg_es) < 1e-6)
stopifnot(all(agg_res$correct))

agg_res

That works.

References

Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-Differences with multiple time periods. Journal of Econometrics, 225(2), 200–230.