Forests for Differences: Beyond Parametric Difference-in-Differences
Organizations: Institute of Mathematics and Computer Sciences at University of S˜ao Paulo, Brazil Av. Trab. S˜ao Carlense 400, 13566-590 S˜ao Carlos (SP), Brazil · Institute of Mathematics and Computer Sciences at University of S˜ao Paulo, Brazil Av. Trab. S˜ao Carlense, 400, S˜ao Carlos, 13566-590, Brazil
Abstract
This paper introduces the Difference-in-Differences Bayesian Causal Forest (DiD-BCF), a novel non-parametric model addressing key challenges in DiD estimation, such as staggered adoption and heterogeneous treatment effects. DiD-BCF provides a unified framework for estimating Average (ATE), Group-Average (GATE), and Conditional Average Treatment Effects (CATE). By structuring separate regression forests for cohort-specific levels, common untreated trends and treatment effects, the model directly embeds conditional parallel trends and no-anticipation restrictions into Bayesian nonparametrics. We establish non-parametric identification and integrated treatment effect consistency of the exact treatment effect posterior distribution. Extensive simulations demonstrate DiD-BCF's superior performance over established benchmarks, particularly under non-linearity, selection biases, and effect heterogeneity. Applied to U.S. minimum wage policy, the model uncovers significant conditional treatment effect heterogeneity related to county population, insights obscured by traditional methods. DiD-BCF offers a robust and versatile tool for more nuanced causal inference in modern DiD applications.
Figures & tables
| Scenario | DiD-BCF | TWFE event study | Synthetic DiD | CS | did2s | DoubleML | grf-DiD |
|---|---|---|---|---|---|---|---|
| A. Size under parallel trends (slope rule) | |||||||
| Unconditional PT | 4.0 | 4.5 | 0.0 | 5.0 | 4.5 | 4.0 | 5.0 |
| Conditional PT only | 8.0 | 75.5 | 5.5 | 7.0 | 75.0 | 5.5 | 10.0 |
| B. Power against linear trends (slope rule) | |||||||
| Slope 0.00 | 4.0 | 4.5 | 0.0 | 5.0 | 4.5 | 4.0 | 5.0 |
| Slope 0.05 | 20.5 | 10.0 | 0.0 | 9.0 | 9.5 | 8.5 | 8.5 |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Estimator | Bias | SD(err) | RMSE | Cov.90 | Cov.95 | Len.95 | ||
|---|---|---|---|---|---|---|---|---|
| Canonical: baseline | ||||||||
| DiD-BCF | 1 | 0.004 | 0.099 | 0.099 | 0.920 | 0.950 | 0.425 | 0.837 |
| TWFE | 1 | -0.024 | 0.177 | 0.179 | 0.970 | 0.980 | 0.837 | 1.067 |
| Callaway–Sant’Anna | 1 | 0.000 | 0.171 | 0.171 | 0.940 | 0.970 | 0.751 | 1.096 |
| Gardner (did2s) | 1 | -0.022 | 0.183 | 0.185 | 0.960 | 0.990 | 0.864 | 1.032 |
| DoubleML DR-DiD | 1 | 0.003 | 0.201 | 0.201 | 0.960 | 0.990 | 0.931 | 1.120 |
| Estimator | Bias | SD(err) | RMSE | Cov.90 | Cov.95 | Len.95 | ||
|---|---|---|---|---|---|---|---|---|
| Canonical: baseline | ||||||||
| DiD-BCF | 1 | -0.019 | 0.110 | 0.112 | 0.950 | 1.000 | 0.559 | 1.003 |
| TWFE | 1 | -0.048 | 0.173 | 0.179 | 0.950 | 0.990 | 0.820 | 1.068 |
| Callaway–Sant’Anna | 1 | -0.031 | 0.219 | 0.221 | 0.870 | 0.970 | 0.902 | 1.031 |
| Gardner (did2s) | 1 | -0.047 | 0.173 | 0.179 | 0.950 | 0.980 | 0.846 | 1.059 |
| DoubleML DR-DiD | 1 | -0.039 | 0.301 | 0.304 | 0.870 | 0.920 | 1.157 | 0.941 |
| Estimator | Bias | SD(err) | Cov.90 | Cov.95 | Size (5%) | MCSE | |
|---|---|---|---|---|---|---|---|
| ATT, ATT | |||||||
| DiD-BCF | 1 | 0.002 | 0.103 | 0.930 | 0.960 | 0.040 | 0.014 |
| TWFE | 1 | -0.022 | 0.193 | 0.930 | 0.955 | 0.045 | 0.015 |
| Callaway–Sant’Anna | 1 | 0.000 | 0.171 | 0.870 | 0.940 | 0.060 | 0.024 |
| Gardner (did2s) | 1 | -0.022 | 0.183 | 0.920 | 0.960 | 0.040 | 0.020 |
| DoubleML DR-DiD | 1 | 0.003 | 0.203 | 0.910 | 0.960 | 0.040 | 0.020 |
| Estimator | Bias | SD(err) | RMSE | Cov.95 | Len.95 | ||
|---|---|---|---|---|---|---|---|
| Sample-size sweep: independent errors | |||||||
| DiD-BCF | 50 | -0.014 | 0.248 | 0.248 | 0.970 | 1.094 | 0.822 |
| DiD-BCF | 100 | 0.018 | 0.165 | 0.166 | 0.980 | 0.782 | 1.037 |
| DiD-BCF | 200 | -0.019 | 0.110 | 0.112 | 1.000 | 0.559 | 1.003 |
| DiD-BCF | 400 | 0.004 | 0.084 | 0.084 | 0.980 | 0.399 | 0.971 |
| DiD-BCF | 800 | 0.001 | 0.068 | 0.068 | 0.960 | 0.280 | 0.776 |
| Estimator | Bias | SD(err) | RMSE | Cov.95 | Len.95 | |||
|---|---|---|---|---|---|---|---|---|
| Sample-size sweep: independent errors | ||||||||
| DiD-BCF | 50 | 2 | -0.019 | 0.259 | 0.260 | 0.960 | 1.117 | 0.816 |
| DiD-BCF | 50 | 3 | -0.026 | 0.269 | 0.270 | 0.950 | 1.130 | 0.970 |
| DiD-BCF | 100 | 2 | 0.015 | 0.181 | 0.182 | 0.990 | 0.803 | 0.993 |
| DiD-BCF | 100 | 3 | 0.002 | 0.187 | 0.187 | 0.970 | 0.816 | 1.077 |
| DiD-BCF | 200 | 2 | -0.025 | 0.119 | 0.122 | 0.980 | 0.572 | 0.994 |
| Estimator | True ATT( ) | Bias | SD(err) | RMSE | Cov.95 | |
|---|---|---|---|---|---|---|
| Staggered adoption (baseline dynamics) | ||||||
| DiD-BCF | 0 | 4.072 | 0.095 | 0.114 | 0.148 | 0.870 |
| DiD-BCF | 1 | 5.701 | -0.025 | 0.122 | 0.125 | 0.970 |
| DiD-BCF | 2 | 6.102 | -0.048 | 0.157 | 0.164 | 0.970 |
| DiD-BCF | 3 | 6.046 | -0.374 | 0.223 | 0.435 | 0.660 |
| TWFE | 0 | 4.069 | 0.196 | 0.146 | 0.245 | 0.930 |
| Estimator | Cells | True | Bias | SD(err) | RMSE | Cov.95 | |
|---|---|---|---|---|---|---|---|
| Staggered adoption (baseline dynamics) | |||||||
| DiD-BCF | 1 | 9 | 5.355 | -0.036 | 0.178 | 0.318 | 0.747 |
| Callaway–Sant’Anna | 1 | 9 | 5.355 | -0.017 | 0.296 | 0.298 | 0.993 |
| DoubleML DR-DiD | 1 | 9 | 5.352 | -0.008 | 0.410 | 0.412 | 0.979 |
| grf-DiD (Wang) | 1 | 9 | 5.352 | -0.012 | 0.299 | 0.301 | 0.949 |
| DiD-BCF | 2 | 9 | 5.355 | -0.044 | 0.188 | 0.329 | 0.732 |
| Estimator | True | Bias | RMSE | Cov.95 | |||
|---|---|---|---|---|---|---|---|
| Staggered adoption (baseline dynamics) | |||||||
| , cohort | |||||||
| DiD-BCF | 4 | 4 | 0 | 2.748 | -0.076 | 0.175 | 0.960 |
| Callaway–Sant’Anna | 4 | 4 | 0 | 2.748 | -0.049 | 0.303 | 0.960 |
| DoubleML DR-DiD | 4 | 4 | 0 | 2.746 | -0.065 | 0.416 | 0.940 |
| grf-DiD (Wang) | 4 | 4 | 0 | 2.746 | -0.050 | 0.300 | 0.870 |
| Estimator | True ATT( ) | Bias | SD(err) | RMSE | Cov.95 | ||
|---|---|---|---|---|---|---|---|
| Staggered adoption (baseline dynamics) | |||||||
| DiD-BCF | 2 | 0 | 4.072 | 0.096 | 0.118 | 0.152 | 0.860 |
| DiD-BCF | 2 | 1 | 5.701 | -0.034 | 0.130 | 0.134 | 0.970 |
| DiD-BCF | 2 | 2 | 6.102 | -0.061 | 0.157 | 0.168 | 0.940 |
| DiD-BCF | 2 | 3 | 6.046 | -0.392 | 0.230 | 0.455 | 0.630 |
| DiD-BCF | 3 | 0 | 4.050 | 0.080 | 0.122 | 0.146 | 0.880 |
| Comparison type | Weight | Contribution | |
|---|---|---|---|
| Staggered adoption (baseline dynamics) | |||
| Earlier vs Later | 0.072 | 3.386 | 0.245 |
| Later vs Earlier | 0.039 | 4.038 | 0.156 |
| Treated vs Untreated | 0.222 | 5.581 | 1.241 |
| TWFE estimate | 4.830 | ||
| true ATT | 5.299 | ||
| Target | Estimator | Bias | RMSE | Cov. | Bias | RMSE | Cov. | Bias | RMSE | Cov. | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Independent errors | |||||||||||||
| DiD-BCF | -0.014 | 0.248 | 0.97 | 0.82 | 0.018 | 0.166 | 0.98 | 1.04 | -0.019 | 0.112 | 1.00 | 1.00 | |
| TWFE | -0.046 | 0.418 | 0.95 | 0.88 | 0.003 | 0.265 | 0.98 | 1.07 | -0.048 | 0.179 | 0.99 | 1.07 | |
| Callaway–Sant’Anna | 0.055 | 0.455 | 0.94 | 0.90 | 0.013 | 0.290 | 0.97 | 1.09 | -0.031 | 0.221 | 0.98 | 1.03 | |
| Gardner (did2s) | -0.046 | 0.418 | 0.95 | 0.87 | 0.003 | 0.265 | 0.98 | 1.06 | -0.047 | 0.179 | 0.98 | 1.06 | |
| Design | Estimator | RMSE | MAPE | Cov. | RMSE | MAPE | Cov. | RMSE | MAPE | Cov. |
|---|---|---|---|---|---|---|---|---|---|---|
| Independent | DiD-BCF | 0.465 | 0.117 | 0.92 | 0.345 | 0.086 | 0.95 | 0.249 | 0.060 | 0.97 |
| TWFE | 1.139 | 0.285 | 0.59 | 1.107 | 0.289 | 0.45 | 0.902 | 0.221 | 0.35 | |
| Callaway–Sant’Anna | 1.126 | 0.291 | 0.57 | 1.094 | 0.288 | 0.41 | 1.069 | 0.285 | 0.29 | |
| Gardner (did2s) | 1.139 | 0.285 | 0.58 | 1.107 | 0.289 | 0.45 | 1.071 | 0.284 | 0.32 | |
| DoubleML DR-DiD | 1.305 | 0.324 | 0.75 | 1.137 | 0.295 | 0.56 | 1.082 | 0.288 | 0.38 | |
| Scenario | DiD-BCF | TWFE event study | Synthetic DiD | CS | did2s | DoubleML | grf-DiD |
|---|---|---|---|---|---|---|---|
| Parallel trends hold | 4.0 / 0.5 / 2.5 | 4.5 / NA / 3.5 | 0.0 / 0.0 / 0.0 | 5.0 / 5.0 / 4.5 | 4.5 / 5.5 / 3.5 | 4.0 / 3.0 / 4.5 | 5.0 / 4.0 / 4.0 |
| Conditional PT only | 8.0 / 0.0 / 3.0 | 75.5 / NA / 71.0 | 5.5 / 0.0 / 3.5 | 7.0 / 8.1 / 7.5 | 75.0 / 8.5 / 71.0 | 5.5 / 2.0 / 6.5 | 10.0 / 3.0 / 9.5 |
| Group slope 0.05 | 20.5 / 0.5 / 9.0 | 10.0 / NA / 6.0 | 0.0 / 0.0 / 0.0 | 9.0 / 5.0 / 9.5 | 9.5 / 6.0 / 6.0 | 8.5 / 1.5 / 4.5 | 8.5 / 5.0 / 6.0 |
| Group slope 0.10 | 52.5 / 0.5 / 36.0 | 25.0 / NA / 20.0 | 0.5 / 0.0 / 0.0 | 32.0 / 5.0 / 27.5 | 24.0 / 5.5 / 19.0 | 18.0 / 2.0 / 15.0 | 28.5 / 4.0 / 23.0 |
| Group slope 0.20 | 93.0 / 0.5 / 86.5 | 70.0 / NA / 69.0 | 17.0 / 0.0 / 6.5 | 83.0 / 5.0 / 76.0 | 70.0 / 5.0 / 68.5 | 54.0 / 3.5 / 49.0 | 79.5 / 3.5 / 71.5 |
| Group slope 0.40 | 100.0 / 0.5 / 100.0 | 100.0 / NA / 100.0 | 85.0 / 0.0 / 76.5 | 100.0 / 5.0 / 100.0 | 100.0 / 5.0 / 100.0 | 98.5 / 2.5 / 99.0 | 100.0 / 4.0 / 100.0 |
| Scenario | DiD-BCF | TWFE event study | Synthetic DiD | CS | did2s | DoubleML | grf-DiD |
|---|---|---|---|---|---|---|---|
| Parallel trends hold | 5.0 / 0.0 / 4.0 | 4.5 / NA / 3.5 | 0.0 / 0.0 / 0.0 | 5.5 / 7.7 / 4.5 | 4.5 / 5.5 / 3.5 | 2.0 / 2.0 / 2.5 | 5.0 / 6.0 / 4.0 |
| Conditional PT only | 8.5 / 0.0 / 4.5 | 75.5 / NA / 71.0 | 5.5 / 0.0 / 3.5 | 6.6 / 13.8 / 7.7 | 75.0 / 8.5 / 71.0 | 4.5 / 1.5 / 5.5 | 8.0 / 3.0 / 6.5 |
| Group slope 0.05 | 20.0 / 0.0 / 10.5 | 10.0 / NA / 6.0 | 0.0 / 0.0 / 0.0 | 10.5 / 7.7 / 8.0 | 9.5 / 6.0 / 6.0 | 6.5 / 3.0 / 4.0 | 11.5 / 6.0 / 8.0 |
| Group slope 0.10 | 48.5 / 0.0 / 34.5 | 25.0 / NA / 20.0 | 0.5 / 0.0 / 0.0 | 30.0 / 7.7 / 22.0 | 24.0 / 5.5 / 19.0 | 19.0 / 1.5 / 12.5 | 31.5 / 6.0 / 25.0 |
| Group slope 0.20 | 90.0 / 0.0 / 83.0 | 70.0 / NA / 69.0 | 17.0 / 0.0 / 6.5 | 76.5 / 7.7 / 72.0 | 70.0 / 5.0 / 68.5 | 53.0 / 0.5 / 44.5 | 75.0 / 6.5 / 70.5 |
| Group slope 0.40 | 100.0 / 0.0 / 100.0 | 100.0 / NA / 100.0 | 85.0 / 0.0 / 76.5 | 100.0 / 7.7 / 100.0 | 100.0 / 5.0 / 100.0 | 98.0 / 3.5 / 97.5 | 100.0 / 6.0 / 100.0 |
| Error process | Prior rate | Bias | RMSE | MAE | 95% coverage | Mean 95% length | |
|---|---|---|---|---|---|---|---|
| AR(1) | 50 | 0.1 | -0.043 | 0.294 | 0.234 | 0.810 | 0.779 |
| AR(1) | 50 | 1.0 | -0.040 | 0.288 | 0.227 | 0.820 | 0.795 |
| AR(1) | 50 | 10.0 | -0.036 | 0.290 | 0.234 | 0.890 | 0.992 |
| AR(1) | 200 | 0.1 | 0.007 | 0.129 | 0.101 | 0.820 | 0.358 |
| AR(1) | 200 | 1.0 | 0.007 | 0.128 | 0.100 | 0.850 | 0.363 |
| AR(1) | 200 | 10.0 | 0.003 | 0.126 | 0.097 | 0.870 | 0.388 |
| Specification | Group | (county-years) | Effect | 95% CI | |
|---|---|---|---|---|---|
| County random intercept | All treated | 291 | 0.001 | ||
| County random intercept | Small population | 72 | 0.020 | ||
| County random intercept | Medium population | 119 | |||
| County random intercept | High population | 100 | 0.071 | ||
| No random intercept | All treated | 291 | 0.009 | ||
| No random intercept | Small population | 72 | 0.075 |
| Prior rate | Specification | Run | ATT (%) | 95% interval (%) | |
|---|---|---|---|---|---|
| 0.1 | No random intercept | 1 | 0.012 | ||
| 0.1 | No random intercept | 2 | 0.007 | ||
| 0.1 | No random intercept | 3 | 0.004 | ||
| 0.1 | County random intercept | 1 | |||
| 0.1 | County random intercept | 2 | |||
| 0.1 | County random intercept | 3 |
| Prior rate | Run | Slope | 95% interval | (slope) | (min lead) | (Bonf. lead) |
|---|---|---|---|---|---|---|
| 0.1 | 1 | 0.157 | 0.021 | 0.064 | ||
| 0.1 | 2 | 0.171 | 0.036 | 0.108 | ||
| 0.1 | 3 | 0.209 | 0.026 | 0.078 | ||
| 1.0 | 1 | 0.206 | 0.046 | 0.138 | ||
| 1.0 | 2 | 0.161 | 0.049 | 0.146 | ||
| 1.0 | 3 | 0.181 | 0.033 | 0.098 |
| Estimator | RMSE (SD) | MAE | MAPE | Cov.95 | Len.95 | |
|---|---|---|---|---|---|---|
| Canonical: baseline | ||||||
| DiD-BCF | 1 | 0.249 (0.058) | 0.191 | 0.060 | 0.968 | 1.027 |
| TWFE | 1 | 0.902 (0.041) | 0.762 | 0.221 | 0.352 | 0.974 |
| Callaway–Sant’Anna | 1 | 1.069 (0.065) | 0.878 | 0.285 | 0.292 | 0.898 |
| Gardner (did2s) | 1 | 1.071 (0.065) | 0.882 | 0.284 | 0.320 | 0.996 |
| DoubleML DR-DiD | 1 | 0.914 (0.049) | 0.770 | 0.225 | 0.404 | 1.153 |
| Object | Guarantee | Result |
|---|---|---|
| Identification | Conditional parallel trends is equivalent to an additive cohort–time baseline; the total effect is the four-cell contrast | Lemma \thechapter.G.1 , Theorem \thechapter.G.3 |
| Full-panel average | Efficient influence function under a Gaussian panel experiment, including treated-composition uncertainty | Theorem \thechapter.G.5 |
| Adaptive posterior mixture | Sufficient bias, selection-complexity, and variance conditions for a Gaussian limit | Theorem \thechapter.G.6 |
| Finite support | Generic-truth BvM with fixed depth caps; simple-truth adaptation and its local nonuniformity | Thm. \thechapter.G.9 Thm. \thechapter.G.11 |
| Unknown scalar variance | Generic finite-support posterior mean and BvM, empirical standardization, feasible cluster variance | Thm. \thechapter.G.16 Cor. \thechapter.G.17 |
| Continuous histograms | Growing balanced saturated partitions; functional bias, posterior moments, cluster and population intervals | Thm. \thechapter.G.21 Cor. \thechapter.G.23 |
Causal Effect Estimation
Causal effect estimation measures how outcomes would change under interventions, using experiments or observational data and explicit assumptions.
See all 287 papers