Where Causal Discovery Breaks on Real Data: Measuring the Assumptions One Property at a Time
Organizations: Intuit AI Research · Microsoft
Abstract
Causal discovery is usually evaluated on simulators or on systems whose ground truth follows from physics or design. When it fails on other real data, the failure is attributed to assumptions that may not hold, but which ones fail, and by how much, is rarely measured. We measure the observable properties of the data they bear on, one property at a time, on a real panel. The case study follows 2,901 US public firms over 31 quarters, with 71 types of corporate action from SEC 8-K filings, 11 financial-statement items, and an expert graph over 651 action-state pairs. The panel breaks measurable requirements of discovery: tails that survive winsorisation, balance-sheet items and revenue close to random walks, and interventions too rare to detect effects of a tenth of a standard deviation in 40 of 71 actions. Even so, a marginal test ranks expert-labelled direct above not-direct pairs within each action (AUROC 0.658) better than every null we built, so the panel does carry information about which pairs are direct. Tuned discovery methods, in configurations feasible at this panel size, recover less of it than a simple pairwise cross-correlation when pairs are ranked within each action, and no more when they are ranked within each target, which removes what the target alone predicts. Most direct pairs (132 of 222) involve actions below the support needed for 80% power against a 0.1-SD effect. For flow targets, conditioning on the target's other expert-labelled parents, often filed alongside the action, erodes the marginal discrimination, and most not-direct associations survive it: even with the graph's parent sets supplied, the expected separation does not emerge. For each property, we give a diagnostic computable before any discovery method is run.
Figures & tables
| Method | Family | AUROC | Excess | Within tgt. |
|---|---|---|---|---|
| PCMCI+ | CB | 0.650 | 0.091 | 0.548 |
| [0.604, 0.695] | [0.045, 0.140] | [0.489, 0.606] | ||
| DYNOTEARS | CO | 0.528 | 0.032 | 0.501 |
| [0.509, 0.547] | [0.011, 0.054] | [0.456, 0.544] | ||
| NTS-NOTEARS | CO | 0.575 | 0.052 | 0.380 |
| [0.540, 0.610] | [0.025, 0.081] | [0.314, 0.447] |
| Marginal | Given parents | |||
|---|---|---|---|---|
| Representation | WA | WT | WA | WT |
| Raw level | 0.647 | 0.622 | 0.533 | 0.523 |
| Scaled level | 0.677 | 0.576 | 0.595 | 0.549 |
| Raw change | 0.584 | 0.603 | 0.541 | 0.556 |
| Scaled change | 0.620 | 0.641 | 0.573 | 0.644 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | Diagnostic | Warning sign | Case study | Response |
| Heavy tails | Kurtosis; sum-of-squares share of top 1% | A few rows carry most of the variation | 27%–76% after winsorising | Report with and without the top rows; rank or robust tests |
| Persistence | Within-firm AR(1), bias-corrected; ICC | Corrected coefficient near one | 5 states at 0.83–0.93; 42%–64% of firms above one after correction | Test changes, not levels; give methods differenced states as well as levels; restrict shift nulls; beware Nickell bias |
| Heteroscedasticity | Auxiliary on scale | A large auxiliary , not a rejection | median 0.0180 | Cluster-robust inference; ignore the rejection rate |
| Collinearity | Computable identities; on the other variables | close to one | up to 0.83 | Drop or combine collinear variables |
| Sparse interventions | Planted-effect power against support | Many interventions below the required support | 40 of 71 below 80% at 0.1 SD | Report power; pool or drop rare interventions |
| Bundling | Share of firings shared with another intervention | Most firings shared | median 89% of firings ( Section 4.4 ) | Merge or test bundled interventions jointly |
| Firms | 2,901 |
|---|---|
| Quarters (2018Q2–2025Q4) | 31 |
| Firm-quarters | 89,931 |
| Actions (8-K items) | 71 (10) |
| States | 11 |
| State values reported | 98.2% |
| Firings per action, median (min–max) | 620 (24–20,826) |
| Item | Event | Subtypes |
|---|---|---|
| 1.01 | Entry into a material definitive agreement | 23 |
| 1.02 | Termination of a material definitive agreement | 12 |
| 2.01 | Completion of an acquisition or disposition | 4 |
| 2.03 | Creation of a direct financial obligation | 5 |
| 2.05 | Exit or disposal activities | 4 |
| 3.02 | Unregistered sale of equity securities | 1 |
| Null | Mean | SD | Excess |
|---|---|---|---|
| Firm-block permutation | 0.501 | 0.033 | 0.157 |
| Label permutation | 0.501 | 0.031 | 0.157 |
| Row permutation | 0.514 | 0.032 | 0.144 |
| Stratified firm permutation | 0.527 | 0.034 | 0.131 |
| Shift per action, distance | 0.531 | 0.027 | 0.127 |
| Shift per firm, distance | 0.538 | 0.027 | 0.120 |
| Feature | Count | Definition | Interpretation |
|---|---|---|---|
| Level association | 4 | Persistent association visible in the state level. | |
| Change association | 4 | Transitory or flow-like association visible in one-period changes. | |
| Cumulative displacement | 3 | , | Association with displacement accumulated over multiple quarters. Lag one is omitted because it duplicates the one-period change. |
| Firm-demeaned level | 4 | Within-firm level association after removing time-invariant firm differences. | |
| Firm-demeaned change | 4 | Within-firm association with one-period changes. | |
| Quarterly-slope mean -statistic | 4 | Strength and consistency of the cross-sectional association across eligible quarters. |
| Method | Searched | Configurations |
|---|---|---|
| PCMCI+ | preprocessing only; swept (4 points) to rank | 8 (32 runs) |
| SVAR-RFCI | preprocessing; swept; PAG marks reported | 32 runs |
| DYNOTEARS | preprocessing, , | 40 (TPE) |
| NTS-NOTEARS | preprocessing, , | 40 (TPE) |
| MiTCD | preprocessing, learning rate, three penalties | 40 (TPE) |
| Longitudinal LiNGAM | preprocessing only | 8 |
| Method | AUROC | Panel transformation | Scale | Tuned hyperparameters |
|---|---|---|---|---|
| PCMCI+ | 0.650 | First differences | Raw | – |
| DYNOTEARS | 0.528 | Firm and quarter effects | Raw | , |
| NTS-NOTEARS | 0.575 | First differences | Raw | , |
| Long. LiNGAM | 0.700 | First differences | Raw | – |
| VAR Granger | 0.620 | Firm and quarter effects | Raw | – |
| MiTCD | 0.541 | Firm effects | Raw | Learning rate ; ; ; |
| Whole PAG | Direct (222) | Not-direct (429) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Transformation | none | none | ||||||||
| Winsorised | ||||||||||
| Pooled | 0.01 | 3,653 | 2 | 0 | 62 | 160 | 107 | 322 | ||
| 0.1 | 6,011 | 2 | 1 | 94 | 128 | 186 | 243 | |||
| 0.5 | 11,122 | 0 | 0 | 172 | 50 | 347 | 82 | |||
| 0.9 | 18,576 | 0 | 0 | 221 | 1 | 426 | 3 | |||
| Cap | ||||
|---|---|---|---|---|
| 2 | 62 / 160 / 0 | 94 / 128 / 0 | 172 / 50 / 0 | 221 / 1 / 0 |
| 3 | 57 / 165 / 0 | 89 / 133 / 0 | 171 / 51 / 0 | 221 / 1 / 0 |
| 4 | 53 / 168 / 1 | 85 / 137 / 0 | 164 / 58 / 0 | 221 / 1 / 0 |
| 5 | 49 / 172 / 1 | 79 / 143 / 0 | 160 / 62 / 0 | 221 / 1 / 0 |
| Conditioning set | Raw level | Raw change | Scaled level | Scaled change |
|---|---|---|---|---|
| None (marginal) | 0.641 / 0.602 | 0.611 / 0.599 | 0.682 / 0.605 | 0.609 / 0.647 |
| Expert parents | 0.505 / 0.469 | 0.562 / 0.588 | 0.587 / 0.533 | 0.568 / 0.622 |
| Direct | Not direct | |||||
|---|---|---|---|---|---|---|
| Specification | Alone | Given parents | Kept | Alone | Given parents | Kept |
| Scaled level | 70.4% | 53.9% | 70% | 50.5% | 43.9% | 83% |
| Scaled change | 40.9% | 33.0% | 74% | 20.7% | 16.0% | 64% |
| Raw level | 73.9% | 56.5% | 76% | 50.5% | 41.7% | 78% |
| Raw change | 26.1% | 25.2% | 83% | 16.9% | 13.8% | 70% |
| Parents | Scaled level | Scaled change | Raw level | Raw change |
|---|---|---|---|---|
| Pre-source | 0.679 / 0.595 (0.87) | 0.611 / 0.626 (1.00) | 0.633 / 0.610 (0.84) | 0.571 / 0.595 (1.00) |
| Same-time | 0.636 / 0.548 (0.76) | 0.592 / 0.644 (1.03) | 0.554 / 0.552 (0.70) | 0.530 / 0.558 (1.01) |
| Post-source | 0.660 / 0.579 (0.87) | 0.622 / 0.639 (1.00) | 0.633 / 0.611 (0.82) | 0.560 / 0.579 (1.00) |
| Specification | Loss | Kept | Ratio |
|---|---|---|---|
| Scaled level | 0.094 0.068 | 83% 79% | 0.59 0.55 |
| Scaled change | 0.047 0.058 | 64% 64% | 1.05 1.07 |
| Raw level | 0.126 0.094 | 81% 78% | 0.56 0.53 |
| Raw change | 0.052 0.058 | 75% 75% | 1.05 1.05 |
| Level | Change | |||
|---|---|---|---|---|
| Conditioning set | WA | WT | WA | WT |
| None (marginal) | 0.647 | 0.649 | 0.549 | 0.601 |
| Expert parents | 0.538 | 0.579 | 0.557 | 0.556 |
| Level | Change | |||
|---|---|---|---|---|
| Conditioning set | WA | WT | WA | WT |
| None (marginal) | 0.452 | 0.395 | 0.516 | 0.597 |
| Expert parents | 0.532 | 0.532 | 0.516 | 0.516 |
| Treated | , df | , df | CR2-BM | Wild boot. | Joint |
|---|---|---|---|---|---|
| 10 | 0.050 | 0.074 | 0.048 | 0.044 | 0.156 |
| 20 | 0.052 | 0.074 | 0.046 | 0.048 | 0.084 |
| 30 | 0.048 | 0.056 | 0.046 | 0.042 | 0.070 |
| 40 | 0.034 | 0.050 | 0.032 | 0.032 | 0.098 |
| 80 | 0.056 | 0.060 | 0.056 | 0.050 | 0.062 |
| Pair score | Within-action | Pooled | Shift null | Excess |
|---|---|---|---|---|
| Minimum | 0.640 | 0.601 | 0.533 | 0.106 |
| Maximum | 0.638 | 0.604 | 0.533 | 0.104 |
| Mean | 0.644 | 0.613 | 0.537 | 0.107 |
| Joint Wald ∗ | 0.658 | 0.609 | 0.538 | 0.120 |
| Labels | Pairs | Direct | Comparisons | Within-action | Label perm. q95 | Within tgt. |
| Final labels (headline) | 651 | 222 | 869 | 0.658 | 0.547 | 0.564 |
| Unanimous pairs only | 442 | 95 | 340 | 0.647 | 0.589 | 0.516 |
| No vote for the other class | 594 | 185 | 652 | 0.687 | 0.555 | 0.592 |
| Direct if direct votes | 651 | 200 | 861 | 0.653 | 0.545 | 0.559 |
| Direct if direct vote | 651 | 240 | 883 | 0.675 | 0.546 | 0.581 |
| Direct only if all 3 vote direct | 651 | 95 | 601 | 0.566 | 0.564 | 0.541 |
| State | IQR | ICC | Firms | |||
|---|---|---|---|---|---|---|
| Equity | 0.93 | [0.84, 0.99] | 1.05 | 64% | 0.78 | 2,901 |
| Noncurrent assets | 0.91 | [0.82, 0.97] | 1.03 | 59% | 0.26 | 2,264 |
| Current assets | 0.87 | [0.74, 0.95] | 0.99 | 47% | 0.37 | 2,901 |
| Current liabilities | 0.84 | [0.68, 0.94] | 0.96 | 42% | 0.80 | 2,901 |
| Revenues | 0.83 | [0.58, 0.96] | 0.95 | 42% | 0.70 | 2,741 |
| Operating income | 0.43 | [0.14, 0.69] | 0.51 | 8% | 0.70 | 2,888 |
| Excess kurtosis | Top 1% | |||
|---|---|---|---|---|
| State | raw | winsorised | Skew | SS share |
| Noncurrent assets | 9,769 | 451 | 16.2 | 59% |
| Financing cash flow | 77,082 | 86 | 7.6 | 65% |
| Current assets | 10,009 | 76 | 5.6 | 27% |
| Current liabilities | 9,495 | 68 | 7.9 | 76% |
| Net income | 81,358 | 66 | -7.6 | 74% |
Causal Discovery
Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.
See all 239 papers