LUCID: Learning Under Confounding for Inference and Discovery in Time Series
Organizations: Bloomberg L.P.
Abstract
Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct edges. We propose LUCID (Learning Under Confounding for Inference and Discovery, a regime-adaptive deconfounding layer that first estimates the confounding regime from data using a Marčenko--Pastur spectral router, then applies a deconfounding strategy matched to that regime. When the spectrum indicates pervasive factor confounding, LUCID attenuates factor-dominated variation and recovers contemporaneous (lag-) structure from the resulting innovations, with edge selection calibrated against a data-driven edge-free null. Rather than being tied to a particular discovery algorithm, it can wrap existing discovery engines; we demonstrate consistent improvements across three such methods. On a diverse synthetic out-of-distribution benchmark spanning changes in confounder strength and sparsity, loading density, lag structure, volatility dynamics, edge heterogeneity, persistence, intermittency, and tail behavior, LUCID achieves the best family-weighted directed, lag-resolved graph (), improving over the strongest baseline by absolute ( relative). Its advantage widens relative to looser lag-collapsed scoring, and remains robust under intermittent and heavy-tailed confounding. Code reproducing the method, the benchmark generators, and every reported experiment is available at https://github.com/bloomberg/causal-ts.
Figures & tables
| Method | Overall | Sparse-ER | Sparse-SF | Sparse-SW | Perv-vol | Perv-struct | GARCH | Perv-dense | Perv-mixedlag | Perv-heterog | Perv-nearunit |
| CDNOTS | |||||||||||
| CDNOTS + tetrad | |||||||||||
| LUCID ( CDNOTS , default) | |||||||||||
| VARLiNGAM | |||||||||||
| LPCMCI † | — | ||||||||||
| TS-ICD ‡ |
| Method | Precision | Recall | SHD |
| LUCID ( CDNOTS ) | |||
| LUCID (PCMCI+) | |||
| LUCID (NTS-NOTEARS) | |||
| LPCMCI † | |||
| TS-ICD | |||
| PCMCI+ |
| P | R | SHD | |||
| (10 seeds) | (50 seeds) | ||||
| CDNOTS | |||||
| LUCID ( CDNOTS ) | |||||
| PCMCI+ | |||||
| LUCID (PCMCI+) |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Role | Value | Source |
| analysis lag horizon | dataset-dep. | required input | |
| CDNOTS CI | skeleton-discovery significance | conventional | |
| S L level | lag- edge-free-null level (§ 4.3 ) | conventional | |
| S L draws | null draws for S L calibration | fixed | |
| (Eq. 2 ) | persistence-gate width (§ 4.3 ) | held-out selection | |
| Orientation lag window | lag used by lead-lag asymmetry (§ 4.3 ) | structural |
| Family | cfg | Loadings | Factor dynamics | Confounding strength | ||||
| Sparse-ER | – | – | local forks | — | — | confounded fraction – | ||
| Sparse-SF | – | – | local forks | — | — | confounded fraction | ||
| Sparse-SW | – | – | local forks | — | — | confounded fraction – | ||
| Perv-vol | , block-disjoint | stoch. vol / regime-switch / static / AR | , edges/group, | |||||
| Perv-struct | as above; overlap / signed / nonlinear / heavy-tail | static | as above | |||||
| GARCH | , block-disjoint | GARCH(1,1) | as above |
| Method | Overall | Sparse-ER | Sparse-SF | Sparse-SW | Perv-vol | Perv-struct | GARCH | Perv-dense | Perv-mixedlag | Perv-heterog | Perv-nearunit |
| CDNOTS | |||||||||||
| CDNOTS + tetrad | |||||||||||
| LUCID ( CDNOTS , default) | |||||||||||
| VARLiNGAM | |||||||||||
| LPCMCI † | — | ||||||||||
| TS-ICD ‡ |
| Method | Median | Mean | p95 | Max |
| CDNOTS | ||||
| LUCID layer | ||||
| PCMCI+ | ||||
| LUCID layer | ||||
| NTS-NOTEARS | ||||
| LUCID layer |
| Contamination | Arm | Lag- recall | |
| Low ( ) | Blanket drop | ||
| Keep all | |||
| S L | |||
| High ( ) | Blanket drop | ||
| Keep all | |||
| S L |
| Overall | P-vol | P-struct | GARCH | P-dense | P-mixedlag | P-heterog | P-nearunit | |
| LUCID (as shipped) | ||||||||
| unconditional filtering |
| duty cycle | 1.00 | 0.50 | 0.25 | 0.10 | 0.05 |
| CDNOTS | |||||
| Tetrad filter | |||||
| Routed (tetrad base) | |||||
| LUCID |
| Method | Gaussian | Laplace | Uniform | Exp. | Het. | Hist. | PNL | |
| CDNOTS | ||||||||
| CDNOTS + tetrad | ||||||||
| DYNOTEARS | ||||||||
| PCMCI+ | ||||||||
| LPCMCI | ||||||||
| VARLiNGAM |
| FPR | |||||||
| Balanced acc. |
| FPR | ||||
| FNR | ||||
| Balanced acc. | ||||
| FNR, loadings |
| Value | Precision | Recall | |
| Sparse-branch PDS | |||
Time-Series Causal Discovery
Time-series causal discovery seeks to infer which variables influence others over time and when those effects occur.
See all 64 papers