LUCID: Learning Under Confounding for Inference and Discovery in Time Series

Mohammad Fesanghary

cs.LGSep 25, 2026

LUCID: Learning Under Confounding for Inference and Discovery in Time Series

Authors: Mohammad Fesanghary

Organizations: Bloomberg L.P.

Abstract

Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct edges. We propose LUCID (Learning Under Confounding for Inference and Discovery, a regime-adaptive deconfounding layer that first estimates the confounding regime from data using a Marčenko--Pastur spectral router, then applies a deconfounding strategy matched to that regime. When the spectrum indicates pervasive factor confounding, LUCID attenuates factor-dominated variation and recovers contemporaneous (lag- $0$ ) structure from the resulting innovations, with edge selection calibrated against a data-driven edge-free null. Rather than being tied to a particular discovery algorithm, it can wrap existing discovery engines; we demonstrate consistent improvements across three such methods. On a diverse synthetic out-of-distribution benchmark spanning changes in confounder strength and sparsity, loading density, lag structure, volatility dynamics, edge heterogeneity, persistence, intermittency, and tail behavior, LUCID achieves the best family-weighted directed, lag-resolved graph $F_1$ ( $0.60$ ), improving over the strongest baseline by $0.19$ absolute ( $\approx\!46\%$ relative). Its advantage widens relative to looser lag-collapsed scoring, and remains robust under intermittent and heavy-tailed confounding. Code reproducing the method, the benchmark generators, and every reported experiment is available at https://github.com/bloomberg/causal-ts.

Figures & tables

Panel of Figure 1 — Figure 1: LUCID pipeline. An MP spectral router selects a sparse or pervasive deconfounding branch. Both use the same base discovery engine; the sparse branch applies the PDS edge filter to every candidate edge, while the pervasive branch recovers the lag- $0$ slice with S $-$ L. We use CDNOTS by default and also wrap PCMCI+ and NTS-NOTEARS.

Method	Overall	Sparse-ER	Sparse-SF	Sparse-SW	Perv-vol	Perv-struct	GARCH	Perv-dense	Perv-mixedlag	Perv-heterog	Perv-nearunit
CDNOTS	$0.331$ $\pm.019$	$0.619$	$0.486$	$0.796$	$0.186$	$0.239$	$0.273$	$0.146$	$0.247$	$0.217$	$0.104$
CDNOTS + tetrad	$0.411$ $\pm.021$	$0.621$	$0.541$	$0.797$	$0.355$	$0.352$	$0.340$	$0.148$	$\mathbf{0.329}$	$0.375$	$0.254$
LUCID ( CDNOTS , default)	$\mathbf{0.601}$ $\pm.025$	$0.810$	$0.586$	$0.843$	$0.626$	$\mathbf{0.781}$	$\mathbf{0.878}$	$0.345$	$\mathbf{0.330}$	$\mathbf{0.465}$	$0.344$
VARLiNGAM	$0.341$ $\pm.020$	$0.397$	$0.276$	$0.635$	$0.381$	$0.372$	$0.429$	$0.168$	$0.131$	$0.426$	$0.197$
LPCMCI †	$0.355$ $\pm.023$	$0.546$	$0.403$	$0.707$	$0.287$	$0.288$	$0.301$	—	$0.274$	$0.247$	$0.146$
TS-ICD ‡	$0.133$ $\pm.024$	$0.046$	$0.007$	$0.049$	$0.352$	$0.240$	$0.317$	$0.005$	$0.045$	$0.143$	$0.121$

Table 1: Out-of-distribution directed, lag-resolved graph

F_{1}^{\text{dir}}

, family-weighted over ten confounding families and

50

seeds. Overall reports mean

\pm

std. The three sparse topologies are treated as separate families rather than collapsed into one. Each LUCID row is placed beneath its corresponding base engine. Bold marks the best method in each column and methods not significantly worse than it (one-sided paired Wilcoxon signed-rank test,

\alpha=0.05

). Pair-level results are in Appendix A.2 .

Figure 2: Per-family out-of-distribution directed, lag-resolved graph-

F_{1}^{\text{dir}}

(

50

seeds, mean

\pm

std), split by confounding regime. LUCID is the only method strong in both: the baselines competitive on the sparse topologies fall to

\leq\!0.43

on every pervasive family, while LUCID leads or ties the best of them on all seven, including the dense-loading stress case. Values are those of Table 1 .

Method	Precision	Recall	SHD
LUCID ( CDNOTS )	$0.614$	$0.571$	$13.4$
LUCID (PCMCI+)	$0.593$	$0.615$	$14.5$
LUCID (NTS-NOTEARS)	$0.491$	$0.601$	$13.6$
LPCMCI †	$0.233$	$0.671$	$22.6$
TS-ICD	$0.168$	$0.123$	$14.0$
PCMCI+	$0.220$	$0.641$	$31.2$

Table 2: Precision, recall, and SHD underlying the OOD results, pooled over all datasets and

50

seeds. LUCID ’s gain is primarily precision-driven, at slightly lower recall than the baselines. Since

F_{1}

is computed per dataset and then averaged, these pooled precision and recall values do not reproduce Table 1 algebraically.

	$T=242$	$T=1000$	P	R	SHD
	(10 seeds)	(50 seeds)
CDNOTS	$0.094$ $\pm.048$	$0.080$ $\pm.052$	$0.044$	$0.530$	$24.0$
LUCID ( CDNOTS )	$0.502$ $\pm.224$	$0.428$ $\pm.322$	$0.442$	$0.530$	$4.4$
PCMCI+	$0.099$ $\pm.055$	$0.093$ $\pm.056$	$0.050$	$0.610$	$25.1$
LUCID (PCMCI+)	$\mathbf{0.538}$ $\pm.263$	$\mathbf{0.441}$ $\pm.303$	$0.404$	$0.610$	$4.6$

Table 3: CDS-calibrated semi-synthetic benchmark. Directed, lag-resolved

F_{1}^{\text{dir}}

(mean

\pm

std) measures recovery of injected lag-

1

edges from a

3

-factor GARCH(1,1)-

t

model calibrated to a real

11

-bank CDS panel.

T=242

matches the weekly-sampled panel length;

T=1000

provides greater statistical power. Precision (P), recall (R), and SHD are reported at

T=1000

. Each LUCID row is placed beneath its corresponding base engine; bold marks the best entry per column.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Role	Value	Source
$L$	analysis lag horizon	dataset-dep.	required input
CDNOTS CI $\alpha$	skeleton-discovery significance	$0.05$	conventional
S $-$ L level $\alpha$	lag- $0$ edge-free-null level (§ 4.3 )	$0.05$	conventional
S $-$ L draws $B$	null draws for S $-$ L calibration	$200$	fixed
$\tau_{g}$ (Eq. 2 )	persistence-gate width (§ 4.3 )	$0.15$	held-out selection
Orientation lag window	lag used by lead-lag asymmetry (§ 4.3 )	$1$	structural

Appendix

Table 4: Hyperparameters used by LUCID .

L

is a standard lag-horizon input; all remaining values are fixed across datasets and experiments. The persistence width

\tau_{g}

is the only value selected empirically, using generators excluded from the ten-family OOD benchmark.

Family	$d$	$T$	$L$	cfg	$k$	Loadings	Factor dynamics	Confounding strength
Sparse-ER	$9$ – $21$	$600$ – $1500$	$3$	$4$	local forks	—	—	confounded fraction $0.3$ – $0.5$
Sparse-SF	$12$ – $16$	$1000$ – $1500$	$3$	$2$	local forks	—	—	confounded fraction $0.5$
Sparse-SW	$11$ – $14$	$600$ – $800$	$3$	$2$	local forks	—	—	confounded fraction $0.3$ – $0.4$
Perv-vol	$12$	$1000$	$1$	$4$	$3$	$\mathcal{U}(0.5,1.5)$ , block-disjoint	stoch. vol / regime-switch / static / AR	$w=0.4$ , $2$ edges/group, $\rho_{\text{obs}}=0.3$
Perv-struct	$12$	$1000$	$1$	$6$	$3$	as above; overlap / signed / nonlinear / heavy-tail	static	as above
GARCH	$12$	$1000$	$1$	$1$	$3$	$\mathcal{U}(0.5,1.5)$ , block-disjoint	GARCH(1,1)	as above

Appendix

Table 5: Data-generating configuration of the ten out-of-distribution families. Ranges span the configurations within a family (“cfg” counts them);

L

is the ground-truth maximum lag and

d

the observed dimension. Loadings are drawn i.i.d. per entry. Only pervasive-dense and pervasive-mixedlag contain ground-truth lag-

0

edges (

158

and

9

respectively at seed

0

); the other eight families have none.

Method	Overall	Sparse-ER	Sparse-SF	Sparse-SW	Perv-vol	Perv-struct	GARCH	Perv-dense	Perv-mixedlag	Perv-heterog	Perv-nearunit
CDNOTS	$0.400$ $\pm.017$	$0.635$	$0.527$	$0.822$	$0.370$	$0.362$	$0.376$	$0.162$	$0.277$	$0.280$	$0.186$
CDNOTS + tetrad	$0.462$ $\pm.020$	$0.637$	$0.578$	$0.822$	$0.458$	$0.426$	$0.428$	$0.164$	$0.366$	$0.419$	$\mathbf{0.316}$
LUCID ( CDNOTS , default)	$\mathbf{0.619}$ $\pm.025$	$0.812$	$0.623$	$0.844$	$0.626$	$\mathbf{0.783}$	$\mathbf{0.878}$	$0.359$	$\mathbf{0.382}$	$\mathbf{0.544}$	$0.344$
VARLiNGAM	$0.402$ $\pm.019$	$0.641$	$0.360$	$0.775$	$0.381$	$0.372$	$0.429$	$0.222$	$0.218$	$0.426$	$0.197$
LPCMCI †	$0.437$ $\pm.025$	$0.571$	$0.493$	$0.747$	$0.441$	$0.394$	$0.398$	—	$0.354$	$0.333$	$0.200$
TS-ICD ‡	$0.200$ $\pm.026$	$0.058$	$0.009$	$0.066$	$0.534$	$0.401$	$0.481$	$0.006$	$0.082$	$0.189$	$0.173$

Appendix

Table 6: OOD graph

F_{1}^{\text{pair}}

under lag-collapsed scoring, using the same runs as Table 1 (

50

seeds; mean

\pm

std for Overall). An ordered pair is credited when an edge is predicted at any lag. Bolding follows the same significance procedure as Table 1 .

Method	Median	Mean	p95	Max
CDNOTS	$1.1$	$8.0$	$59.6$	$161$
$+$ LUCID layer	$0.7$	$0.7$	$1.3$	$2.5$
PCMCI+	$3.3$	$61.6$	$424.7$	$1206$
$+$ LUCID layer	$0.7$	$0.7$	$1.4$	$3.5$
NTS-NOTEARS	$31.4$	$47.6$	$140.3$	$507$
$+$ LUCID layer	$0.6$	$0.7$	$1.3$	$1.8$

Appendix

Table 7: Per-dataset runtime in seconds, pooled over the ten scored families and

50

seeds. “

+

LUCID layer” rows give the cost of the layer alone: each reuses its base engine’s stored skeleton, so base discovery is excluded. All runs used identical machines (10-core CPU, one core per dataset, NVIDIA L4). CDNOTS and NTS-NOTEARS are GPU-backed; the remaining methods, and the LUCID layer itself, are CPU-only, so differences between engines reflect implementation as well as algorithm.

Contamination	Arm	$F_{1}$	Lag- $0$ recall
Low ( $0.2$ )	Blanket drop	$0.658$	$0.000$
	Keep all	$0.163$	$0.732$
	S $-$ L	$\mathbf{0.776}$	$\mathbf{0.980}$
High ( $0.5$ )	Blanket drop	$0.458$	$0.000$
	Keep all	$0.148$	$0.563$
	S $-$ L	$\mathbf{0.719}$	$\mathbf{0.948}$

Appendix

Table 8: Lag-

0

handling ablation on injected genuine contemporaneous edges (

20

seeds, averaged across five factor processes).

F_{1}

is directed and lag-resolved; lag-

0

recall is computed only over the injected contemporaneous edges.

	Overall	P-vol	P-struct	GARCH	P-dense	P-mixedlag	P-heterog	P-nearunit
LUCID (as shipped)	$\mathbf{0.592}$	$\mathbf{0.600}$	$\mathbf{0.761}$	$\mathbf{0.877}$	$\mathbf{0.342}$	$\mathbf{0.317}$	$\mathbf{0.456}$	$\mathbf{0.341}$
$+$ unconditional filtering	$0.451$	$0.244$	$0.387$	$0.575$	$0.150$	$0.249$	$0.424$	$0.233$

Appendix

Table 9: Ablation of pervasive-branch lag-

\ell\geq 1

filtering on the full ten-family OOD suite (

20

seeds). Overall is the ten-family weighted mean; the seven pervasive-family columns are shown individually.

duty cycle	1.00	0.50	0.25	0.10	0.05
CDNOTS	$0.161$	$0.160$	$0.130$	$0.149$	$0.177$
Tetrad filter	$0.487$	$0.586$	$0.660$	$0.615$	$0.596$
Routed (tetrad base)	$0.496$	$0.649$	$\mathbf{0.715}$	$0.697$	$0.701$
LUCID	$\mathbf{0.623}$	$\mathbf{0.683}$	$0.599$	$\mathbf{0.705}$	$\mathbf{0.818}$

Appendix

Table 10: Directed, lag-resolved

F_{1}

under transient factor confounding (

d=15

T=1500

20

seeds) as a function of factor duty cycle. “Routed (tetrad base)” uses the routed pipeline with tetrad-based pervasive correction and no S

-

L lag-

0

recovery. Best per column is bold.

Method	Gaussian	Laplace	$t_{3}$	Uniform	Exp.	Het.	Hist.	PNL
CDNOTS	$0.269$	$0.272$	$0.266$	$0.269$	$0.270$	$0.264$	$0.268$	$0.288$
CDNOTS + tetrad	$0.280$	$0.280$	$0.279$	$0.282$	$0.280$	$0.285$	$0.286$	$0.291$
DYNOTEARS	$0.282$	$0.283$	$0.282$	$0.285$	$0.285$	$0.296$	$0.294$	$0.179$
PCMCI+	$0.336$	$0.337$	$0.331$	$0.336$	$0.332$	$0.344$	$0.350$	$0.342$
LPCMCI	$0.250$	$0.247$	$0.271$	$0.270$	$0.282$	$0.290$	$0.298$	$0.354$
VARLiNGAM	$0.380$	$0.362$	$0.360$	$0.367$	$0.366$	$0.353$	$0.333$	$0.335$

Appendix

Table 11: Robustness to idiosyncratic noise distribution on a fixed pervasive-factor generator (

d=12

T=1000

50

seeds, communality

c=0.3

t_{3}

denotes Student-

t

noise with

\nu=3

; Het. is heteroscedastic location-scale, Hist. is ARCH-type history dependence, and PNL is post-nonlinear noise. Best per column is bold.

$\gamma$	$0.8$	$1.0$	$1.15$	$1.3^{\ast}$	$1.5$	$1.8$	$2.2$
FPR	$1.00$	$0.77$	$0.43$	$0.33$	$0.17$	$0.03$	$0.00$
Balanced acc.	$0.50$	$0.62$	$0.78$	$0.83$	$0.92$	$0.98$	$1.00$

Appendix

Table 12: Router diagnostics. Top: FPR and balanced accuracy versus the MP margin

\gamma

; the default

\gamma=1.3

is marked by ∗ . Bottom: FPR across dimension and sample-size settings at

\gamma=1.3

. FNR is zero for all configurations shown.

$k$	$1$	$2^{\ast}$	$3$	$4$
FPR	$0.40$	$0.33$	$0.20$	$0.13$
FNR	$0.00$	$0.00$	$0.00$	$0.00$
Balanced acc.	$0.800$	$0.833$	$0.900$	$0.933$
FNR, loadings $(0.2,0.4)$	$0.00$	$0.00$	$0.00$	$0.16$

Appendix

Table 13: Router sensitivity to the number of top eigenvalues

k

, on the same

55

-dataset battery as Table 12 (

\gamma=1.3

; every dataset is scored at every

k

, so the comparison is paired). The

k=2^{\ast}

column reproduces the

\gamma=1.3

entries of Table 12 . The last row reports FNR at the weak loading range

(0.2,0.4)

; all other rows use the benchmark range

(0.5,1.5)

Value	$F_{1}$	Precision	Recall
Sparse-branch PDS $\alpha$
$10^{-12}$	$0.588$	$0.659$	$0.606$
$10^{-10,\ast}$	$0.592$	$0.657$	$0.613$
$10^{-8}$	$0.596$	$0.655$	$0.621$
$10^{-6}$	$0.597$	$0.652$	$0.627$
$10^{-4}$	$0.598$	$0.646$	$0.634$

Appendix

Table 14: Sensitivity of LUCID to downstream fixed constants (family-weighted directed, lag-resolved

F_{1}

;

20

seeds). One constant is varied at a time with all others fixed at their defaults ( ∗ ).

\tau_{g}=0

disables the persistence gate.

Time-Series Causal Discovery

Time-series causal discovery seeks to infer which variables influence others over time and when those effects occur.

See all 64 papers

Provable Guarantees and Efficient Learning of Structural Equation Models with Latent Confounders
Sep 16, 2026Weijian Yu, Jean HonorioStructural Causal Models Low-Rank Matrix Decomposition
CIS, The University of Melbourne
Causal Lag Structure Discovery in Confounded Time Series via Orthogonalized Adaptive Estimation
Oct 4, 2026Hong Kiat Tan, Isaac-Neil Zanoria, James Chen +2Debiased ML Time-Series Causal Discovery
Department of Mathematics, University of California, Los Angeles · Department of Electrical and Computer Engineering, University of California, Los Angeles · Oxford-Man Institute of Quantitative Finance, University of Oxford +1
FoundCause: Causal Discovery with Latent Confounders from Observational Data
Jun 16, 2026Patrick Blöbaum, Krishnakumar Balasubramanian, Shiva Prasad KasiviswanathanStructural Causal Models Amortized Inference
Amazon Web Services · Department of Statistics, University of California, Davis

Sep 16, 2026J/K move · Enter open · S save

Provable Guarantees and Efficient Learning of Structural Equation Models with Latent Confounders

Weijian Yu, Jean Honorio

CIS, The University of Melbourne

Structural Causal Models Low-Rank Matrix Decomposition Causal Structure Learning Causal Discovery with Latent Confounding Causal Inference with Unobserved Confounding

Open paper AskPDF arXiv

Causal discovery aims to recover causal relationships from observed data. In various fields, exploring causal relationships among variables remains an important topic, but this task becomes challenging due to the existence of latent confounders. Ignoring such confounders can lead to false associations and incorrect edge directions. In this paper, we study the linear structural equation model with latent confounders. We propose an algorithm that iteratively identifies terminal (observed) nodes and reconstructs the directed acyclic graph of the observed variables. To do this, we recover the precision matrix of the observed variables as a sparse plus low-rank matrix: a sparse matrix captures the conditional dependencies among observed variables, while a low-rank matrix captures the combined influence of a few latent confounders. We establish that for

p

observed variables,

r

latent confounders and

s

edges, our procedure correctly identifies the directed causal relationship among observed variables, for

n \gtrsim \max\{s\log p,\ r p\}

samples. Experimental results validate our theoretical contributions.

Abstract

Time-Series Causal Discovery

Explore similar work

Provable Guarantees and Efficient Learning of Structural Equation Models with Latent Confounders

Causal Lag Structure Discovery in Confounded Time Series via Orthogonalized Adaptive Estimation

FoundCause: Causal Discovery with Latent Confounders from Observational Data