seq2cause: One Autoregressive Backbone, Four Causal Discovery Tasks in Event Sequences

Hugo Math

cs.LGSep 25, 2026

seq2cause: One Autoregressive Backbone, Four Causal Discovery Tasks in Event Sequences

Authors: Hugo Math

Organizations: BMW Group

Abstract

Complex systems such as vehicles, patients, or genomes emit discrete event sequences whose operative question is causal, not predictive: which events cause which other events, and which cause higher-level outcomes such as failures or diseases? This question decomposes along two axes -- dependency type (event $\to$ event vs.\ event $\to$ outcome) and causal scope (single sequence vs.\ population) -- yielding four structurally distinct regimes with different identifiability conditions. No existing method addresses more than one, because all assume multi-stream structure with low vocabulary, and none scales beyond a few hundred event types. We present Seq2Cause, a unified framework that resolves all four regimes through a single shared primitive: a pretrained autoregressive model repurposed as an amortized conditional independence testing engine requiring no task-specific retraining. We establish a prediction--causality duality: the model's excess cross-entropy simultaneously bounds causal identification error across all four regimes, so that every improvement in next-token prediction tightens causal guarantees for free. On nonlinear SCMs (vocabularies up to $8{,}000$ types) and real-world vehicle diagnostic logs ( $29$ K event types, $474$ failure outcomes), Seq2Cause is the first method to populate all four regimes at scale with a single frozen backbone. Existing methods are either inapplicable, inaccurate, or computationally intractable in this setting.

Figures & tables

Panel of Figure 1 — Figure 1: seq2cause : one frozen backbone, four causal regimes. Any pretrained AR model — LLaMA, GPT, Mamba — is repurposed via teacher forcing as a parallelised causal discovery engine on GPUs, with no fine-tuning. Two heads share the same backbone: the event conditional ( blue ) and the outcome likelihood ( orange ). Using a single sequence, the sample-level graphs (a, b) follow from CMI testing; for multiple sequences, population-level graphs (c, d) are obtained from Minimum Description Length (MDL) [ 17 ] . Their union enables root-cause identification in sequences.

	Event-to-Event: $\mathcal{X}\!\to\!\mathcal{X}$	Events-to-Outcomes: $\mathcal{X}\!\to\!\mathcal{Y}$
Sample-Level (Single Sequence)	$A$ $B$ $A$ $C$ $t_{1}$ $t_{2}$ $t_{3}$ $t_{4}$ $t$ (a) Sample Time Graph Recovery Recover the time causal graph specific to a single sequence of events with lagged effects.	$A$ $B$ $C$ $y$ $\mathcal{MB}(y)$ (b) Sample Markov Boundary Identify the direct causes of each outcome $y$ (e.g., defects, diseases) from a single sequence.
Population-Level (Full Dataset)	$A$ $B$ $s^{(1)}$ $A$ $C$ $s^{(2)}$ $A$ $B$ $C$ (c) Population Summary Graph Summary causal graph over event types $\mathcal{X}$ : which event caused other events across samples?	$A$ $B$ $y$ $s^{(1)}$ $A$ $C$ $y$ $s^{(2)}$ $B$ $C$ $y$ (d) Population-Markov Boundary Identify the Causes of outcomes aggregated across sequences: What events lead to this product defect?

Table 1: The four regimes of causal discovery in discrete event sequences. Rows separate causal scope — a single observed sequence (sample-level) vs. an entire dataset (population-level). Columns separate dependency type — event-to-event (

\mathcal{X}\!\to\!\mathcal{X}

, blue ) vs. event-to-outcome (

\mathcal{X}\!\to\!\mathcal{Y}

, orange ). These four cells have different identifiability conditions, different estimands, and different failure modes. No existing method addresses more than one at scale.

Symbol	Name	Nodes	Edge meaning
$\mathcal{G}^{s}_{\mathcal{XX}}$	Sample-time event-to-event graph	Time indices $\{0,\ldots,L{-}1\}$	$(t{-}\ell)\!\to\!t$ : realization $x_{t-\ell}$ caused $x_{t}$ in $s$
$\mathcal{G}^{s}_{\mathcal{XY}}$	Sample-time event-to-outcome graph	Time indices $\{0,\ldots,L{-}1\}$ $\cup$ outcomes $\mathcal{Y}_{s}$	$t\!\to\!Y_{j}$ : realization $x_{t}$ caused $Y_{j}$ in $s$
$\mathcal{G}^{s}=\mathcal{G}^{s}_{\mathcal{XX}}\cup\mathcal{G}^{s}_{\mathcal{XY}}$	Sample-time causal graph	Time indices $\{0,\ldots,L{-}1\}$ $\cup$ outcomes $\mathcal{Y}_{s}$	Joint DAG over all events and outcomes in $s$
$\bar{\mathcal{G}}=\pi(\mathcal{G}^{s})$	Sample-summary graph	Event types $\mathcal{X}_{s}\subseteq\mathcal{X}$ , outcomes $\mathcal{Y}_{s}\subseteq\mathcal{Y}$	$u\!\to\!v$ : type $u$ caused type $v$ in $s$
$\mathcal{G}$	Population summary graph	Event types $\mathcal{X}$ , outcomes $\mathcal{Y}$	$u\!\to\!v$ : type $u$ causes type $v$ across $\mathcal{D}$

Table 2: Graph objects in seq2cause . All time graphs are DAGs by temporal precedence. Summary graphs may contain cycles [ 1 ] . The sample-time graph decomposes into two subgraphs (

\mathcal{G}^{s}_{\mathcal{XX}}

and

\mathcal{G}^{s}_{\mathcal{XY}}

) by dependency type; their union is the complete sample-level causal graph

\mathcal{G}^{s}

	Event-to-Event $\mathcal{X}\to\mathcal{X}$				Events-to-Outcomes $\mathcal{X}\to\mathcal{Y}$
	(a) Sample-Summary Graph $\;\mathcal{G}^{s}_{\mathcal{XX}}$ (nonlinear-SCM)				(b) Sample-Markov Boundary $\;\mathcal{G}^{s}_{\mathcal{XY}}$ (Vehicle Diagnostics)
Sample-Level	Method	SHD $\downarrow$	F1 $\uparrow$	Prec. $\uparrow$	Method	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
	Attention (BERT)	321.0 $\pm$ 15	50.1 $\pm$ .01	35.0 $\pm$ .01	IAMB [ 57 ]	$-^{\dagger}$	$-^{\dagger}$	$-^{\dagger}$
	Saliency (IG, LLaMA) [ 49 ]	160.2 $\pm$ 6.6	67.3 $\pm$ .01	51.4 $\pm$ .01	CMB [ 13 ]	$-^{\dagger}$	$-^{\dagger}$	$-^{\dagger}$
	Shapley (LLaMA) [ 25 ]	148.0 $\pm$ 5.1	60.0 $\pm$ .01	55.5 $\pm$ .01	MB-by-MB [ 65 ]	$-^{\dagger}$	$-^{\dagger}$	$-^{\dagger}$
	Neural Granger (LLaMA) [ 16 ]	100.2 $\pm$ 14.6	69.2 $\pm$ .04	71.6 $\pm$ .04	PCDbyPCD [ 69 ]	$-^{\dagger}$	$-^{\dagger}$	$-^{\dagger}$

Table 3: Unified evaluation of seq2cause across all four causal regimes. (a)

|\mathcal{X}|=1{,}000

L=64

\epsilon{=}0.05

\tau{=}3{\times}10^{-5}

N{=}128,H(P)=2.4

nats. (b, d) Diagnostic vehicle sequences leading to critical failures as outcomes,

|\mathcal{X}|{=}29{,}100

|\mathcal{Y}|{=}474

L{\approx}150

n=50,000

sequences, weighted avg,

\dagger

: timeout

{>}3

days (c)

|\mathcal{X}|=1{,}000

L=64

\epsilon{=}0.05

n{=}50,000\;\text{sequences},H(P)=2.4

nats.

Panel of Figure 2 — Figure 2: Robustness and scalability of seq2cause . (a) Phase transition at $\epsilon{\approx}0.1$ : Precision stays near 0.95 as $\epsilon$ increases (conservative failure mode). (b) Stable performance across vocabularies from $|\mathcal{X}|{=}10^{2}$ to $8{\times}10^{3}$ ; Shannon Redundancy ( Pred = $1-H(P)/H_{max}$ ) accounts for modeling difficulty ( $\hat{\epsilon}=0.01,L=64,N=64,\tau=10^{-4}$ ).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Robustness to Hidden Confounding. Evaluation of seq2cause under violations of causal sufficiency. (a) Measurement Error: Random noise is injected into the context. Precision stays high, indicating resistance to hallucination. (b) Temporal Drops: Time steps are randomly deleted, thus conditioning sets are broken. seq2cause still recovers the correct sample-summary causal graph structure despite missing intermediaries but the discovery scores quickly decrease.

Method	Discrete	Non-Param.	Single Stream	$\mathcal{X}{\to}\mathcal{X}$	$\mathcal{X}{\to}\mathcal{Y}$	Scalable	RCA
Constraint-based (PCMCI, FCI)	$\times$	✓	$\times$	✓	$\times$	$\times$	$\times$
Score-based (DYNOTEARS)	$\times$	$\times$	$\times$	✓	$\times$	$\times$	$\times$
Granger (TCDF, CAUSE)	$\times$	$\times$	$\times$	✓	$\times$	$\times$	$\times$
Noise-based (VarLiNGAM)	$\times$	$\times$	$\times$	✓	$\times$	$\times$	$\times$
Hawkes / TPP (THP, SHTP)	✓	$\times$	$\times$	✓	$\times$	$\times$	$\times$
Info-Theoretic (NPHC, CASCADE)	✓	✓	$\times$	✓	$\times$	$\times$	$\times$

Appendix

Table 4: Comparison of causal discovery methods for event sequences . Discrete : Operates on symbolic sequences. Non-Param. : No assumed functional form. Single Stream : Inference from one observed sequence.

\mathcal{X}{\to}\mathcal{X}

: Event-to-event dependencies.

\mathcal{X}{\to}\mathcal{Y}

: Event-to-outcome. Scalable : Tractable at

|\mathcal{X}|{>}10^{3}

(linear complexity in

|\mathcal{X}|

). RCA : Sample-level root-cause identification from a single sequence without topology assumptions or interventional data.

Feature	Traditional Approach	Ours (seq2cause)
	(e.g., Hawkes, Granger, PCMCI)	(Autoregressive Model)
Input Format	Long / Vertical Stream	Wide / Horizontal Batches
Structure	Time Event_Type 0.0 E_A 0.4 E_B 1.2 E_C ... ...	Seq_ID Sequence (Tokens) Time 0 [E_A, E_B, E_A, ...] [0.0, 1.3, 1.4, ...] 1 [E_C, E_D, E_G, ...] [0.0, 1.2, 5.9, ...] ... ...
Dimensionality	Low $\|\mathcal{X}\|\leq 100$	Massive $\|\mathcal{X}\|>1,000$
Discovery Scope	Global (Graph of all processes)	Local / Sample (Summary Graph of specific trace)
Vocabulary Complexity	Often $O(\|\mathcal{X}\|^{2})$ or $O(\|\mathcal{X}\|^{3})$	$O(L\cdot\|\mathcal{X}\|)$ (Inference)

Appendix

Table 5: Data Paradigm Comparison. Contrast between traditional event sequence causal discovery (multivariate) and our single sequence setting (Session-based/NLP-like). Standard methods require the structure on the left and scale poorly to the structure on the right.

Panel of Figure 4 — Figure 4: An example of a sample-level causal graph event sequence leading to outcomes where $\text{MB}_{1}$ represents the Markov Boundary of $Y_{1}$ and $\text{MB}_{2}$ the Markov Boundary of $Y_{2}$ . $X_{1}$ is the root-cause of $Y_{2}$

Figure 5: Sample Time Causal Graph

\mathcal{G}^{s}_{\mathcal{XX}}

. Temporal evolution of a diagnostic defect cascade in a vehicle (

|\mathcal{X}|\approx 29,100

). seq2cause effectively captures causal relationships, revealing distinct error clusters at different time steps (e.g., initial sensor failures at

t=3

triggering mechanical faults at

t=12

, battery at issue

t=17

). This enables actionable root-cause analysis by isolating the specific onset of a failure mechanism and their strength using The CMI

\hat{I}_{N}

Panel of Figure 6 — Figure 6: Overview of seq2cause Parallel CI-tests. We construct a single broadcasted tensor $\mathbf{X}_{do}$ where each row $j$ incrementally fixes the history $x_{\leq j}$ while randomizing the future (staircase pattern). The model processes this tensor in parallel to produce raw probabilities $\mathbf{P}_{raw}$ (grey). We then compute the CMI by comparing adjacent rows: the distribution at row $j-1$ serves as the baseline ( $\mathbf{P}_{base}$ , blue) for the intervention at row $j$ ( $\mathbf{P}_{do}$ , red).

Figure 7: Population-level

\mathcal{X}\to\mathcal{X}

baseline comparison with oracle threshold selection. F1, Precision, and Recall as a function of the number of observed sequences

N

for each method on the nonlinear SCM benchmark (

|\mathcal{V}|=1000

m=6

). For PMI and TF-IDF, scores are binarised at the threshold

\tau^{*}

that maximises F1 on the held-out ground truth ; this constitutes an upper bound on their achievable performance, since such oracle calibration is unavailable at test time. seq2cause uses its MDL-derived threshold (Corollary 2 ) with no access to the ground truth. Despite this advantage, PMI and TF-IDF remain below seq2cause across all sample sizes, confirming that the performance gap is not an artefact of threshold miscalibration but reflects a fundamental limitation of association-based scores for causal discovery.

Figure 8: Population-level

\mathcal{X}\!\to\!\mathcal{X}

discovery as a function of number of sequences

n

(

|\mathcal{X}|{=}1{,}000

, nonlinear-SCM,

\epsilon{=}0.05

). The green dashed line marks the CASCADE timeout (

N{=}100

); SHP times out at the same scale and is omitted beyond that point. Correlation baselines (PMI, TF-IDF): Recall saturates near 1.0 early as heuristic thresholds admit all co-occurring pairs, while Precision remains near zero — confirming that frequency-based methods cannot distinguish causal structure from spurious association regardless of evidence volume. seq2cause w/o correction (dashed red): Precision peaks near

N{=}200

then collapses to 4.2% as oracle approximation errors

\sqrt{\epsilon_{T}/2}\cdot\eta_{uv}

accumulate without correction, empirically validating Lem. 1 — the uncorrected threshold becomes equivalent to no threshold at scale. seq2cause w/ correction (solid red): the adaptive

\tau^{*}_{uv}

absorbs error accumulation, maintaining Precision above 0.80 across the full range while Recall grows steadily as genuine weak causal signals clear the corrected threshold — the only method achieving balanced F1 at any scale. (

L{=}64

m{=}6

\tau^{*}_{uv}

from Cor. 2 .)

Figure 9: Sample-Level (

\mathcal{X}\to\mathcal{X}

) Robustness and Scalability Analysis (

|\mathcal{X}|=1000,N=128,\tau=10^{-4},L=64

). (a) Robustness to Generative Error: Performance as a function of the model’s oracle score

\epsilon

. seq2cause exhibits a phase transition, recovering structure even for imperfect models (

\epsilon<0.1

) and maintaining high Precision even as fidelity degrades. (b) Scalability to Length: Performance and GPU memory usage vs. sequence length

L

. The Sparse variant (Appendix. F.3 ) demonstrates linear memory scaling (

O(mL)

), enabling inference on sequences far exceeding the training length (

L=64

), whereas the Full variant scales quadratically. (c) Long-Range Dependencies: Robustness to increasing delayed-effects

m

. seq2cause maintains F1

>0.8

even as dependencies span one third of the sequence (

m=20

), confirming the method’s ability to capture distant causal mechanisms.

$\|\mathcal{X}\|$	$n$	IAMB	CMB	MB-by-MB	PCDbyPCD	MI-MCF	seq2cause
500	500	0.0	0.0	0.0	0.0	0.0	58.4
500	5,000	2.1	1.8	2.4	0.0	0.0	59.7
2,000	500	0.0	0.0	0.0	0.0	0.0	52.2
2,000	5,000	0.0	0.0	0.0	– †	0.0	53.1
2,000	50,000	– †	– †	– †	– †	– †	55.3
29,100	50,000	– †	– †	– †	– †	– †	46.1

Appendix

Table 6: Double degeneracy of classical MB algorithms on the Sample-Level

\mathcal{X}\!\to\!\mathcal{Y}

. Weighted F1 (%) on the vehicle diagnostic Markov-boundary recovery task, evaluated on restricted vocabularies (

|\mathcal{X}|

= top-

K

most frequent event types) and reduced sample sizes

n

. At low

(K,n)

, classical LSL algorithms run to completion but collapse to

\text{F1}=0

because expected co-occurrence

\mathbb{E}[\#(u,Y_{j})]\approx n/(K|\mathcal{Y}|)

falls below the threshold required for any asymptotic CI test to reject independence. Scaling up either

K

n

to alleviate this sample-starvation regime immediately triggers the dual vocabulary-driven intractability regime ( † : timeout

>

3 days on 4

\times

A10G) GPUs. seq2cause amortizes conditional density estimation across all

(u,Y_{j})

pairs through a single AR forward pass and is the only method viable in both regimes.

Figure 10: Self-correcting MDL threshold

\tau^{*}_{uv}

prevents precision collapse as evidence accumulates. Population-level causal discovery metrics as a function of the number of sequences

N

, comparing the uncorrected MDL threshold (dashed, w/o corr. ) against the generative-error-corrected threshold of Cor. 2 (solid, w/ corr. ). Without correction: Precision degrades catastrophically beyond

N{\approx}1{,}000

sequences — each additional sequence accumulates approximation error from the

\epsilon

-oracle, progressively lowering the effective threshold until spurious edges flood the graph (Recall

\to 1

, Precision

\to 0

). With correction: The adaptive term

\sqrt{\epsilon_{T}/2}\cdot\eta_{uv}

\tau^{*}_{uv}

absorbs this drift, maintaining Precision above 0.80 across the full range

N\in[10,200{,}000]

while Recall grows steadily as genuine weak causal signals accumulate evidence. The crossover at

N{\approx}2{,}000

marks the point where uncorrected evidence accumulation overtakes the model cost

c_{\mathrm{type}}

(synthetic non-linear SCMs:

|\mathcal{X}|{=}1{}000,L=64

Figure 11: Evolution of the Sample-Level

\mathcal{X}\to\mathcal{X}

Causal Discovery in Function of the Number of Particles

N

|\mathcal{X}|=1000,m=6,L=64

on non-linear SCMs.

Figure 12: Evolution of the Population Causal Discovery in Function of the Number of Counterfactuals sampled as Mediators

M

|\mathcal{X}|=1000,m=6,L=66,n=2000

Causal Discovery

Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.

See all 221 papers

Sep 30, 2026

Inferring Causal Relations between Two Sequences of Events with Language Models

Causal AI is a branch of Artificial Intelligence which helps understand and reason about cause and effect relationships, not just patterns or correlations. Causal discovery aims to infer elements of the underlying causal structure--often represented as a directed graph--from observational and, when available, interventional data. While causal discovery is the fundamental step for moving beyond mere associations toward genuine understanding, and thus the basic building block of causal AI, it becomes intrinsically difficult when causal relations must be inferred from single observations. In such situations, standard causal discovery methods cannot be used and one has to identify causal relations from limited amount of information. This is typically the case for, e.g., sequences of events produced by different alarms which need to be analyzed on the fly to detect abnormal phenomena, which are usually rare. We show in this study that it is possible to leverage the predictive power of Large Language Models (LLMs) to infer causal relations between only two sequences of events. This approach, which is validated on both synthetic and real data, provides better results than standard causal discovery algorithms on several time series data, even though these data were converted into smaller, single observed sequences.

Nishchal Prasad, Eric Gaussier, Emilie Devijver +3

INRIA, Univ. Grenoble Alpes, CNRS, LIG · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG · Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG +1

Time-Series Causal Discovery Causal Discovery Causal Reasoning in Language Models

Ask PDF

May 8, 2026

Mask2Cause: Causal Discovery via Adjacency Constrained Causal Attention

Leveraging deep learning for causal discovery in time series remains challenging because existing neural methods predominantly rely on component-wise architectures that fail to capture shared system dynamics or employ decoupled post-hoc graph extraction that risks overfitting to spurious correlations. We propose

\textbf{Mask2Cause}

, an end-to-end framework that recovers the underlying causal graph directly during the forecasting forward pass. Our approach introduces an Inverted Variable Embedding and an Adjacency-Constrained Masked Attention mechanism, trained with homoscedastic or heteroscedastic objectives to capture causal influences in both mean and variance. Empirical results on diverse benchmarks, from synthetic chaotic dynamics to realistic biological simulations, demonstrate state-of-the-art causal discovery with significantly reduced parameter complexity compared to standard baselines. We further show that inferred causal structures can be used to reduce parameter count of forecasting models by more than 70% on average while maintaining predictive accuracy.

Omar Muhammad, Pasupuleti Dhruv Shivkant, Deepak N. Subramani

1Indian Institute of Science, Bengaluru, India

Time-Series Causal Discovery Causal Structure Learning Causal Attention

Ask PDF

Mar 17, 2026

Learning to Predict, Discover, and Reason in High-Cardinality Event Sequences

Electronic control units (ECUs) embedded within modern vehicles generate a large number of asynchronous events known as diagnostic trouble codes (DTCs). These discrete events form complex temporal sequences that reflect the evolving health of the vehicle's subsystems. In the automotive industry, domain experts manually group these codes into higher-level error patterns (EPs) using Boolean rules to characterize system faults and ensure safety. However, as vehicle complexity grows, this manual process becomes increasingly costly, error-prone, and difficult to scale. Notably, the number of unique DTCs in a modern vehicle is on the same order of magnitude as the vocabulary of a natural language, often numbering in the tens of thousands. This observation motivates a paradigm shift: treating diagnostic sequences as a language that can be modeled, predicted, and ultimately explained. Traditional statistical approaches fail to capture the rich dependencies and do not scale to high-cardinality datasets characterized by thousands of nodes, large sample sizes, and long sequence lengths. Specifically, the high cardinality of categorical event spaces in industrial logs poses a significant challenge, necessitating new machine learning architectures tailored to such event-driven systems. This thesis addresses automated fault diagnostics by unifying event sequence modeling, causal discovery, and large language models (LLMs) into a coherent framework for event streams. It is structured in three parts, reflecting a progressive transition from prediction to causal understanding and finally to reasoning for vehicle diagnostics. Consequently, we introduce several Transformer-based architectures for predictive maintenance, scalable sample- and population-level causal discovery frameworks and a multi-agent system that automates the synthesis of Boolean EP rules.

Hugo Math

Faculty of Applied Computer Science, University of Augsburg

Fault Diagnosis Predictive Maintenance Sequence Modeling Time-Series Causal Discovery

Ask PDF

Abstract

Causal Discovery

Explore similar work

Inferring Causal Relations between Two Sequences of Events with Language Models

Mask2Cause: Causal Discovery via Adjacency Constrained Causal Attention

Learning to Predict, Discover, and Reason in High-Cardinality Event Sequences