A Synthetic Benchmark for Target-Centric Causal Discovery in Industrial Processes
Organizations: Svelaro GmbH
Abstract
Causal discovery promises to find which recorded variables drive a process target such as product quality, yield or energy use. Process data with known causes are rare, and in existing benchmarks the difficulty of identifying the causes of a target is either fixed by the recorded data or set for the whole graph. We present a synthetic benchmark for target-centric causal discovery in industrial processes. Its generator is a two-level structural causal model inspired by batch production, assembled from components that each model one phenomenon of process data, such as a control loop or a delayed effect. The graph is built around the target: without sensor error, every non-cause can be distinguished from the direct causes by conditioning on them or by time order. Each component is set independently, so the difficulty can be varied one condition at a time. We evaluate eight methods, from association measures to effect estimation, with an error analysis by the role of the selected candidate; all degrade as the difficulties combine, each in a way consistent with its assumptions.
Figures & tables
| Component | In the generator | In a batch plant |
| Once per batch | ||
| batch profile | 2–4 motifs per variable (step, ramp, spike, plateau); their onsets, durations, sizes and the level form | a temperature or flow rises, holds and falls in every batch, at different times and by different amounts |
| batch conditions | act on the profiles of many variables | conditions that differ between batches and affect many sensors, e.g. coolant or outside temperature |
| confounders , | move a direct cause and its siblings together, also within the batch | an unmeasured property of a raw material, e.g. its purity, that moves several sensors together |
| batch-level links | between profile parameters, without lag; seen only across batches | one batch quantity sets another, e.g. a larger charge gives a higher temperature peak |
| At every time step within a batch | ||
| configurations | 119 (the settings scored in Section 6 ) | |||||
|---|---|---|---|---|---|---|
| plants | 1,190 (10 per configuration) | |||||
| batches time steps | 25, 50, 100 or 200 512 | |||||
| recorded variables | 20 or 40 | |||||
| candidates per plant | 2.6 0.9 direct causes, 6.8 3.4 ancestors, 9.8 6.0 siblings, 2.5 1.1 descendants | |||||
| shares: setting measured | reference | other settings | ||||
| signal | 0.7 0.69 | 0.05 0.02 | 0.1 0.07 | 0.2 0.18 | 0.4 0.39 | 0.9 0.88 |
| selected list | ranking | ||||||||
| Method | first pick | precision | lift | length | nothing selected | R-precision | mAP | MRR | nDCG@3 |
| All configurations (1,190 plants) | |||||||||
| random | 0.13 | 0.13 | 1.0 | n/a | n/a | 0.13 | 0.24 | 0.31 | 0.15 |
| Correlation | 0.64 ±.09 | 0.15 ±.03 | 1.1 ±.1 | 16.9 ±.8 | 0.00 ±.00 | 0.47 ±.09 | 0.61 ±.09 | 0.76 ±.06 | 0.57 ±.11 |
| Granger | 0.79 ±.12 | 0.21 ±.06 | 1.7 ±.3 | 13.7 ±1.7 | 0.10 ±.04 | 0.59 ±.13 | 0.69 ±.11 | 0.88 ±.07 | 0.67 ±.12 |
| VARLiNGAM | 0.66 ±.05 | 0.30 ±.04 | 2.3 ±.7 | 7.6 ±1.6 | 0.05 ±.02 | 0.53 ±.08 | 0.63 ±.07 | 0.77 ±.04 | 0.60 ±.08 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Value |
|---|---|
| Batch level | |
| batch conditions | 8–11 (log-uniform); random directed tree, built by adding the conditions one by one, each linked to a random earlier one or, with a small probability, to a neighbour of it, which creates a few hubs (growing network with redirection, Krapivsky and Redner, 2001 ); conditions without parents ; weights uniform in , noise s.d. |
| confounders | one per direct cause, ; the recorded variables that are not direct causes, the actuator included, are assigned to the causes in blocks (19, 9, 5–6 or 4 per cause at for to 4); enters each profile parameter of a variable in its block with probability 0.7 |
| profile parameters (Eq. 4 ) | equal weights except on routes with a share; noise s.d. ; onset 0–0.5 and duration 0.2–0.5 of the slot; size 0.5–2 (times a base size 1.5); level – |
| batch-level links | expected in-degree 1; 2 upstream variables |
| lots and drift (Eq. 7 ) | lot process: AR(1) over batches, persistence 0.9; drift: linear in the batch index |
| Share | Setting | Expected | Measured, median [IQR] |
| 0.05 | 0.05 | 0.02 [0.01, 0.04] | |
| 0.1 | 0.1 | 0.07 [0.06, 0.10] | |
| 0.2 | 0.2 | 0.18 [0.16, 0.21] | |
| 0.4 | 0.4 | 0.39 [0.36, 0.42] | |
| 0.7 | 0.7 | 0.69 [0.68, 0.71] | |
| 0.9 | 0.9 | 0.88 [0.87, 0.89] |
| Wrong candidates | number | not separable | by averaging | also at 4 steps |
|---|---|---|---|---|
| descendants | 3,117 | 14% | 12% | 2% |
| disturbance of the controlled cause | 1,150 | 23% | 18% | 5% |
| other ancestors | 7,190 | 9% | 7% | 2% |
| siblings | 12,010 | 7% | 4% | 3% |
| all | 23,467 | 9% | 7% | 2% |
| wrong selections not separable | as scored without them | ||||||
|---|---|---|---|---|---|---|---|
| Method | all | by averaging | also at 4 steps | share of list | first pick | precision | lift |
| Correlation | 11% | 8% | 3% | 10% | 0.64 0.69 | 0.15 0.16 | 1.1 1.2 |
| Granger | 13% | 10% | 4% | 11% | 0.79 0.83 | 0.21 0.24 | 1.7 1.7 |
| VARLiNGAM | 16% | 12% | 4% | 12% | 0.66 0.70 | 0.30 0.34 | 2.3 2.4 |
| DYNOTEARS | 2 wrong selections in all | 0.28 0.28 | 1.00 1.00 | 7.3 6.6 | |||
| PC | 17% | 12% | 5% | 12% | 0.25 0.29 | 0.30 0.35 | 2.3 2.5 |
| share of the wrong selections | ||||||
|---|---|---|---|---|---|---|
| Method | wrong per plant | descendants | disturbance of the controlled cause | other ancestors | siblings | nothing selected |
| random choice | 13% | 5% | 31% | 51% | ||
| Correlation | 14.7 | 16% | 5% | 30% | 48% | 0% |
| Granger | 11.6 | 12% | 6% | 33% | 49% | 10% |
| VARLiNGAM | 5.7 | 23% | 8% | 28% | 42% | 5% |
| DYNOTEARS | 0.0 | 2 wrong selections in all | 72% | |||
| first pick | mAP | precision | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Configuration | Correlation | Granger | VARLiNGAM | DYNOTEARS | PC | PCMCI+ | SpaceTime | DoubleML | Correlation | Granger | VARLiNGAM | DYNOTEARS | PC | PCMCI+ | SpaceTime | DoubleML | Correlation | Granger | VARLiNGAM | DYNOTEARS | PC | PCMCI+ | SpaceTime | DoubleML |
| Reference (5) | ||||||||||||||||||||||||
| 0 (reference) | 10 | 10 | 10 | 6 | 3 | 10 | 8 | 10 | 0.79 | 0.78 | 0.84 | 0.94 | 0.31 | 0.96 | 0.77 | 0.94 | 0.14 | 0.15 | 0.33 | 1.00 | 0.35 | 0.79 | 0.85 | 0.70 |
| 100 batches | 9 | 9 | 10 | 6 | 2 | 10 | 8 | 10 | 0.83 | 0.76 | 0.87 | 0.93 | 0.31 | 0.96 | 0.71 | 0.88 | 0.15 | 0.17 | 0.27 | 1.00 | 0.37 | 0.78 | 0.94 | 0.83 |
| 50 batches | 9 | 8 | 10 | 8 | 3 | 9 | 8 | 9 | 0.82 | 0.75 | 0.83 | 0.95 | 0.44 | 0.89 | 0.63 | 0.87 | 0.16 | 0.25 | 0.46 | 1.00 | 0.50 | 0.88 | 0.94 | 0.82 |
| 25 batches | 9 | 9 | 9 | 6 | 2 | 7 | 9 | 6 | 0.80 | 0.70 | 0.67 | 0.88 | 0.39 | 0.81 | 0.73 | 0.61 | 0.20 | 0.48 | 0.42 | 0.92 | 0.45 | 0.65 | 1.00 | 0.60 |
| Method | Threshold | length | precision | lift | first pick | nothing selected |
|---|---|---|---|---|---|---|
| Correlation | BH, (used) | 16.9 | 0.15 | 1.1 | 64% | 0% |
| BH, | 13.2 | 0.17 | 1.3 | 64% | 2% | |
| BH, | 9.6 | 0.22 | 1.7 | 63% | 6% | |
| BH, | 7.1 | 0.26 | 2.0 | 61% | 11% | |
| Bonferroni, | 14.2 | 0.16 | 1.3 | 64% | 0% | |
| first 3 by score | 3.0 | 0.42 | 3.3 | 64% | 0% |
Causal Discovery
Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.
See all 239 papers