Causal discovery is increasingly used to analyze telemetry from interactive systems, but a plausible graph does not guarantee an identifiable or reliable product decision. We study this problem for one query: whether early competitive gameplay increases Day-1 retention in a deployed Role-playing game (RPG). We propose an admissibility-first framework that treats discovered graphs as structural hypotheses, keeps only graphs that identify the target effect and have adequate treatment-control overlap, and validates admissible effects through cross-algorithm stability, placebo, subsampling, and E-value diagnostics. On real telemetry, under minimal constraints the admissible runs that return Directed Acyclic Graphs (DAGs) are those whose graphs leave player-versus-player (PvP) with no parents, so they estimate the confounded raw difference; under the full constraint set all are admissible. Runs returning Partial Ancestral Graphs (PAGs) are not evaluated by the DAG gate and are reported separately. The admissible DAGs agree on one adjustment set and return a positive, cohort-conditional risk-difference average treatment effect (ATE), down from a raw retention gap; the estimate keeps its sign under placebo and subsampling checks, but its magnitude changes with the adjustment set and the constraint specification. A known-truth simulation shows that the checks screen identification and stability given the measured variables and cannot certify the absence of unmeasured confounding, and a second game-telemetry dataset shows that identifiability rates do not transfer. Graph-level recovery is therefore a weak proxy for causal reliability; discovery pipelines for decision support should be evaluated at the query and effect level.
Figures & tables
Figure 1: Two graphs with the same skeleton yet opposite implications for the query T→Y . Graph A: W is an observed confounder; backdoor adjustment identifies E[Y∣do(T)] . Graph B: T→Y coexists with a latent common cause T↔Y ; adjusting for W cannot block the hidden path T←H→Y , so no observed adjustment set point-identifies the effect.
Constraint
Identifiable rate (%)
Mean ATE (RD)
SD (ATE)
C0
47.62
0.239
0.000
C1
0
NaN
NaN
C2
100
0.174
0.000
C3
100
0.174
0.000
Table 1: Constraint ladder ablation on the telemetry (DAG-returning algorithms PC, GRaSP and BOSS; 7 thresholds each, 21 runs per level; full sample, n=1,280 ; seeded estimator). The C0 runs return the unadjusted risk difference. SD is zero because every admissible run at a level implies the same adjustment set. C0 is one run; GRaSP varies over repeats (Appendix P ).
Table 2: Sensitivity of the PvP effect to the adjustment set (common-support sample, n=1,170 ; seeded). REG: regression adjustment; IPW: inverse-probability weighting; DR: doubly robust (DRLearner) with 95% CI. { Web3 } is the minimal set returned by DoWhy under C3; { Web3 , Total_PvE_Battle , Total_Session } is the parent set of PvP in every completed C4 run.
Regime
True ATE
Gate pass
Safe
Mean abs. error
Wrong despite every check
Identifiable
0.101
90.1%
85.4%
1.8 pp
10.0%
Latent confounder
0.102
99.6%
0%
13.9 pp
100%
Orientation ambiguous
0.116
98.3%
1.3%
11.1 pp
98.6%
Null (no causal path)
0.000
2.6%
—
—
0 runs cleared all checks
Table 3: Known-truth evaluation (DAG-returning runs; Appendix N ). “Wrong despite every check”: among the runs that cleared the gate and the placebo, subsampling and E-value checks together, the share that were nevertheless not safe. The gate rejects any unresolved edge, as on the telemetry.
Cohort
C0
C1
C2
C3
WoWAH 24h ( n=9,592 )
0/0%
0/0%
0/0%
0/0%
WoWAH 168h ( n=9,436 )
19.0/9.5%
33.3/23.8%
61.9/33.3%
61.9/33.3%
WoWAH 336h ( n=9,338 )
4.8/0%
81.0/66.7%
66.7/33.3%
66.7/33.3%
Table 4: DAG-returning runs (PC, GRaSP and BOSS; 7 thresholds each, 21 runs per cell, seeded) on WoWAH. Each cell reports the percentage producing a defined effect / the percentage also lying within the logical bounds [−1,1] of a risk difference. BOSS runs hitting the upstream iteration cap are retained as nonconverged, contributing no estimate.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method family
Typical output
Used in main stability claims
PC / GRaSP / BOSS
DAG
Yes
FCI, CFCI, FCI-MAX, GFCI, BOSS-FCI, GRaSP-FCI
PAG
No
Appendix
Table 5: Summary of causal discovery method families. PAG outputs are separated from the DAG-based main analysis because partial orientation can imply a different target effect or identification rule. The key point is that methods producing valid causal outputs need not contribute to the same stability claim.
Method
Runs with an estimate (of 7)
Range of estimates
FCI
0
—
GFCI
0
—
BOSS-FCI
0
—
GRaSP-FCI
0
—
FCI-MAX
0
—
CFCI
5
0.00–0.39
Appendix
Table 6: PAG-calculus estimates for the six PAG-returning algorithms (seven significance levels each; common-support sample; C1 constraints; identical across two runs). A run is counted as defined when the PAG-calculus estimator returns a point estimate.
Variable
Type
Definition
R1 (Outcome)
Binary
Day-1 retention: 1 if user returned during hours 24–48, 0 otherwise
PvP (Treatment)
Binary
Early PvP exposure: participation in competitive mode during first 24 hours
Web3
Binary
Wallet connection status (proxy for platform commitment)
Time_Play_Level1
Continuous
Time spent on Level 1 (onboarding friction proxy)
Total_PvE_Battle
Integer
Number of PvE battles (baseline progression)
Total_Session
Integer
Total login sessions (engagement intensity)
Appendix
Table 7: Variables used in the causal analysis: treatment, outcome, and four pre-treatment covariates covering platform commitment, onboarding friction, baseline progression, and engagement intensity.
Variable
Overall
PvP=1
PvP=0
R1 (Retention)
0.70
0.89
0.65
Web3 (Wallet)
0.48
0.73
0.42
Time_Play_Level1
7.83
8.07
7.77
Total_PvE_Battle
10.76
18.87
8.67
Total_Session
3.87
5.32
3.50
Appendix
Table 8: Descriptive statistics by treatment status. The raw 24 pp retention gap is heavily confounded by self-selection: PvP-exposed users differ strongly on Web3 status, PvE progression, and session count.
Panel A: PC, GFCI, FCI, and BOSS-FCI
Metric
PC
GFCI
FCI
BOSS-FCI
precision
0.30
0.36
0.50
0.42
recall
0.33
0.44
0.78
0.56
tp
3.00
4.00
7.00
5.00
fp
7.00
7.00
7.00
7.00
fn
6.00
5.00
2.00
4.00
Appendix
Table 9: Graph deviation metrics of unconstrained discovery (no domain knowledge, default significance level) relative to the nine-edge domain-consistent reference graph (Figure 2 ), using the six analysis variables. Precision, recall, tp, fp and fn count arrowheads (CFCI returns no arrowhead, so its precision is undefined); adj_* count adjacencies. GRASP is stochastic and shown as the mean of 20 runs (SHD 7–9); the other columns were identical in every run. Metrics are descriptive: low SHD does not imply effect reliability, and high SHD does not preclude admissible effect estimation.
Figure 2: Reference graph from domain constraints. Not ground truth, one admissible hypothesis supporting backdoor identification of the target estimand.
Node
LogReg
DT
HistGB
Assessment
R1
0.163
0.223
0.180
Very Good
PvP
0.166
0.169
0.170
Very Good
Time_Play_Level1
0.502
0.509
0.501
Fair
Total_PvE_Battle
0.528
0.527
0.523
Fair
Total_Session
0.477
0.476
0.474
Fair
Appendix
Table 10: Calibration performance (CRPS) for each modeled node under three mechanism classes. Lower values indicate better calibration. The key pattern is that retention and PvP are calibrated well across mechanisms, while the continuous telemetry variables are only fair, with logistic regression performing best overall for the retention node. The Assessment column is a qualitative label per node, not a formal threshold on CRPS.
Mechanism
Intrinsic (%)
Parents (%)
Logistic Regression
61.6
38.4
Decision Tree
37.8
62.2
HistGradientBoosting
58.5
41.5
Appendix
Table 11: Intrinsic causal contribution (ICC) to the variance of retention, by mechanism class (seeded). “Intrinsic” is the share attributed to R1’s own noise term and “Parents” the share attributed to its modeled parents (Web3, PvP, Time_Play_Level1, Total_PvE_Battle, Total_Session); the two sum to 100. The decision-tree mechanism attributes the most variance to the parents, and logistic regression the least.
Figure 3: Propensity score overlap between treated and control groups. Common support spans [0.032, 0.935]; trimming 8.6% (concentrated among controls) leaves overlap for estimation.
Metric
Before Trim
After Trim
Sample Size
Total N
1,280
1,170
Treated
263
261
Control
1,017
909
Propensity Score Range
Treated
[0.032, 0.949]
[0.032, 0.910]
Appendix
Table 12: Overlap and balance diagnostics before and after trimming. Trimming reduces sample size modestly, narrows common support to regions shared by treated and control users, and keeps post-IPW covariate balance near the conventional 0.1 threshold (the largest absolute SMD after trimming is 0.104, for Web3 ). Overall effective sample sizes use stabilized weights; the treated and control rows are invariant to that choice. What to notice is that positivity is imperfect but remains adequate for the main analysis.
Claim
Assumption
Evidence
Failure condition
Backdoor identification
Temporal order; no unblocked hidden path after adjustment
Constraint table; DoWhy ID check
No valid adjustment set exists
Positivity satisfied
Common support of propensity scores
Overlap plot; standardized mean difference (SMD); effective sample size (ESS)
Effect undefined or driven by extrapolation
Artifact check
No residual confounding or model misspecification
Placebo test ( p=0.96 ); subsampling sign stability
Placebo non-null; subsample sign flip
Hidden-bias sensitivity
Unmeasured confounding weaker than the E-value
E-value ( =1.66 , bound 1.48 ), a sensitivity summary, not exclusion of confounding
A confounder with risk ratio ≥1.66 to both T and Y would explain the effect away; moderate confounding cannot be ruled out
Cross-algorithm agreement
Graphs differ only where the estimand does not depend on them
Agreement is on one adjustment set, not independent recovery; a different adjustment set or constraint specification gives a different estimate
PAG caution
Partial orientation reflects latent-aware caution
PAG appendix; separate reporting
PAG and DAG agree only when query resolved
Appendix
Table 13: Claim–assumption matrix. Each row states what is being evaluated, the required assumptions, how the paper tests it, and what failure looks like.
Bin
Fraction
Algorithms
Admissible — convergent
100% (21/21)
PC, GRaSP, BOSS (all α )
Inadmissible — identification failure
0%
None at C3 (21 of 21 at C1; 11 of 21 at C0)
Appendix
Table 14: Classification of the 21 DAG-returning discovery runs (PC, GRaSP, BOSS; 7 thresholds each) under full constraints (C3), scoped to the main backdoor admissibility gate. A run is one (algorithm, α ) tuple. Positivity is assessed at dataset level, and no run fails this gate after passing identification (0% positivity failures); no admissible run is unstable (0%), since all return the same estimate. PAG-returning methods (FCI, CFCI, FCI-MAX, GFCI, BOSS-FCI, GRaSP-FCI; 42 runs) are not evaluated under the DAG backdoor gate (Section 3.4 ) and are reported separately in Appendix B .
Figure 4: The three admissible graphs for the telemetry query under C2 and C3. Grey edges are shared by all three. Red edges appear only in graph A ( Total_PvE_Battle → Time_Play_Level1 and Time_Play_Level1 → R1), blue in graphs B and C ( Time_Play_Level1 → Total_PvE_Battle , reversed), green only in graph C ( Web3 → Total_PvE_Battle ). TP1 = Time_Play_Level1 , PvE = Total_PvE_Battle , Ses = Total_Session . In all three, parents(PvP) ={Web3} .
Cohort
Admissible runs
Distinct graphs
Pairwise SHD
Distinct adjustment sets
Telemetry, C2–C3
42
3
1–4
1 ({ Web3 })
Telemetry, C0–C3
52
5
1–9
2 ({}, { Web3 })
WoWAH 336h, C0–C3
46
7
1–7
2
WoWAH 168h, C0–C3
37
6
1–9
3
Appendix
Table 15: Admissible DAG runs (a converged run with a defined estimate, pooled over the constraint levels shown), their structural distinctness (pairwise SHD, a reversed edge counting as two errors), and the adjustment sets they imply. The 42 telemetry records at C2–C3 are the same 21 PC, GRaSP and BOSS runs at each level. On telemetry, estimates are identical within an adjustment set and differ across sets; on WoWAH this holds for the estimates within the risk-difference bounds [−1,1] , while the other admissible runs return out-of-bounds values.
Constraint dropped
Telemetry identified
WoWAH identified
WoWAH within bounds
none (full set)
100
66.7
33.3
semantic
100
66.7
33.3
temporal
66.7
33.3
0.0
platform
47.6
4.8
0.0
measurement
0.0
81.0
66.7
Appendix
Table 16: Leave-one-constraint-out over 21 DAG-returning runs (PC, GRaSP and BOSS; 7 thresholds each). The WoWAH columns separate graph-identified runs from those whose estimates also lie within the risk-difference bounds [−1,1] . BOSS iteration-cap runs remain in the denominator as nonconverged. On telemetry the runs admitted without the platform constraint return the unadjusted difference 0.239 instead of 0.174. Estimator seeded. GRaSP is stochastic in the platform-dropped telemetry variant: over 20 repeats the admissible share ranges from 33.3 to 52.4 (mean 41.7), against the 47.6; the measurement-dropped value (0) was identical in all 140 GRaSP repeats.
Causal-discovery algorithms return a directed graph, yet provide no principled means of distinguishing edge directions identified by the data from those assigned without an identifying assumption. Under the standard Markov and faithfulness conditions, the observational distribution identifies only a Markov equivalence class; orientations within that class are not determined by the joint distribution and cannot be recovered from additional samples alone, but require either a functional restriction or an intervention. We introduce a protocol for observational causal discovery on continuous data that attaches to each candidate edge a discrete impossibility certificate: a RESOLVED code records the identifiability theorem under which the direction was committed, while an IMPOSSIBLE code records the failure mode together with the specific question a domain expert must answer to resolve it. The bivariate cascade is extended with five gated identifiability tiers LSNM, IGCI, Stein, MDL, and PEIT that abstain when their precondition test rejects. Two oracle primitives, the meta-hub query and the node-children query, jointly establish an upper bound of 1+K expert interactions sufficient to recover any DAG, where K denotes the number of non-leaf vertices. Under an ideal-oracle assumption, the bound is met exactly on the asia, sachs, child, and alarm benchmarks.
Causal discovery is challenging in general dynamical systems because, without strong structural assumptions, the underlying causal graph may not be identifiable even from interventional data. However, many real-world systems exhibit directional, cascade-like structure, in which components activate sequentially and upstream failures suppress downstream effects. We study causal discovery in such chain-reaction systems and show that the causal structure is uniquely identifiable from blocking interventions that prevent individual components from activating. We propose a minimal estimator with finite-sample guarantees, achieving exponential error decay and logarithmic sample complexity. Experiments on synthetic models and diverse chain-reaction environments demonstrate reliable recovery from a few interventions, while observational heuristics fail in regimes with delayed or overlapping causal effects.
Panayiotis Panayiotou, Özgür Şimşek
Department of Computer Science, University of Bath, UK
Observational causal discovery often leaves causal di- rections unresolved or unstable. Large language mod- els (LLMs) can provide semantic information to resolve these directions, but incorrect proposals can exclude the true causal graph and invalidate downstream inference. We study three separate decisions: where to allocate semantic queries, which proposals to admit, and which causal graphs to retain. Our framework distinguishes population non-identifiability from finite-sample insta- bility, evaluates semantic orientation against statistical and random controls, and examines calibrated admis- sion and error-tolerant DAG retention. Experiments across benchmark networks show that semantic infor- mation improves recovery over coin-flip orientations, but targeting unstable edges has no established ad- vantage over matched-random allocation. The tested admission certificate cannot support nontrivial admis- sion with the available calibration tasks. Moreover, downstream retention depends on reference adequacy: learned references contain the true DAG in only three of 80 planned tasks. These findings separate the value of semantic information from the reliability of its alloca- tion, admission, and retention, and identify calibration size and reference adequacy as key limitations.
Muhammad Ahmed Mohsin, Saad Bin Ather, Valentin Six +3
Stanford University · FAST-NUCES Lahore, Pakistan · University of Virginia Charlottesville, USA +3