Effect-Level Validation for Causal Discovery in Interactive Telemetry
Organizations: Independent Researcher, Vietnam · RMIT University, Australia · Florida Atlantic University, USA
Abstract
Causal discovery is increasingly used to analyze telemetry from interactive systems, but a plausible graph does not guarantee an identifiable or reliable product decision. We study this problem for one query: whether early competitive gameplay increases Day-1 retention in a deployed Role-playing game (RPG). We propose an admissibility-first framework that treats discovered graphs as structural hypotheses, keeps only graphs that identify the target effect and have adequate treatment-control overlap, and validates admissible effects through cross-algorithm stability, placebo, subsampling, and E-value diagnostics. On real telemetry, under minimal constraints the admissible runs that return Directed Acyclic Graphs (DAGs) are those whose graphs leave player-versus-player (PvP) with no parents, so they estimate the confounded raw difference; under the full constraint set all are admissible. Runs returning Partial Ancestral Graphs (PAGs) are not evaluated by the DAG gate and are reported separately. The admissible DAGs agree on one adjustment set and return a positive, cohort-conditional risk-difference average treatment effect (ATE), down from a raw retention gap; the estimate keeps its sign under placebo and subsampling checks, but its magnitude changes with the adjustment set and the constraint specification. A known-truth simulation shows that the checks screen identification and stability given the measured variables and cannot certify the absence of unmeasured confounding, and a second game-telemetry dataset shows that identifiability rates do not transfer. Graph-level recovery is therefore a weak proxy for causal reliability; discovery pipelines for decision support should be evaluated at the query and effect level.
Figures & tables
| Constraint | Identifiable rate (%) | Mean ATE (RD) | SD (ATE) |
|---|---|---|---|
| C0 | 47.62 | 0.239 | 0.000 |
| C1 | 0 | NaN | NaN |
| C2 | 100 | 0.174 | 0.000 |
| C3 | 100 | 0.174 | 0.000 |
| Adjustment set | REG | IPW | DR | 95% CI (DR) |
|---|---|---|---|---|
| None (raw difference) | 0.194 | — | — | — |
| { Web3 } (main analysis) | 0.155 | 0.146 | 0.139 | [0.081, 0.198] |
| { Web3 , Total_PvE_Battle } | 0.064 | 0.082 | 0.075 | [ , 0.158] |
| { Web3 , Total_PvE_Battle , Total_Session } (C4-motivated sensitivity) | 0.044 | 0.053 | 0.043 | [ , 0.133] |
| All four pre-treatment covariates | 0.035 | 0.049 | 0.056 | [ , 0.144] |
| Regime | True ATE | Gate pass | Safe | Mean abs. error | Wrong despite every check |
|---|---|---|---|---|---|
| Identifiable | 0.101 | 90.1% | 85.4% | 1.8 pp | 10.0% |
| Latent confounder | 0.102 | 99.6% | 0% | 13.9 pp | 100% |
| Orientation ambiguous | 0.116 | 98.3% | 1.3% | 11.1 pp | 98.6% |
| Null (no causal path) | 0.000 | 2.6% | — | — | 0 runs cleared all checks |
| Cohort | C0 | C1 | C2 | C3 |
|---|---|---|---|---|
| WoWAH 24h ( ) | 0/0% | 0/0% | 0/0% | 0/0% |
| WoWAH 168h ( ) | 19.0/9.5% | 33.3/23.8% | 61.9/33.3% | 61.9/33.3% |
| WoWAH 336h ( ) | 4.8/0% | 81.0/66.7% | 66.7/33.3% | 66.7/33.3% |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Method family | Typical output | Used in main stability claims |
|---|---|---|
| PC / GRaSP / BOSS | DAG | Yes |
| FCI, CFCI, FCI-MAX, GFCI, BOSS-FCI, GRaSP-FCI | PAG | No |
| Method | Runs with an estimate (of 7) | Range of estimates |
|---|---|---|
| FCI | 0 | — |
| GFCI | 0 | — |
| BOSS-FCI | 0 | — |
| GRaSP-FCI | 0 | — |
| FCI-MAX | 0 | — |
| CFCI | 5 | 0.00–0.39 |
| Variable | Type | Definition |
|---|---|---|
| R1 (Outcome) | Binary | Day-1 retention: 1 if user returned during hours 24–48, 0 otherwise |
| PvP (Treatment) | Binary | Early PvP exposure: participation in competitive mode during first 24 hours |
| Web3 | Binary | Wallet connection status (proxy for platform commitment) |
| Time_Play_Level1 | Continuous | Time spent on Level 1 (onboarding friction proxy) |
| Total_PvE_Battle | Integer | Number of PvE battles (baseline progression) |
| Total_Session | Integer | Total login sessions (engagement intensity) |
| Variable | Overall | PvP=1 | PvP=0 |
|---|---|---|---|
| R1 (Retention) | 0.70 | 0.89 | 0.65 |
| Web3 (Wallet) | 0.48 | 0.73 | 0.42 |
| Time_Play_Level1 | 7.83 | 8.07 | 7.77 |
| Total_PvE_Battle | 10.76 | 18.87 | 8.67 |
| Total_Session | 3.87 | 5.32 | 3.50 |
| Panel A: PC, GFCI, FCI, and BOSS-FCI | ||||
|---|---|---|---|---|
| Metric | PC | GFCI | FCI | BOSS-FCI |
| precision | 0.30 | 0.36 | 0.50 | 0.42 |
| recall | 0.33 | 0.44 | 0.78 | 0.56 |
| tp | 3.00 | 4.00 | 7.00 | 5.00 |
| fp | 7.00 | 7.00 | 7.00 | 7.00 |
| fn | 6.00 | 5.00 | 2.00 | 4.00 |
| Node | LogReg | DT | HistGB | Assessment |
|---|---|---|---|---|
| R1 | 0.163 | 0.223 | 0.180 | Very Good |
| PvP | 0.166 | 0.169 | 0.170 | Very Good |
| Time_Play_Level1 | 0.502 | 0.509 | 0.501 | Fair |
| Total_PvE_Battle | 0.528 | 0.527 | 0.523 | Fair |
| Total_Session | 0.477 | 0.476 | 0.474 | Fair |
| Mechanism | Intrinsic (%) | Parents (%) |
|---|---|---|
| Logistic Regression | 61.6 | 38.4 |
| Decision Tree | 37.8 | 62.2 |
| HistGradientBoosting | 58.5 | 41.5 |
| Metric | Before Trim | After Trim |
|---|---|---|
| Sample Size | ||
| Total N | 1,280 | 1,170 |
| Treated | 263 | 261 |
| Control | 1,017 | 909 |
| Propensity Score Range | ||
| Treated | [0.032, 0.949] | [0.032, 0.910] |
| Claim | Assumption | Evidence | Failure condition |
|---|---|---|---|
| Backdoor identification | Temporal order; no unblocked hidden path after adjustment | Constraint table; DoWhy ID check | No valid adjustment set exists |
| Positivity satisfied | Common support of propensity scores | Overlap plot; standardized mean difference (SMD); effective sample size (ESS) | Effect undefined or driven by extrapolation |
| Artifact check | No residual confounding or model misspecification | Placebo test ( ); subsampling sign stability | Placebo non-null; subsample sign flip |
| Hidden-bias sensitivity | Unmeasured confounding weaker than the E-value | E-value ( , bound ), a sensitivity summary, not exclusion of confounding | A confounder with risk ratio to both and would explain the effect away; moderate confounding cannot be ruled out |
| Cross-algorithm agreement | Graphs differ only where the estimand does not depend on them | Constraint-based (PC) and score-based (GRaSP, BOSS) searches; graph comparison (Table 15 ) | Agreement is on one adjustment set, not independent recovery; a different adjustment set or constraint specification gives a different estimate |
| PAG caution | Partial orientation reflects latent-aware caution | PAG appendix; separate reporting | PAG and DAG agree only when query resolved |
| Bin | Fraction | Algorithms |
|---|---|---|
| Admissible — convergent | 100% (21/21) | PC, GRaSP, BOSS (all ) |
| Inadmissible — identification failure | 0% | None at C3 (21 of 21 at C1; 11 of 21 at C0) |
| Cohort | Admissible runs | Distinct graphs | Pairwise SHD | Distinct adjustment sets |
|---|---|---|---|---|
| Telemetry, C2–C3 | 42 | 3 | 1–4 | 1 ({ Web3 }) |
| Telemetry, C0–C3 | 52 | 5 | 1–9 | 2 ({}, { Web3 }) |
| WoWAH 336h, C0–C3 | 46 | 7 | 1–7 | 2 |
| WoWAH 168h, C0–C3 | 37 | 6 | 1–9 | 3 |
| Constraint dropped | Telemetry identified | WoWAH identified | WoWAH within bounds |
|---|---|---|---|
| none (full set) | 100 | 66.7 | 33.3 |
| semantic | 100 | 66.7 | 33.3 |
| temporal | 66.7 | 33.3 | 0.0 |
| platform | 47.6 | 4.8 | 0.0 |
| measurement | 0.0 | 81.0 | 66.7 |
Causal Effect Estimation
Causal effect estimation measures how outcomes would change under interventions, using experiments or observational data and explicit assumptions.
See all 287 papers