Authors: Harald Kugler, Leonard Henckel, Sebastian Weichwald
Organizations: Department of Mathematical Sciences University of Copenhagen, Denmark · School of Mathematics and Statistics University College Dublin, Ireland
There are many causal discovery algorithms, but few guiding principles for choosing between them. Methods are usually compared through three lenses: identifiability, consistency, and benchmarks. Each introduces assumptions about the data-generating process that are often untestable. We argue that there are no shared standards for comparing, prioritizing, or even discussing those assumptions. We support this claim by showing that (i) identifiability assumptions are often preferred by convention rather than by principle, (ii) uniform consistency requires further untestable assumptions, while the weaker pointwise consistency has arguably narrowed algorithm design, and (iii) benchmarks are not neutral tests but carry an epistemic role of their own: they quietly fix which regimes matter, which guarantees come into play, and what counts as success. In doing so, they shape what algorithm comparisons are taken to show and thus perform some of the same epistemic work as theory, but less transparently. In this position paper we argue for a more transparent, integrated practice: Make the epistemic standards for discussing identifiability and consistency assumptions explicit, especially their failure modes and scope conditions; design benchmarks to operationalize those standards, by stress-testing and ablating both. Then, theoretical and empirical results jointly clarify when, why, and how algorithms may work rather than accumulate as disconnected advances.
Figures & tables
Figure 1: FLOP vs. DirectLiNGAM vs. VarSortRegress on linear ANMs with p=50 nodes, n=1000 samples, and generalized normal noise ( β=1.4 ); 50 sampled graphs per setting. (a) MEC size vs. number of edges on ER graphs (one point per graph): small across the considered range. (b) SHD to the true DAG on ER graphs: although FLOP targets the MEC rather than the DAG, the small MEC leaves few edges undirected, and FLOP outperforms DirectLiNGAM despite the latter’s assumptions being satisfied. (c) The same comparison on chains without colliders (MEC size of 50 ): FLOP recovers the skeleton (SHD ≤49 ) but orients edges at random, and is therefore outperformed by DirectLiNGAM. See Figure 3 for the corresponding ancestor AID results.
Figure 2: FLOP vs. DirectLiNGAM vs. VarSortRegress on linear ANMs with n=1000 samples and generalized normal noise ( β controls peakedness and tails). (a) Varying average degree dˉ on ER graphs ( p=50 , β=1.2 ): FLOP’s performance degrades with density, while DirectLiNGAM and VarSortRegress remain stable. (b) Varying the noise distribution parameter β on ER graphs ( p=50 , dˉ=10 ): DirectLiNGAM is sensitive to β and fails close to β=2 (Gaussian noise, violating its assumptions); FLOP and VarSortRegress are stable. (c) Varying edge weights on chains ( p=20 , β=1.2 ): VarSortRegress improves with larger absolute weights; FLOP and DirectLiNGAM are stable. See Figure 4 for the corresponding ancestor AID results.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: As Figure 1 (b) and (c), but reporting ancestor AID instead of SHD.
Figure 4: As Figure 2 , but reporting ancestor AID instead of SHD.
Figure 5: SHD and ancestor AID vs. number of restarts in FLOP, on linear ANMs with p=50 nodes, n=1000 samples, and uniform noise; results across 50 sampled graphs per setting, with average degree dˉ=8 (left) and dˉ=16 (right). Lower BIC is better, so points with BIC(Gest)≤BIC(Gtrue) mark runs in which the search found a graph scoring at least as well as the true graph. With more restarts FLOP increasingly does so, but for dˉ=16 this does not translate to recovering the true graph: the search succeeds while the graph remains wrong.
Figure 6: (a) : The implied edge weights of an iSCM tend to be smaller than the originally drawn weights, where implied weights are computed via Algorithm 2 in Ormaniec et al. [2025] . (b) and (c) : FLOP exhibits a slightly worse performance on the iSCMs compared to standard ER graphs. Both use p=50 nodes, dˉ=8 , and Gaussian noise.
Figure 7: FLOP vs. DirectLiNGAM across two different metrics, SHD and ancestor AID, on linear ANMs with p=50 node ER graphs with dˉ=8 , n=1000 samples, and uniform noise (variances uniformly sampled from [0.5,2.0] ); results across 50 sampled graphs. FLOP outperforms DirectLiNGAM with respect to SHD, but DirectLiNGAM outperforms FLOP with respect to ancestor AID, illustrating that the choice of metric influences results. VarSortRegress is omitted because its performance is not informative here.
In graphical causal model, causal discovery aims to construct a causal graph based on numerical data and domain knowledge in plain text. However, the evaluation of causal discovery methods remains a challenge in the area as the progress of domain researches often makes benchmark causal graphs contain mis-aligned knowledge. This problem especially affects the evaluation of large language model (LLM) based causal discovery methods as they are sensitive to the new discoveries in the literature. This work is the first to systematically study the quality of benchmark causal graphs. Specifically, we design a pipeline that automatically retrieves relevant research papers from scientific databases, and prompts LLMs to check the consistency between the benchmark causal graphs and domain research papers. We evaluate 11 popular real-world benchmarks, for which our pipeline in total proceeds 38,081 domain papers. Our results show that popular benchmarks vary significantly in their consistency with domain research, with clear implications for causal discovery research.
Learning causal structure from data requires assumptions about mechanisms, noises, observation schemes, and interventions. Historically, causal discovery has relied primarily on conditional independence. This review focuses on modern causal discovery methods that exploit properties of distributions beyond conditional independence. Our organizing principle separates the source of identification from the statistical estimator and its computational implementation. This perspective connects conditional-independence methods with distributional asymmetries, conditional-moment restrictions, and variation across environments or individuals. We examine discovery under latent confounding and feedback, distinguishing causal information about observed variables from reconstruction of latent structure and emphasizing the additional semantics required by cyclic models. We also distinguish population identification, frequentist and Bayesian consistency, uncertainty quantification, and algorithmic guarantees. Throughout, examples show why the same observational distribution can support different conclusions under different model classes. We conclude with guidance for comparing methods at their identifiable targets and a research agenda for uncertainty, heterogeneous populations, non-scalar variables, and the intersection of discrete data, feedback, and hidden confounding.
Yang Ni
Department of Statistics and Data Science, The University of Texas at Austin
Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification, formulating a valid research design under stated assumptions, and estimation, implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 174 queries across 139 real-world datasets, curated from 73 peer-reviewed research papers and three widely used causal-inference textbooks. For each query, a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 81% of cases, full identification-specification correctness drops to only 38%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.