Due to the fundamental problem of causal inference -- the impossibility of observing counterfactual outcomes -- there is always a gap between the training goal (outcome, propensity) and the inference target (CATE, ATE). Traditional causal effect estimation revolves around setting assumptions, selecting identification strategies, and choosing causal estimators, while more recently, Causal Foundation Models (CFMs), built on Prior-Data-Fitted Networks (PFNs), ease this process by pretraining on massive synthetic prior datasets. Despite their success, such amortization makes causal inference less transparent, obscuring which assumptions the model implicitly relies on for a given dataset. The high diversity of real-world data-generating processes (DGPs) and Structural Causal Models (SCMs) makes it hard to know, a priori, which causal inference strategy is compatible with a given dataset. In this work, we focus on this compatibility between an observational dataset and method-specific assumptions, and find that geometric characteristics of data embeddings -- specifically, the variogram -- provide a useful signal of whether a given causal estimation strategy is appropriate. We develop MoE-Learner, a mixture-of-causal-experts framework that embeds a dataset under multiple assumption-specific views and uses the variogram as a counterfactual-free router to assess each expert's validity. Rather than eliminating causal assumptions, this shifts their validation from a fixed, a priori choice to a criterion that can be checked directly from data geometry. MoE-Learner achieves the best average rank on benchmark datasets (IHDP, ACIC, Lalonde). Being orthogonal to (causal) foundation models, our method builds a bridge from data characteristics to causal estimation strategy rather than proposing another CFM.
Figures & tables
Figure 1: Comparison of different paradigms.
Figure 2: Illustration of the MoE-Learner workflow using ACIC data as the example. The observational data (X,T,Y) is converted to four kinds of embeddings by modeling different relationships ( Y∼(X,T) , Y∼X , T∼X , and a covariate modeling of X ). Each expert makes its own causal estimation. The Router selects the expert by evaluating the variograms of the embeddings.
Figure 3: Expert activation percentages for CATE (upper panel) and ATE (lower panel).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset Group
IHDP
ACIC 2016
Lalonde CPS
Lalonde PSID
Realizations
100
10
100
100
Data Sizes
747
4802
16177
2675
Treated Group Sizes
129
1326
156
144
Control Group Sizes
618
3476
16021
2531
Treated Fractions
17.27%
27.61%
0.96%
5.38%
Covariate Dimensions
25
58
8
8
Appendix
Table 2: Statistics for the benchmarking datasets used in the experiment. The numbers of the treated examples are highlighted , showing a diverse range of percentages of the treated group.
Figure 4: (A) ICE from the percentile interval, standard-error interval, and CausalPFN. (B) Sensitivity of ICE to K . (C) Different distribution of the p-values from Shapiro–Wilk test.
Figure 5: The distributions of PEHE scores from MoE-Learner, CausalPFN, TabPFN, and TabICL on four datasets, respectively. Lalonde PEHE is in thousands.
Figure 6: The distributions of the relative ATE errors from MoE-Learner, CausalPFN, TabPFN, and TabICL on four datasets, respectively.
Figure 7: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on IHDP dataset for CATE.
Figure 8: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on ACIC 2016 dataset for CATE.
Figure 9: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on Lalonde CPS dataset for CATE.
Figure 10: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on Lalonde PSID dataset for CATE.
Figure 11: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on IHDP dataset for ATE.
Figure 12: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on ACIC 2016 dataset for ATE.
Figure 13: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on Lalonde CPS dataset for ATE.
Figure 14: The t-SNE visualizations of embeddings (left column), the variograms (central column), and the distribution of the correlations in the variograms (right column) from the four experts on Lalonde PSID dataset for ATE.
Causal inference is the practice of estimating the effect of a treatment or intervention from data. It traditionally requires a bespoke pipeline for every new problem: first proposing a causal mechanism, selecting a compatible estimator, and finally training it. Meanwhile, across diverse settings and modalities, much of machine learning has shifted to the paradigm of foundation models: networks pretrained once at scale and applied to new tasks without fine-tuning. Causal foundation models (CFMs) bring this paradigm to causal inference. CFMs are pretrained neural networks that estimate causal quantities, such as the average treatment effect, on entirely new datasets using in-context learning without requiring model updates. This work provides a practical introduction to this emerging area. We summarize the necessary background in causal inference and machine learning before discussing CFMs. Throughout, we include example code and Jupyter notebooks.
Christopher Stith, Hossein Rahmani, Jesse C. Cresswell
Estimating causal quantities traditionally relies on bespoke estimators tailored to specific assumptions. Recently proposed Causal Foundation Models (CFMs) promise a more unified approach by amortising causal discovery and inference in a single step. However, in their current state, they do not allow for the incorporation of any domain knowledge, which can lead to suboptimal predictions. We bridge this gap by introducing methods to condition CFMs on causal information, such as the causal graph or more readily available ancestral information. When access to complete causal graph information is too strict a requirement, our approach also effectively leverages partial causal information. We systematically evaluate conditioning strategies and find that injecting learnable biases into the attention mechanism, together with a graph-convolutional encoder, is a highly effective method to utilise full and partial causal information. Our experiments show that this conditioning allows a general-purpose CFM to match the performance of specialised models trained on specific causal structures. Overall, our approach addresses a central hurdle on the path towards all-in-one causal foundation models: the capability to answer causal queries in a data-driven manner while effectively leveraging any amount of domain expertise.
Arik Reuter, Anish Dhir, Cristiana Diaconu +6
University of Cambridge, Cambridge, United Kingdom · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Gatsby Computational Neuroscience Unit, University College London +5
Causal foundation models (CFMs) pre-trained on data generated from various structural causal models (SCMs) have been proposed for estimating causal effects from observational data. However, differences in pre-training environments and evaluation protocols make it difficult to assess how their performance depends on the information available for causal identification. To enable controlled comparisons, we introduce CausalIDView, a multi-view benchmark that holds fixed SCM realization and target estimand while varying only the observational view available to the estimator. Each observational view corresponds to a distinct identification regime under the benchmark's maintained causal assumptions. Across these matched views, no CFM consistently performs best and model rankings vary substantially. Under controlled structural changes, CFMs exhibit model-specific failures to maintain stable estimates when true effects are unchanged and to track genuine effect changes. We also examine whether combining explicit identification with strong predictive estimation is effective. A modular approach that pairs a predictive tabular foundation model with regime-specific identification procedures is competitive with CFMs and outperforms several of them. These findings motivate cross-regime comparisons to assess the empirical value of CFMs.
Heejin Jung, Gyeongdeok Seo, Hoyoon Byun +2
Yonsei University · University of Illinois Urbana–Champaign