CIDER-FM: Foundation Models for Causal Inference from Diverse Experimental Regimes

Yuche Gao; Arik Reuter; Siyuan Guo; Anish Dhir; Bernhard Schölkopf; Adrian Weller

cs.LGSep 30, 2026

CIDER-FM: Foundation Models for Causal Inference from Diverse Experimental Regimes

Authors: Yuche Gao, Arik Reuter, Siyuan Guo, Anish Dhir, Bernhard Schölkopf, Adrian Weller

Organizations: University of Cambridge, Cambridge, United Kingdom · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Prior Labs · Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom · ELLIS Institute, Tübingen, Germany · The Alan Turing Institute, London, United Kingdom

Abstract

Causal foundation models (CFMs) amortise causal inference over priors of synthetic structural causal models (SCMs), predicting the effect of an experiment on a specific variable. However, observational data alone may leave multiple causal models compatible with available evidence, while experimental data with interventions on exactly the variable of interest might be unavailable. This work studies CFMs as a method to combine finite observational and surrogate-interventional datasets in order to predict a target conditional interventional distribution (CID) more accurately than with observational data alone. We first formalise the conceptual benefits of surrogate experiments. Building on this analysis, we introduce \textsc{Foundation Models for Causal Inference from Diverse Experimental Regimes} (\emph{CIDER-FM}), a causal foundation model that uses an intervention-aware representation and hierarchical three-axis attention to exchange information across variables, samples, and experimental regimes. We evaluate CIDER-FM against a wide range of baselines across diverse synthetic graph and mechanism families, as well as on both simulated and real-world data from Causal Chambers. Our results demonstrate strong CID prediction performance and show that incorporating experimental context can improve predictions over observational data alone.

Figures & tables

Figure 1: Conceptual comparison: observational data (top) and multi-regime experimental data (bottom) in Causal Foundation Models (CFMs). With observational data alone, multiple compatible SCMs may imply different CIDs, producing a broad or multimodal posterior predictive distribution. Adding data from different experimental regimes can exclude or downweight incompatible SCMs and thereby refine the predictive distribution. Compared to existing TFMs, the stacked tables in the lower panel make explicit a third structural axis, namely the different experimental regime, alongside variables and samples along which CIDER-FM performs the attention operation.

Figure 2: Case study: surrogate experiments sharpen conditional interventional predictions. Model predictive densities are visualised using Gaussian KDEs. Left: the same CIDER-FM checkpoint receives 256 observational samples, 320 observational samples, or 256 observational samples plus 64 randomized $\mathrm{do}(Z)$ samples. Right: comparison with observational-only Graphs4CFM ( Reuter et al., 2026 ) without graph input. Details of the task construction and quantitative results are in Appendix H .

Panel of Figure 3 — Figure 3: Case studies: Three roles of surrogate experiments for $p(y\mid\operatorname{do}\!\left(T=t\right),x)$ . In Case 0, the CID is identifiable from observational data alone. In Case 1, it becomes identifiable after incorporating experiments on $Z$ . In Case 2, it remains non-identifiable but with a potentially smaller identified set.

Method		CIDER-FM	CIDER-FM-Obs	Bayesian LR	TabPFN v2	Graph4CFM
Data & Info		fusion	obs_total	obs_total	obs_total	obs_total
Data & Info						all ancestry
Linear-Gaussian $Y_{\mathrm{mean}}$	MSE $\downarrow$	95.03 $\,\pm\,35.10$	471.72 $\,\pm\,167.56$	795.25 $\,\pm\,247.80$	386.37 $\,\pm\,213.81$	160.10 $\,\pm\,60.11$
	$R^{2}_{\mathrm{mean}}\uparrow$	0.675 $\,\pm\,0.045$	-0.016 $\,\pm\,0.030$	-21.719 $\,\pm\,7.873$	-3.608 $\,\pm\,1.736$	0.474 $\,\pm\,0.082$
	$R^{2}_{\mathrm{med}}\uparrow$	0.947 $\,\pm\,0.006$	-0.016 $\,\pm\,0.006$	0.843 $\,\pm\,0.032$	0.897 $\,\pm\,0.016$	0.811 $\,\pm\,0.019$
	$R^{2}_{\mathrm{pool}}\uparrow$	0.895 $\,\pm\,0.067$	0.479 $\,\pm\,0.144$	0.122 $\,\pm\,0.633$	0.573 $\,\pm\,0.347$	0.823 $\,\pm\,0.101$

Table 1: Performance on general random graphs: 560 linear–Gaussian and 512 ComplexMech test SCMs. Entries report the indicated statistic with its SCM-level standard error conditional on fixed checkpoints. Best point estimates are in bold. Infinite NLL arises when the predicted density is numerically zero for a query outcome. ComplexMech

Y_{\mathrm{mean}}

is omitted because reliable ground-truth conditional means are unavailable.

Panel of Figure 4 — Figure 4: Selected baseline comparisons on restricted graph families in fusion_id and fusion_non_id . Error bars indicate 95% confidence intervals.

Mechanism	Rank 1	Top 3
Linear–Gaussian	9/15 (60%)	15/15 (100%)
Nonlinear–Gaussian	7/15 (46.7%)	13/15 (86.7%)

Table 2: Rank-1 and top-3 frequencies of CIDER-FM over five metrics in each of three settings, giving 15 setting–metric combinations per mechanism family.

Method	Data & Info	$\bm{Y_{q}}$ NLL $\downarrow$	$\bm{Y_{q}}$ MSE $\downarrow$	$\bm{Y_{q}\,R^{2}_{\mathrm{mean}}}\uparrow$	$\bm{Y_{q}\,R^{2}_{\mathrm{med}}}\uparrow$	$\bm{Y_{q}\,R^{2}_{\mathrm{pool}}}\uparrow$
CIDER-FM	fusion	5.772 $\,\pm\,0.031$	6810 $\,\pm\,399$	0.367 $\,\pm\,0.026$	0.380 $\,\pm\,0.038$	0.374 $\,\pm\,0.026$
CIDER-FM-Obs	obs_total	5.770 $\,\pm\,0.036$	6861 $\,\pm\,340$	0.360 $\,\pm\,0.026$	0.351 $\,\pm\,0.029$	0.370 $\,\pm\,0.026$
CIDER-FM-Obs	obs_fixed	5.816 $\,\pm\,0.042$	7057 $\,\pm\,392$	0.343 $\,\pm\,0.028$	0.318 $\,\pm\,0.038$	0.352 $\,\pm\,0.028$
CIDER-FM	obs_total	5.792 $\,\pm\,0.029$	6956 $\,\pm\,375$	0.352 $\,\pm\,0.025$	0.350 $\,\pm\,0.026$	0.361 $\,\pm\,0.026$
Bayesian LR	obs_total	5.850 $\,\pm\,0.027$	7099 $\,\pm\,400$	0.338 $\,\pm\,0.029$	0.352 $\,\pm\,0.051$	0.348 $\,\pm\,0.029$
TabPFN v2	obs_total	5.930 $\,\pm\,0.062$	9042 $\,\pm\,670$	0.156 $\,\pm\,0.062$	0.172 $\,\pm\,0.078$	0.169 $\,\pm\,0.052$

Table 3: Real-world performance on Causal Chambers. Values are estimates

\pm

standard errors. Bold indicates the best point estimate in each column. For Graph4CFM, arrows denote supplied ancestral relations; R, G, and B denote the red, green, and blue channels, respectively.

Rows per regime	Same width		Parameter matched
	Inference	Training	Inference	Training
256	0.814	0.865	0.940	1.002
512	0.941	1.001	1.112	1.167
1024	1.108	1.199	1.297	1.383
2048	1.357	1.484	1.561	1.684

Table 4: Throughput ratios (three-axis / two-axis) from short-run profiling on an NVIDIA A100, with 10 variables,

R=3

, and eight memory tokens per regime and variable. Same width:

d_{\mathrm{3D}}=d_{\mathrm{2D}}=128

. Parameter matched:

d_{\mathrm{3D}}=128

d_{\mathrm{2D}}=160

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Data fusion example: diverse experimental regimes with continuous, row-wise interventions. The upper panel shows one possible latent-confounded data-generating SCM. An actual task contains an observational table and/or single-target interventional tables, while the right table contains CID queries. During pre-training, many SCMs are sampled and used to generate finite context and query data. At inference, only these data are provided; the underlying SCM remains unknown, and CIDER-FM predicts the CID in a single forward pass.

Full method name	Context policy	Data supplied	Causal information supplied
CIDER-FM and observational controls
CIDER-FM	fusion	$N_{r}$ obs rows and $(R-1)N_{r}$ int rows	Intervention targets and values
CIDER-FM	obs_total	$RN_{r}$ obs rows	None
CIDER-FM-Obs	obs_total	$RN_{r}$ obs rows	None
CIDER-FM-Obs	obs_fixed	$N_{r}$ obs rows	None
Predictive methods

Appendix

Table 5: Methods and context configurations used across the synthetic and real-world experiments.

\mathcal{S}\leftarrow\left\{P(V),P\!\left(V\setminus\{Z\}\mid do(Z)\right)\right\}.

Appendix

Algorithm 2 Restricted Graph-Family Sampling

Setting	Target	Metric	CIDER-FM mean $\bm{\pm}$ SE	Rank		Best method	Best mean $\bm{\pm}$ SE
obs_id	$Y_{q}$	NLL	0.9456 $\pm$ 0.0440	2 /	11	CIDER-FM-Obs ( obs_total )	0.9338 $\pm$ 0.0394
		MSE	0.9097 $\pm$ 0.0922	3 /	19	CIDER-FM-Obs ( obs_total )	0.8679 $\pm$ 0.0766
		$R^{2}$	0.7330 $\pm$ 0.0174	1 /	19	CIDER-FM	0.7330 $\pm$ 0.0174
	$Y_{\mathrm{mean}}$	MSE	0.4131 $\pm$ 0.0769	3 /	19	CIDER-FM-Obs ( obs_total )	0.3711 $\pm$ 0.0592
		$R^{2}$	0.8498 $\pm$ 0.0241	1 /	19	CIDER-FM	0.8498 $\pm$ 0.0241
fusion_id	$Y_{q}$	NLL	1.6150 $\pm$ 0.0403	1 /	11	CIDER-FM	1.6150 $\pm$ 0.0403

Appendix

Table 6: Restricted-family linear–Gaussian detailed ranking. Values are mean

\pm

standard error. Rank gives the position of CIDER-FM among 19 methods, excluding all

c\text{-}g\text{ID}

methods.

Setting	Target	Metric	CIDER-FM mean $\bm{\pm}$ SE	Rank		Best method	Best mean $\bm{\pm}$ SE
obs_id	$Y_{q}$	NLL	0.9537 $\pm$ 0.0323	1 /	11	CIDER-FM	0.9537 $\pm$ 0.0323
		MSE	1.5096 $\pm$ 0.2347	3 /	19	TabPFN v2 ( obs_total )	1.3793 $\pm$ 0.2732
		$R^{2}$	0.3737 $\pm$ 0.0154	2 /	19	TabPFN v2 ( obs_total )	0.3936 $\pm$ 0.0156
	$Y_{\mathrm{mean}}$	MSE	1.1354 $\pm$ 0.2335	3 /	19	TabPFN v2 ( obs_total )	1.0021 $\pm$ 0.2722
		$R^{2}$	0.5057 $\pm$ 0.0176	2 /	19	TabPFN v2 ( obs_total )	0.5299 $\pm$ 0.0182
fusion_id	$Y_{q}$	NLL	1.3479 $\pm$ 0.0306	1 /	11	CIDER-FM	1.3479 $\pm$ 0.0306

Appendix

Table 7: Restricted-family nonlinear–Gaussian detailed ranking. Values are mean

\pm

standard error. Rank gives the position of CIDER-FM among 19 methods, excluding all

c\text{-}g\text{ID}

methods.

Figure 6: Case studies: Effect of observational–interventional context allocation on linear–Gaussian CIDER-FM for a fixed four-node fusion_id graph. The total context budget is fixed at 192 rows, while the horizontal axis varies the fraction of randomised interventional samples, $n_{\operatorname{do}(X_{0})}/192$ , from entirely observational data ( $0$ ) to entirely interventional data ( $1$ ).

Method	$Y_{q}$			$Y_{\mathrm{mean}}$
	NLL $\downarrow$	MSE $\downarrow$	$R^{2}$ $\uparrow$	MSE $\downarrow$	$R^{2}$ $\uparrow$
obs_id
CIDER-FM	0.9456 $\pm$ 0.0440	0.9097 $\pm$ 0.0922	0.7330 $\pm$ 0.0174	0.4131 $\pm$ 0.0769	0.8498 $\pm$ 0.0241
CIDER-FM-Obs ( obs_total )	0.9338 $\pm$ 0.0394	0.8679 $\pm$ 0.0766	0.7299 $\pm$ 0.0190	0.3711 $\pm$ 0.0592	0.8345 $\pm$ 0.0305
LR ( obs_total )	N/A	1.1995 $\pm$ 0.2220	0.6022 $\pm$ 0.0768	0.7032 $\pm$ 0.2122	0.5799 $\pm$ 0.1247
LR ( fusion )	N/A	1.2662 $\pm$ 0.2240	0.5798 $\pm$ 0.0787	0.7703 $\pm$ 0.2143	0.5556 $\pm$ 0.1255

Appendix

Table 8: Restricted-family linear–Gaussian validation: all comparison methods. Mean

\pm

SE across 520 SCMs per setting (130 each at 4, 6, 8 and 10 nodes). Bold denotes the best applicable mean in each panel, excluding gray

c\text{-}g\text{ID}

rows. N/A : no density or identification route.

Method	$Y_{q}$			$Y_{\mathrm{mean}}$
	NLL $\downarrow$	MSE $\downarrow$	$R^{2}$ $\uparrow$	MSE $\downarrow$	$R^{2}$ $\uparrow$
obs_id
CIDER-FM	0.9537 $\pm$ 0.0323	1.5096 $\pm$ 0.2347	0.3737 $\pm$ 0.0154	1.1354 $\pm$ 0.2335	0.5057 $\pm$ 0.0176
CIDER-FM-Obs ( obs_total )	1.0179 $\pm$ 0.0337	1.7253 $\pm$ 0.3161	0.3430 $\pm$ 0.0167	1.3486 $\pm$ 0.3153	0.4258 $\pm$ 0.0229
LR ( obs_total )	N/A	4.4135 $\pm$ 0.9504	0.2465 $\pm$ 0.0185	4.0367 $\pm$ 0.9513	0.3103 $\pm$ 0.0285
LR ( fusion )	N/A	4.4630 $\pm$ 0.9372	0.2212 $\pm$ 0.0171	4.0890 $\pm$ 0.9378	0.2764 $\pm$ 0.0240

Appendix

Table 9: Restricted-family nonlinear–Gaussian validation: all comparison methods. Mean

\pm

SE across 520 SCMs per setting (130 each at 4, 6, 8 and 10 nodes). Bold denotes the best applicable mean in each panel, excluding gray

c\text{-}g\text{ID}

rows. N/A : no density or identification route.

Context	Obs rows	$\operatorname{do}\!\left(\mathrm{green}\right)$ rows	$\operatorname{do}\!\left(\mathrm{blue}\right)$ rows
$\operatorname{do}\!\left(\mathrm{green}\right)$	0	192	0
$\operatorname{do}\!\left(\mathrm{blue}\right)$	0	0	192
$\operatorname{do}\!\left(\mathrm{green}\right)+\operatorname{do}\!\left(\mathrm{blue}\right)$	0	96	96
Obs + $\operatorname{do}\!\left(\mathrm{green}\right)$	96	96	0
Obs + $\operatorname{do}\!\left(\mathrm{blue}\right)$	96	0	96
Obs + $\operatorname{do}\!\left(\mathrm{green}\right)+\operatorname{do}\!\left(\mathrm{blue}\right)$	64	64	64

Appendix

Table 10: Candidate contexts for

\mathrm{red}\to\mathrm{vis}_{3}\mid\mathrm{green},\mathrm{blue}

, each with 192 rows. Bold marks the context selected by simulator NLL.

Model	Context	NLL $\downarrow$	CRPS $\downarrow$	90% width	Coverage
Graphs4CFM	$256O$	$1.5301\pm.0619$	$.5864\pm.0287$	$3.0828\pm.0702$	$.7903\pm.0298$
Graphs4CFM	$320O$	$1.3926\pm.0500$	$.5622\pm.0245$	$3.1862\pm.0596$	$.8539\pm.0223$
CIDER-FM	$256O$	$1.0544\pm.0186$	$.4404\pm.0113$	$3.3288\pm.0233$	$.9260\pm.0058$
CIDER-FM	$320O$	$1.0383\pm.0142$	$.4313\pm.0085$	$3.3268\pm.0256$	$.9287\pm.0046$
CIDER-FM	$256O+64I_{Z}$	$\mathbf{.7852}\pm.0124$	$\mathbf{.2977}\pm.0055$	$2.5854\pm.0467$	$.9277\pm.0049$
True CID	—	$.5886$	$.2459$	$1.4340$	$.9000$

Appendix

Table 11: Fixed-query case study (mean

\pm

SE over 16 context draws).

O

denotes observational rows and

I_{Z}

randomised

\operatorname{do}\!\left(Z\right)

rows. All CIDER-FM rows use the same checkpoint. Nominal coverage is 0.90; the true CID provides an analytic reference.

Explore similar work

Sep 2, 2026cs.LG

Causal Foundation Models

Causal inference is the practice of estimating the effect of a treatment or intervention from data. It traditionally requires a bespoke pipeline for every new problem: first proposing a causal mechanism, selecting a compatible estimator, and finally training it. Meanwhile, across diverse settings and modalities, much of machine learning has shifted to the paradigm of foundation models: networks pretrained once at scale and applied to new tasks without fine-tuning. Causal foundation models (CFMs) bring this paradigm to causal inference. CFMs are pretrained neural networks that estimate causal quantities, such as the average treatment effect, on entirely new datasets using in-context learning without requiring model updates. This work provides a practical introduction to this emerging area. We summarize the necessary background in causal inference and machine learning before discussing CFMs. Throughout, we include example code and Jupyter notebooks.

Christopher Stith, Hossein Rahmani, Jesse C. Cresswell

Layer 6 AI, Toronto, Canada · TD Bank Group, Toronto, Canada

Causal Inferences Foundation Model Causal Attention

Ask PDF

Jun 25, 2026cs.LG

A Causal Foundation Model for Structure and Outcome Prediction

We introduce TabPFN-CFM, a causal foundation model that can handle multiple causal problems. TabPFN-CFM predicts both causal structure and outcomes from observational data, supports queries on all three levels of Pearl's Causal Hierarchy and uses known graph structure when available to improve predictions. TabPFN-CFM is trained on synthetic datasets, and generalises to real datasets, demonstrating improved performance over both structural and outcome prediction baselines.

Max Zhu, Martino Mansoldo, Ching-Hao Wang +1

University of Cambridge, United Kingdom · GSK.ai.

Tabular Prior-Data Fitted Network Tabpfn

Ask PDF

Feb 16, 2026cs.LG

Use What You Know: Causal Foundation Models with Partial Graphs

Estimating causal quantities traditionally relies on bespoke estimators tailored to specific assumptions. Recently proposed Causal Foundation Models (CFMs) promise a more unified approach by amortising causal discovery and inference in a single step. However, in their current state, they do not allow for the incorporation of any domain knowledge, which can lead to suboptimal predictions. We bridge this gap by introducing methods to condition CFMs on causal information, such as the causal graph or more readily available ancestral information. When access to complete causal graph information is too strict a requirement, our approach also effectively leverages partial causal information. We systematically evaluate conditioning strategies and find that injecting learnable biases into the attention mechanism, together with a graph-convolutional encoder, is a highly effective method to utilise full and partial causal information. Our experiments show that this conditioning allows a general-purpose CFM to match the performance of specialised models trained on specific causal structures. Overall, our approach addresses a central hurdle on the path towards all-in-one causal foundation models: the capability to answer causal queries in a data-driven manner while effectively leveraging any amount of domain expertise.

Arik Reuter, Anish Dhir, Cristiana Diaconu +6

University of Cambridge, Cambridge, United Kingdom · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Gatsby Computational Neuroscience Unit, University College London +5

Causal Graph Causal Causal Attention Acyclic Graphs

Ask PDF