Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining

Zhiheng Zhang

stat.MLSep 22, 2026

Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining

Authors: Zhiheng Zhang

Organizations: School of Statistics and Data Science Shanghai University of Finance and Economics

Abstract

Causal tabular foundation models amortize effect estimation across synthetic mechanisms, but latent-effect supervision rewards posterior shrinkage rather than encoding the repeated-sample response needed in a fixed deployment population. We introduce fluctuation-supervised pretraining (FSP): each synthetic table is labeled by its average treatment effect plus its efficient influence-function fluctuation; deployment remains a frozen forward pass. Along the path $T_{λ,P}=θ(P)+λP_nψ_P$ , we prove an endpoint transition: every fixed $λ<1$ retains label ambiguity of order $(1-λ)^2/n$ , whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order $n^{-2}$ . A finite-pretraining bound combines label, network, episode-sampling, and optimization errors; its sampling defect controls fixed-mechanism bias, mean squared error, variance, Gaussian approximation, and, with variance-head accuracy, studentized coverage. Complementary lower bounds separate local $n^{-1}$ ATE risk from the $\log N/M$ excess risk of generic finite-dictionary episode learning. Experiments trace the learned sampling response. Across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells; validation-selected Summary FSP deploys $11.6\times$ faster per table in our warm one-thread benchmark. Under effect shift, matched Raw FSP lowers mean-checkpoint RMSE by 54.2% and teacher defect by 99.0% versus latent-effect supervision, and RMSE by 10.2% versus the released CausalPFN-S checkpoint. Known-effect semisynthesis tests coverage; two randomized-study evaluations show that lower RMSE can coexist with residual attenuation.

Figures & tables

Figure 1 : Teach the sampling response once, reuse it across new tables. Panel a shows why latent-effect supervision can have the wrong repeated-sampling center under prior shift; Panel b shows that FSP moves EIF structure from per-dataset correction to reusable synthetic supervision; Panel c shows why the full-fluctuation endpoint is statistically special and how finite pretraining error propagates into inferential guarantees.

Figure 2 : Second-order label learnability supports first-order inference. (a) FSP labels follow each realized synthetic table. (b) Allocation imbalance multiplies centered cell-mean error; Theorem 3.2 also controls rare cells. (c) Training controls the task average of

d(P)=\mathfrak{D}_{n}(P;\widehat{f})

; mechanism transfer and variance-head accuracy yield fixed-population inference. This schematic fixes

K,p_{*},\epsilon

and distinguishes label-prediction risk from deployment ATE risk.

Figure 3 : Point accuracy and sampling-law fidelity separate. Within each panel, all methods use the same 300 tables at

n=256

. Blue headlines compare Summary FSP with CausalPFN-S throughout. Raw FSP and Raw latent-effect share architecture,

M=32768

, table streams, optimizer and five seeds, differing only in supervision. The first two axes are logarithmic and lower is better; the dashed response target is one. Classical estimators refit per table; Do-PFN is an out-of-domain check. Appendix N gives protocols and paired intervals.

Figure 4 : Frozen transfer trades variance against residual attenuation. (a–b) National Supported Work at $n=256$ : 1,500 re-randomized-null and post-treatment estimates; white marks are medians and thick/thin bars are interquartile/5th–95th percentiles. (c) Known-effect semisynthesis at $n=256$ . Appendix N gives checkpoint budgets and protocols.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Assumption	Why it is needed	Observable diagnostic	Failure consequence
Consistency and exchangeability	Identify the ATE in ( 2 )	Scientific design and sensitivity analysis; not testable from one table	The same algebra targets an observed standardized contrast, not necessarily a causal effect
$p_{s}\geq p_{*}$ and $e_{s}\in[\epsilon,1-\epsilon]$	Control empty cells and efficient-score moments	Stratum and arm counts; estimated propensity tails	Constants or variance diverge; uniform root- $n$ claim is unavailable
Bounded outcomes and outputs	Concentrate finite-episode squared losses	Outcome support; enforced output clipping	Replace by robust losses and explicit tail conditions
Independent synthetic episodes	Make $M$ the effective task sample size	Generator seeds and episode provenance	Dependent queries cannot be counted as independent tasks
Accurate simulator labels	Match the intended influence geometry	Teacher-corruption and nuisance diagnostics	Proposition L.1 adds bias and variance to the native defect
Architecture and optimization control	Separate representability, estimation, and achieved fit	Norm/width audit; held-out target loss; optimization gap	The corresponding terms in ( 12 ) persist

Appendix

Table 1 : Assumptions are paired with their role, an observable diagnostic, and the mathematical consequence of failure.

Pattern	Representative examples	Where inferential structure enters	Deployment object or operation
Amortized causal or interventional prediction	CausalPFN; Do-PFN; CausalFM [ Balazadeh Meresht et al., 2025 , Robertson et al., 2025 , Ma et al., 2026 ]	Synthetic causal or interventional target	Frozen forward prediction of an effect, uncertainty object, or interventional outcome
Per-data-set orthogonal or targeted estimation	DML; targeted regularization [ Chernozhukov et al., 2018 , Shi et al., 2019 ]	Nuisance fits plus an orthogonal score and cross-fitting, or a targeted neural objective	Construct an estimate anew on the analyzed data set
Downstream inferential construction	MP-OSPC; WALDO [ Melnychuk et al., 2026 , Masserano et al., 2023 ]	After nuisance posteriors, a predictor, or a posterior estimator has been learned	EIF-based one-step ATE posterior, or simulation-calibrated critical values and Neyman inversion
FSP	This work	Efficient table fluctuation and $V(P)$ in synthetic supervision	Reuse native fixed heads; guarantees require stated mechanism-transfer and variance conditions, with a finite-pretraining upper bound and separate hard-family lower bounds

Appendix

Table 2 : Where inferential structure enters. Representative, non-exclusive workflows differ in what is learned once and what a new data set still requires. “Native” denotes the uncorrected output of FSP’s fixed mean and variance heads.

Figure 5 : Changing supervision changes the sampling response. (a–c) Exact Gaussian MSE relative to FSP ( $s=.035$ , $v=1$ ); the white contour marks equal MSE. (d) All 105 seedwise coefficients at the typical mechanism: seven targets, five seeds, three context lengths, $M=8192$ . Lines are means; bands are $95\%$ across-seed $t$ intervals. Appendix N.1 defines the coefficient.

Figure 6 : Seven-target finite-network diagnostics. Panels a–c report own-label risk and its

n

- and

n^{2}

-scaled versions; panels d–e report fixed-mechanism bias and sampling variance. Every mark averages five independently trained checkpoints over 1,000 common held-out tables. Intervals are percentile intervals from all

5^{5}

ordered seed-bootstrap resamples. These finite-network diagnostics complement the population order transition in Theorem 3.2 .

Method	Typical RMSE	Shifted RMSE	Boundary RMSE	Shifted $\mathfrak{D}_{n}$	Shifted slope
Summary FSP	0.0634	0.0627	0.0696	0.042	0.985
Raw FSP	0.0621	0.0624	0.0650	0.061	0.972
Raw latent-effect	0.0176	0.1364	0.0161	5.876	-0.058
CausalPFN-S	0.0702	0.0695	0.0802	0.200	1.021
S-learner	0.0576	0.0565	0.0644	0.061	0.879
T-learner	0.0671	0.0664	0.0776	0.157	0.988

Appendix

Table 3: Paired fixed-mechanism baselines. All estimates use the same 300 held-out tables in each scenario (

n=256

). Summary FSP, Raw FSP and Raw latent-effect each use five

M=32768

checkpoints; the raw pair shares architecture, table streams, optimizer and seeds and differs only in the label. CausalPFN-S and Do-PFN each use one released checkpoint, and classical rows refit once per table. Teacher defect

\mathfrak{D}_{n}=n\mathbb{E}(\widehat{\theta}-T_{P})^{2}

and the through-origin response slope use the oracle teacher only for evaluation. † Do-PFN is tested outside its published continuous-input/output domain. ‡ The DR empirical-

X

average equals AIPW exactly under this intercept fit.

Figure 7 : Statistical quality and deployment cost define an empirical Pareto frontier. (a–c) Mean-checkpoint RMSE versus warm one-checkpoint wall latency on one CPU thread (300 common

n=256

tables); “warm” excludes one-time model loading but includes preprocessing and all per-table work. Solid charcoal segments connect nondominated methods. (d–e) Effect RMSE and scaled teacher defect

n\mathbb{E}(\widehat{\theta}-T_{P})^{2}

on the fixed boundary-overlap mechanism. The factor

n

makes

O(n^{-1})

teacher MSE flat on panel e, so monotone decay is not implied. Summary and Raw FSP use

M=32768

; open FSP marks and grey shading denote frozen

n=1024

extrapolation.

Mechanism	FSP map	Bias	$\mathfrak{D}_{n}$	$\widehat{V}/V$	Coverage	Kolmogorov
Typical	Summary	.0098	.095	.906	.952/.965	.081/.082
Typical	Raw	.0414	.715	.882	.953/.972	.352/.345
Large effect	Summary	$-.0046$	.115	.909	.963/.971	.064/.071
Large effect	Raw	$-.0039$	.399	.864	.993/.995	.186/.199
Weak overlap	Summary	.0388	1.444	.398	.955/1.000	.287/.311
Weak overlap	Raw	.0828	3.286	.328	.706/.998	.591/.514

Appendix

Table 4: Joint inference diagnostics at

n=256

M=8192

. Means over five checkpoints, each evaluated on 1,000 common tables. Coverage and Kolmogorov columns list native/oracle-

V

values; smaller Kolmogorov distance means closer agreement with

N(0,1)

Figure 8 : Two distinct scales, and the cost of forgetting confounding information. Left: empirical comparator defect on continuous data; cyan rings identify the smallest observed defect at each $n$ , without asserting optimality beyond the tested grid. Coarse bins retain confounding bias; excessively fine bins create sparse-cell error. Right: analytic polynomial bounds and the exact Bernoulli nuisance-prior label lower bound. The exponentially small rare-cell term is omitted only in this explicitly labeled scale diagram, not in the theorem. The $n^{-1}$ reference concerns causal parameter estimation, whereas the $n^{-2}$ curve concerns predicting the efficient training label.

Figure 9 : Neural FSP on unbinned continuous covariates. The 12 mechanisms use 400 common tables each at $n=256$ ; neural metrics average three checkpoints. Panel A resolves dimension, smoothness and overlap; B shows the point-risk cost of moving toward the efficient sampling response; C compares native defect with response slope. Neural latent-effect supervision has lower point RMSE, whereas FSP recovers more efficient fluctuation.

Figure 10 : Complete nonlinear and deployment-cost evidence. (a) FSP’s checkpoint-mean RMSE change relative to S-learner in all 48 continuous cells; blue negative entries favor FSP. Row suffixes denote smooth/rough response (S/R) and strong/weak overlap (S/W). (b–c) Equal-cell-weight macro RMSE and sampling-response slope; the grey region and open neural marks denote frozen extrapolation. (d) Median measured cumulative one-thread CPU wall time, including one cold load; FSP curves use direct batches, whereas comparison curves accumulate observed per-table fits. No panel filters seeds or executed cells.

Figure 11 : Gaussian dictionary companion to the Bernoulli proof-family experiment. The executed $N\times M$ surface (left) collapses against $\log N/M$ with near-unit log–log slope (center). Error bars are $\pm 1.96$ standard errors across five simulation seeds. Right: each violin pools 30 seed–budget mean risks per $N$ , normalized by $\log N/M$ .

Figure 12 : Seed-level raw-backbone stress test. RMSE at

n=256

M=8192

, using all 1,000 common tables per mechanism; the aligned main comparison instead uses its disjoint locked 300-table stream. Open marks show five training seeds, filled marks their mean, and intervals are

95\%

across-seed

t

intervals.

Figure 13 : Mean-head and variance-head diagnostics. Typical mechanism,

n=256

, 1,000 common tables. (a) Tablewise variance ratios averaged over five checkpoints, with median/interquartile marks. (b) Paired checkpoint-specific native/oracle coverage. (c) Quantiles averaged over checkpoints, rather than quantiles of ensemble predictions. QQ shape distinguishes centering and scale even when marginal coverage is close to nominal.

Figure 14 : Full randomized-data distributions. National Supported Work at

n=256

, with 1,500 resamples for each endpoint: the re-randomized pre-treatment null and the post-treatment empirical benchmark. Neural predictions average five checkpoints. White marks show medians, thick bars interquartile ranges, and thin bars 5th–95th percentile ranges.

Figure 15 : Independent randomized validation on voter turnout. (a) Full-trial neighbors-mailing versus control contrast. (b–c) RMSE and bias over 1,500 common arm-stratified subsamples at each context length. Open symbols show five learned checkpoints; filled symbols show their prediction ensemble or the unaveraged design-based estimator. Every error targets the full-trial observed-arm difference.

Figure 16 : Changing the target changes the native repeated-sample bias. Fixed large-effect mechanism,

n=256

M=32768

, seed zero, same deployment tables. Intervals bootstrap table indices and quantify Monte Carlo uncertainty of the signed bias. They do not quantify variation across training seeds. The blended target partially retains shrinkage, and the deliberately biased teacher visibly shifts the learned estimator.

Figure 17 : Exact Gaussian objective map. Fixed-task sampling distributions and the analytic latent-effect-to-FSP MSE ratio from Proposition 3.1 ; the equality contour separates prior-center shrinkage from effect-shift attenuation.

Figure 18 : Deployment rows and pretraining episodes are separate resources. Same-scale maps of $n\mathbb{E}(f-T_{P})^{2}$ on the large-effect mechanism over the executed $M\times n$ grid. Asterisks identify the extrapolation length $n=1024$ .

Figure 19 : One executed $M\times n$ grid, four inferential consequences. Summary-backbone FSP on the fixed large-effect mechanism, with three checkpoint seeds and 2,000 common tables per cell. Panels show native defect, $\sqrt{n}$ times the absolute seed-averaged bias, oracle-normalized $n\operatorname{MSE}$ , and native $95\%$ coverage. The outlined executed-cell region is where mean defect is at least one; every displayed value is an executed cell.

Figure 20 : Repeated-sample shape at a fixed mechanism. Peak-normalized ridges and QQ diagnostics use the large-effect mechanism,

n=256

, 2,000 common tables, and seed-zero neural models trained at

M=32768

. QQ outputs are standardized with oracle

V

; the grey band is an approximate pointwise

95\%

normal-quantile envelope.

Figure 21 : Bias, sampling spread, and native coverage. At

n=256

M=32768

, learned methods show three checkpoint seeds and baselines one deterministic-model point. Right-panel

95\%

Wilson intervals use 2,000 deployment repetitions per checkpoint. Centering, spread and coverage assess complementary aspects of inference.

Figure 22 : Observed-data stability and known-effect calibration. Top: the empirical birthweight contrast, $n=256$ , seed zero; 160 displayed dots accompany density and median/interquartile/2.5th–97.5th percentile summaries of all 2,000 bootstrap outputs. Bottom: known-effect semisynthesis uses 2,000 tables per context length and effect, with neural coverage averaged over three checkpoints trained at $M=32768$ .

Target / setting	Method	Bias	RMSE	Coverage / inclusion
Fixed large effect	Latent-PFN	-0.1262	0.1271	0.506
	FSP-PFN	+0.0056	0.0645	0.953
	Smoothed-stratified	-0.0010	0.0628	0.956
	Stratified	+0.0006	0.0667	0.942
Fixed typical effect	Latent-PFN	-0.0143	0.0234	1.000
	FSP-PFN	+0.0062	0.0632	0.947

Appendix

Table 5: Accuracy and inferential behavior on the original summary architecture. All fixed-mechanism rows use

n=256

and learned methods use

M=32768

. The final block reports inclusion of an empirical descriptive contrast.

Theory	Implemented check	Diagnostic readout
$\lambda$ -phase transition	Seven targets, five seeds, identical generator and compute	Scaled own-label risk, FSP defect, response slope
Finite pretraining defect	Summary-backbone $M\times n$ maps and aligned nine-method fixed-mechanism $n$ -grid	Bias, RMSE, $n\mathbb{E}(f-T_{P})^{2}$
Bias, variance, CLT	Fixed-mechanism repetitions; native output quantiles	Centering, spread, Kolmogorov/QQ shape
Studentization	Native/oracle- $V$ intervals and Kolmogorov distances	$\widehat{V}/V$ , coverage, centering and shape
Pretraining lower bound	Exact Bernoulli proof family; separate Gaussian dictionary companion	Excess risk versus $\log N/M$
Teacher robustness	Blended and shifted labels on common tables	Signed bias response

Appendix

Table 6: Theorem–evidence contract. Proof establishes each mathematical statement; the paired experiment probes its observable implication.

Explore similar work

May 29, 2026cs.LG

TabCausal: Pretraining Across Causal Environments for Tabular Causal Discovery

Causal discovery aims to recover directed causal relations from observational and interventional data, providing a basis for mechanistic understanding and reliable decision-making. Causal discovery foundation models (CDFMs) seek to amortize this problem by mapping a dataset directly to a causal graph in a single forward pass, avoiding per-dataset testing, search, or optimization. However, existing CDFMs remain limited, often failing to consistently match strong classical methods, and we find that a key bottleneck is how causal pretraining tasks are constructed. Based on this observation, we propose TabCausal, a data-driven CDFM trained with broad causal pretraining over diverse graph priors, structural mechanisms, noise models, dimensions, sample sizes, and intervention regimes. A dynamic task construction strategy composes these causal environments into varied discovery tasks, enabling more transferable structural learning from observational and mixed-interventional data. On large-scale synthetic benchmarks, TabCausal achieves better macro-averaged performance than a diverse set of causal discovery baselines. To further bridge abstract synthetic generators and realistic causal reasoning scenarios, we introduce a protocol-guided and LLM-audited semantic causal environment benchmark, where domain-grounded SCMs generate interpretable observational and interventional datasets for out-of-distribution analysis. Across both synthetic and semantic environments, TabCausal demonstrates robust structure recovery, especially under interventional evidence, highlighting broad causal pretraining as a key ingredient for transferable amortized causal discovery.

Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang +1

Nanjing University

Causal Discovery Methods Causal Model Discovery Causal Attention Data-Driven Environment

Ask PDF

May 19, 2026cs.AI

When Tabular Foundation Models Meet Strategic Tabular Data: A Prior Alignment Approach

Tabular foundation models based on pretrained prior-data fitted networks~(PFNs) have shown strong generalization on diverse tabular tasks, but they are typically designed for \emph{non-strategic} settings where data distributions are independent of deployed classifiers. In many real-world decision scenarios, however, individuals may strategically modify their features after deployment to obtain favorable outcomes, inducing a post-deployment distribution shift. This paper studies whether PFN-style tabular foundation models can generalize to such \emph{strategic} tabular data. We show that strategic manipulation creates a mismatch between the non-strategic prior learned during pretraining and the post-manipulation strategic prior, which leads to systematic prediction bias. To address this issue, we propose \textbf{Strategic Prior-data Fitted Network}~\textit{(SPN)}, an inference-time strategy-aware framework that adapts tabular foundation models to strategic environments without retraining. SPN constructs strategic in-context examples to approximate post-manipulation inputs and aligns PFN predictions with the induced strategic distribution. Experiments on real-world and synthetic tabular datasets show that SPN consistently improves robustness and predictive performance under strategic manipulation compared with both tabular foundation models and classical tabular methods.

Xinpeng Lv, Yunxin Mao, Renzhe Xu +13

College of Computer, National University of Defense Technology, Changsha, China · Institute for Theoretical Computer Science, Shanghai University of Finance and Economics, Shanghai, China · State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Beijing, China +3

Tabular Foundation Models Tabular Prior-Data Fitted Network Tabular Data Inference-Time

Ask PDF

Sep 2, 2026cs.LG

Causal Foundation Models

Causal inference is the practice of estimating the effect of a treatment or intervention from data. It traditionally requires a bespoke pipeline for every new problem: first proposing a causal mechanism, selecting a compatible estimator, and finally training it. Meanwhile, across diverse settings and modalities, much of machine learning has shifted to the paradigm of foundation models: networks pretrained once at scale and applied to new tasks without fine-tuning. Causal foundation models (CFMs) bring this paradigm to causal inference. CFMs are pretrained neural networks that estimate causal quantities, such as the average treatment effect, on entirely new datasets using in-context learning without requiring model updates. This work provides a practical introduction to this emerging area. We summarize the necessary background in causal inference and machine learning before discussing CFMs. Throughout, we include example code and Jupyter notebooks.

Christopher Stith, Hossein Rahmani, Jesse C. Cresswell

Layer 6 AI, Toronto, Canada · TD Bank Group, Toronto, Canada

Causal Inferences Foundation Model Causal Attention

Ask PDF