LLM knowledge for improved Out-of-Distribution Tabular Prediction - A Cross-Validated Regularisation Approach

Sølve Langseth; Kristoffer Herland Hellton; Hugo L. Hammer

cs.LGOct 27, 2026

LLM knowledge for improved Out-of-Distribution Tabular Prediction - A Cross-Validated Regularisation Approach

Authors: Sølve Langseth, Kristoffer Herland Hellton, Hugo L. Hammer

Organizations: Department of Computer Science, Oslo Metropolitan University, Oslo, Norway · Department of Holistic Systems, SimulaMet, Oslo, Norway · Department of Plastic and Reconstructive Surgery, Oslo University Hospital, Oslo, Norway

Abstract

Prediction models degrade when the deployment population differs from the one used to train them, and training data cannot say which feature-response relationships will transfer. Expert elicitation is slow and unscalable. We ask whether coefficient priors elicited from large language models (LLMs) improve out-of-distribution (OOD) prediction as the shrinkage target of a regularised regression instead of zero, across five tabular datasets split on real covariates, selecting providers and prior structures by source-domain cross-validation alone. The priors help on four of five datasets, by up to 7.9% OOD mean squared error, and ten of twenty-four prior-informed results resolve the mean improvement at conventional confidence; single-deployment intervals are much wider. On classification the gain is in probabilistic accuracy, not ranking, and control tests with corrupted priors attribute the benefit to feature-specific content rather than scale. Two findings temper this. Variation across candidate priors exceeds the average benefit, yet cross-validation margins fall below their own noise, so selection avoids the worst prior rather than finding the best. On three datasets the prior improves in-domain accuracy as much as out-of-domain, making it better regularisation than shift correction. Only where the training data are large enough that the in-domain gain disappears does elicited knowledge demonstrably improve transfer. Code, priors and results: https://github.com/hugohammer/llm-priors-ood.

Figures & tables

Figure 1: Pipeline overview. The data branch (top) fixes the ID/OOD split and fits preprocessing on source data only. The elicitation branch (middle) feeds dataset context into prompt construction, draws repeated samples per provider, validates them against a JSON schema, and estimates $\widehat{\mu}$ , $\widehat{\sigma}^{2}$ and $\widehat{\Sigma}$ as in Eq. 6 . The modelling branch (bottom) fits each family, tunes $\lambda$ by 5-fold CV on the source training pool, selects the winning variant per family by mean CV score, and evaluates on the held-out ID test and OOD sets. The OOD set never enters any selection step.

Family	$\mu$	$P$	Elicited quantities
Baseline (ridge, $L_{2}$ -logistic)	$0$	$I_{p}$	–
Prior-Mean	$\widehat{\mu}$	$I_{p}$	$p$
Independent-Prior	$\widehat{\mu}$	$\mathrm{diag}(1/\widehat{\sigma}^{2}_{j})$	$2p$
Correlated-Prior	$\widehat{\mu}$	$\widehat{\Sigma}^{-1}$	$p+p(p{+}1)/2$
Mixed	blend of the two rows above, Eq. Eq. 7		$p$
Unregularised ref.	$\lambda=0$ ; the prior drops out		–

Table 1: The fitted families as choices of

(\mu,P)

in Eq. 5 . The intercept is never penalised.

\widehat{\mu}

\widehat{\sigma}^{2}_{j}

and

\widehat{\Sigma}

are the elicited quantities of Eq. 6 ; “elicited quantities” counts how many numbers from the elicitation enter the penalty.

Dataset	Domain split (source $\rightarrow$ OOD)	$p$	$n_{\text{tr}}$	$n_{\text{OOD}}$	Shifted
Regression
Boston Housing	housing stock: newer $\rightarrow$ older	11	284	151	9/11
Cancer	US region: South $\rightarrow$ non-South	13	1,106	1,664	5/13
Classification
Heart Disease	cohort: Cleveland $\rightarrow$ Hungary/Switz.	18	242	417	2/18
Diabetes	patient age: $\leq 29\rightarrow\ >29$	7	316	372	2/7

Table 2: Datasets, domain splits and resulting partitions. “Shifted” counts features with

|

Cohen’s

d|>0.5

between source and OOD.

Label	Provider	Model	Mode	Stored as
GPT-5.1	OpenAI	GPT-5.1	plain	openai__gpt-5-1
GPT-5.1 (R)	OpenAI	GPT-5.1	reasoning	openai__gpt-5-1__reasoning
GPT-5-mini	OpenAI	GPT-5-mini	plain	openai__gpt-5-mini
GPT-5-mini (R)	OpenAI	GPT-5-mini	reasoning	openai__gpt-5-mini__reasoning
Opus 4.5	Anthropic	Claude Opus 4.5	plain	anthropic__opus-4-5
Opus 4.5 (R)	Anthropic	Claude Opus 4.5	reasoning	anthropic__opus-4-5__reasoning

Table 3: The twelve elicitation variants. “Mode” distinguishes a plain completion from one in which the model was asked to reason before emitting the coefficient block; the two modes use different JSON schemas, described in Appendix D . “Stored as” is the directory under priors/<dataset>/ in the repository. Dated API snapshot identifiers were not recorded at elicitation time; see the note below.

Family	Selected variant	$\lambda$	OOD MSE	Lift	95% CI (mean)	95% PI (one run)	Below
Boston Housing ( $n_{\text{train}}=284$ , $n_{\text{OOD}}=151$ )
Unregularised ref.	–	–	44.99	–	–	–	–
Baseline ridge	–	25.1	44.10	–	–	–	–
Prior-Mean	Gemini Pro	251	42.92	$-0.18$	[ $-8.29$ , $+7.92$ ]	[ $-12.97$ , $+10.54$ ]	48%
Independent-Prior	Opus 4.5	10.0	39.56	$+7.40$	[ $+0.57$ , $+14.24$ ] †	[ $-1.04$ , $+18.23$ ]	4%
Correlated-Prior	Opus 4.5	15.8	39.18	$+7.85$	[ $+1.18$ , $+14.51$ ] †	[ $-0.61$ , $+18.87$ ]	3%

Table 4: Regression datasets. The selected variant and

\lambda

are those chosen by mean 5-fold CV on the source training pool of the reference split, and the OOD MSE column is that deployed model’s error on the OOD partition. The MSE and the lift columns come from two different computations: the MSE is a single number from the reference split, whereas the lift and both intervals are averaged over repeated source splits, so the lift is not recoverable by comparing the two MSE entries. Lift is the percentage reduction in OOD MSE relative to the ridge baseline. “CI” is a

95\%

confidence interval for the mean lift; “PI” is a

95\%

prediction interval for the lift of a single deployment. Both propagate source-split,

\lambda

-selection and target-sample variability; they differ in whether the source component is averaged over the

S

splits. “Below” is the fraction of prediction draws in which the prior-informed model is worse than baseline. The variant named is the CV-best one the pipeline deploys, not the OOD-best one. A dagger ( † ) marks a confidence interval excluding zero.

Family	Variant	Metric	Value	Lift	95% CI (mean)	95% PI (one run)	Below
Heart Disease ( $n_{\text{train}}=242$ , $n_{\text{OOD}}=417$ )
Baseline	–	AUC	0.8779	–	–	–	–
Prior-Mean	Opus 4.5	AUC	0.8860	$+0.08$	[ $-0.75$ , $+0.91$ ]	[ $-1.13$ , $+2.15$ ]	51%
Independent-Prior	Gemini Pro (R)	AUC	0.8845	$-0.07$	[ $-1.13$ , $+1.00$ ]	[ $-1.50$ , $+1.90$ ]	57%
Correlated-Prior	Opus 4.5	AUC	0.8862	$+0.18$	[ $-0.68$ , $+1.04$ ]	[ $-1.03$ , $+2.47$ ]	47%
Baseline	–	LogLoss	0.4709	–	–	–	–

Table 5: Classification datasets. “Value” is the deployed model’s metric on the reference split; “lift” is the percentage improvement over the

L_{2}

-logistic baseline. “CI” is a

95\%

confidence interval for the mean lift and “PI” a

95\%

prediction interval for a single deployment; both propagate source-split,

\lambda

-selection and target-sample variability and differ only in whether the source component is averaged over the

S

splits. “Below” is the fraction of prediction draws in which the prior-informed model is worse than baseline. The Mixed family is omitted and tracks Prior-Mean closely (

\gamma=1.00

and

0.97

). A dagger ( † ) marks a confidence interval excluding zero; on Adult the two such entries point in the direction of harm, at magnitudes of no practical consequence.

Figure 2: Placebo priors. Each row is one (dataset, prior family, placebo type) combination. Vermillion points are individual placebo draws (

200

for regression,

100

for classification) with the grey bar marking their

2.5

–

97.5

percentile range and the tick their median; the bluish-green marker is the real elicited prior.

p

is the empirical fraction of placebo draws reaching at least the real prior’s lift. Lift is OOD MSE reduction for Boston Housing and Cancer, and OOD log-loss improvement for Heart Disease and Diabetes, the metrics the priors actually move, as established in Tables 4 and 5 . The horizontal axis is symmetric-log beyond

\pm 10\%

, because corrupted priors on the classification datasets can be worse than baseline by several hundred per cent.

Figure 3: Per-dataset OOD lift over baseline across all

48

(12 elicitation variants

\times

4 prior model families) candidates at their CV-tuned

\lambda

. Grey bar: the range from worst to best candidate. Blue circles: the four models deployed by the within-family rule, which are the models reported in Tables 4 and 5 . Green diamond: the single model selected by a cross-family rule, the best of all

48

candidates by source-domain cross-validation. Every quantity is computed from source data alone; the distance from each marker to the top of its bar is the benefit that source-domain selection does not reach.

Figure 4: Cross-variant coefficient agreement on the four datasets where priors help. Each panel shows the prior-mean coefficient for all twelve variants per feature in source-standardised space. Filled circles: non-reasoning. Open circles: reasoning. Grey tick: across-variant mean. Heart Disease shows numeric features only. Off-panel points are annotated at the boundary.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: OOD lift over baseline across ID resplits, with

\lambda

re-tuned on every resplit. Boxes show the interquartile range and median; points are individual resplits. Lift is MSE reduction for Boston Housing and Cancer and AUC gain for the classification datasets. The single-split values reported in Table 4 are in the upper part of these distributions.

Panel of Figure 11 — Figure S2: OOD calibration on Heart Disease (left) and Diabetes (right). Top panels: 10-quantile reliability curves for the deployed baseline and the deployed best-prior family, with the dotted diagonal marking perfect calibration. Bottom panels: histograms of predicted probabilities. AUC is essentially unchanged in both cases while Brier and log loss improve.

Dataset	Family	Single split	Mean	SD	[2.5, 97.5]	Below
Boston Housing	Prior-Mean	$+2.7\%$	$-0.02\%$	4.02	[ $-7.22$ , $+6.55$ ]	42%
Boston Housing	Independent-Prior	$+10.3\%$	$+7.52\%$	3.20	[ $+2.66$ , $+14.43$ ]	0%
Boston Housing	Correlated-Prior	$+11.2\%$	$+7.94\%$	3.48	[ $+2.09$ , $+15.27$ ]	0%
Boston Housing	Mixed	$+2.7\%$	$-0.02\%$	4.02	[ $-7.22$ , $+6.55$ ]	42%
Cancer	Prior-Mean	$+3.9\%$	$+2.37\%$	2.05	[ $-0.30$ , $+7.25$ ]	12%
Cancer	Independent-Prior	$+2.8\%$	$+1.13\%$	1.08	[ $-1.11$ , $+3.33$ ]	10%

Appendix

Table S1: Supplementary. OOD lift over baseline across ID resplits (

n=50

for Boston Housing and Cancer,

30

for Heart Disease and Diabetes,

10

for Adult). “Single split” is the value from Tables 4 and 5 . “Below” is the fraction of resplits in which the prior-informed model is worse than baseline.

Figure S3:

F_{1}

on the OOD set as a function of the classification threshold, for the deployed

L_{2}

baseline (blue) and the deployed Prior-Mean model (orange). Crosses mark

F_{1}

at the fixed

0.5

threshold; filled circles mark

F_{1}

at the threshold chosen by 5-fold cross-validation on source data only, which is the operating point a practitioner would actually deploy. Panel titles give the prevalence shift from the source training pool to the OOD set. Both curves peak well below

0.5

because prevalence rises between the domains, so at

0.5

the baseline has already fallen off its peak while the prior-informed model has not. The prior-informed model’s apparent

F_{1}

advantage at

0.5

is therefore an operating-point effect, and it disappears once the baseline is given the same threshold freedom.

Dataset	Family	Zero-spread	$10^{-8}$	$10^{-6}$	$10^{-4}$	$10^{-2}$	$10^{-1}$	Ledoit–Wolf
Boston Housing	Independent-Prior	4/11	$+10.29$	$+10.29$	$+10.30$	$+9.27$	$+7.53$	–
Boston Housing	Correlated-Prior	4/11	$+11.15$	$+11.15$	$+11.15$	$+9.70$	$+7.69$	$+11.10$
Cancer	Independent-Prior	0/13	$+2.78$	$+2.78$	$+2.78$	$+2.75$	$+2.44$	–
Cancer	Correlated-Prior	0/13	$+2.12$	$+2.12$	$+2.12$	$+2.12$	$+2.08$	$+2.20$
Heart Disease	Independent-Prior	0/18	$+0.75$	$+0.75$	$+0.73$	$+0.41$	$+0.09$	–
Heart Disease	Correlated-Prior	2/18	$+0.94$	$+0.95$	$+0.90$	$+0.93$	$+0.93$	$+0.93$

Appendix

Table S2: OOD lift (%) for the two families whose precision depends on the elicited spread, as the variance floor

\varepsilon

is varied and

\lambda

re-tuned at each setting. “Zero-spread” counts features for which the deployed variant returned bit-identical values across all

M=50

draws, so that

\widehat{\sigma}^{2}_{j}=0

exactly. The Ledoit–Wolf column replaces the jittered pseudo-inverse of

\widehat{\Sigma}

with a shrinkage estimator and applies only to Correlated-Prior. The deployed configuration is

\varepsilon=10^{-8}

. Lift is MSE reduction for the regression datasets and AUC gain for the classification ones.

OOD Generalization

Out-of-distribution generalization studies how machine-learning models perform when test data, domains, tasks, or environments differ from training.

See all 395 papers

Strengthening LLMs for Tabular Prediction with Structural Priors
Oct 20, 2025Pengxiang Cai, Zihao Gao, Wanchen Lian +2Tabular Foundation Models RL for Language Models
The Hong Kong University of Science and Technology (Guangzhou) · Zhejiang University
Marginal Response Surface Elicitation for Zero-Label Tabular Learning
Sep 30, 2026Liangyu Teng, Yicheng Ding, Jing Liu +5Tabular Prediction Tabular ML
Fudan Institute on Networking Systems of AI
Why Large Language Models Fail at Tabular Prediction
Aug 3, 2026Marta Garnelo, Wojciech M. CzarneckiLLM Evaluation Tabular Classification
Fundamental Technologies

Oct 20, 2025J/K move · Enter open · S save

Strengthening LLMs for Tabular Prediction with Structural Priors

Pengxiang Cai, Zihao Gao, Wanchen Lian, Guocong Li+1

The Hong Kong University of Science and Technology (Guangzhou) · Zhejiang University

Tabular Foundation Models RL for Language Models Permutation-Invariant Learning LLM Post-Training Tabular Prediction

Open paper AskPDF arXiv

Tabular prediction has long been dominated by gradient-boosted decision trees and specialized deep tabular models, while large language models (LLMs) remain difficult to make competitive despite their cross-task adaptability and transparent reasoning traces. We address this gap by incorporating tabular structural priors into LLM post-training. Specifically, we propose Permutation Relative Policy Optimization (PRPO), which operationalizes column-permutation invariance through label-preserving column permutations and two-level advantage estimation. This design converts sparse outcome rewards into denser and more stable optimization signals. Extensive experiments on 139 OpenML datasets show that our 8B model reaches a genuinely competitive regime against strong specialized tabular baselines. It achieves strong fully supervised performance, dominates zero-shot settings, and performs on par with 32-shot strong baselines. Moreover, it substantially outperforms much larger general-purpose and reasoning LLMs, including up to a 53.17% improvement over DeepSeek-R1 (685B). These results show that structural-prior RL post-training is an effective route for making LLMs competitive in tabular prediction.

Abstract

OOD Generalization

Explore similar work

Strengthening LLMs for Tabular Prediction with Structural Priors

Marginal Response Surface Elicitation for Zero-Label Tabular Learning

Why Large Language Models Fail at Tabular Prediction