Marginal Response Surface Elicitation for Zero-Label Tabular Learning

Every tabular classification task requires sufficient labeled data. When labels are scarce, practitioners can collect more examples, encode domain expertise into features, or accept lower performance. Large language models offer an alternative by leveraging knowledge acquired during pretraining. This raises three questions: (i) can zero-shot inference reduce the need for labeled data; (ii) can a richer prompt reduce the need for more examples; and (iii) can encoded domain knowledge compensate for missing data? To answer each under explicit, testable conditions, we introduce TabLens, a benchmark of 91 real-world and synthetic datasets across eight domains, plus eight industrial datasets with free-text columns. First, below a complexity threshold estimable from a dataset's intrinsic dimensionality before running an LLM, zero-shot inference matches the accuracy of up to about 16 labeled examples. This holds whether the model generates an answer or, more cheaply, scores candidate labels by log-probabilities, which loses almost no accuracy except for the smallest models. Above the threshold, the advantage disappears. On datasets with free-text columns, however, zero-shot LLMs outperform 16-shot TabPFN-3 Plus without manual feature engineering. Second, a richer prompt reduces the number of required demonstrations only when it includes both real column names and a plain-language task description; providing only one can be worse than neither, particularly at higher shot counts. Third, on LLM-synthetic datasets, explicit if--then expert rules help only when the prompt provides enough task context to interpret them, and even then, they substitute for at most about 16 real examples. Together, these results establish testable conditions under which LLMs can reduce data-collection and feature-engineering requirements for tabular classification.

Tabular prediction has long been dominated by gradient-boosted decision trees and specialized deep tabular models, while large language models (LLMs) remain difficult to make competitive despite their cross-task adaptability and transparent reasoning traces. We address this gap by incorporating tabular structural priors into LLM post-training. Specifically, we propose Permutation Relative Policy Optimization (PRPO), which operationalizes column-permutation invariance through label-preserving column permutations and two-level advantage estimation. This design converts sparse outcome rewards into denser and more stable optimization signals. Extensive experiments on 139 OpenML datasets show that our 8B model reaches a genuinely competitive regime against strong specialized tabular baselines. It achieves strong fully supervised performance, dominates zero-shot settings, and performs on par with 32-shot strong baselines. Moreover, it substantially outperforms much larger general-purpose and reasoning LLMs, including up to a 53.17% improvement over DeepSeek-R1 (685B). These results show that structural-prior RL post-training is an effective route for making LLMs competitive in tabular prediction.

Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning. TabFM produces calibrated zero-shot predictions in a single forward pass without task-specific tuning. Trained entirely on synthetic tables generated from structural causal models, TabFM learns general tabular representations that transfer zero-shot to real-world tasks. Across all 51 benchmark datasets in TabArena (38 classification and 13 regression), zero-shot TabFM ranks first among default tabular foundation models and outperforms tuned AutoML pipelines. Two extensions over the same frozen weights improve performance further on both tracks: multi-view feature expansion with ensembling and post-hoc calibration (TabFM+), and LLM-guided, dataset-specific data processing and feature engineering (TabFM-Auto).