GIST: Generator-state-Informed Supervision for Tabular Foundation Models

Zhangyang Peng; Jingze Shi; Tongyi Zhang; Yuyu Luo

cs.LGOct 9, 2026

GIST: Generator-state-Informed Supervision for Tabular Foundation Models

Authors: Zhangyang Peng, Jingze Shi, Tongyi Zhang, Yuyu Luo

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Abstract

Tabular foundation models (TFMs) based on prior-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining. A key observation is that intermediate generator states can reveal label uncertainty beyond a sampled outcome. We propose Generator-state-Informed Supervision for TFMs (GIST), which uses these states to average query losses over conditional label distributions. GIST jointly designs task generation and query supervision, using entropy-calibrated encodings to obtain tractable, input-consistent targets on supported label-dependent branches. We show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution. Under matched pretraining schedules, GIST improves mean rank across three architectures on both TabArena and TALENT without changing architecture or inference. Across these benchmarks and architectures, GIST wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.

Figures & tables

Figure 1: Overview of GIST. (I) Task generation retains state and calibrates supported category encodings. (II) Query targets use conditional probabilities, with hard-label fallback. (III) The TFM uses standard inputs and hard context labels. Objective and raw-gradient covariance comparisons use the same generating distribution at fixed parameters.

Figure 2: Gaussian encoding and entropy-budget calibration. (I) Observing label-dependent features updates class probabilities. (II) Routing all query-label effects through a local Gaussian channel makes posterior targets tractable. (III) Calibration selects the largest noise scale certified by an entropy budget; nearby codes require a smaller scale in the illustrated example.

Method	TabArena (38)				TALENT (188)
Method	AUC $\uparrow$ (30)	LL $\downarrow$ (8)	$I\downarrow$	Rank $\downarrow$	AUC $\uparrow$ (120)	LL $\downarrow$ (68)	$I\downarrow$	Rank $\downarrow$
CatBoost	0.8496	0.2611	11.47	6.03	0.8536	0.4273	26.08	7.09
LightGBM	0.8480	0.2614	12.52	6.20	0.8606	0.4503	26.57	7.36
XGBoost	0.8473	0.2610	13.08	6.89	0.8481	0.4735	28.79	7.61
RealMLP	0.8491	0.2543	10.86	5.98	0.8593	0.4572	29.73	8.14
TabM	0.8488	0.2615	11.43	5.93	0.8513	0.4422	28.58	7.70

Table 1: IID performance of the complete GIST design. AUC: binary ROC-AUC; LL: multiclass log loss;

I

: Improvability (%); Rank: mean rank among 11 methods. Bold marks the best value per column; blue / green marks improvement/deterioration against same-architecture Hard.

Figure 3: Component contributions to clipped-gradient variance reduction. Segments show contributions

c_{m}

in percentage points of whole-model Hard variance; hollow diamonds show their sum

R

with pointwise 95% batch-pair bootstrap intervals. Shading marks Stage 2; checkpoints are equally spaced for display, and panels use separate linear

y

-axis ranges.

Table 5

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Architecture	Datasets	W/L/T	Win (%)
TabArena	TabICLv2	38	21/17/0	55.3
TabArena	TabSwift	38	25/13/0	65.8
TabArena	TabPFN-2.5	38	27/11/0	71.1
TALENT	TabICLv2	200	132/65/3	66.0
TALENT	TabSwift	200	137/62/1	68.5
TALENT	TabPFN-2.5	188	101/85/2	53.7

Appendix

Table 4: Stage 2 paired results on each architecture’s complete supported range. These ranges differ across architectures and are not used for cross-architecture rankings. W/L/T denotes wins/losses/ties of GIST against Hard.

Architecture	Hard	GIST
TabICLv2	50.0%	50.0%
TabSwift	25.0%	75.0%
TabPFN-2.5	41.7%	58.3%

Appendix

Table 5: Paired OOD win rates on 12 BeyondArena datasets. Each column reports its dataset-level wins against the other condition. There are no ties.

Method	AUC $\uparrow$	LL $\downarrow$	$I\downarrow$	Rank $\downarrow$	Win (%)
CatBoost	0.7806	0.8537	20.25	4.85	—
LightGBM	0.7830	0.8638	21.42	6.06	—
XGBoost	0.7791	0.9670	26.63	6.59	—
RealMLP	0.7943	0.8508	18.64	5.37	—
TabM	0.7894	0.8133	16.75	4.87	—
TabICLv2 (Hard)	0.7527	0.8655	27.45	7.21	—

Appendix

Table 6: Stage 2 OOD results on the 12 common BeyondArena datasets (six binary, six multiclass). AUC is binary ROC-AUC; LL is multiclass log loss;

I

is mean Improvability (%); Rank is mean rank among 11 methods. Scores are averaged within each dataset and then equally across datasets. Win compares GIST with its same-architecture Hard control. Bold marks the best AUC, LL,

I

, and Rank.

Architecture	Pool	$L$	Reduction (%)	RB (%)
TabICLv2	Full	1,024	$+1.37\;[-0.44,+3.23]$	26.2
TabICLv2	Channel	1,024	$+2.07\;[-0.77,+5.06]$	98.5
TabICLv2	Tractable targets	1,024	$+11.52\;[+6.72,+15.93]$	99.3
TabICLv2	Tractable targets, $Q=128$	1,024	$+13.77\;[+9.50,+18.32]$	99.4
TabICLv2	Full	8,192	$+1.69\;[+0.81,+3.07]$	30.7
TabICLv2	Channel	8,192	$+1.99\;[-0.81,+5.68]$	98.3

Appendix

Table 7: Gradient diagnostics at the final checkpoints after global norm-one clipping of each whole-batch gradient. RB coverage is the fraction of supervised query rows using conditional targets. Intervals are pointwise 95% batch-pair bootstrap intervals.

Component	Hard share (%)	Relative reduction (%)	Contribution (pp)
TabICLv2
TFcol	86.95 [84.15, 89.29]	9.05 [3.57, 13.94]	7.87 [3.07, 12.10]
TFrow	1.62 [1.29, 2.01]	10.14 [-3.97, 19.36]	0.16 [-0.07, 0.32]
TFicl	8.23 [6.67, 10.16]	26.85 [23.55, 29.96]	2.21 [1.72, 2.85]
Prediction head	3.20 [2.62, 3.94]	39.90 [32.18, 47.73]	1.28 [0.94, 1.73]
TabSwift

Appendix

Table 8: Functional decomposition at the final checkpoints for

L=1024

. Entries are estimates with pointwise 95% batch-pair bootstrap intervals. Shares and relative reductions are percentages; contributions are percentage points of whole-model reduction. Values are rounded only for display.

Component	Hard share (%)	Relative reduction (%)	Contribution (pp)
TabICLv2
TFcol	93.89 [91.93, 95.20]	5.72 [1.93, 10.24]	5.37 [1.82, 9.54]
TFrow	1.01 [0.79, 1.29]	8.35 [5.78, 11.23]	0.08 [0.06, 0.12]
TFicl	3.88 [3.02, 5.17]	19.73 [17.02, 22.79]	0.77 [0.57, 1.07]
Prediction head	1.23 [0.92, 1.71]	47.23 [40.06, 54.10]	0.58 [0.40, 0.87]
TabSwift

Appendix

Table 9: Functional decomposition at the final checkpoints for

L=8192

	TabArena (38)			TALENT (188)
Budget	AUC $\uparrow$	LL $\downarrow$	Win $\uparrow$	AUC $\uparrow$	LL $\downarrow$	Win $\uparrow$
Hard	0.8420	0.2848	—	0.8713	0.4267	—
$\beta=0.05$	0.8427	0.2824	68.4	0.8731	0.4250	64.9
$\beta=0.15$	0.8425	0.2811	65.8	0.8726	0.4243	67.0
$\beta=0.30$	0.8423	0.2822	60.5	0.8727	0.4240	63.3

Appendix

Table 10: TabSwift-GIST entropy-budget sensitivity at Stage 2. AUC and LL denote mean binary ROC-AUC and multiclass log loss; each dataset receives equal weight. Bold marks the best mean for each metric. Win is the percentage of datasets won against Hard, including ties in the denominator. Dataset sets match Table 1 .

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Aug 19, 2026Qi Qin, Jiajie Zhu, Dali Chen +6Knowledge Distillation Tabular Foundation Models
Renmin University of China, Center for Applied Statistics and School of Statistics, Beijing, China · Ant Digital Technologies, Ant Group, Hangzhou, China · Nanjing University, Nanjing, China
TabGift: A Context-Efficient Foundation Model for Tabular Data Generation
Oct 12, 2026Yu-Sheng Li, Jun-Peng Jiang, Han-Jia YeSynthetic Tabular Data Generation Generative Modeling
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
TabFORGE: A Tabular Foundation Model for Generative Modelling
May 10, 2026Xiangjian Jiang, Mingxuan Liu, Nikola Simidjievski +2Tabular Foundation Models Synthetic Tabular Data Generation
Department of Computer Science and Technology, University of Cambridge, UK · SAP SE · Télécom Paris, Institut Polytechnique de Paris, France

Aug 19, 2026J/K move · Enter open · S save

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang+5

Renmin University of China, Center for Applied Statistics and School of Statistics, Beijing, China · Ant Digital Technologies, Ant Group, Hangzhou, China · Nanjing University, Nanjing, China

Knowledge Distillation Tabular Foundation Models Cost-Aware Inference Tabular ICL

Open paper AskPDF arXiv

Tabular foundation models (TFMs) achieve strong performance through in-context learning, but context-dependent inference imposes substantial latency and memory costs, hindering large-scale deployment. We propose GEAR (Generative Expansion and Real Anchoring), a modular two-stage framework that distills TFMs into lightweight MLP or tree-based predictors that can be deployed on commodity CPUs. Stage 1 uses synthetic covariates solely as teacher-query locations and trains the student on soft TFM targets, expanding coverage beyond observed rows. Stage 2 re-anchors the student to the target distribution using real labels and out-of-fold teacher predictions, whitch avoids self-labeling leakage. We further derive a risk certificate characterizing the trade-off between generated-query volume and generator fidelity. Experiments on TALENT and TabArena demonstrate the broad applicability of GEAR. Two-stage MLPs outperform supervised MLPs by 1.81--2.00 AUC points on binary tasks and 1.19--1.35 points on multiclass tasks, with additional gains over real-data-only distillation of 1.76--2.19 and 2.09--2.40 points, respectively. On binary tasks, the gains also transfer to LightGBM and XGBoost, and all three student families outperform CatBoost, the strongest non-TFM baseline, in mean AUC. Ablations show gains beyond longer training or alternative warm starts, greater stability from staged than mixed optimization, and generator-dependent diminishing returns as query volume increases. Finally, GEAR reduces median inference time by 57--2866 times and peak prediction memory by 1.9--3.3 times, while retaining higher AUC than matched supervised baselines.

Abstract

Tabular Foundation Models

Explore similar work

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

TabGift: A Context-Efficient Foundation Model for Tabular Data Generation

TabFORGE: A Tabular Foundation Model for Generative Modelling