cs.LGAug 14, 2026

Training Fair Tabular Foundation Models

Authors: Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini, Samira Ebrahimi Kahou, Ulrich Aïvodji

Organizations: ÉTS Montréal Mila - Quebec AI Institute · Layer 6 AI Toronto, Canada · University of Calgary Mila - Quebec AI Institute, CIFAR

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Tabular Foundation Models (TFMs) have emerged as leading methods for tabular predictive tasks, leveraging in-context learning to predict on new data without task-specific training. Despite the increased use of TFMs in high-stakes decision-making, their fairness properties remain largely unexplored. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in-context learning paradigm. We propose FairTFM, a scalable training strategy based on synthetic fairness tasks and a fairness-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes. Experiments on 132 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy.

Figures & tables

Figure 1: (a) We construct fairness tasks from a prior data generator by first sampling a dataset (X′,y)(X^{\prime},y) , then randomly removing one feature to serve as the sensitive attribute ss and converting it to a categorical variable, yielding the triplet (X,y,s)(X,y,s) . The resulting data are split into context (Xctx,yctx,sctx)(X_{\text{ctx}},y_{\text{ctx}},s_{\text{ctx}}) and query (Xqy,yqy,sqy)(X_{\text{qy}},y_{\text{qy}},s_{\text{qy}}) sets for ICL. (b) FairTFM jointly processes context and query inputs with a transformer. In addition to the main prediction head for the target label y^qy\hat{y}_{\text{qy}} , a second head predicts the sensitive attribute s^qy\hat{s}_{\text{qy}} . A gradient reversal layer on the sensitive branch (red arrow) encourages the model to learn representations that are invariant to ss , thereby reducing reliance on sensitive-correlated features while preserving predictive performance.
Figure 2: Pareto front between accuracy and fairness for various models. ↑\uparrow indicates higher is better (accuracy) and ↓\downarrow indicates smaller is better (unfairness).
Figure 3: Pareto front between accuracy and fairness for various models on the 12 fairness tasks beyond ACS PUMS (Appendix E.6 ). ↑\uparrow indicates higher is better (accuracy) and ↓\downarrow indicates smaller is better (unfairness).
Figure 4: Training dynamics comparison between nanoTabPFN (without fairness constraint) and FairTFM with λ∈{0.7,25}\lambda\in\{0.7,25\} .
Figure 5: Fairness-accuracy trade-off comparison with task-specific classical models trained with fairness constraints using Exponentiated Gradient (EG), on ACSIncome (Alabama) with gender as the sensitive attribute. Results for race and age are shown in Figure 11 of Appendix E .
Figure 6: Pareto front of XGBoost models trained with different data representations, with gender as the sensitive attribute. Results for race and age, and for RF and LR as downstream models, are shown in Appendix E.4 .
Probe target (AUC)Raw inputnanoTabPFNFairTFM -0.7FairTFM -25
Sensitive attribute ss0.783 ±\pm 0.110.746 ±\pm 0.100.646 ±\pm 0.140.542 ±\pm 0.11
Label yy0.834 ±\pm 0.020.823 ±\pm 0.030.805 ±\pm 0.040.787 ±\pm 0.03
Table 1: Linear probe AUC for predicting the sensitive attribute ss and the label yy on different representations. We show AUC mean ±\pm std over three seeds and the three sensitive attributes on ACSIncome, Alabama. FairTFM displays a controllable trade-off of reducing the predictability of the sensitive attribute with some obfuscation of the label.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Pareto front between AUCROC and fairness for various models. ↑\uparrow indicates higher is better (accuracy) and ↓\downarrow indicates smaller is better (unfairness).
ModelAccuracyAUCROCDP DiffEOD DiffEOP Diff
TabICLv20.771±0.05\mathbf{0.771\pm 0.05}0.802±0.07\mathbf{0.802\pm 0.07}0.120±0.110.120\pm 0.110.140±0.100.140\pm 0.100.096±0.090.096\pm 0.09
TabPFNv2.50.770±0.050.770\pm 0.050.801±0.070.801\pm 0.070.121±0.120.121\pm 0.120.145±0.110.145\pm 0.110.098±0.100.098\pm 0.10
XGB0.744±0.060.744\pm 0.060.760±0.090.760\pm 0.090.116±0.110.116\pm 0.110.144±0.110.144\pm 0.110.097±0.100.097\pm 0.10
KNN0.716±0.070.716\pm 0.070.707±0.100.707\pm 0.100.106±0.110.106\pm 0.110.139±0.100.139\pm 0.100.093±0.090.093\pm 0.09
RF0.745±0.060.745\pm 0.060.767±0.080.767\pm 0.080.113±0.110.113\pm 0.110.147±0.110.147\pm 0.110.100±0.100.100\pm 0.10
LR0.728±0.060.728\pm 0.060.733±0.100.733\pm 0.100.107±0.120.107\pm 0.120.134±0.110.134\pm 0.110.094±0.090.094\pm 0.09
Appendix
Table 2: Average accuracy, AUCROC, and fairness metrics across the 120 benchmark tasks, reported as mean ±\pm standard deviation over three random seeds. This table complements Figure 2 by summarizing the final-checkpoint performance of FairTFM models. Recent unconstrained TFMs attain the highest average accuracy but also the largest fairness gaps, classical baselines are generally less accurate without being substantially fairer, and FairTFM variants provide better fairness–accuracy trade-offs, with larger λ\lambda yielding progressively lower unfairness at the cost of reduced accuracy.
Table 3: Fairness improvement (positive numbers mean fairer) and accuracy and AUCROC change of each FairTFM variant relative to (a) the best TFM (TabPFNv2.5), (b) the best classical model (XGB), and (c) LR, averaged over the 120 benchmark tasks and derived from Table 2 .
Figure 8: Pareto front of XGBoost models trained with different data representations, for race and age as sensitive attributes (gender is shown in Figure 6 of the main text).
Figure 9: Pareto front of Random Forest models trained with different data representations.
Figure 10: Pareto front of Logistic Regression models trained with different data representations.
Figure 11: Fairness-accuracy trade-off comparison with task-specific classical models trained with fairness constraints using Exponentiated Gradient (EG), on ACSIncome (Alabama) with race and age as sensitive attributes (gender is shown in Figure 5 of the main text).
Figure 12: Pareto front of classical models trained with fairness constraints using Exponentiated Gradient (EG). FairTFM show best overall pareto front when performance are measured with AUCROC.
ModelAccuracyAUCROCDP DiffEOD DiffEOP Diff
TabICLv20.796±0.09\mathbf{0.796\pm 0.09}0.829±0.09\mathbf{0.829\pm 0.09}0.101±0.090.101\pm 0.090.097±0.060.097\pm 0.060.087±0.060.087\pm 0.06
TabPFNv2.50.793±0.100.793\pm 0.100.828±0.090.828\pm 0.090.097±0.090.097\pm 0.090.106±0.070.106\pm 0.070.092±0.080.092\pm 0.08
KNN0.732±0.100.732\pm 0.100.664±0.140.664\pm 0.140.065±0.060.065\pm 0.060.078±0.050.078\pm 0.050.062±0.060.062\pm 0.06
XGB0.780±0.100.780\pm 0.100.803±0.100.803\pm 0.100.100±0.090.100\pm 0.090.099±0.070.099\pm 0.070.085±0.070.085\pm 0.07
LR0.784±0.090.784\pm 0.090.803±0.100.803\pm 0.100.099±0.090.099\pm 0.090.115±0.080.115\pm 0.080.087±0.070.087\pm 0.07
RF0.786±0.090.786\pm 0.090.806±0.090.806\pm 0.090.102±0.090.102\pm 0.090.112±0.060.112\pm 0.060.084±0.050.084\pm 0.05
Appendix
Table 4: Average fairness and accuracy metrics across the 12 fairness tasks beyond ACS PUMS, reported as mean ±\pm standard deviation over three random seeds. The same pattern as on the 120 ACS tasks holds: unconstrained TFMs are the most accurate but least fair, while FairTFM variants sharply reduce disparities at a moderate predictive cost.
Figure 13: Pareto front between AUCROC and fairness for various models on the 12 fairness tasks beyond ACS PUMS. ↑\uparrow indicates higher is better and ↓\downarrow indicates smaller is better (unfairness).
Table 5: Fairness improvement (positive numbers mean fairer) and accuracy and AUCROC change of each FairTFM variant relative to (a) TabPFNv2.5, (b) XGB, and (c) LR, averaged over the 12 fairness tasks beyond ACS PUMS and derived from Table 4 .
ModelAccuracyAUCROCDP DiffEOD DiffEOP Diff
XGB (unconstrained)0.7800.7800.8030.8030.1000.1000.0990.0990.0850.085
XGB-EG-DP (best ε\varepsilon )0.7770.7770.6950.6950.062\mathbf{0.062}0.0750.0750.0600.060
XGB-EG-EOD (best ε\varepsilon )0.7780.7780.7020.7020.0840.0840.083\mathbf{0.083}0.0680.068
XGB-EG-EOP (best ε\varepsilon )0.7770.7770.7000.7000.0850.0850.0840.0840.070\mathbf{0.070}
FairTFM -0.70.7520.7520.8030.8030.0490.0490.0560.0560.0560.056
FairTFM -100.7340.7340.7800.7800.0210.0210.0250.0250.0250.025
Appendix
Table 6: Summary comparison on the 12 non-ACS tasks. For each EG constraint type, we report the configuration achieving the best (lowest) value of its targeted metric across the ε\varepsilon sweep (targeted metric in bold); unconstrained XGBoost is included as a reference. Each FairTFM variant is fairer on every targeted metric at substantially higher AUCROC.
Modelε\boldsymbol{\varepsilon}AccuracyAUCROCDP DiffEOD DiffEOP Diff
XGB-EG-DP0.010.778 ±\pm 0.100.695 ±\pm 0.070.068 ±\pm 0.060.073 ±\pm 0.030.061 ±\pm 0.03
0.020.778 ±\pm 0.100.695 ±\pm 0.070.064 ±\pm 0.050.065 ±\pm 0.030.054 ±\pm 0.03
0.030.778 ±\pm 0.100.694 ±\pm 0.070.064 ±\pm 0.050.069 ±\pm 0.030.058 ±\pm 0.03
0.050.778 ±\pm 0.100.692 ±\pm 0.060.067 ±\pm 0.050.070 ±\pm 0.030.059 ±\pm 0.03
0.070.777 ±\pm 0.100.694 ±\pm 0.060.062 ±\pm 0.050.072 ±\pm 0.030.062 ±\pm 0.03
0.100.777 ±\pm 0.100.692 ±\pm 0.060.068 ±\pm 0.050.071 ±\pm 0.020.059 ±\pm 0.03
Appendix
Table 7: Full Exponentiated Gradient results on the 12 non-ACS tasks: average fairness and accuracy metrics for each fairness constraint (DP, EOD, EOP) and tolerance ε\varepsilon , mean ±\pm standard deviation over three random seeds.
More in this topic

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

Explore similar work

CardsList

  1. FTFM: Factorized Tabular Foundation Models

    Oct 18, 2026Josif GrabockaTabular Foundation ModelsTabular ICL

    University of Technology Nuremberg

  2. Efficient and Interpretable Transformer for Counterfactual Fairness

    Apr 29, 2026Panyi Dong, Zhiyu QuanTransformer InterpretabilityTransformer

    Acturial and Risk Mangament Sciences, University of Illinois Urbana-Champaign, Champaign, Illinois, USA.

  3. Deep Fair Learning: Task-Aware Fair Representations via Joint Distance-Covariance Regularization

    Apr 8, 2025Enze Shi, Yiqun Xiao, Linglong Kong +1Representation LearningAlgorithmic Fairness

    Department of Mathematical and Statistical Science, University of Alberta.