cs.LGAug 14, 2026

Training Fair Tabular Foundation Models

Authors: Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini, Samira Ebrahimi Kahou, Ulrich Aïvodji

Organizations: ÉTS Montréal Mila - Quebec AI Institute · Layer 6 AI Toronto, Canada · University of Calgary Mila - Quebec AI Institute, CIFAR

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Tabular Foundation Models (TFMs) have emerged as leading methods for tabular predictive tasks, leveraging in-context learning to predict on new data without task-specific training. Despite the increased use of TFMs in high-stakes decision-making, their fairness properties remain largely unexplored. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in-context learning paradigm. We propose FairTFM, a scalable training strategy based on synthetic fairness tasks and a fairness-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes. Experiments on 132 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy.

Figures & tables

Figure 1: (a) We construct fairness tasks from a prior data generator by first sampling a dataset (X′,y)(X^{\prime},y) , then randomly removing one feature to serve as the sensitive attribute ss and converting it to a categorical variable, yielding the triplet (X,y,s)(X,y,s) . The resulting data are split into context (Xctx,yctx,sctx)(X_{\text{ctx}},y_{\text{ctx}},s_{\text{ctx}}) and query (Xqy,yqy,sqy)(X_{\text{qy}},y_{\text{qy}},s_{\text{qy}}) sets for ICL. (b) FairTFM jointly processes context and query inputs with a transformer. In addition to the main prediction head for the target label y^qy\hat{y}_{\text{qy}} , a second head predicts the sensitive attribute s^qy\hat{s}_{\text{qy}} . A gradient reversal layer on the sensitive branch (red arrow) encourages the model to learn representations that are invariant to ss , thereby reducing reliance on sensitive-correlated features while preserving predictive performance.
Figure 2: Pareto front between accuracy and fairness for various models. ↑\uparrow indicates higher is better (accuracy) and ↓\downarrow indicates smaller is better (unfairness).
Figure 3: Pareto front between accuracy and fairness for various models on the 12 fairness tasks beyond ACS PUMS (Appendix E.6 ). ↑\uparrow indicates higher is better (accuracy) and ↓\downarrow indicates smaller is better (unfairness).
Figure 4: Training dynamics comparison between nanoTabPFN (without fairness constraint) and FairTFM with λ∈{0.7,25}\lambda\in\{0.7,25\} .
Figure 5: Fairness-accuracy trade-off comparison with task-specific classical models trained with fairness constraints using Exponentiated Gradient (EG), on ACSIncome (Alabama) with gender as the sensitive attribute. Results for race and age are shown in Figure 11 of Appendix E .
Figure 6: Pareto front of XGBoost models trained with different data representations, with gender as the sensitive attribute. Results for race and age, and for RF and LR as downstream models, are shown in Appendix E.4 .
Probe target (AUC)Raw inputnanoTabPFNFairTFM -0.7FairTFM -25
Sensitive attribute ss0.783 ±\pm 0.110.746 ±\pm 0.100.646 ±\pm 0.140.542 ±\pm 0.11
Label yy0.834 ±\pm 0.020.823 ±\pm 0.030.805 ±\pm 0.040.787 ±\pm 0.03
Table 1: Linear probe AUC for predicting the sensitive attribute ss and the label yy on different representations. We show AUC mean ±\pm std over three seeds and the three sensitive attributes on ACSIncome, Alabama. FairTFM displays a controllable trade-off of reducing the predictability of the sensitive attribute with some obfuscation of the label.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Pareto front between AUCROC and fairness for various models. ↑\uparrow indicates higher is better (accuracy) and ↓\downarrow indicates smaller is better (unfairness).
ModelAccuracyAUCROCDP DiffEOD DiffEOP Diff
TabICLv20.771±0.05\mathbf{0.771\pm 0.05}0.802±0.07\mathbf{0.802\pm 0.07}0.120±0.110.120\pm 0.110.140±0.100.140\pm 0.100.096±0.090.096\pm 0.09
TabPFNv2.50.770±0.050.770\pm 0.050.801±0.070.801\pm 0.070.121±0.120.121\pm 0.120.145±0.110.145\pm 0.110.098±0.100.098\pm 0.10
XGB0.744±0.060.744\pm 0.060.760±0.090.760\pm 0.090.116±0.110.116\pm 0.110.144±0.110.144\pm 0.110.097±0.100.097\pm 0.10
KNN0.716±0.070.716\pm 0.070.707±0.100.707\pm 0.100.106±0.110.106\pm 0.110.139±0.100.139\pm 0.100.093±0.090.093\pm 0.09
RF0.745±0.060.745\pm 0.060.767±0.080.767\pm 0.080.113±0.110.113\pm 0.110.147±0.110.147\pm 0.110.100±0.100.100\pm 0.10
LR0.728±0.060.728\pm 0.060.733±0.100.733\pm 0.100.107±0.120.107\pm 0.120.134±0.110.134\pm 0.110.094±0.090.094\pm 0.09
Appendix
Table 2: Average accuracy, AUCROC, and fairness metrics across the 120 benchmark tasks, reported as mean ±\pm standard deviation over three random seeds. This table complements Figure 2 by summarizing the final-checkpoint performance of FairTFM models. Recent unconstrained TFMs attain the highest average accuracy but also the largest fairness gaps, classical baselines are generally less accurate without being substantially fairer, and FairTFM variants provide better fairness–accuracy trade-offs, with larger λ\lambda yielding progressively lower unfairness at the cost of reduced accuracy.
Table 3: Fairness improvement (positive numbers mean fairer) and accuracy and AUCROC change of each FairTFM variant relative to (a) the best TFM (TabPFNv2.5), (b) the best classical model (XGB), and (c) LR, averaged over the 120 benchmark tasks and derived from Table 2 .
Figure 8: Pareto front of XGBoost models trained with different data representations, for race and age as sensitive attributes (gender is shown in Figure 6 of the main text).
Figure 9: Pareto front of Random Forest models trained with different data representations.
Figure 10: Pareto front of Logistic Regression models trained with different data representations.
Figure 11: Fairness-accuracy trade-off comparison with task-specific classical models trained with fairness constraints using Exponentiated Gradient (EG), on ACSIncome (Alabama) with race and age as sensitive attributes (gender is shown in Figure 5 of the main text).
Figure 12: Pareto front of classical models trained with fairness constraints using Exponentiated Gradient (EG). FairTFM show best overall pareto front when performance are measured with AUCROC.
ModelAccuracyAUCROCDP DiffEOD DiffEOP Diff
TabICLv20.796±0.09\mathbf{0.796\pm 0.09}0.829±0.09\mathbf{0.829\pm 0.09}0.101±0.090.101\pm 0.090.097±0.060.097\pm 0.060.087±0.060.087\pm 0.06
TabPFNv2.50.793±0.100.793\pm 0.100.828±0.090.828\pm 0.090.097±0.090.097\pm 0.090.106±0.070.106\pm 0.070.092±0.080.092\pm 0.08
KNN0.732±0.100.732\pm 0.100.664±0.140.664\pm 0.140.065±0.060.065\pm 0.060.078±0.050.078\pm 0.050.062±0.060.062\pm 0.06
XGB0.780±0.100.780\pm 0.100.803±0.100.803\pm 0.100.100±0.090.100\pm 0.090.099±0.070.099\pm 0.070.085±0.070.085\pm 0.07
LR0.784±0.090.784\pm 0.090.803±0.100.803\pm 0.100.099±0.090.099\pm 0.090.115±0.080.115\pm 0.080.087±0.070.087\pm 0.07
RF0.786±0.090.786\pm 0.090.806±0.090.806\pm 0.090.102±0.090.102\pm 0.090.112±0.060.112\pm 0.060.084±0.050.084\pm 0.05
Appendix
Table 4: Average fairness and accuracy metrics across the 12 fairness tasks beyond ACS PUMS, reported as mean ±\pm standard deviation over three random seeds. The same pattern as on the 120 ACS tasks holds: unconstrained TFMs are the most accurate but least fair, while FairTFM variants sharply reduce disparities at a moderate predictive cost.
Figure 13: Pareto front between AUCROC and fairness for various models on the 12 fairness tasks beyond ACS PUMS. ↑\uparrow indicates higher is better and ↓\downarrow indicates smaller is better (unfairness).
Table 5: Fairness improvement (positive numbers mean fairer) and accuracy and AUCROC change of each FairTFM variant relative to (a) TabPFNv2.5, (b) XGB, and (c) LR, averaged over the 12 fairness tasks beyond ACS PUMS and derived from Table 4 .
ModelAccuracyAUCROCDP DiffEOD DiffEOP Diff
XGB (unconstrained)0.7800.7800.8030.8030.1000.1000.0990.0990.0850.085
XGB-EG-DP (best ε\varepsilon )0.7770.7770.6950.6950.062\mathbf{0.062}0.0750.0750.0600.060
XGB-EG-EOD (best ε\varepsilon )0.7780.7780.7020.7020.0840.0840.083\mathbf{0.083}0.0680.068
XGB-EG-EOP (best ε\varepsilon )0.7770.7770.7000.7000.0850.0850.0840.0840.070\mathbf{0.070}
FairTFM -0.70.7520.7520.8030.8030.0490.0490.0560.0560.0560.056
FairTFM -100.7340.7340.7800.7800.0210.0210.0250.0250.0250.025
Appendix
Table 6: Summary comparison on the 12 non-ACS tasks. For each EG constraint type, we report the configuration achieving the best (lowest) value of its targeted metric across the ε\varepsilon sweep (targeted metric in bold); unconstrained XGBoost is included as a reference. Each FairTFM variant is fairer on every targeted metric at substantially higher AUCROC.
Modelε\boldsymbol{\varepsilon}AccuracyAUCROCDP DiffEOD DiffEOP Diff
XGB-EG-DP0.010.778 ±\pm 0.100.695 ±\pm 0.070.068 ±\pm 0.060.073 ±\pm 0.030.061 ±\pm 0.03
0.020.778 ±\pm 0.100.695 ±\pm 0.070.064 ±\pm 0.050.065 ±\pm 0.030.054 ±\pm 0.03
0.030.778 ±\pm 0.100.694 ±\pm 0.070.064 ±\pm 0.050.069 ±\pm 0.030.058 ±\pm 0.03
0.050.778 ±\pm 0.100.692 ±\pm 0.060.067 ±\pm 0.050.070 ±\pm 0.030.059 ±\pm 0.03
0.070.777 ±\pm 0.100.694 ±\pm 0.060.062 ±\pm 0.050.072 ±\pm 0.030.062 ±\pm 0.03
0.100.777 ±\pm 0.100.692 ±\pm 0.060.068 ±\pm 0.050.071 ±\pm 0.020.059 ±\pm 0.03
Appendix
Table 7: Full Exponentiated Gradient results on the 12 non-ACS tasks: average fairness and accuracy metrics for each fairness constraint (DP, EOD, EOP) and tolerance ε\varepsilon , mean ±\pm standard deviation over three random seeds.
More in this topic

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

Explore similar work

CardsList
Oct 18, 2026

FTFM: Factorized Tabular Foundation Models

Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across rows and columns, while faster models collapse each row into a fixed embedding and lose this fine-grained state. We introduce Factorized Tabular Foundation Models (FTFM), which preserve cell representations while factorizing their computation through one contextual latent per row and one per column. Cells no longer attend to one another directly; instead, each cell is updated through the interaction of its row and column latents, reducing repeated attention while retaining feature-specific information. FTFM achieves competitive predictive performance in tabular classification on major benchmarks such as TabArena and BeyondArena.

Josif Grabocka

University of Technology Nuremberg

Tabular Foundation ModelsTabular ICLTabular ClassificationTransformer Attention
AskPDF
Apr 29, 2026

Efficient and Interpretable Transformer for Counterfactual Fairness

The growing reliance of machine learning models in high-stakes, highly regulated domains such as finance and insurance has created a growing tension between predictive performance, interpretability, and regulatory fairness requirements. In these settings, models are expected not only to deliver reliable predictions but also to provide transparent decision rationales and comply with strict fairness requirements. Attention-based transformers offer powerful mechanisms for modeling complex data relationships as demonstrated in various language tasks, yet their attention mechanisms alone do not ensure counterfactually fair predictions, even when combined with fairness-aware techniques. To address these limitations, we propose the Feature Correlation Transformer (FCorrTransformer), an attention-light architecture tailored for tabular data. In this design, the attention matrix admits a direct statistical interpretation as pairwise feature dependencies, enhancing both interpretability and efficiency. Leveraging this structure, we introduce Counterfactual Attention Regularization (CAR), a framework that enforces group-invariant fair representations of sensitive features at the attention level, promoting counterfactually fair predictions without relying on explicit causal assumptions. Empirical evaluations on imbalanced classification and regression benchmarks demonstrate that FCorrTransformer combined with CAR achieves strong counterfactual fairness while maintaining competitive predictive performance and substantially reducing model complexity compared with standard transformer-based baselines. Overall, this work bridges a critical gap between fairness theory and machine learning models, offering a practical framework for responsible AI in regulatory-sensitive domains.

Panyi Dong, Zhiyu Quan

Acturial and Risk Mangament Sciences, University of Illinois Urbana-Champaign, Champaign, Illinois, USA.

Transformer InterpretabilityTransformerAlgorithmic FairnessCounterfactual Fairness
AskPDF
Apr 8, 2025

Deep Fair Learning: Task-Aware Fair Representations via Joint Distance-Covariance Regularization

Ensuring fairness is essential as machine learning increasingly informs consequential decisions. However, many fairness-aware methods focus on the outputs of individual predictors, without directly controlling sensitive information retained in the underlying representations. We propose Deep Fair Learning (DFL), which combines distance covariance regularization with predictive loss to jointly learn representations and downstream predictors, promoting fairness at both levels while preserving task-relevant information. Its marginal and class-conditional formulations target independence and separation, respectively. Under suitable regularity conditions, we establish non-asymptotic joint excess-risk rates and convergence of the learned representation up to natural invariances. We further derive fairness-inheritance bounds linking representation-level dependence to downstream disparities over suitable predictor classes, extending fairness guarantees beyond the jointly trained predictor. Experiments on tabular, text, and image benchmarks show that DFL achieves lower fairness gaps than competing methods in many evaluated settings while maintaining competitive predictive accuracy, with fairness gains largely preserved after downstream retraining.

Enze Shi, Yiqun Xiao, Linglong Kong +1

Department of Mathematical and Statistical Science, University of Alberta.

Representation LearningAlgorithmic FairnessPredictive Representation Learning
AskPDF