Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

Yael S. Elmatad

cs.LGMar 8, 2026

Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

Authors: Yael S. Elmatad

Abstract

Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.

Figures & tables

Model	Mean MAE	Median MAE	Mode MAE	Median RMSE
Full model ( $\sigma=3$ )	36.54	35.94	38.50	71.83
XGBoost (tuned)	40.31	40.31	40.31	73.15
Naive mean	52.72	52.72	52.72	88.16
Riegel formula	49.74	49.74	49.74	94.71

Table 1: Main results

Parameter	Value
Transformer width	512
Inner feed forward size	2,048
Dropout after attention/FFN	0.12
Max races to consider	30
Attention heads	8
Decoder-only Transformer layers	6

Table 2: Training configuration for the fixed

\sigma=3

sweep. RunTime is a decoder-style causal Transformer.

Configuration	Mean MAE	Median MAE	Mode MAE	Wall clock (h)
Time token ablation	37.24	36.58	39.28	107
Time token ablation, shuffled sequence	37.23	36.65	39.73	145
Full model ( $\sigma=3$ )	36.54	35.94	38.50	60

Table 3: Ablation MAE statistics (

\sigma=3

) and wall-clock costs to best MAE.

Figure 1: Top: MAE vs. history length. Middle: normalized MAE showing the ablation gap relative to the full model. Bottom: MAE vs. week gap between the penultimate and target race, highlighting temporal staleness.

Figure 2: Q–Q plot showing distributions across various percentiles from $\sigma=0$ (cross entropy) to $\sigma=35$ (wide Gaussian).

Figure 3: Quantile calibration emphasizing where predicted densities concentrate. Note that small $\sigma$ values concentrate predictions at the extremes, suggesting model overconfidence, whereas large $\sigma$ values appear underconfident, with larger probability mass in the middle.

Figure 4: Calibration plots highlighting how different smoothing choices affect residual drift across percentiles.

	KS	MAE			RMSE
Model	Statistic	Mean	Median	Mode	Mean	Median	Mode
Adaptive $\sigma$	0.0264	36.74	36.22	38.34	70.89	72.12	77.82
Adaptive $\sigma$ , best KS	0.0045	37.34	36.36	38.88	71.05	71.99	76.50
Cross-entropy $\sigma=0$	0.0293	37.00	36.32	38.74	71.02	72.14	77.65
Cross-entropy $\sigma=0$ , best KS	0.0107	37.44	36.43	39.04	71.15	72.06	77.62
$\sigma=1$	0.0263	36.83	36.22	38.98	70.89	72.01	82.29

Table 4: Calibration sweep leaderboard (seconds). Lower is better; values were collected on short, cost-aware sweeps and should only be compared internally.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: YE (short-career) runner trajectory with predicted PDFs illustrating tight uncertainty as the history grows.

Figure 6: RY (median-career) runner trajectory showing moderate calibration and widening distributions for longer distances.

Figure 7: RW (long-career) runner trajectory that exhibits sustained variability and broad PDFs even late in the history.

Panel of Figure 8 — Figure 8: Activation and attention diagnostics for a high-uncertainty prediction. Top: attention weights for the final transformer block. Middle: per-layer contribution magnitudes. Bottom: pace distribution histogram showing the predicted softmax.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 540 papers

TabPFN-3.5: Technical Report
Sep 15, 2026Benjamin Jäger, Nick Erickson, Léo Grinsztajn +44Tabular Foundation Models Inference-Time Optimization
Towards Pretraining Text Encoders for TabPFN
Jun 3, 2026Mustafa Tajjar, Alexander Pfefferle, Lennart Purucker +1Tabular Foundation Models Fine-Tuning
Department of Computer Science, University of Freiburg, Freiburg, Germany · ELLIS Institute Tübingen, Tübingen, Germany · Prior Labs, Freiburg, Germany.
LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
Jun 3, 2026Yuanrui Wang, Xingxuan Zhang, Han Yu +7Tabular Foundation Models Tabular Representation Learning
Stable AI · Tsinghua University

Sep 15, 2026J/K move · Enter open · S save

TabPFN-3.5: Technical Report

Benjamin Jäger, Nick Erickson, Léo Grinsztajn, Felix Birkel+43

Tabular Foundation Models Inference-Time Optimization Tabular ICL Tabular Prediction

Open paper AskPDF arXiv

We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faster than TabPFN-3 while keeping most of the accuracy gains. In addition, we upgrade TabPFN-3.5-Plus, expanding our multimodal capabilities with advanced text and date handling alongside proprietary inference optimizations. Finally, we release a new version of our Thinking mode, TabPFN-3.5-Thinking, which scales inference-time computation to push the state of the art further. It benefits from our stronger base model and from inference-time improvements that make it up to 12x faster than TabPFN-3-Thinking.

Abstract

Transformer

Explore similar work

TabPFN-3.5: Technical Report

Towards Pretraining Text Encoders for TabPFN

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models