Benchmarking Attention for Tabular Foundation Models

Maximilian Schambach; Clemens Biehl; Sam Thelin

cs.LGSep 25, 2026

Benchmarking Attention for Tabular Foundation Models

Authors: Maximilian Schambach, Clemens Biehl, Sam Thelin

Organizations: SAP SE, Germany

Abstract

Tabular in-context learners such as TabPFN, Mitra, or ConTextTab rely on alternating row and column attention over 2D sequences of latent embeddings. These attention patterns differ markedly from the one-dimensional case in language models: row attention involves longer sequences while column attention operates on much shorter ones, and the strided memory layout of tabular data makes producing contiguous tensors costly. Moreover, the hidden dimensions used in current models are small compared to recent language models. Yet efficient attention has been studied mostly for one-dimensional sequences, leaving the two-dimensional tabular setting unexplored. To this end, we create a reproducible benchmarking setup and study the unique characteristics of tabular attention across several backends -- Torch SDPA (efficient and cuDNN), FlashAttention-2/3/4, and the inference-only backends vLLM and SageAttention -- measuring forward and backward throughput across realistic tabular shapes on three GPU generations (A100, H100, B200). We find that the optimal backend choice differs between column and row attention and varies across hardware as well as model specifics: While the FlashAttention implementations tailored for each GPU generation perform overall best, they are at times outperformed by CuDNN in the case of column attention at longer sequences with cross-over points depending on the head dimension. Among inference-only backends, SageAttention performs well for row attention and large sequences beyond 16,k rows. Our reproducible benchmark lays the foundation for future improvements to table-native attention. The self-contained benchmarking and evaluation code is openly available at: https://github.com/SAP-samples/tabular-attention-benchmark

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

$R$	$C$	Layout A	Layout B	Layout A	Layout B	Ratio A/B
		Time ( $\mathrm{\SIUnitSymbolMicro s}$ )		Bandwidth ( $\mathrm{GB}\text{/}\mathrm{s}$ )
$1024$	$20$	65.0	64.9	1290.6	1293.1	1.002
$2048$	$50$	310.3	310.8	1351.6	1349.4	0.998
$4096$	$50$	619.5	619.7	1354.1	1353.6	1.000
$8192$	$100$	2479.5	2481.8	1353.3	1352.0	0.999
$8192$	$16$	396.5	397.6	1354.0	1350.1	0.997

Appendix

Table 1 : Microbenchmark of .contiguous() for the row-first layout (Layout A)

(B,R,C,H,D)

vs. the column-first layout (Layout B)

(B,C,R,H,D)

with

B{=}2

H{=}8

D{=}64

. The transpose(1,2) call makes the non-contiguous dimension contiguous. Times in

\mathrm{\SIUnitSymbolMicro s}

; bandwidth in

\mathrm{GB}\text{/}\mathrm{s}

. All ratios are within

0.3

% of unity, confirming that layout choice does not affect copy cost.

$N$	Total size ( $\mathrm{MB}$ )	Time ( $\mathrm{\SIUnitSymbolMicro s}$ )	Bandwidth ( $\mathrm{GB}\text{/}\mathrm{s}$ )
$32$	9	24.1	783
$64$	18	33.2	1138
$128$	36	56.8	1330
$256$	72	115.4	1308
$512$	144	220.5	1370
$1024$	288	422.7	1429

Appendix

Table 2 : Measured .contiguous() bandwidth for three copies (Q, K, V) on the H100 at the row attention shapes used in the benchmark. Configuration:

B_{\text{eff}}{=}64,H{=}12,N,D{=}64

in bfloat16 .

Explore similar work

TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
Jun 5, 2026Si-Yang Liu, Han-Jia YeTabular Foundation Models Tabular Prior-Data Fitted Network
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Attention Quantization for Tabular Foundation Models
Sep 14, 2026Jonas M. Kübler, Benjamin Jäger, Klemens Flöge +2Tabular Foundation Models Key-Value Cache Quantization
Prior Labs
TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
Aug 2, 2026Rasa Hosseinzadeh, Alex Labach, Zexin Xue +3Tabular Foundation Models In-Context Learning
Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada

Jun 5, 2026, cs.LGJ/K move · Enter open · S save

TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention

Si-Yang Liu, Han-Jia Ye

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.

Tabular Foundation Models Tabular Prior-Data Fitted Network In-Context Learning

Open paper AskPDF arXiv

Tabular foundation models, exemplified by TabPFN, perform prediction via in-context learning, inferring test labels directly from labeled training examples. They have demonstrated competitive performance, particularly on small-to-medium datasets. However, recent tabular foundation models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment. In this work, we revisit the original TabPFN design and show that a lightweight row-wise attention-only backbone can remain highly competitive with two simple enhancements: a gated attention stabilization mechanism and a small set of learnable register tokens that provide global context and improve pretraining quality. The resulting model, TabSwift, supports both classification and regression, and is competitive with stronger tabular foundation models (e.g., TabPFN v2 and TabICL) while being more efficient at inference. For latency-sensitive serving, we further introduce an adaptive layer-wise early-exit mechanism that dynamically adjusts inference depth per sample. Overall, TabSwift enables efficient and anytime tabular in-context learning for practical deployments.

Abstract

Explore similar work

TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention

Attention Quantization for Tabular Foundation Models

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction