Tabular foundation models achieve strong performance by conditioning on labelled examples in context, but softmax attention limits their use on large datasets. Existing linear-time alternatives, however, are mostly causal, and their potential for tabular in-context learning (ICL) remains underexplored. To address this, we (1) revisit causal training setups, (2) compare linear sequence mixers, and (3) investigate their ICL generalisation beyond the pretraining context length. First, we show that the best training setup for causal models resembles next-token prediction. Then, perhaps surprisingly, the most promising linear sequence mixer is causal: DeltaNet outperforms even non-causal linear attention. However, it degrades beyond 2-4× the pretraining context length, and existing mitigation strategies such as bidirectionality defer the problem at best. A hidden-state oracle shows that this is not a capacity problem. Instead, our analysis points to an instability in the recurrent state, which drifts in deeper layers of causal models. Since DeltaNet's learned write rates overfit to the pretraining regime, we modulate them with a time-dependent decay schedule intervention to stabilise length generalisation. Finally, re-introducing non-causality by reading out from the final state allows us to closely match a controlled softmax attention baseline on OpenML-CC18 and TabArena.
Figures & tables
Figure 1: Training strategies given by label embedding and target formulation. Combined-label variants represent features and labels in one token; interleaved variants represent them separately. Single-target variants predict each query independently from the context; multi-target variants process target positions sequentially. Combined-label single-target is the standard (Tab)PFN setup.
OpenML-CC18 ∗
TabArena ∗
Training setup
Len.
ROC-AUC ↑
ACC ↑
ROC-AUC ↑
ACC ↑
Interleaved Multi-Target
2×
0.8837
0.8039
0.8038
0.8398
Combined Single-Target
1×
0.8824
0.8020
0.8004
0.8370
Interleaved Single-Target
2×
0.8800
0.7980
0.8005
0.8350
Combined Multi-Target
1×
0.8793
0.7952
0.7978
0.8338
Table 1: Aggregated training setup performance on OpenML-CC18 ∗ and TabArena ∗ , the preprocessed benchmark variants from Section 3.1 . Values are averaged across splits, datasets, and the four model families supporting all four setups: Gated Linear Attention, DeltaNet, Gated DeltaNet, and Mamba-2. Len. is the sequence length relative to combined-label models. Bold marks the best setup per metric. The vertical bars denote insignificance classes per benchmark metric; setups sharing a bar are not significantly different under the paired Wilcoxon signed-rank test with Holm correction at α=5% .
OpenML-CC18 ∗
TabArena ∗
ROC-AUC ↑
ACC ↑
ROC-AUC ↑
ACC ↑
TabPFN v2.5
0.8993
0.8298
0.8383
0.8642
TabICLv2
0.8971
0.8286
0.8375
0.8631
TabFlex
0.8859
0.8093
0.8089
0.8463
CatBoost
0.8833
0.8062
0.8231
0.8562
Softmax Non-Causal
0.8900
0.8166
0.8172
0.8506
Table 2: Model comparison on the preprocessed OpenML-CC18 ∗ and TabArena ∗ benchmark variants (Section 3.1 ). The upper block reports full-scale released baselines, not matched in parameter count or pretraining budget, while the lower block reports our controlled backbones, all using Combined Single-Target. Values are averaged over 5-fold stratified CV splits and datasets. Bold and underline mark the best and second-best controlled models per metric. The vertical bars denote pairwise insignificance classes per metric; models sharing a bar are not significantly different.
Figure 2: Length generalisation beyond the 1k pretraining length. We report normalised ROC-AUC averaged over 500 synthetic datasets with 95% confidence intervals. (a) Length Generalisation Gap: Causal models underperform their non-causal counterparts. DeltaNet is initially competitive but degrades more strongly, while subsampled DeltaNet remains stable; the oracle is evaluated only at 128k and shown as a horizontal reference. (b) Mitigation Strategies: None of the applicable strategies eliminates DeltaNet’s long-context degradation (Section 4.2 ).
Figure 3: Effective write rates βteff over sequence positions across network depth. We report means over 50 datasets, including the 95% CIs. The dashed vertical line marks the 1000-row pretraining limit, with additional layer plots provided in Figure 9 of the appendix. Standard DeltaNet learns a decreasing write-rate profile, but compared to the (not compute-matched) long-sequence pretrained variant, it lowers the write rates less aggressively at very deep layers. The decay variant is our proposed solution, which imposes the decay explicitly within the recurrence relation.
OpenML-CC18 ∗
TabArena ∗
ROC-AUC ↑
ACC ↑
ROC-AUC ↑
ACC ↑
Softmax Non-Causal
0.8900
0.8166
0.8172
0.8506
Delta-FS + Decay
0.8884
0.8145
0.8139
0.8468
Delta + Decay (Int-MT)
0.8873
0.8120
0.8141
0.8459
Delta + Decay
0.8864
0.8091
0.8100
0.8438
Delta-FS
0.8874
0.8137
0.8022
0.8352
Table 3: Benchmark performance of the DeltaNet variants and the non-causal attention baselines (protocol of Section 3.1 ). FS denotes final-state readout, Decay denotes the time-dependent decay intervention, and Int-MT denotes the Interleaved Multi-Target configuration (Appendix B.2 ). All other models use Combined Single-Target. Bold and underline mark the best and second-best models per metric. Significance bars mark non-significant differences between models within one group.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Recurrence
Readout
Linear Attention
St=St−1+vtkt⊤
ot=Stqt
+ Kernel
St=St−1+vtϕ(kt)⊤
ot=Stϕ(qt)
Linear Attn. Gated
St=St−1Diag(αt)+vtkt⊤
ot=Stqt
Delta Rule
St=St−1(I−βtktkt⊤)+βtvtkt⊤
ot=Stqt
Delta Rule Gated
St=St−1(αt(I−βtktkt⊤))+βtvtkt⊤
ot=Stqt
Mamba-2
St=γtSt−1+vtkt⊤
ot=Stqt
Appendix
Table 4: Unified recurrent-memory view of the linear-time sequence mixers considered in this work. The table is adapted from Yang et al. (2024b) . St denotes the recurrent memory state, and ot is the unnormalised memory readout. We omit normalisation terms, output projections, and implementation-specific details for clarity.
Model
Parameters
Softmax Non-Causal
12,507,210
Delta
12,489,162
Delta Gated
12,504,426
Linear Non-Causal
12,473,482
Linear Gated
12,567,882
Linear Causal
12,473,802
Appendix
Table 5: Parameter counts for the models in the controlled comparisons of Tables 2 and 3 . Counts include the encoders, backbone, and prediction head. Shared weights are counted once. Delta-FS, Delta + Decay, and Delta-FS + Decay have the same parameter count as Delta.
Hyperparameter
Value
Synthetic prior
TabPFN v1 prior
Rows per dataset
1000
Context/query split
uniformly sampled, min. 64 context rows
Input features
uniformly sampled from 2-20
Classes
up to 10
Training budget
200 epochs, 4000 minibatches/epoch
Appendix
Table 6: Shared pretraining hyperparameters for the size-matched real-world comparisons.
Figure 4: Model ranks as a function of dataset size in TabArena. The ranks for each model are smoothed in log10 -space with a Gaussian kernel of σlog10=0.1 . At moderate sizes, DeltaNet variants are competitive with softmax attention. With increasing dataset size nc , non-causal softmax and linear attention outperform the causal models.
Dataset
Rows
DeltaNet
Hidden-State Oracle
Softmax Non-Causal
Linear Non-Causal
GiveMeSomeCredit
150,000
0.8446
0.8615
0.8559
0.8546
Airline satisfaction
129,880
0.9314
0.9931
0.9765
0.9486
SDSS17
78,053
0.9851
0.9938
0.9915
0.9889
APSFailure
76,000
0.9828
0.9862
0.9845
0.9816
Diabetes130US
71,518
0.5587
0.6165
0.5887
0.5418
Mean
0.8605
0.8902
0.8794
0.8631
Appendix
Table 7: Hidden-state oracle comparison on the five largest TabArena classification datasets, reporting ROC-AUC averaged over stratified 5-fold cross-validation splits. Bold marks the best value per dataset and best mean.
Figure 5: Comparison of layer-local ridge model with DeltaNet variants and non-causal linear attention over increasing sequence lengths. Curves show globally normalised ROC-AUC with 95% confidence intervals. The layer-local ridge model is pretrained from scratch using the same protocol, but replaces the recurrent state update with a closed-form ridge regression solve.
Figure 6: Length generalisation of DeltaNet models pretrained with longer context lengths. The reference model is pretrained with at most 1000 rows, while the other models sample the row number from a log-uniform distribution between 200 and the maximum shown in the legend. Longer-context pretraining improves long-context performance but mainly shifts degradation to larger sequences. The comparison is not compute-matched, since all models see the same number of datasets but a different total number of rows. A compute-matched version is in Section 4 of the main paper.
Figure 7: Length generalisation for GLA beyond the 1k pretraining length. We report normalised ROC-AUC averaged over 500 synthetic datasets with 95% confidence intervals. (a) Length Generalisation Gap: Compared to other models, GLA exhibits strong degradation starting at 4× the pretraining context length. (b) Mitigation Attempts: State passing and mimetic initialisation mitigate but do not eliminate long-sequence degradation, while bidirectionality is the strongest mitigation for GLA at very long contexts. Final-state readout is strongest up to around 16× the pretraining length but degrades substantially beyond it. State weaving slightly improves performance in the pretraining context range, but shows no significant improvement in addressing degradation.
Figure 8: Length generalisation performance of standard and Frobenius-normalised causal and non-causal linear attention variants (top panel), and Frobenius norm evolution for the first head, together with directional drift averaged over heads and repetitions, across sequence lengths, for the unnormalised (second and third panels) and Frobenius-normalised (bottom two panels) models. Constraining the state magnitude does not remove the degradation of the causal model.
Figure 9: Effective DeltaNet write rate βt across sequence positions and its relation to length generalisation. The top panel shows globally normalised ROC-AUC as the number of in-context samples increases. The lower panels show effective write rates βt across sequence positions for selected layers of DeltaNet, a variant pretrained on longer sequences, and the write-rate decay intervention. For the long-sequence pretraining variants, dataset sizes are sampled log-uniformly between 200 and 64k rows. Write-rate values are averaged across 50 datasets, with means and 95% confidence intervals visualised. Standard DeltaNet learns a decreasing write-rate profile in deeper layers, where long-sequence pretraining further lowers write rates at later positions. The write-rate decay explicitly decays the effective write rate towards zero across the sequence length in every layer.
Figure 10: Length generalisation of standard DeltaNet (left) and final-state readout DeltaNet (right) under the decay intervention and long-sequence pretraining; eval only applies the decay to the reference checkpoint at evaluation time. Long-sequence pretrained models are not compute-matched and serve only as analysis references.
Figure 11: Contribution to the final DeltaNet hidden state across lookback distances. The top row compares DeltaNet with its long-sequence pretrained variant, and the bottom row compares the corresponding final-state readout variants. Panels (a,b) show a lower bound on the contribution Cs=βs∏j>s(1−βj) of each source position s to the final hidden state, averaged across 50 synthetic datasets and for each layer. Panel (c) shows the log-ratio log10(Clong/Cstandard) where red indicates a larger contribution for the long-sequence pretrained model and blue indicates a larger contribution for the standard model.
Mean
Adjusted p (ROC-AUC above diagonal, ACC below)
#
Model
AUC
ACC
1
2
3
4
5
6
7
8
9
10
11
1
Softmax Non-Causal
0.8900
0.8166
–
1.00
.60
1.00
1.00
1.00
.43
.17
.25
.25
.05
2
Delta-FS + Decay
0.8884
0.8145
1.00
–
1.00
1.00
1.00
1.00
1.00
.21
.27
.27
.77
3
Delta (Int-MT)
0.8880
0.8098
.51
1.00
–
1.00
1.00
1.00
1.00
1.00
.99
.99
1.00
4
Delta-FS
0.8874
0.8137
1.00
1.00
.04
–
1.00
1.00
1.00
1.00
.99
.99
1.00
5
Delta-FS (Subs. 3K)
0.8874
0.8137
1.00
1.00
.04
1.00
–
1.00
1.00
1.00
.99
.99
1.00
Appendix
Table 8: Full comparison on the 30 OpenML-CC18 ∗ datasets. Models are ordered by ROC-AUC. FS denotes final-state readout, Decay denotes the time-dependent decay intervention, Subs. denotes the subsampled and ensembled configuration, and Int-MT denotes the Interleaved Multi-Target configuration (Appendix B.2 ). All other models use Combined Single-Target. Bold and underline mark the best and second-best mean per metric, while bold p -values mark adjusted p≤0.05 indicating significant differences. The pairwise significance comparison uses two-sided Wilcoxon signed-rank tests with Holm adjustment like in Table 3 . The 1K sample cap of OpenML-CC18 ∗ makes each Subs. 3K variant identical to its full-context counterpart.
Mean
Adjusted p (ROC-AUC above diagonal, ACC below)
#
Model
AUC
ACC
1
2
3
4
5
6
7
8
9
10
11
1
Softmax Non-Causal
0.8172
0.8506
–
1.00
.79
.03
< .01
< .01
< .001
< .001
< .001
.08
< .001
2
Delta + Decay (Int-MT)
0.8141
0.8459
.74
–
1.00
.74
< .01
< .001
< .001
< .001
< .01
.07
< .001
3
Delta-FS + Decay
0.8139
0.8468
1.00
1.00
–
.01
< .01
< .001
< .001
< .001
< .001
1.00
< .001
4
Delta-FS (Subs. 3K)
0.8123
0.8456
1.00
1.00
.60
–
.55
.01
.01
< .001
.43
1.00
< .001
5
Delta + Decay
0.8100
0.8438
.38
.60
.12
1.00
–
1.00
.11
.01
1.00
1.00
< .001
Appendix
Table 9: Full comparison on the 38 TabArena ∗ datasets. Here we use the identical setup from Table 8 .
Model
1k
8k
32k
128k
DeltaNet, two-axis
0.7808
0.7813
0.7592
0.7059
DeltaNet, single-axis
0.7791
0.7806
0.7719
0.7569
Causal Linear Attention, two-axis
0.7777
0.7863
0.7852
0.7812
Causal Linear Attention, single-axis
0.7775
0.7805
0.7781
0.7723
Appendix
Table 10: Length generalisation under two-axis versus single-axis (row-wise) processing, reporting mean ROC-AUC over the 500 synthetic evaluation datasets by context length. Two-axis processing increases long-context degradation substantially for DeltaNet, while causal linear attention remains comparatively stable. The two-axis models are not parameter-matched to the row-wise models.
Figure 12: Runtime scaling with the number of in-context samples. Softmax attention scales quadratically during context processing and linearly during query processing with a cached context. Linear attention and DeltaNet scale linearly during context processing and remain constant during query processing from a fixed-size recurrent state.
Figure 13: Context-state size across increasing context lengths. Softmax attention stores a KV cache that grows linearly with the number of context samples, whereas linear-time recurrent models store a fixed-size hidden state.
Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.
Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance. Instead of discarding context, we propose activation alignment, a method that leverages the full context to teach a model how to behave when seeing only a subset. This is achieved by training a lightweight linear transformation on synthetic unlabeled data to map the intermediate activations of a data-constrained "student" (using partial context) toward those of a full-context "teacher" (using all data). Training the aligner requires no GPU and converges in seconds to minutes on commodity hardware. We evaluate on 38 classification datasets from the TabArena benchmark using the leading two tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student yields broad, statistically significant improvements over the unaligned baseline for both models. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage. The method provides a practical, low-overhead approach to achieving the inference speed of compact contexts while closing a significant fraction of the performance gap to the full-context teacher.
Transformer-based tabular foundation models such as TabPFN achieve strong predictive performance but incur quadratic computational cost with context length. On the other hand, subquadratic SSM-based alternatives such as Hydra trade away accuracy for efficiency. To balance both, we introduce Tydra, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers. Across 30 OpenML datasets, Tydra reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance. Tydra also outperforms an approximately ten-times-larger Hydra model while providing faster inference. The results indicate that hybrid architectures are a promising direction for tabular foundation models.
Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2
Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany +1