Adapting Linear-Time Architectures for Tabular In-Context Learning

David Schnurr; Felix Sarnthein; Thomas Hofmann; Imanol Schlag

cs.LGSep 28, 2026

Adapting Linear-Time Architectures for Tabular In-Context Learning

Authors: David Schnurr, Felix Sarnthein, Thomas Hofmann, Imanol Schlag

Organizations: ETH Zürich · ELLIS Institute Tübingen · MPI-IS · ETH AI Center

Abstract

Tabular foundation models achieve strong performance by conditioning on labelled examples in context, but softmax attention limits their use on large datasets. Existing linear-time alternatives, however, are mostly causal, and their potential for tabular in-context learning (ICL) remains underexplored. To address this, we (1) revisit causal training setups, (2) compare linear sequence mixers, and (3) investigate their ICL generalisation beyond the pretraining context length. First, we show that the best training setup for causal models resembles next-token prediction. Then, perhaps surprisingly, the most promising linear sequence mixer is causal: DeltaNet outperforms even non-causal linear attention. However, it degrades beyond $2$ - $4\times$ the pretraining context length, and existing mitigation strategies such as bidirectionality defer the problem at best. A hidden-state oracle shows that this is not a capacity problem. Instead, our analysis points to an instability in the recurrent state, which drifts in deeper layers of causal models. Since DeltaNet's learned write rates overfit to the pretraining regime, we modulate them with a time-dependent decay schedule intervention to stabilise length generalisation. Finally, re-introducing non-causality by reading out from the final state allows us to closely match a controlled softmax attention baseline on OpenML-CC18 and TabArena.

Figures & tables

Figure 1: Training strategies given by label embedding and target formulation. Combined-label variants represent features and labels in one token; interleaved variants represent them separately. Single-target variants predict each query independently from the context; multi-target variants process target positions sequentially. Combined-label single-target is the standard (Tab)PFN setup.

		OpenML-CC18 ∗		TabArena ∗
Training setup	Len.	ROC-AUC $\uparrow$	ACC $\uparrow$	ROC-AUC $\uparrow$	ACC $\uparrow$
Interleaved Multi-Target	$2\times$	0.8837	0.8039	0.8038	0.8398
Combined Single-Target	$1\times$	0.8824	0.8020	0.8004	0.8370
Interleaved Single-Target	$2\times$	0.8800	0.7980	0.8005	0.8350
Combined Multi-Target	$1\times$	0.8793	0.7952	0.7978	0.8338

Table 1: Aggregated training setup performance on OpenML-CC18 ∗ and TabArena ∗ , the preprocessed benchmark variants from Section 3.1 . Values are averaged across splits, datasets, and the four model families supporting all four setups: Gated Linear Attention, DeltaNet, Gated DeltaNet, and Mamba-2. Len. is the sequence length relative to combined-label models. Bold marks the best setup per metric. The vertical bars denote insignificance classes per benchmark metric; setups sharing a bar are not significantly different under the paired Wilcoxon signed-rank test with Holm correction at

\alpha=5\%

	OpenML-CC18 ∗		TabArena ∗
	ROC-AUC $\uparrow$	ACC $\uparrow$	ROC-AUC $\uparrow$	ACC $\uparrow$
TabPFN v2.5	0.8993	0.8298	0.8383	0.8642
TabICLv2	0.8971	0.8286	0.8375	0.8631
TabFlex	0.8859	0.8093	0.8089	0.8463
CatBoost	0.8833	0.8062	0.8231	0.8562
Softmax Non-Causal	0.8900	0.8166	0.8172	0.8506

Table 2: Model comparison on the preprocessed OpenML-CC18 ∗ and TabArena ∗ benchmark variants (Section 3.1 ). The upper block reports full-scale released baselines, not matched in parameter count or pretraining budget, while the lower block reports our controlled backbones, all using Combined Single-Target. Values are averaged over 5-fold stratified CV splits and datasets. Bold and underline mark the best and second-best controlled models per metric. The vertical bars denote pairwise insignificance classes per metric; models sharing a bar are not significantly different.

Figure 2: Length generalisation beyond the 1k pretraining length. We report normalised ROC-AUC averaged over 500 synthetic datasets with 95% confidence intervals. (a) Length Generalisation Gap: Causal models underperform their non-causal counterparts. DeltaNet is initially competitive but degrades more strongly, while subsampled DeltaNet remains stable; the oracle is evaluated only at 128k and shown as a horizontal reference. (b) Mitigation Strategies: None of the applicable strategies eliminates DeltaNet’s long-context degradation (Section 4.2 ).

Figure 3: Effective write rates

\beta_{t}^{\mathrm{eff}}

over sequence positions across network depth. We report means over 50 datasets, including the 95% CIs. The dashed vertical line marks the 1000-row pretraining limit, with additional layer plots provided in Figure 9 of the appendix. Standard DeltaNet learns a decreasing write-rate profile, but compared to the (not compute-matched) long-sequence pretrained variant, it lowers the write rates less aggressively at very deep layers. The decay variant is our proposed solution, which imposes the decay explicitly within the recurrence relation.

	OpenML-CC18 ∗		TabArena ∗
	ROC-AUC $\uparrow$	ACC $\uparrow$	ROC-AUC $\uparrow$	ACC $\uparrow$
Softmax Non-Causal	0.8900	0.8166	0.8172	0.8506
Delta-FS + Decay	0.8884	0.8145	0.8139	0.8468
Delta + Decay (Int-MT)	0.8873	0.8120	0.8141	0.8459
Delta + Decay	0.8864	0.8091	0.8100	0.8438
Delta-FS	0.8874	0.8137	0.8022	0.8352

Table 3: Benchmark performance of the DeltaNet variants and the non-causal attention baselines (protocol of Section 3.1 ). FS denotes final-state readout, Decay denotes the time-dependent decay intervention, and Int-MT denotes the Interleaved Multi-Target configuration (Appendix B.2 ). All other models use Combined Single-Target. Bold and underline mark the best and second-best models per metric. Significance bars mark non-significant differences between models within one group.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Recurrence	Readout
Linear Attention	$\bm{S}_{t}=\bm{S}_{t-1}+\bm{v}_{t}\bm{k}_{t}^{\top}$	$\bm{o}_{t}=\bm{S}_{t}\bm{q}_{t}$
+ Kernel	$\bm{S}_{t}=\bm{S}_{t-1}+\bm{v}_{t}\phi(\bm{k}_{t})^{\top}$	$\bm{o}_{t}=\bm{S}_{t}\phi(\bm{q}_{t})$
Linear Attn. Gated	$\bm{S}_{t}=\bm{S}_{t-1}\mathrm{Diag}(\alpha_{t})+\bm{v}_{t}\bm{k}_{t}^{\top}$	$\bm{o}_{t}=\bm{S}_{t}\bm{q}_{t}$
Delta Rule	$\bm{S}_{t}=\bm{S}_{t-1}\big(\mathbf{I}-\beta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\big)+\beta_{t}\bm{v}_{t}\bm{k}_{t}^{\top}$	$\bm{o}_{t}=\bm{S}_{t}\bm{q}_{t}$
Delta Rule Gated	$\bm{S}_{t}=\bm{S}_{t-1}\big(\alpha_{t}(\mathbf{I}-\beta_{t}\bm{k}_{t}\bm{k}_{t}^{\top})\big)+\beta_{t}\bm{v}_{t}\bm{k}_{t}^{\top}$	$\bm{o}_{t}=\bm{S}_{t}\bm{q}_{t}$
Mamba-2	$\bm{S}_{t}=\gamma_{t}\bm{S}_{t-1}+\bm{v}_{t}\bm{k}_{t}^{\top}$	$\bm{o}_{t}=\bm{S}_{t}\bm{q}_{t}$

Appendix

Table 4: Unified recurrent-memory view of the linear-time sequence mixers considered in this work. The table is adapted from Yang et al. (2024b) .

\bm{S}_{t}

denotes the recurrent memory state, and

\bm{o}_{t}

is the unnormalised memory readout. We omit normalisation terms, output projections, and implementation-specific details for clarity.

Model	Parameters
Softmax Non-Causal	12,507,210
Delta	12,489,162
Delta Gated	12,504,426
Linear Non-Causal	12,473,482
Linear Gated	12,567,882
Linear Causal	12,473,802

Appendix

Table 5: Parameter counts for the models in the controlled comparisons of Tables 2 and 3 . Counts include the encoders, backbone, and prediction head. Shared weights are counted once. Delta-FS, Delta + Decay, and Delta-FS + Decay have the same parameter count as Delta.

Hyperparameter	Value
Synthetic prior	TabPFN v1 prior
Rows per dataset	1000
Context/query split	uniformly sampled, min. 64 context rows
Input features	uniformly sampled from 2-20
Classes	up to 10
Training budget	200 epochs, 4000 minibatches/epoch

Appendix

Table 6: Shared pretraining hyperparameters for the size-matched real-world comparisons.

Figure 4: Model ranks as a function of dataset size in TabArena. The ranks for each model are smoothed in

\mathrm{log10}

-space with a Gaussian kernel of

\sigma_{\mathrm{log10}}=0.1

. At moderate sizes, DeltaNet variants are competitive with softmax attention. With increasing dataset size

n_{c}

, non-causal softmax and linear attention outperform the causal models.

Dataset	Rows	DeltaNet	Hidden-State Oracle	Softmax Non-Causal	Linear Non-Causal
GiveMeSomeCredit	150,000	0.8446	0.8615	0.8559	0.8546
Airline satisfaction	129,880	0.9314	0.9931	0.9765	0.9486
SDSS17	78,053	0.9851	0.9938	0.9915	0.9889
APSFailure	76,000	0.9828	0.9862	0.9845	0.9816
Diabetes130US	71,518	0.5587	0.6165	0.5887	0.5418
Mean		0.8605	0.8902	0.8794	0.8631

Appendix

Table 7: Hidden-state oracle comparison on the five largest TabArena classification datasets, reporting ROC-AUC averaged over stratified 5-fold cross-validation splits. Bold marks the best value per dataset and best mean.

Figure 5: Comparison of layer-local ridge model with DeltaNet variants and non-causal linear attention over increasing sequence lengths. Curves show globally normalised ROC-AUC with 95% confidence intervals. The layer-local ridge model is pretrained from scratch using the same protocol, but replaces the recurrent state update with a closed-form ridge regression solve.

Figure 6: Length generalisation of DeltaNet models pretrained with longer context lengths. The reference model is pretrained with at most 1000 rows, while the other models sample the row number from a log-uniform distribution between 200 and the maximum shown in the legend. Longer-context pretraining improves long-context performance but mainly shifts degradation to larger sequences. The comparison is not compute-matched, since all models see the same number of datasets but a different total number of rows. A compute-matched version is in Section 4 of the main paper.

Figure 7: Length generalisation for GLA beyond the 1k pretraining length. We report normalised ROC-AUC averaged over 500 synthetic datasets with 95% confidence intervals. (a) Length Generalisation Gap: Compared to other models, GLA exhibits strong degradation starting at

4\times

the pretraining context length. (b) Mitigation Attempts: State passing and mimetic initialisation mitigate but do not eliminate long-sequence degradation, while bidirectionality is the strongest mitigation for GLA at very long contexts. Final-state readout is strongest up to around

16\times

the pretraining length but degrades substantially beyond it. State weaving slightly improves performance in the pretraining context range, but shows no significant improvement in addressing degradation.

Figure 8: Length generalisation performance of standard and Frobenius-normalised causal and non-causal linear attention variants (top panel), and Frobenius norm evolution for the first head, together with directional drift averaged over heads and repetitions, across sequence lengths, for the unnormalised (second and third panels) and Frobenius-normalised (bottom two panels) models. Constraining the state magnitude does not remove the degradation of the causal model.

Panel of Figure 9 — Figure 9: Effective DeltaNet write rate $\beta_{t}$ across sequence positions and its relation to length generalisation. The top panel shows globally normalised ROC-AUC as the number of in-context samples increases. The lower panels show effective write rates $\beta_{t}$ across sequence positions for selected layers of DeltaNet, a variant pretrained on longer sequences, and the write-rate decay intervention. For the long-sequence pretraining variants, dataset sizes are sampled log-uniformly between 200 and 64k rows. Write-rate values are averaged across 50 datasets, with means and 95% confidence intervals visualised. Standard DeltaNet learns a decreasing write-rate profile in deeper layers, where long-sequence pretraining further lowers write rates at later positions. The write-rate decay explicitly decays the effective write rate towards zero across the sequence length in every layer.

Figure 10: Length generalisation of standard DeltaNet (left) and final-state readout DeltaNet (right) under the decay intervention and long-sequence pretraining; eval only applies the decay to the reference checkpoint at evaluation time. Long-sequence pretrained models are not compute-matched and serve only as analysis references.

Panel of Figure 11 — Figure 11: Contribution to the final DeltaNet hidden state across lookback distances. The top row compares DeltaNet with its long-sequence pretrained variant, and the bottom row compares the corresponding final-state readout variants. Panels (a,b) show a lower bound on the contribution $C_{s}=\beta_{s}\prod_{j>s}(1-\beta_{j})$ of each source position $s$ to the final hidden state, averaged across 50 synthetic datasets and for each layer. Panel (c) shows the log-ratio $\log_{10}(C_{\mathrm{long}}/C_{\mathrm{standard}})$ where red indicates a larger contribution for the long-sequence pretrained model and blue indicates a larger contribution for the standard model.

		Mean		Adjusted $p$ (ROC-AUC above diagonal, ACC below)
#	Model	AUC	ACC	1	2	3	4	5	6	7	8	9	10	11
1	Softmax Non-Causal	0.8900	0.8166	–	1.00	.60	1.00	1.00	1.00	.43	.17	.25	.25	.05
2	Delta-FS + Decay	0.8884	0.8145	1.00	–	1.00	1.00	1.00	1.00	1.00	.21	.27	.27	.77
3	Delta (Int-MT)	0.8880	0.8098	.51	1.00	–	1.00	1.00	1.00	1.00	1.00	.99	.99	1.00
4	Delta-FS	0.8874	0.8137	1.00	1.00	.04	–	1.00	1.00	1.00	1.00	.99	.99	1.00
5	Delta-FS (Subs. 3K)	0.8874	0.8137	1.00	1.00	.04	1.00	–	1.00	1.00	1.00	.99	.99	1.00

Appendix

Table 8: Full comparison on the 30 OpenML-CC18 ∗ datasets. Models are ordered by ROC-AUC. FS denotes final-state readout, Decay denotes the time-dependent decay intervention, Subs. denotes the subsampled and ensembled configuration, and Int-MT denotes the Interleaved Multi-Target configuration (Appendix B.2 ). All other models use Combined Single-Target. Bold and underline mark the best and second-best mean per metric, while bold

p

-values mark adjusted

p\leq 0.05

indicating significant differences. The pairwise significance comparison uses two-sided Wilcoxon signed-rank tests with Holm adjustment like in Table 3 . The 1K sample cap of OpenML-CC18 ∗ makes each Subs. 3K variant identical to its full-context counterpart.

		Mean		Adjusted $p$ (ROC-AUC above diagonal, ACC below)
#	Model	AUC	ACC	1	2	3	4	5	6	7	8	9	10	11
1	Softmax Non-Causal	0.8172	0.8506	–	1.00	.79	.03	$<$ .01	$<$ .01	$<$ .001	$<$ .001	$<$ .001	.08	$<$ .001
2	Delta + Decay (Int-MT)	0.8141	0.8459	.74	–	1.00	.74	$<$ .01	$<$ .001	$<$ .001	$<$ .001	$<$ .01	.07	$<$ .001
3	Delta-FS + Decay	0.8139	0.8468	1.00	1.00	–	.01	$<$ .01	$<$ .001	$<$ .001	$<$ .001	$<$ .001	1.00	$<$ .001
4	Delta-FS (Subs. 3K)	0.8123	0.8456	1.00	1.00	.60	–	.55	.01	.01	$<$ .001	.43	1.00	$<$ .001
5	Delta + Decay	0.8100	0.8438	.38	.60	.12	1.00	–	1.00	.11	.01	1.00	1.00	$<$ .001

Appendix

Table 9: Full comparison on the 38 TabArena ∗ datasets. Here we use the identical setup from Table 8 .

Model	1k	8k	32k	128k
DeltaNet, two-axis	0.7808	0.7813	0.7592	0.7059
DeltaNet, single-axis	0.7791	0.7806	0.7719	0.7569
Causal Linear Attention, two-axis	0.7777	0.7863	0.7852	0.7812
Causal Linear Attention, single-axis	0.7775	0.7805	0.7781	0.7723

Appendix

Table 10: Length generalisation under two-axis versus single-axis (row-wise) processing, reporting mean ROC-AUC over the 500 synthetic evaluation datasets by context length. Two-axis processing increases long-context degradation substantially for DeltaNet, while causal linear attention remains comparatively stable. The two-axis models are not parameter-matched to the row-wise models.

Figure 12: Runtime scaling with the number of in-context samples. Softmax attention scales quadratically during context processing and linearly during query processing with a cached context. Linear attention and DeltaNet scale linearly during context processing and remain constant during query processing from a fixed-size recurrent state.

Figure 13: Context-state size across increasing context lengths. Softmax attention stores a KV cache that grows linearly with the number of context samples, whereas linear-time recurrent models store a fixed-size hidden state.

Tabular ICL

Tabular in-context learning (ICL) uses examples in a table to adapt pretrained models to prediction tasks without task-specific weight updates.

See all 108 papers

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
Aug 2, 2026Rasa Hosseinzadeh, Alex Labach, Zexin Xue +3Tabular Foundation Models Tabular ICL
Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada
Closing the Context Gap: Activation Alignment for Tabular In-Context Learning
Oct 5, 2026Yoel ZeldesTabular Foundation Models Tabular ICL
Independent Researcher
Tydra: An Efficient Hybrid Model for Tabular Data
Aug 21, 2026Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2Hybrid Transformer-State Space Model Architectures Tabular Foundation Models
Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany +1

Aug 2, 2026J/K move · Enter open · S save

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han+2

Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada

Tabular Foundation Models Tabular ICL Tabular Prediction

Open paper AskPDF arXiv

Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.

Abstract

Tabular ICL

Explore similar work

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Tydra: An Efficient Hybrid Model for Tabular Data