Authors: Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian Kersting
Organizations: Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany · German Research Center for AI (DFKI)
Transformer-based tabular foundation models such as TabPFN achieve strong predictive performance but incur quadratic computational cost with context length. On the other hand, subquadratic SSM-based alternatives such as Hydra trade away accuracy for efficiency. To balance both, we introduce Tydra, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers. Across 30 OpenML datasets, Tydra reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance. Tydra also outperforms an approximately ten-times-larger Hydra model while providing faster inference. The results indicate that hybrid architectures are a promising direction for tabular foundation models.
Figures & tables
Figure 1: Tydra is fast on tabular data without sacrificing predictive performance. Inference speed and predictive performance of Tydra compared to TabPFN and Hydra on OpenML datasets. Color encodes a combined accuracy–speed score (F2, weighted toward speed). Shaded ellipses span one standard deviation around each model’s mean. The star marks the ideal corner: fastest and most accurate. Hydra 16M is fast but trails substantially in predictive performance, while TabPFN and Hydra 160M are accurate but slow; Tydra achieves predictive performance close to TabPFN and Hydra 160M at over 2× their inference speed.
Figure 2: Tydra’s Hybrid Architecture. Feature vectors are projected into row-level embeddings, with label embeddings added for training rows. The resulting sequence passes through four Hydra–Transformer encoder pairs (eight layers total), each ordered as Hydra followed by attention. An MLP decoder produces class logits for the test rows, which are converted into class probabilities by softmax.
Figure 3: The Tydra Family of Architectures. Architectures of the evaluated Hydra and Tydra models. Blocks are ordered from the input layer at the bottom to the output layer at the top. T denotes a Transformer attention layer, and H denotes a Hydra layer.
Figure 4: Tydra scales far better than TabPFN and Hydra 160M, staying close to Hydra 16M’s speed well into the large-context regime. Inference speed of Tydra, Hydra 160M, and TabPFN relative to Hydra 16M on the synthetic benchmark. Tydra is competitive with or faster than Hydra 16M up to 212 samples, then falls increasingly behind at larger scales, reaching roughly a third of Hydra 16M’s speed at 215 – still far ahead of TabPFN and Hydra 160M. Hydra 160M is substantially slower than Hydra 16M across the entire range, while TabPFN degrades from being competitive at small sizes (up to 1.06× faster) to roughly to roughly the same speed as Hydra 160M at 215 .
Figure 5: Tydra matches TabPFN’s accuracy at substantially lower inference cost. Inference speedup relative to TabPFN (top) and change in predictive performance ( ΔAUROC ) relative to TabPFN (bottom), for Tydra 4HT and two Hydra baselines (16M, 160M), across 30 OpenML datasets. Datasets are sorted by Tydra’s inference speedup.
Figure 6: The Tydra family consistently speeds up inference over TabPFN, with balanced hybrid variants also preserving predictive performance. Inference speedup (top) and AUC-ROC change (bottom) relative to TabPFN across 30 OpenML datasets, for six Tydra configurations that vary the ratio and placement of attention (T) and Hydra (H) layers. All achieve similar inference speedups over TabPFN (up to 1.4 × ), but the one that concentrate Hydra layers together, such as T{6H}T, show substantially larger accuracy drops on individual datasets (e.g., -15.7 points on dataset 50) than more balanced interleavings. Datasets are sorted by Tydra {4HT}’s inference speedup.
Method
AUROC (%)
Time (ms)
TabPFN
88.90
27.12
Tydra {4 HT}
88.61 (-0.29)
19.99 (-7.13)
Tydra {4 TH }
88.41 (-0.49)
23.41 (-3.71)
Tydra H{6T}H
88.73 (-0.17)
21.98 (-5.14)
Tydra {2H}{4T}{2H}
88.71 (-0.19)
22.57 (-4.55)
Tydra T{6H}T
87.66 (-1.24)
23.59 (-3.63)
Table 1: Hybrid Model Comparison vs. TabPFN. Mean test AUC-ROC and mean inference time (per batch, in milliseconds) across all OpenML datasets. 5/6 hybrid variants achieve faster inference without sacrificing much accuracy.
Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.
Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany
Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.
We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faster than TabPFN-3 while keeping most of the accuracy gains. In addition, we upgrade TabPFN-3.5-Plus, expanding our multimodal capabilities with advanced text and date handling alongside proprietary inference optimizations. Finally, we release a new version of our Thinking mode, TabPFN-3.5-Thinking, which scales inference-time computation to push the state of the art further. It benefits from our stronger base model and from inference-time improvements that make it up to 12x faster than TabPFN-3-Thinking.