Authors: Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian Kersting
Organizations: Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany · German Research Center for AI (DFKI)
Transformer-based tabular foundation models such as TabPFN achieve strong predictive performance but incur quadratic computational cost with context length. On the other hand, subquadratic SSM-based alternatives such as Hydra trade away accuracy for efficiency. To balance both, we introduce Tydra, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers. Across 30 OpenML datasets, Tydra reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance. Tydra also outperforms an approximately ten-times-larger Hydra model while providing faster inference. The results indicate that hybrid architectures are a promising direction for tabular foundation models.
Figures & tables
Figure 1: Tydra is fast on tabular data without sacrificing predictive performance. Inference speed and predictive performance of Tydra compared to TabPFN and Hydra on OpenML datasets. Color encodes a combined accuracy–speed score (F2, weighted toward speed). Shaded ellipses span one standard deviation around each model’s mean. The star marks the ideal corner: fastest and most accurate. Hydra 16M is fast but trails substantially in predictive performance, while TabPFN and Hydra 160M are accurate but slow; Tydra achieves predictive performance close to TabPFN and Hydra 160M at over 2× their inference speed.
Figure 2: Tydra’s Hybrid Architecture. Feature vectors are projected into row-level embeddings, with label embeddings added for training rows. The resulting sequence passes through four Hydra–Transformer encoder pairs (eight layers total), each ordered as Hydra followed by attention. An MLP decoder produces class logits for the test rows, which are converted into class probabilities by softmax.
Figure 3: The Tydra Family of Architectures. Architectures of the evaluated Hydra and Tydra models. Blocks are ordered from the input layer at the bottom to the output layer at the top. T denotes a Transformer attention layer, and H denotes a Hydra layer.
Figure 4: Tydra scales far better than TabPFN and Hydra 160M, staying close to Hydra 16M’s speed well into the large-context regime. Inference speed of Tydra, Hydra 160M, and TabPFN relative to Hydra 16M on the synthetic benchmark. Tydra is competitive with or faster than Hydra 16M up to 212 samples, then falls increasingly behind at larger scales, reaching roughly a third of Hydra 16M’s speed at 215 – still far ahead of TabPFN and Hydra 160M. Hydra 160M is substantially slower than Hydra 16M across the entire range, while TabPFN degrades from being competitive at small sizes (up to 1.06× faster) to roughly to roughly the same speed as Hydra 160M at 215 .
Figure 5: Tydra matches TabPFN’s accuracy at substantially lower inference cost. Inference speedup relative to TabPFN (top) and change in predictive performance ( ΔAUROC ) relative to TabPFN (bottom), for Tydra 4HT and two Hydra baselines (16M, 160M), across 30 OpenML datasets. Datasets are sorted by Tydra’s inference speedup.
Figure 6: The Tydra family consistently speeds up inference over TabPFN, with balanced hybrid variants also preserving predictive performance. Inference speedup (top) and AUC-ROC change (bottom) relative to TabPFN across 30 OpenML datasets, for six Tydra configurations that vary the ratio and placement of attention (T) and Hydra (H) layers. All achieve similar inference speedups over TabPFN (up to 1.4 × ), but the one that concentrate Hydra layers together, such as T{6H}T, show substantially larger accuracy drops on individual datasets (e.g., -15.7 points on dataset 50) than more balanced interleavings. Datasets are sorted by Tydra {4HT}’s inference speedup.
Method
AUROC (%)
Time (ms)
TabPFN
88.90
27.12
Tydra {4 HT}
88.61 (-0.29)
19.99 (-7.13)
Tydra {4 TH }
88.41 (-0.49)
23.41 (-3.71)
Tydra H{6T}H
88.73 (-0.17)
21.98 (-5.14)
Tydra {2H}{4T}{2H}
88.71 (-0.19)
22.57 (-4.55)
Tydra T{6H}T
87.66 (-1.24)
23.59 (-3.63)
Table 1: Hybrid Model Comparison vs. TabPFN. Mean test AUC-ROC and mean inference time (per batch, in milliseconds) across all OpenML datasets. 5/6 hybrid variants achieve faster inference without sacrificing much accuracy.
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany