cs.LGOct 30, 2026

ElTab: Elastically Looped Tabular Transformers

Authors: Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian Kersting

Organizations: Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany

Abstract

Tabular transformer models such as TabPFN achieve strong predictive performance but incur substantial inference costs, making them difficult to deploy in resource-constrained settings. While looped transformers reduce parameter count through weight sharing, they can lose significant predictive power when stopped early. To address this challenge, we introduce the Elastically Looped Tabular Transformer (ElTab), a single model trained to operate across different computational budgets through supervision at randomly sampled early exits. Across 30 OpenML classification datasets, a single ElTab model matches six depth-specific looped models while cutting training cost by more than half, and matches a 12-layer TabPFN-style baseline with 25% fewer inference FLOPs. These results show that elastic-depth training yields a single model that adapts to the deployment budget at hand, offering a favorable trade-off among predictive performance, storage, and computational cost.

Explore similar work

CardsList
  1. LoopICL: Looping a single transformer block to solve tabular tasks

    Sep 28, 2026Amir Rezaei Balef, Katharina EggenspergerTabular Foundation ModelsCost-Aware Inference

  2. Tabular Foundation Models Are Effectively Shallow

    Oct 9, 2026Irene Cannistraci, Julia E. VogtTransformer PruningTabular Foundation Models

  3. Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

    May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation ModelsTransformer Interpretability