cs.AINov 5, 2026

AlterNet: Adaptive-Depth Mixture-of-Experts with Learned Halting for Elastic Inference

Authors: Alexander Kastius

Organizations: SAP Palo Alto, CA 94304

Abstract

Most LLM architectures have fixed compute demands determined at training time. We propose a novel architecture that combines the Mixture-of-Experts architecture with shallow experts and a novel dynamic-depth system to make the extent of a forward pass dynamically adjustable. Our approach lets the model automatically determine the compute needed for a given task, paired with a system to scale it down after training without retraining when the execution environment demands faster or cheaper inference. To achieve this, we introduce an iterative computation mechanism, sequence-level depth-based routing steered by a halting mechanism within the experts, a depth encoding that lets the model track its progress in the chain, and a regularization mechanism that incentivizes the model to use allocated compute efficiently. We present initial benchmarks and discuss future research directions.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Speculating Experts Accelerates Inference for Mixture-of-Experts

    Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts InferenceLLM Inference Acceleration

  2. BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference

    Jun 8, 2026Yuhua Zhou, Shaoqi Yu, Shichao Weng +4LLM PruningCost-Aware Inference