cs.LGNov 3, 2026

Towards an Optimal Control Foundation of Early Exits - Turnpikes and ResNets

Authors: Jens Püttschneider, Simon Heilig, Asja Fischer, Timm Faulwasser

Organizations: Institute of Control Systems, Hamburg University of Technology, Hamburg, Germany · Faculty of Computer Science, Ruhr University Bochum, Bochum, Germany

Abstract

The feed-forward propagation through neural networks is crucial for training and inference. To trade off efficiency and accuracy, it has been proposed to consider intermediate outputs of hidden states as data-dependent regularization during training and, ultimately, as early-exits during inference. In this work, we analyze the joint optimization of loss functions computed for the output layer and intermediate layers using optimal control theory. We prove that ResNets trained in this joint fashion exhibit the so-called turnpike phenomenon, this implies that the number of layers required to reach a prescribed loss level is bounded from above independently of the overall network depth. This induces a similarity property across the internal representations of early-exiting networks of varying depths. We connect the turnpike to the overthinking phenomenon, and provide theoretical insights into the success of early-exit criteria derived from softmax probabilities, specifically entropy and max-probability. In a range of experiments, we demonstrate the existence and effect of turnpikes and the resulting similarity in the early-exit performance for ResNets of varying depths.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Early learning in large residual networks

    Oct 22, 2026Vladimir Narovlansky, Boris HaninNeural Network Training DynamicsResidual Learning

  2. When Do Early-Exit Networks Generalize? A PAC-Bayesian Theory of Adaptive Depth

    Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuNeural Network GeneralizationPAC-Bayesian Generalization Bounds

  3. Deep learning from a control vantage point

    Oct 9, 2026Álvaro Rodríguez Abella, João Pedro Silvestre, Bahman Gharesifard +1Transformer