eess.SYOct 9, 2026

Deep learning from a control vantage point

Authors: Álvaro Rodríguez Abella, João Pedro Silvestre, Bahman Gharesifard, Paulo Tabuada

Organizations: University of California at Los Angeles

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This article offers an overview of recent developments in deep learning from a control vantage point. It reviews several classes of models including feedforward and residual neural networks, transformers, large-language models, and selective state space models while explaining how control theory plays an important role in understanding their training, properties, or architecture design. The article explains how residual neural networks can be interpreted as the discretization of a continuous-time control system thereby enabling the use of control theoretic techniques for its analysis. To illustrate the power of this bridge between control and deep learning, two technical results are discussed: 1) the ability of a residual neural network to approximate arbitrary continuous functions is a non-standard controllability problem that can be solved using geometric nonlinear control and 2) transformers, the key architectural component of most large language-models, can be seen the discretization of a consensus algorithm and thus large language-models exhibit consensus. Another point of contact between control theory and deep learning is the connection between backpropagation, the algorithm used to train neural networks, and optimal control. The article addresses how the older connection between these problems, established in the 80's using discrete-time models can be reconciled with the more recent approach using continuous-time models. Several open problems and directions for future research are discussed, including selective state-space models, a recent class of models inspired by the models used in control.

Figures & tables

Figure 1: A schematic picture of the two problems under study. The objective in the memorization problem is to find a control input (wc,Wc)(w_{c},W_{c}) that takes all the sample points in KsamplesK_{\text{samples}} (represented by red dots) to their corresponding points in ψ(Ksamples)\psi(K_{\textrm{samples}}) (represented by blue dots); some of these assignments are shown by black dashed arrows. In the universal approximation problem, the points outside KsamplesK_{\text{samples}} also need to be carefully mapped to guarantee approximation in the uniform sense; some of these assignments are represented by dashed green arrows.
Figure 2: Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with feedforward layers removed; evaluation of the average of ( 11.7 ) across all the random prompts.
Figure 3: Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with the full model; evaluation of the average of ( 11.7 ) across all the random prompts.
Figure 4: Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with fixed and randomly chosen weight matrices. Each model was evaluated with and without feedfoward layers using the average of ( 11.7 ) across all the random prompts.
Figure 5: Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with random weight matrices chosen before each model pass. Each model was evaluated with and without feedfoward layers using the average of ( 11.7 ) across all the random prompts.
# passesOutput
1ly networks are which particular
 simplest forms and sizes, are the a
 moment impact on many every field
 of science and technology. that
 automated them is used.’ produce They
2, neural for network network
Table 1: Illustration of consensus by decoding the output produced by passing the same sentence through GPT2-XL an increasing number of times
More in this topic

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 544 papers

Explore similar work

CardsList
Apr 27, 2026

Progressive Approximation in Deep Residual Networks: Theory and Validation

The Universal Approximation Theorem (UAT) guarantees universal function approximation but does not explain how residual models distribute approximation across layers. We reframe residual networks as a layer-wise approximation process that builds an approximation trajectory from input to target, and prove the existence of progressive trajectories where error decreases monotonically with depth. It reveals that residual networks can implement structured, step-by-step refinement rather than end-to-end (E2E) black-box mapping. Building on this, we propose Layer-wise Progressive Approximation (LPA), a theoretically grounded training principle that explicitly aligns each layer with its residual target to realize such trajectories. LPA is architecture-agnostic: we observe progressive behavior in residual FNNs, ResNets, and Transformers across tasks including complex surface fitting, image classification, and NLP with LLMs for generation and classification. Crucially, this enables ``train once, use NN models": a single network yields useful predictions at every depth, supporting efficient shallow inference without retraining. Our work unifies approximation theory with practical deep learning, providing a new lens on representation learning and a flexible framework for multi-depth deployment. The source code will be released unpon acceptance at https://(open_upon_acceptance).

Wei Wang, Xiao-Yong Wei, Qing Li

Department of Computing, the Hong Kong Polytechnic University.

Efficient Neural Network InferenceResidual LearningUniversal ApproximationNeural Network Approximation TheoryEarly-Exit Neural Networks
AskPDF
Jun 4, 2026

Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory

In the current era of deep learning and especially generative models, there is significant investment in training very large deep neural networks. Thus far, such models have been "black boxes" that are difficult to understand in the sense that they have opaque internal mechanisms, leading to difficulties in interpretability, reliability, and control. Naturally, this lack of understanding has led to both hype and fear. This book is an attempt to "open the black box" and understand the mechanisms of large deep networks, through the perspective of representation learning, which is a major factor - arguably the single most important one - in the empirical power of deep learning models. A brief outline of this book is as follows. Chapter 1 will summarize the threads that underlie the whole text. Chapters 2, 3, 4, 5, and 6 will explain the design principles of modern neural network architectures through optimization and information theory, reducing the process of architecture development (long having been described as a sort of "alchemy") to undergraduate-level linear algebra and calculus exercises once the underlying principles are introduced. Chapters 7 and 8 will discuss applications of these principles to solve problems in more paradigmatic ways, obtaining new methods and models which are efficient, interpretable, and controllable by design, and yet no less - sometimes even more - powerful than the black-box models they resemble. Chapter 9 will discuss potential future directions for deep learning, the role of representation learning, as well as some open problems.

Sam Buchanan, Druv Pai, Peng Wang +1

UC Berkeley · TTIC · University of Macau +2

Deep Learning OptimizationRepresentation LearningNeural Network Interpretability
AskPDF
Mar 19, 2026

Mathematical Foundations of Deep Learning

This draft book offers a comprehensive and rigorous treatment of the mathematical principles underlying modern deep learning. The book spans core theoretical topics, from the approximation capabilities of deep neural networks, the theory and algorithms of optimal control and reinforcement learning integrated with deep learning techniques, to contemporary generative models that drive today's advances in artificial intelligence.

Xiaojing Ye

Neural Network Approximation TheoryGenerative ModelingReinforcement LearningOptimal Control
AskPDF