Deep Learning Optimization

Ask about this topic

Overview

It studies how parameter updates, learning-rate and batch schedules, regularization, and model structure shape neural-network training. Common approaches range from SGD and AdamW to adaptive preconditioners such as Shampoo, curvature-aware and sharpness-aware methods, and matrix or spectral optimizers such as Muon; theory examines convergence, stability, and geometry, while systems work reduces memory, communication, and step time. Evaluation spans vision and tabular models, language-model pretraining, diffusion, and continual learning, measuring accuracy and loss alongside time to target and memory use. Optimizer rankings depend on architecture, batch size, training horizon, and compute budget, leaving a tension between per-step progress, wall-clock efficiency, generalization, and reliable tuning.

Latest papers

CardsList

208 papers39 papers in the last four weeks, up 457% on the four weeks before. 0.2% of all new papers.

Apr 25, 2026

Learn&Drop: Fast Learning of CNNs based on Layer Dropping

This paper proposes a new method to improve the training efficiency of deep convolutional neural networks. During training, the method evaluates scores to measure how much each layer's parameters change and whether the layer will continue learning or not. Based on these scores, the network is scaled down such that the number of parameters to be learned is reduced, yielding a speed up in training. Unlike state-of-the-art methods that try to compress the network to be used in the inference phase or to limit the number of operations performed in the backpropagation phase, the proposed method is novel in that it focuses on reducing the number of operations performed by the network in the forward propagation during training. The proposed training strategy has been validated on two widely used architecture families: VGG and ResNet. Experiments on MNIST, CIFAR-10 and Imagenette show that, with the proposed method, the training time of the models is more than halved without significantly impacting accuracy. The FLOPs reduction in the forward propagation during training ranges from 17.83% for VGG-11 to 83.74% for ResNet-152. These results demonstrate the effectiveness of the proposed technique in speeding up learning of CNNs. The technique will be especially useful in applications where fine-tuning or online training of convolutional models is required, for instance because data arrive sequentially.

Giorgio Cruciata, Luca Cruciata, Liliana Lo Presti +2

Engineering Department, University of Palermo, V.le delle Scienze, Ed. 6, Palermo, 90128, Italy. · Computer Vision Lab, Delft University of Technology, Van Mourik Broekmanweg 6,2026 Delft, 2628 XE, The Netherlands.

Deep Learning OptimizationConvolutional Neural NetworksNeural Network Pruning
AskPDF
Apr 25, 2026

A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning

This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks.

Yaru Liu, Michael K. Ng, Yiqi Gu

School of Mathematical Sciences, University of Electronic Science and Technology of China, Sichuan 611731, China · Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

Deep Learning OptimizationNonconvex Optimization
AskPDF
Apr 17, 2026

Training Time Prediction for Mixed Precision-based Distributed Training

Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.

Minchul Kang, Changyong Shin, Jinwoo Jeong +5

Korea University · KT Corporation

Deep Learning OptimizationDistributed Training
AskPDF
Apr 16, 2026

Natural gradient descent with momentum

We consider the problem of approximating a function by an element of a nonlinear manifold which admits a differentiable parametrization, typical examples being neural networks with differentiable activation functions or tensor networks. Natural gradient descent (NGD) for the optimization of a loss function can be seen as a preconditioned gradient descent where updates in the parameter space are driven by a functional perspective. In a spirit similar to Newton's method, a NGD step uses, instead of the Hessian, the Gram matrix of the generating system of the tangent space to the approximation manifold at the current iterate, with respect to a suitable metric. This corresponds to a locally optimal update in function space, following a projected gradient onto the tangent space to the manifold. Still, both gradient and natural gradient descent methods get stuck in local minima. Furthermore, when the model class is a nonlinear manifold or the loss function is not ideally conditioned (e.g., the KL-divergence for density estimation, or a norm of the residual of a partial differential equation in physics informed learning), even the natural gradient might yield non-optimal directions at each step. This work introduces a natural version of classical inertial dynamic methods like Heavy-Ball or Nesterov and show how it can improve the learning process when working with nonlinear model classes.

Anthony Nouy, Agustín Somacal

Centrale Nantes, Nantes Universit´e, Laboratoire de Math´ematiques Jean Leray UMR CNRS 6629

Deep Learning OptimizationRiemannian OptimizationMomentum MethodsGradient DescentNatural Gradient MethodsNeural Network Optimization
AskPDF
Apr 16, 2026

Benchmarking Optimizers for MLPs in Tabular Deep Learning

MLP is a heavily used backbone in modern deep learning (DL) architectures for supervised learning on tabular data, and AdamW is the go-to optimizer used to train tabular DL models. Unlike architecture design, however, the choice of optimizer for tabular DL has not been examined systematically, despite new optimizers showing promise in other domains. To fill this gap, we benchmark 15 optimizers on 17 tabular datasets for training MLP-based models in the standard supervised learning setting under a shared experiment protocol. Our main finding is that the Muon optimizer consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers, if the associated training efficiency overhead is affordable. Additionally, we find exponential moving average of model weights to be a simple yet effective technique that improves AdamW on vanilla MLPs, though its effect is less consistent across model variants.

Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov +1

α Yandex · β HSE University

Multilayer PerceptronsDeep Learning OptimizationNeural Network OptimizationMuon OptimizerTabular ML
AskPDF
Apr 16, 2026

CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization

Lion optimizer is a popular learning-based optimization algorithm in machine learning, which shows impressive performance in training many deep learning models. Although convergence property of the Lion optimizer has been studied, its generalization analysis is still missing. To fill this gap, we study generalization property of the Lion via algorithmic stability based on the mathematical induction. Specifically, we prove that the Lion has a generalization error of O(1NτT)O(\frac{1}{Nτ^T}), where NN is training sample size, and τ>0τ>0 denotes the smallest absolute value of non-zero element in gradient estimator, and TT is the total iteration number. In addition, we obtain an interesting byproduct that the SignSGD algorithm has the same generalization error as the Lion. To enhance generalization of the Lion, we design a novel efficient Cautious Lion (i.e., CLion) optimizer by cautiously using sign function. Moreover, we prove that our CLion has a lower generalization error of O(1N)O(\frac{1}{N}) than O(1NτT)O(\frac{1}{Nτ^T}) of the Lion, since the parameter ττ generally is very small. Meanwhile, we study convergence property of our CLion optimizer, and prove that our CLion has a fast convergence rate of O(dT1/4)O(\frac{\sqrt{d}}{T^{1/4}}) under ℓ1\ell_1-norm of gradient for nonconvex stochastic optimization, where dd denotes the model dimension. Extensive numerical experiments demonstrate effectiveness of our CLion optimizer.

Feihu Huang, Guanyi Zhang, Songcan Chen

College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

Stochastic OptimizationSign-Based OptimizationNeural Network GeneralizationDeep Learning OptimizationNonconvex Stochastic Optimization
AskPDF
Apr 14, 2026

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Tong Zhang, Jiangning Zhang, Zhucun Xue +9

Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1

Deep Learning OptimizationAdaptive Gradient MethodsStochastic Gradient DescentSecond-Order OptimizationZeroth-Order Optimization
AskPDF
Apr 14, 2026

Evolution-Inspired Sample Competition for Deep Neural Network Optimization

Conventional deep network training generally optimizes all samples under a largely uniform learning paradigm, without explicitly modeling the heterogeneous competition among them. Such an oversimplified treatment can lead to several well-known issues, including bias under class imbalance, insufficient learning of hard samples, and the erroneous reinforcement of noisy samples. In this work, we present Natural Selection (NS), a novel evolution-inspired optimization method that explicitly incorporates competitive interactions into deep network training. Unlike conventional sample reweighting strategies that rely mainly on predefined heuristics or static criteria, NS estimates the competitive status of each sample in a group-wise context and uses it to adaptively regulate its training contribution. Specifically, NS first assembles multiple samples into a composite image and rescales it to the original input size for model inference. Based on the resulting predictions, a natural selection score is computed for each sample to characterize its relative competitive variation within the constructed group. These scores are then used to dynamically reweight the sample-wise loss, thereby introducing an explicit competition-driven mechanism into the optimization process. In this way, NS provides a simple yet effective means of moving beyond uniform sample treatment and enables more adaptive and balanced model optimization. Extensive experiments on 12 public datasets across four image classification tasks demonstrate the effectiveness of the proposed method. Moreover, NS is compatible with diverse network architectures and does not depend on task-specific assumptions, indicating its strong generality and practical potential. The code will be made publicly available.

Ying Zheng, Yiyi Zhang, Yi Wang +1

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China.

Instance ReweightingDeep Learning OptimizationAdaptive Loss Weighting
AskPDF
Apr 13, 2026

VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation

Deep learning models may converge to suboptimal solutions despite strong validation accuracy, masking an optimization failure we term Trajectory Deviation. This is because as training proceeds, models can abandon high generalization states for specific data sub-populations, thus discarding previously learned latent features without triggering classical overfitting signals. To address this problem we introduce VISTA, an online self-distillation framework that enforces consistency along the optimization trajectory. Using a validation-informed Marginal Coverage score, VISTA identifies expert anchors, which are earlier model states that retain specialized competence over distinct data regions. A coverage-weighted ensemble of these anchors is integrated online during training, regularizing the loss landscape and preserving mastered knowledge. When evaluated across multiple benchmarks, VISTA demonstrates improved robustness and generalization over standard training and prior self-distillation methods, while a lightweight implementation reduces storage overhead by 90% without performance loss.

Eli Corn, Daphna Weinshall

Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.

Neural Network GeneralizationDeep Learning OptimizationSelf-DistillationNeural Network Training Dynamics
AskPDF
Apr 13, 2026

A Faster Path to Continual Learning

Continual Learning (CL) aims to train neural networks on a dynamic stream of tasks without forgetting previously learned knowledge. Among optimization-based approaches, C-Flat has emerged as a promising solution due to its plug-and-play nature and its ability to encourage uniformly low-loss regions for both new and old tasks. However, C-Flat requires three additional gradient computations per iteration, imposing substantial overhead on the optimization process. In this work, we propose C-Flat Turbo, a faster yet stronger optimizer that significantly reduces the training cost. We show that the gradients associated with first-order flatness contain direction-invariant components relative to the proxy-model gradients, enabling us to skip redundant gradient computations in the perturbed ascent steps. Moreover, we observe that these flatness-promoting gradients progressively stabilize across tasks, which motivates a linear scheduling strategy with an adaptive trigger to allocate larger turbo steps for later tasks. Experiments show that C-Flat Turbo is 1.0×\times to 1.25×\times faster than C-Flat across a wide range of CL methods, while achieving comparable or even improved accuracy.

Wei Li, Hangjie Yuan, Zixiang Zhao +3

College of Computer Science, Sichuan University, China · College of Computer Science and Technology, Zhejiang University, China · Photogrammetry and Remote Sensing Lab, ETH Zürich, Switzerland +3

Continual LearningNeural Network OptimizationDeep Learning Optimization
AskPDF
Apr 10, 2026

AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.

Ioannis Tsingalis, Constantine Kotropoulos, Corentin Briat

Aristotle University of Thessaloniki, Greece · University of Applied Sciences Northwestern Switzerland, Muttenz, Switzerland

Deep Learning Optimization
AskPDF
Apr 10, 2026

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

The foundational capabilities of large language models are acquired during pretraining on internet-scale, highly heterogeneous data mixtures. In this work, we investigate an interesting geometric question regarding the converged state of pretraining: Does the model converge to a common minimizer across all data sources (e.g., \cref{fig:cwa_illustration:close}), or merely a minimizer of the summed loss (e.g., \cref{fig:cwa_illustration:distant})? We hypothesize that the geometric "closeness" of task-specific minima is intrinsically linked to downstream generalization. We reveal that standard optimizers (e.g., AdamW) often converge to points where task-specific minima are distant from each other. To address this, we propose the Nexus optimizer, which encourages the closeness of these minima by maximizing gradient similarity during optimization. Experiments across models ranging from 130M to 3B parameters, various data mixtures and hyperparameter schedules, show that Nexus significantly boosts downstream performance, despite achieving the same pretraining loss (see \cref{fig:demo:benchmark}). Notably, on the 3B model, Nexus reduces the out-of-distribution loss by 0.012 and yields up to a 15.0% accuracy improvement on complex reasoning tasks (e.g., GSM8k). This finding challenges the reliance on pretraining loss as the sole proxy for model evaluation and demonstrates the importance of implicit biases in unlocking downstream generalization.

Huanran Chen, Huaqing Zhang, Xiao Li +3

Tsinghua University · ByteDance Seed

Language Model PretrainingNeural Network GeneralizationDeep Learning OptimizationGradient AlignmentOOD Generalization
AskPDF
Apr 10, 2026

A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay

Adaptive optimizers like AdamW apply uniform hyperparameters across all parameter groups, ignoring heterogeneous optimization dynamics across layers and modules. We address this limitation by proposing MetaAdamW - a new optimizer that integrates a self-attention mechanism to dynamically modulate per-group learning rates and weight decay. The modulation factors are produced by a lightweight Transformer encoder that operates on statistical features (gradient norms, momentum norms, correlations) extracted from each parameter group. To train the attention module, we introduce a meta-learning objective that combines gradient alignment, loss decrease, and generalization gap. A key novel contribution is the extension of homoscedastic uncertainty weighting (HUW) with task-specific priorities that directly scale the regularization terms - enabling domain knowledge to guide automatic loss balancing. Extensive experiments on five diverse tasks-time series forecasting (ETT), language modeling (WikiText-2), machine translation (Multi30k), image classification (CIFAR-10), and sentiment analysis (IMDB) - demonstrate that MetaAdamW consistently outperforms the standard AdamW baseline in terms of validation loss, accuracy, or perplexity. Depending on the task, MetaAdamW either reduces overall training time (by up to 17.11%) or improves performance (by up to 11.08%) while introducing only moderate overhead; in some cases, it can also mitigate issues of insufficient convergence caused by premature early stopping. Ablation studies validate the effectiveness of each component, including feature versions, grouping strategies, and the proposed priority-injected uncertainty weighting.

JiangBo Zhao, ZhaoXin Liu

Deep Learning OptimizationLearning to OptimizeLayer-Wise Learning Rate AdaptationAdaptive Loss Weighting
AskPDF
Apr 3, 2026

Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability

Empirically, modern deep learning training often occurs at the Edge of Stability (EoS), where the sharpness of the loss exceeds the threshold below which classical convergence analysis applies. Despite recent progress, existing theoretical explanations of EoS either rely on restrictive assumptions or focus on specific squared-loss-type objectives. In this work, we introduce and study a structural property of loss functions that we term product-stability. We show that for losses with product-stable minima, gradient descent applied to objectives of the form (x,y)↦l(xy)(x,y) \mapsto l(xy) can provably converge to the local minimum even when training in the EoS regime. This framework substantially generalizes prior results and applies to a broad class of losses, including binary cross entropy. Using bifurcation diagrams, we characterize the resulting training dynamics, explain the emergence of stable oscillations, and precisely quantify the sharpness at convergence. Together, our results offer a principled explanation for stable EoS training for a wider class of loss functions.

Eric Gan

Independent Researcher.

Deep Learning OptimizationEdge of StabilityGradient Descent DynamicsNeural Network StabilityLoss Landscape Sharpness
AskPDF
Apr 1, 2026

Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point's residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to find the minimum-norm parameter update that best satisfies all conditions at once. In practice, this pseudoinverse is approximated via a truncated singular value decomposition, retaining only the kk most significant directions. We show that Sven can be understood as a natural gradient method generalized to the overparametrized regime, recovering natural gradient descent in the underparametrized limit. We test Sven on a variety of regression and classification tasks, including small-scale language modeling with transformers, and find that it is competitive with leading baselines such as Adam, Muon, and K-FAC. We also discuss Sven's memory overhead, which presents a barrier to scaling under a naive implementation, and introduce an optimized implementation that keeps memory usage on par with standard baselines under mild restrictions on model architecture. Beyond standard machine learning benchmarks, we anticipate that Sven will find natural application in scientific computing settings where custom loss functions decompose into several conditions.

Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1

Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford +2

Deep Learning OptimizationNatural Gradient MethodsNeural Network Optimization
AskPDF
Mar 26, 2026

Second-Order, First-Class: A Composable Stack for Curvature-Aware Training

Second-order methods promise improved stability and faster convergence, yet they remain underused due to implementation overhead, tuning brittleness, and the lack of composable APIs. We introduce Somax, a composable Optax-native stack that treats curvature-aware training as a single JIT-compiled step governed by a static plan. Somax exposes first-class modules -- curvature operators, estimators, linear solvers, preconditioners, and damping policies -- behind a single step interface and composes with Optax by applying standard gradient transformations (e.g., momentum, weight decay, schedules) to the computed direction. This design makes typically hidden choices explicit and swappable. Somax separates planning from execution: it derives a static plan (including cadences) from module requirements, then runs the step through a specialized execution path that reuses intermediate results across modules. We report system-oriented ablations showing that (i) composition choices materially affect scaling behavior and time-to-accuracy, and (ii) planning reduces per-step overhead relative to unplanned composition with redundant recomputation.

Mikalai Korbit, Mario Zanon

IMT School for Advanced Studies Lucca Lucca, Italy

Curvature-Aware OptimizationDeep Learning OptimizationNeural Network OptimizationSecond-Order Optimization
AskPDF
Mar 25, 2026

Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

Deep neural networks (DNNs), particularly those using Rectified Linear Unit (ReLU) activation functions, have achieved remarkable success across diverse machine learning tasks, including image recognition, audio processing, and language modeling. Despite this success, the non-convex nature of DNN loss functions complicates optimization and limits theoretical understanding. In this paper, we highlight how recently developed convex equivalences of ReLU NNs and their connections to sparse signal processing models can address the challenges of training and understanding NNs. Recent research has uncovered several hidden convexities in the loss landscapes of certain NN architectures, notably two-layer ReLU networks and other deeper or varied architectures. This paper seeks to provide an accessible and educational overview that bridges recent advances in the mathematics of deep learning with traditional signal processing, encouraging broader signal processing applications.

Emi Zeger, Mert Pilanci

Department of Electrical Engineering, Stanford University

ReLU Neural NetworksDeep Learning OptimizationNeural Network Optimization
AskPDF
Mar 23, 2026

dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

This paper proposes dynActivation\mathrm{dynActivation}, a per-layer trainable activation defined as fi(x)=BaseAct(x)(αi−βi)+βixf_i(x) = \mathrm{BaseAct}(x)(α_i - β_i) + β_i x, where αiα_i and βiβ_i are lightweight learned scalars that interpolate between the base nonlinearity and a linear path and BaseAct(x)\mathrm{BaseAct}(x) resembles any ReLU-like function. The static and dynamic ReLU-like variants are then compared across multiple vision tasks, language modeling tasks, and ablation studies. The results suggest that dynActivation variants tend to linearize deep layers while maintaining high performance, which can improve training efficiency by up to +54%+54\% over ReLU. On CIFAR-10, dynActivation(Mish) improves over static Mish by up to +14.02%+14.02\% on AttentionCNN with an average improvment by +6.00%+6.00\%, with a 24%24\% convergence-AUC reduction relative to Mish (2120 vs. 2785). In a 1-to-75-layer MNIST depth-scaling study, dynActivation never drops below 95%95\% test accuracy (95.395.3--99.3%99.3\%), while ReLU collapses below 80%80\% at 25 layers. Under FGSM at ε=0.08\varepsilon{=}0.08, dynActivation(Mish) incurs a 55.39%55.39\% accuracy drop versus 62.79%62.79\% for ReLU (7.40%7.40\% advantage). Transferred to language modeling, a new proposed dynActGLU-variant achieves a 10.3%10.3\% relative perplexity reduction over SwiGLU at 5620 steps (4.047 vs. 4.514), though the gap vanishes at 34300 steps.

Alois Bachmann

Ruprecht-Karls-Universität Heidelberg

Neural Network Activation FunctionsNeural Network Training DynamicsDeep Learning Optimization
AskPDF
Mar 21, 2026

Democratizing AI: A Comparative Study in Deep Learning Efficiency and Future Trends in Computational Processing

The exponential growth in data has intensified the demand for computational power to train large-scale deep learning models. However, the rapid growth in model size and complexity raises concerns about equal and fair access to computational resources, particularly under increasing energy and infrastructure constraints. GPUs have emerged as essential for accelerating such workloads. This study benchmarks four deep learning models (Conv6, VGG16, ResNet18, CycleGAN) using TensorFlow and PyTorch on Intel Xeon CPUs and NVIDIA Tesla T4 GPUs. Our experiments demonstrate that, on average, GPU training achieves speedups ranging from 11x to 246x depending on model complexity, with lightweight models (Conv6) showing the highest acceleration (246x), mid-sized models (VGG16, ResNet18) achieving 51-116x speedups, and complex generative models (CycleGAN) reaching 11x improvements compared to CPU training. Additionally, in our PyTorch vs. TensorFlow comparison, we observed that TensorFlow's kernel-fusion optimizations reduce inference latency by approximately 15%. We also analyze GPU memory usage trends and projecting requirements through 2025 using polynomial regression. Our findings highlight that while GPUs are essential for sustaining AI's growth, democratized and shared access to GPU resources is critical for enabling research innovation across institutions with limited computational budgets.

Lisan Al Amin, Md Ismail Hossain, Rupak Kumar Das +2

Cognitive Links, Maryland, USA · North South University, Bangladesh · Pennsylvania State University, USA +2

GPU AccelerationDeep Learning OptimizationEfficient Neural Network Inference
AskPDF
Mar 21, 2026

LassoFlexNet: Flexible Neural Architecture for Tabular Data

Despite their dominance in vision and language, deep neural networks often underperform relative to tree-based models on tabular data. To bridge this gap, we incorporate five key inductive biases into deep learning: robustness to irrelevant features, axis alignment, localized irregularities, feature heterogeneity, and training stability. We propose LassoFlexNet, an architecture that evaluates the linear and nonlinear marginal contribution of each input via Per-Feature Embeddings, and sparsely selects relevant variables using a Tied Group Lasso mechanism. Because these components introduce optimization challenges that destabilize standard proximal methods, we develop a Sequential Hierarchical Proximal Adaptive Gradient optimizer with exponential moving averages (EMA) to ensure stable convergence. Across 5252 datasets from three benchmarks, LassoFlexNet matches or outperforms leading tree-based models, achieving up to a 1010% relative gain, while maintaining Lasso-like interpretability. We substantiate these empirical results with ablation studies and theoretical proofs confirming the architecture's enhanced expressivity and structural breaking of undesired rotational invariance.

Kry Yik Chau Lui, Cheng Chi, Kishore Basu +1

RBC Borealis · RBC

Tabular MLTabular Representation LearningSparse Neural NetworksDeep Learning Optimization
AskPDF
Mar 18, 2026

Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

Orthogonalized-momentum optimizers such as Muon improve transformer training by approximately whitening/orthogonalizing matrix-valued momentum updates via a short polar-decomposition iteration. However, polar-factor approximations typically require multiple large matrix multiplications, and the resulting overhead can be substantial and hardware-dependent. We introduce MUD (MomentUm Decorrelation), a complementary whitening approach that replaces Muon's polar update with a triangular (Cholesky-like) whitening surrogate inspired by classical Gram--Schmidt and Gauss-Seidel ideas. We show that row-orthonormal matrices are fixed points of the MUD map, relate the inner step to symmetric Gauss-Seidel preconditioning of the Gram matrix, and prove quadratic local convergence near the fixed point. In terms of time-to-perplexity, MUD yields consistent 10-50% wall-clock improvements over tuned AdamW and Muon in time-to-perplexity, typically converging slightly slower per step than Muon but with substantially lower optimizer overhead -- relative to Muon, MUD improves peak tokens/s by roughly 1.3−2.6×1.3-2.6\times across most settings and up to nearly 3×3\times on GPT-2 large on an A100. We also demonstrate training a ESM-2 150M protein language model, where MUD matches Muon-level validation perplexity in significantly less wall-clock time.

Ben S. Southworth, Stephen Thomas

Theoretical Division, Los Alamos National Laboratory, USA. · Lehigh University, Computer Science

Deep Learning OptimizationMuon OptimizerTransformer
AskPDF
Mar 13, 2026

Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy

Optimization analyses for cross-entropy training rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective. These surrogates are reliable only inside the Taylor convergence radius of the true loss along the update direction. That radius is set not by real-line curvature alone but by the nearest complex singularity. For cross-entropy, the softmax partition function F=∑jexp⁡(zj)F=\sum_j \exp(z_j) has complex zeros -- ``ghosts of softmax'' -- that induce logarithmic singularities in the loss and cap this radius. To make this geometry usable, we derive closed-form expressions under logit linearization along the proposed update direction. In the binary case, the exact radius is ρ∗=δ2+π2/Δaρ^*=\sqrt{δ^2+ π^2}/Δ_a. In the multiclass case, we obtain the lower bound ρa=π/Δaρ_a=π/Δ_a, where Δa=max⁡kak−min⁡kakΔ_a=\max_k a_k-\min_k a_k is the spread of directional logit derivatives ak=∇zk⋅va_k=\nabla z_k\cdot v. This bound costs one Jacobian-vector product and reveals what makes a step fragile: samples that are both near a decision flip and highly sensitive to the proposed direction tighten the radius. The normalized step size r=τ/ρar=τ/ρ_a separates safe from dangerous updates. Across six tested architectures and multiple step directions, no model fails for r<1r<1, yet collapse appears once r≥1r\ge 1. Temperature scaling confirms the mechanism: normalizing by ρaρ_a shrinks the onset-threshold spread from standard deviation 0.9920.992 to 0.1640.164. A controller that enforces τ≤ρaτ\leρ_a survives learning-rate spikes up to 10,000×10{,} 000\times in our tests, where gradient clipping still collapses. Together, these results identify a geometric constraint on cross-entropy optimization that operates through Taylor convergence rather than Hessian curvature.

Piyush Sao

Oak Ridge National Laboratory, Oak Ridge, TN, USA

Deep Learning OptimizationConvergence Analysis
AskPDF
Mar 12, 2026

A Geometrically-Grounded Drive for MDL-Based Optimization in Deep Learning

This paper introduces a novel optimization framework that fundamentally integrates the Minimum Description Length (MDL) principle into the training dynamics of deep neural networks. Moving beyond its conventional role as a model selection criterion, we reformulate MDL as an active, adaptive driving force within the optimization process itself. The core of our method is a geometrically-grounded cognitive manifold whose evolution is governed by a coupled Ricci flow, enriched with a novel MDL Drive term derived from first principles. This drive, modulated by the task-loss gradient, creates a seamless harmony between data fidelity and model simplification, actively compressing the internal representation during training. We establish a comprehensive theoretical foundation, proving key properties including the monotonic decrease of description length (Theorem~\ref{thm:convergence}), a finite number of topological phase transitions via a geometric surgery protocol (Theorems~\ref{thm:surgery}, \ref{thm:ultimate_fate}), and the emergence of universal critical behavior (Theorem~\ref{thm:universality}). Furthermore, we provide a practical, computationally efficient algorithm with O(Nlog⁡N)O(N \log N) per-iteration complexity (Theorem~\ref{thm:complexity}), alongside guarantees for numerical stability (Theorem~\ref{thm:stability}) and exponential convergence under convexity assumptions (Theorem~\ref{thm:convergence_rate}). Empirical validation on synthetic regression and classification tasks confirms the theoretical predictions, demonstrating the algorithm's efficacy in achieving robust generalization and autonomous model simplification. This work provides a principled path toward more autonomous, generalizable, and interpretable AI systems by unifying geometric deep learning with information-theoretic principles.

Ming Lei, Shufan Wu, Christophe Baehr

School of Aeronautics and Astronautics, Shanghai JiaoTong University, Shanghai China · Météo-France/CNRS CNRM/GAME UMR 3589 and the Mathematical Institute of Toulouse, France

Deep Learning OptimizationMinimum Description LengthStatistical Learning Theory
AskPDF
Mar 10, 2026

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning

Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improve generalization. However, Muon implicitly assumes an isotropic optimization landscape, enforcing a uniform spectral update norm across all eigen-directions. We argue that this "egalitarian" constraint is suboptimal for Deep Neural Networks, where the curvature spectrum is known to be highly heavy-tailed and ill-conditioned. In such landscapes, Muon risks amplifying instabilities in high-curvature directions while limiting necessary progress in flat directions. In this work, we propose Mousse (Muon Optimization Utilizing Shampoo's Structural Estimation), a novel optimizer that reconciles the structural stability of spectral methods with the geometric adaptivity of second-order preconditioning. Instead of applying Newton-Schulz orthogonalization directly to the momentum matrix, Mousse operates in a whitened coordinate system induced by Kronecker-factored statistics (derived from Shampoo). Mathematically, we formulate Mousse as the solution to a spectral steepest descent problem constrained by an anisotropic trust region, where the optimal update is derived via the polar decomposition of the whitened gradient. Empirical results across language models ranging from 160M to 800M parameters demonstrate that Mousse consistently outperforms Muon, achieving around ∼\sim12% reduction in training steps with negligible computational overhead.

Yechen Zhang, Shuhao Xing, Junhao Huang +5

Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University

Deep Learning OptimizationMuon OptimizerStiefel Manifold OptimizationTrust-Region Optimization
AskPDF
Mar 9, 2026

Improving Continual Learning for Gaussian Splatting based Environments Reconstruction on Commercial Off-the-Shelf Edge Devices

Novel view synthesis (NVS) is increasingly relevant for edge robotics, where compact and incrementally updatable 3D scene models are needed for SLAM, navigation, and inspection under tight memory and latency budgets. Variational Bayesian Gaussian Splatting (VBGS) enables replay-free continual updates for the 3DGS algorithm by maintaining a probabilistic scene model, but its high-precision computations and large intermediate tensors make on-device training impractical. We present a precision-adaptive optimization framework that enables VBGS training on resource-constrained hardware without altering its variational formulation. We (i) profile VBGS to identify memory/latency hotspots, (ii) fuse memory-dominant kernels to reduce materialized intermediate tensors, and (iii) automatically assign operation-level precisions via a mixed-precision search with bounded relative error. Across the Blender, Habitat, and Replica datasets, our optimised pipeline reduces peak memory from 9.44 GB to 1.11 GB and training time from ~234 min to ~61 min on an A5000 GPU, while preserving (and in some cases improving) reconstruction quality of the state-of-the-art VBGS baseline. We also enable for the first time NVS training on a commercial embedded platform, the Jetson Orin Nano, reducing per-frame latency by 19x compared to 3DGS.

Ivan Zaino, Matteo Risso, Daniele Jahier Pagliari +3

Politecnico di Torino · VERSES

3DGS Training3D Gaussian SplattingNovel View SynthesisDeep Learning OptimizationGPU Kernel OptimizationEdge Computing
AskPDF
Mar 9, 2026

Beyond the Markovian Assumption: Robust Optimization via Fractional Weyl Integrals in Imbalanced Data

Standard Gradient Descent and its modern variants assume local, Markovian weight updates, making them highly susceptible to noise and overfitting. This limitation becomes critically severe in extremely imbalanced datasets such as financial fraud detection where dominant class gradients systematically overwrite the subtle signals of the minority class. In this paper, we introduce a novel optimization algorithm grounded in Fractional Calculus. By isolating the core memory engine of the generalized fractional derivative, the Weighted Fractional Weyl Integral, we replace the instantaneous gradient with a dynamically weighted historical sequence. This fractional memory operator acts as a natural regularizer. Empirical evaluations demonstrate that our method prevents overfitting in medical diagnostics and achieves an approximately 40 percent improvement in PR-AUC over classical optimizers in financial fraud detection, establishing a robust bridge between pure fractional topology and applied Machine Learning.

Gustavo A. Dorrego

Department of Mathematics, Universidad Nacional del Nordeste

Deep Learning OptimizationClass-Imbalanced Learning
AskPDF
Mar 9, 2026

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, \uline{when applied to the current parameters}, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, while addressing the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts.

Jianlong Chen, Zhiming Zhou

Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics

Sharpness-Aware MinimizationDeep Learning OptimizationLoss Landscape SharpnessMin-Max Optimization
AskPDF
Mar 6, 2026

Dynamic Momentum Recalibration in Online Gradient Learning

Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers.

Zhipeng Yao, Rui Yu, Guisong Chang +3

Shenyang University of Chemical Technology · University of Louisville · Northeastern University

Deep Learning OptimizationStochastic Gradient Descent
AskPDF
Mar 4, 2026

When to restart? Exploring escalating restarts on convergence

Learning rate scheduling plays a critical role in the optimization of deep neural networks, directly influencing convergence speed, stability, and generalization. While existing schedulers such as cosine annealing, cyclical learning rates, and warm restarts have shown promise, they often rely on fixed or periodic triggers that are agnostic to the training dynamics, such as stagnation or convergence behavior. In this work, we propose a simple yet effective strategy, which we call Stochastic Gradient Descent with Escalating Restarts (SGD-ER). It adaptively increases the learning rate upon convergence. Our method monitors training progress and triggers restarts when stagnation is detected, linearly escalating the learning rate to escape sharp local minima and explore flatter regions of the loss landscape. We evaluate SGD-ER across CIFAR-10, CIFAR-100, and TinyImageNet on a range of architectures including ResNet-18/34/50, VGG-16, and DenseNet-101. Compared to standard schedulers, SGD-ER improves test accuracy by 0.5-4.5%, demonstrating the benefit of convergence-aware escalating restarts for better local optima.

Ayush K. Varshney, Šarūnas Girdzijauskas, Konstantinos Vandikas +1

Ericsson Research, Ericsson AB, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden · RISE Research Institutes of Sweden, Stockholm, Sweden

Deep Learning OptimizationLearning Rate SchedulingStochastic Gradient Descent
AskPDF
Feb 28, 2026

Leveraging Arbitrary Data Sources for AI-Generated Image Detection Without Sacrificing Generalization

The accelerating advancement of generative models has introduced new challenges for detecting AI-generated images, especially in real-world scenarios where novel generation techniques emerge rapidly. Existing learning paradigms are likely to make classifiers data-dependent, resulting in narrow decision margins and, consequently, limited generalization ability to unseen generative models. We observe that both real and generated images intend to form clustered low-dimensional manifolds within high-level feature spaces extracted by pre-trained visual encoders. Building on this observation, we propose a single-class attribution modeling framework that first amplifies the intrinsic differences between real and generated images by constructing a compact attribution space from any single-class training set, either composed of real images or generated ones, and then establishes a more stable decision boundary upon the enlarged separation. This process enhances class distinction and mitigates the reliance on generator-specific artifacts, thereby improving cross-model generalization. Extensive experiments show that our method generalizes well across various unseen generative models, outperforming existing detectors by as much as 7.21% in accuracy and 7.20% in cross-model generalization.

Qinghui He, Haifeng Zhang, Xiuli Bi +3

Chongqing Key Laboratory of Image Cognition, School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China · Chongqing Key Laboratory of Image Cognition, School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, China · Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China

AI-Generated Image DetectionDeep Learning Optimization
AskPDF