Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

Aug 12, 2026

MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

Multi-objective optimization (MOO) has demonstrated significant success in multi-task learning by mitigating task conflicts through gradient manipulation. However, most existing methods flatten model parameters into vectors and perform gradient manipulation under Euclidean geometry, thereby overlooking the matrix structure prevalent in modern architectures such as Transformers. In this paper, we show that gradient manipulation in Euclidean space does not generally yield the steepest descent direction under matrix geometry, potentially limiting optimization efficiency. Drawing from the theory of steepest descent for matrix-valued parameters, we propose MOON (Multi-Objective OrthoNormalized Updates), which performs gradient manipulation under spectral--nuclear norm geometry and uses the orthonormalized manipulated gradient for parameter updates. Theoretically, for smooth non-convex objectives, we establish convergence of the averaged Pareto-stationarity measure at rates of O(T−1/2)\mathcal{O}(T^{-1/2}) in the deterministic setting and O(T−1/4)\mathcal{O}(T^{-1/4}) under stochastic gradients. Empirical results across various benchmarks show that MOON consistently improves both optimization efficiency and final multi-task performance. Our code is available at https://github.com/KunlinLyu/MOON.

Shiji Zhou, Kunlin Lyu, Lei Zhang +2

Institute of Artificial Intelligence, Beihang University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · Beijing Academy of Artificial Intelligence (BAAI) +1

Neural Network OptimizationMulti-Task LearningMatrix OptimizationMulti-Task OptimizationMulti-Objective Optimization
AskPDF
Aug 11, 2026

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

We establish the first convergence guarantees of gradient descent for general feedforward neural networks of any width or depth, with any initialization or dataset. We only assume that the activation functions are linearly bounded, Lipschitz continuous, and Lipschitz smooth---properties that hold for linear, tanh, softplus, sigmoid, and smoothed ReLU functions---and that the loss function is Lipschitz smooth in the model outputs, a mild condition satisfied by mean squared error and binary cross-entropy loss. By relating the Lipschitz properties of one layer to the next, we obtain a novel generalized Lipschitz smoothness condition for an LL-layer neural network where the change in gradient is upper bounded by the change in the parameter space, multiplied by a polynomial of the parameter norms at both endpoints of degree 2L−22L - 2. This yields a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the leading term of the polynomial grows at a controlled rate, we prove that the minimum squared gradient norm converges to zero in TT iterations at rate O(1/T1/L)O(1/T^{1/L}).

Siqiao Mu, Diego Klabjan

Gradient DescentNeural Network OptimizationFeedforward Neural Networks
AskPDF
Aug 11, 2026

Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network

In this work we present a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN). This optimization procedure introduces contextual features into the first layer of a DNN. The parameters of DNN are optimized via standard gradient descent while keeping the input-feature basis fixed. After optimization of the DNN parameters, the feature layer is provided a chance to update and change before DNN optimization resumes. The feature layer has two types of functions: those that can be evaluated quickly in a matrix-free way on the domain (i.e. rank-1 features) and more complex features that must first be decomposed using tensor network (TN) decomposition strategies (tensor features). In particular, we study the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy. To efficiently decompose high-dimensional functions constructed from discretized DNN, we leverage a randomized tensor decomposition strategy. Using randomization, we are able to reduce the storage cost of decomposing high dimensional functions by at least 8 orders of magnitude. Using this approach, we are able to efficiently train models between 5 and 40 dimensions.

Karl Pierce, Yuehaw Khoo, Haizhao Yang

† The University of Maryland, College Park, College Park MD 20742 USA · ∗The University of Chicago, Chicago, IL 60637 USA

Neural Network OptimizationNeural Network Approximation Theory
AskPDF
Aug 10, 2026

Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training

Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory does not fully explain this success: conventional analyses rest on differentiability, convexity, and smoothness, assumptions that DNN objectives routinely violate. We close this gap by generalizing convexity and smoothness, through Legendre functions and convex conjugation, into a unified optimization framework for DNN training. We introduce H(ψ)\mathcal{H}(ψ)-convexity and H(Ψ)\mathcal{H}(Ψ)-smoothness, a single formalism that encompasses convex and non-convex as well as smooth and non-smooth objectives and reveals a natural duality between generalized convexity and smoothness. Building on these notions, we derive generalized gradient descent (GD) and generalized SGD via convex conjugation. We prove that generalized GD attains an optimal learning rate of exactly 11, and we establish rigorous gradient-energy-based convergence rates for both optimizers. We further recast DNN training as a composite optimization problem and show that its convergence hinges on jointly reducing the gradient energy and controlling the extreme eigenvalues of the structure matrix; a model capacity risk quantifies the residual error that no optimizer can remove. Experiments on six datasets (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, TREC, and SST-2) and on convolutional, residual, and Transformer architectures, with five loss functions, are consistent with the predicted training dynamics.

Binchuan Qi

College of Electronics and Information Engineering Tongji University Shanghai, 200092, China · Sihao Street Zhejiang Yuying College of Vocational Technology Zhejiang, 310018, China

Gradient DescentNeural Network OptimizationHypergradient OptimizationStochastic Gradient Descent
AskPDF
Aug 8, 2026

ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources of edge devices, they commonly rely on backpropagation (BP) training, which has a fundamental limitation of update locking and could experience severe throughput and memory bottlenecks. In this work, we propose a BP-free algorithm, called ZeroLock, that decouples the model updates into independent chunk updates by local objective construction. It breaks the update locking of BP and hence can improve throughput at the algorithm level and lower memory usage by reducing activation storage. To the best of our knowledge, we provide the first theoretical framework for such local objective construction-based approaches under general model chunk division by mapping local objectives to the global objective. We prove that ZeroLock has a convergence rate of O~(1/T)\tilde{\mathcal{O}}(1/\sqrt{T}), which differs from BP only by polylogarithmic factors. We design a system for ZeroLock and build real-world prototypes, incorporating techniques such as early forwarding and failure recovery for efficient and robust implementation. Experiments on the prototype show that compared to BP-based baselines, ZeroLock reduces the memory by 26.5% and improves throughput by 4.9%.

Wentao Dai, Xuanran Li, Yuxiang Zhang +2

Department of Computer Science and Engineering, Southern University of Science and Technology, China · Department of Mathematics, Southern University of Science and Technology, China · School of Computing, Montclair State University, New Jersey, USA

On-Device TrainingMemory-Efficient Fine-TuningLLM Fine-TuningNeural Network OptimizationMemory-Efficient OptimizationMemory-Efficient Training
AskPDF
Aug 5, 2026

MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training

Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable. We study this failure in a controlled 110M-parameter nanowhale DeepSeek-style MoE pretraining setting. A SAGE/Sinkhorn hybrid reduces optimizer state from 0.883GB to 0.331GB but degrades evaluation loss to 3.8265, far above the AdamW baselines observed in the same setup (3.58--3.64 across the seeds we study). We show that routed MoE expert matrices are the dominant failure point: their gradients are conditional, temporally varying, and poorly served by stateless Sinkhorn normalization. We propose MESH, a hidden-momentum Sinkhorn update for MoE experts. MESH restores a temporal first-moment signal through the gradient-buffer lifecycle, without storing the expert first moment as optimizer state. MESH is an optional block-preconditioned variant that adds a coarse neuron/block inverse-RMS multiplier. Across ablations, temporal smoothing before matrix normalization is the primary causal ingredient; block/neuron preconditioning can improve the memory-quality frontier, but is not established as universally necessary. In two additional seeds, MESH and MESH-B reduce optimizer-state memory by 62.5% and peak PyTorch CUDA allocation by about 12.6% relative to AdamW, with a modest evaluation-loss gap. Full-state diagnostic variants recover AdamW-like performance in ablations, supporting the conclusion that MoE experts need temporal smoothing, but not necessarily full coordinate-wise AdamW state.

Masato Fujitake

FA Research, Fast Accounting Co., Ltd.

Neural Network OptimizationMemory-Efficient OptimizationMatrix OptimizationMixture-of-Experts ModelsMemory-Efficient Training
AskPDF
Aug 4, 2026

Muon Meets Mamba: Spectral Optimization for State Space Models

Muon is a recent optimizer that orthogonalizes the update to each weight matrix with a Newton-Schulz iteration, which performs steepest descent under the spectral norm. Almost all the evidence for it comes from Transformer models, and its behavior on state-space models is largely unreported. We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon. The benefit is localized. Muon on the output projection alone beats Muon on the input projection or on both. The advantage is mainly one of token efficiency. It holds on two corpora and two token budgets, and persists when training continues well past the compute-optimal point. Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.

Arslan Battalov, Karim Kramin, Alexander Markotenko +1

HSE University · Moscow, Russia

Neural Network OptimizationMuon OptimizerState Space Models
AskPDF
Aug 4, 2026

Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon

Matrix spectral optimizers reshape weight-update spectra but usually delegate vector-valued biases to a separate optimizer. We study whether this separation is neutral. We formulate each affine layer as a joint momentum matrix A=[MW,αmb]A=[M_W,αm_b] and apply a capped regularized-inverse spectral map to the complete matrix, producing both the weight and physical bias updates. A strict five-seed ablation on a four-layer BERT-mini trained from scratch on IMDb compares exact-SVD Muon, weight-only inverse shaping, affine-probe inverse shaping, and the proposed joint regularized inverse (JRI). Weight-only inverse shaping raises validation-loss-selected test accuracy from 84.903±0.242%84.903\pm0.242\% to 85.562±0.308%85.562\pm0.308\% and lowers selected test loss from 0.34790.3479 to 0.33450.3345. Allowing bias to alter the joint SVD while retaining an independent Adam bias update does not improve over weight-only inverse shaping. Using the transformed bias jointly raises selected test accuracy to 85.738±0.180%85.738\pm0.180\% and lowers test loss to 0.32910.3291, with all five seeds improving relative to the probe baseline. During the peak-performance window, JRI preserves the eligible weight-update norm while reducing the bias-update norm from 0.020950.02095 to 0.003010.00301, lowers boundary-function share from 86.58%86.58\% to 78.97%78.97\%, and changes the cosine between weight-induced boundary motion and explicit bias from +0.030+0.030 to −0.137-0.137. An independent 22-seed replication yields 85.743±0.203%85.743\pm0.203\% selected test accuracy. These results identify joint affine spectral allocation as a small but consistent extension to weight-only spectral optimization.

Gongyue Zhang, Honghai Liu

State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.

Neural Network OptimizationMuon OptimizerMatrix OptimizationSpectral MethodsSpectral Optimization
AskPDF
Jul 30, 2026

New Synchronous Computation Dynamics for Hopfield Networks

The dynamics of the original Hopfield network is asynchronous (sequential) (updates the state of only one neuron per time step). In this paper, we propose a new tool and a new dynamics to reduce the processing time by updating one or more neurons simultaneously per instant while ensuring process convergence and aiming for the maximum energy decrease at each step, thus guaranteeing the shortest total processing time. From the point of view of synchronous dynamics, calculating the next network state at which energy decreases the most from the current state while ensuring convergence is itself a combinatorial optimization problem. We develop and use a new tool to solve it. We call this new tool Discrete Differential Filter (DDF) and, based upon it, we develop a new synchronous dynamics which we call SD-DDF (Synchronous Dynamics based upon Discrete Differential Filter). In this paper, we review the original asynchronous dynamics for Hopfield networks and present a new tool and a new synchronous dynamics with its theoretical justification and four computational experiments to assess the speed up in processing time empirically.

Francisco Requena-Domínguez, Rafaela Benítez-Rochel, Ezequiel López-Rubio

Department of Computer Languages and Computer Science University of Málaga Bulevar Louis Pasteur 35, 29071 Málaga, Spain · ITIS Software, University of Málaga C/ Arquitecto Francisco Peñalosa 18, 29010 Málaga, Spain

Neural Network OptimizationHopfield Networks
AskPDF
Jul 27, 2026

Greedy dynamical meta-learning

Gradient descent scales well to large models, but becomes unstable over long time horizons. Gradient-free optimizers can scale to arbitrary timespans, but are hobbled by high dimensions. Since learning occurs in large models over long timescales, neither of these approaches is likely to produce traits which can accelerate the learning process. Instead, we propose a meta-learning algorithm in which the agent learns to modify its own weights and biases. Our algorithm consists of an inner loop, wherein the agent performs some high-dimensional optimization upon itself, and an outer loop, wherein we perform some low-dimensional optimization upon the inner loop. Since the outer loop handles very few parameters, standard zeroth-order methods may be used.

Aria Yom

Rhodora

Meta-LearningNeural Network OptimizationLearning to Optimize
AskPDF
Jul 26, 2026

Scale Weight Decay and Train Better

The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data. This is typically done with a constant decoupled weight decay which causes the network weights to shrink steadily over the course of training. Taking inspiration from the Robbins--Monro conditions, we propose to scale weight decay by the fraction of the peak learning rate η/ηmax⁡η/η_{\max}. We prove that this scaled weight decay preserves the asymptotic stationarity guarantees of the corresponding unregularized methods for both stochastic gradient descent and the non-Euclidean spectral optimizer Muon, thereby avoiding the additional asymptotic bias introduced by constant decoupled weight decay. This retains the stability benefits of weight decay without changing the asymptotic optimization target. Using a steady-state analysis, we explain why under standard weight decay the weight norm shrinks steadily as training proceeds, whereas under scaled weight decay it settles to a roughly constant value. When applied to the training of mixture-of-experts models, Muon with scaled weight decay (Muon-SW) consistently outpaces Muon with identical hyperparameters, reaching the same validation loss 30%\mathbf{30\%} faster at our largest scale across models from 72−93072 - 930 million parameters trained at ∼600\sim 600 tokens per active parameter. If this trend continues to hold, the method promises to substantially accelerate the pre-training of frontier models while requiring only a few lines of code to implement.

Anuj Apte

Global Technology Applied Research, JPMorganChase, New York, NY 10001

Weight DecayNeural Network OptimizationStochastic Gradient Descent
AskPDF
Jul 24, 2026

Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

Weights and biases are normally optimized as separate parameter tensors, yet they do not represent separate functions when the input to an affine layer has nonzero mean. For an affine map z=Wx+bz=Wx+b with input mean μμ, a weight update contains a sample-independent displacement ΔWμΔWμ that is functionally indistinguishable from a bias update. We call this hidden contribution boundary motion and decompose each update into a centered, sample-varying shape component and a shared boundary component. On a four-layer Transformer trained from scratch on IMDb, the bias-like term gbμ⊤g_bμ^\top has a median norm equal to 0.664 of the raw weight-gradient norm across affine layers and training checkpoints. More strikingly, the median ratio \normΔWμ/\normΔb\norm{ΔWμ}/\norm{Δb} is 134.7, while \normΔWμ/\normΔb+ΔWμ\norm{ΔWμ}/\norm{Δb+ΔWμ} is 0.994. Thus, under AdamW, the observed boundary motion is almost entirely realized through the weight matrix rather than the explicit bias. We implement a diagnostic optimizer, Shape--Boundary Orthogonal AdamW (SBO-AdamW), that optimizes gW−gbμ⊤g_W-g_bμ^\top and gbg_b with independent Adam states and compensates the weight-induced boundary displacement. In a single-seed experiment, SBO-AdamW raises validation accuracy from 81.68% to 85.81% and validation-selected test accuracy from 78.73% to 82.73%, with the best validation checkpoint occurring at step 800 instead of step 3000. However, the moving-batch-center compensation produces severe bias-coordinate drift and strongly reduces boundary energy. The present evidence therefore supports hidden boundary motion as an important optimization mechanism, but it does not yet establish a final general-purpose optimizer. A stable centered-affine parameterization is identified as the required next step.

Zhang Gongyue, Sheng Yixuan, Liu donghan +3

arXiv:2607.22927v1

TransformerNeural Network OptimizationAdaptive Gradient MethodsTransformer Training Dynamics
AskPDF
Jul 24, 2026

Hyperball May Not Be a Free Lunch

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

Yihao Xiao, Jialong Sun, Zitian Gao +5

1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2

Learning Rate SchedulingNeural Network OptimizationMuon OptimizerNeural Network Training Dynamics
AskPDF
Jul 23, 2026

Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

We investigate whether symbolic regression can discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks. Candidate update rules are represented as fixed-depth symbolic expressions over operands derived from common optimizers, including gradient, momentum, adaptive-gradient, and moment-estimate quantities. Across 30 benchmark/neural network combinations, the symbolic regression procedure found an update rule outperforming the best hyperparameter-tuned established optimizer in 25 cases, with an aggregate MSE reduction of 44.47% over the improved cases. The discovered rules do not all share a single common symbolic form, but many combine adaptive normalization, momentum-like quantities, nonlinear transformations, and rational expressions. These results suggest that symbolic regression can serve as a lightweight mechanism for discovering compact optimizer variants, while also highlighting the need for larger-scale validation.

Charles Brum, Edward Finkelstein

San Diego, California, USA · UCI, USA

Neural Network OptimizationLearning to OptimizeFeedforward Neural NetworksSymbolic Regression
AskPDF
Jul 23, 2026

Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

Most explanations of training instability focus on learning-rate criticality, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. We argue that, in practical deep neural network training, there is an additional and often overlooked weight-norm criticality. This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay (which persistently shrinks parameter norms). As the weight decay coefficient increases, the norms of scale-invariant weights are progressively driven toward zero. Meanwhile, the sharpness of the loss landscape increases rapidly, destabilizing the optimization dynamics and resulting in abrupt loss spikes. This perspective provides a rationale for why weight penalties can improve generalization yet cannot be made arbitrarily strong: excessive decay drives scale-invariant weight norms past a critical boundary and destabilizes training. Our work provides a new mechanistic understanding of loss spikes through the lens of weight-norm criticality. Moreover, weight-norm criticality yields testable predictions that we validate empirically in networks with scale-invariant components, providing empirical support for the proposed mechanism.

Xiaolong Li, Zhangchen Zhou, Zhi-Qin John Xu

Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University.

Weight DecayNeural Network OptimizationNeural Network Training Dynamics
AskPDF
Jul 23, 2026

New Complexity-Theoretic Frontiers of Tractability for Neural Network Training

In spite of the fundamental role of neural networks in contemporary machine learning research, our understanding of the computational complexity of optimally training neural networks remains incomplete even when dealing with the simplest kinds of activation functions. Indeed, while there has been a number of very recent results that establish ever-tighter lower bounds for the problem under linear and ReLU activation functions, less progress has been made towards the identification of novel polynomial-time tractable network architectures. In this article we obtain novel algorithmic upper bounds for training linear- and ReLU-activated neural networks to optimality which push the boundaries of tractability for these problems beyond the previous state of the art. In particular, for ReLU networks we establish the polynomial-time tractability of all architectures where hidden neurons have an out-degree of 11, improving upon the previous algorithm of Arora, Basu, Mianjy and Mukherjee. On the other hand, for networks with linear activation functions we identify the first non-trivial polynomial-time solvable class of networks by obtaining an algorithm that can optimally train network architectures satisfying a novel data throughput condition.

Cornelius Brand, Robert Ganian, Mathis Rocton

Algorithms & Complexity Group Vienna University of Technology Favoritenstraße 9-11, 1040 Vienna, Austria

ReLU Neural NetworksNeural Network OptimizationDeep Linear Networks
AskPDF
Jul 19, 2026

On the Depth Scalability of Logic Gate Networks

Logic Gate Networks (LGNs) compute through compositions of Boolean operations, yet existing LGNs do not reliably benefit from increased depth. We identify two causes: optimization collapse and topology-induced degradation of output-specific credit that persists even after skip-biased initialization and straight-through estimation stabilize training. We introduce Input-Anchored Logic Gate Networks (IALGNs), in which each gate combines a private hidden spine with a direct input anchor. This topology prevents output-path merging while retaining input access at every layer. Credit diagnostics show that random wiring dilutes or conflicts output-specific gradients, whereas IALGN maintains usable and coherent credit. Random-kxk_x relaxation improves anchor selection without relaxing the spine. Across MNIST, CIFAR-10, and CIFAR-100, IALGN exhibits consistent fixed-width depth--accuracy scaling up to 150 layers, while alternative topologies saturate or degrade. Linear probes, topology ablations, and operation-aware analysis show that trained IALGNs preserve private states and apply sparse anchor-conditioned updates. These results indicate that scalable LGN depth requires both stable optimization and credit-preserving information access.

Taegun An, Dohun kim, Haebeom Lee +1

Jung Woonoh IT & General Education Center, Korea University 145, Anam-ro, Seongbuk-gu, Seoul 02841, Republic of Korea

Neural Network OptimizationCredit Assignment
AskPDF
Jul 18, 2026

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space. We measure 8 such properties with the same harness around a multitask LoRA operating point, on 9 transformers (82M-7B), with a prospectively registered property list, thresholds, and test split. We find a shared one-direction validity window up to the tested scale 10−210^{-2}, but no universal radius for pairwise composition or update ordering. Along individual directions, changes of the probe loss remain first-order predictable throughout the grid: a perturbation's effect on the loss is essentially its projection onto the gradient, which is also what makes local random search work. Pairwise structure, however, proves to be far more fragile: on over a third of the measured (model, task pair) combinations, two-update order sensitivity sets in strictly inside that window; task-gradient subspaces rotate within tens of steps; additivity under our fixed activation probe fails at full task-vector scale on several models, including both held-out 7B models; and no model median passes the registered global mean-vector weight-to-steering correspondence bar. For two sequential task-gradient steps, the leading order-dependent term is the Lie bracket HBgA−HAgBH_B\textbf{g}_A-H_A\textbf{g}_B; its normalized prediction c(η)=ηκ+O(η2)c(η)=ηκ+O(η^2) tracks the measured defect at median ratio 1.002, while the onset scale η†≈0.10/κη^\dagger\approx0.10/κ spans three orders of magnitude across models and task pairs.

Irina Piontkovskaia, Sergey Nikolenko

1DAIMLD, Moscow, Russia · 2St. Petersburg Department of the Steklov Institute of Mathematics, St. Petersburg, Russia · 3St. Petersburg State University

Fine-TuningNeural Network OptimizationTask ArithmeticActivation Steering
AskPDF
Jul 18, 2026

Effects of width-dependent model hyperparameters and ℓ2\ell_2-regularization on the loss landscape of two-layer ReLU networks

Understanding deep neural networks remains a central challenge in machine learning. In particular, the theoretical properties of even two-layer ReLU networks, especially in the presence of weight decay, remain poorly understood. To this end, we derive a sufficient condition on the hyperparameter settings under which the global minima collapse to the zero solution. Interestingly, our experiments reveal that using AdamW as an optimizer prevents the collapse of the learned parameters, whereas using SGD does not, which may help explain the success of AdamW in deep learning training. In addition, when restricting the input dimension to one, we derive an analytical solution for the globally optimal parameter sets of two-layer ReLU networks and show that ℓ2\ell_2-regularization has a width-invariant effect on connectivity, but its dimensionality-reducing effect becomes stronger as the network width increases. These results provide insight into how width-dependent hyperparameters influence the geometry of regularized loss landscapes.

Haruka Eshima, Makoto Yamada

Okinawa Institute of Science and Technology, Okinawa, Japan

ReLU Neural NetworksWeight DecayNeural Network OptimizationLoss Landscape GeometryHyperparameter Optimization
AskPDF
Jul 18, 2026

Backpropagation-Free Trunk Training via the Split Forward Gradients

Backpropagation makes training deep networks memory intensive because it must store intermediate activations. Forward-mode methods avoid this cost, but their gradient estimates become increasingly noisy as the number of trained parameters grows. We introduce Split Forward Gradient (Split-FG), which splits a network at an intermediate representation: it computes the output head gradient exactly and estimates only the trunk gradient with a Jacobian--vector product. This reduces estimator variance and requires no backward pass through the trunk, while retaining an Adam-style convergence guarantee. Our experiments reveal an important practical failure mode. On WikiText-103, naive forward-gradient training of the trunk performs worse than leaving a randomly initialized trunk frozen, likely because Adam updates every noisy, under-determined trunk coordinate too aggressively. Simply using a much smaller learning rate for the trunk reverses this result: a 1616M-parameter GPT-2-style model reaches validation perplexity 387387, compared with 668668 for the frozen-trunk control and 2,8852{,}885 for a matched pure forward-gradient baseline (backpropagation reaches 150150). Split-FG also produces the strongest backprop-free results on our tabular benchmarks and reaches 60.5%60.5\% on CIFAR-10 and 35.2%35.2\% on CIFAR-100 with a heavy-head design. It reduces peak memory by up to 35%35\% relative to matched backpropagation, although the performance gap widens as the forward-mode trunk grows.

Tian Qin, Wei-Min Huang

Math department Lehigh University Bethlehem, PA 18015, USA

BackpropagationGradient DescentNeural Network OptimizationNeural Network Training DynamicsMemory-Efficient Training
AskPDF
Jul 17, 2026

Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss in a local parameter neighborhood. Standard SAM implicitly allocates its global perturbation budget across parameter blocks according to instantaneous minibatch gradient norms. Such an allocation can be noisy and may not reflect the sensitivity that blocks accumulate throughout training. We propose Gradient-Energy Adaptive Radius SAM (GEAR-SAM), which maintains an exponential moving average (EMA) of squared block gradients as a lightweight, curvature-related sensitivity signal and allocates the fixed SAM budget through a closed-form constrained optimization. GEAR-SAM preserves the global SAM radius, requires no Hessian-vector products or explicit Fisher estimation, and adds only scalar state beyond SAM. Experiments on image classification, transfer learning, noisy-label learning, and partition studies demonstrate improved generalization and robustness across architectures and tasks. More broadly, GEAR-SAM provides a dynamic view of sharpness-aware optimization: a fixed perturbation budget should be redistributed as the sensitivity of functional network blocks evolves during training.

Zhen Huang, Jiaxin Deng, Junbiao Pang

Faculty of Information Technology, Beijing University of Technology, Beijing, China

Sharpness-Aware MinimizationNeural Network Optimization
AskPDF
Jul 16, 2026

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks. We build a representative Adam variant (Bogacki-Shampine 3(2) RK pair, FSAL reuse, local-error step control) and evaluate it under a strict compute-matched protocol giving every method the same gradient-evaluation budget - an accounting this literature rarely enforces. Under it the RK variant loses to plain Adam on training loss in both minibatch and full-batch (RK's best-case) training. Instrumenting it shows the "adaptivity" is illusory: normalized error stays far below tolerance, the step size pins at its growth cap from step one (98-100 percent of steps), and no rtol x hmax x h0 setting makes it act; tolerances spanning 100x give bit-identical trajectories. The method is exactly fixed-step Adam with an averaged gradient at 3-4x cost. Repairing it (true reject branch; error on the applied map) reverses the full-batch result - about 40x lower training loss than tuned Adam - and a fixed-step control isolates adaptivity (an emergent warmup-and-growth schedule) as the mechanism. But the gain is fragile to the initial step size and does not reach test accuracy. A pre-registered follow-up rules out the obvious explanations: deeper minimization does not overfit, and an explicit temperature knob only hurts - leaving a trajectory effect, the controller selecting a minimum generalizing 1.3-3.4 points below first-order descent at equal depth. An n=10 study confirms one secondary effect: gradient averaging is a genuine implicit regularizer, beating lr-matched Adam and AdamW on 10/10 seeds - yet RMSprop and NAdam match or beat it at a third the per-step cost. Higher-order adaptive integration buys deeper deterministic minimization and a small regularization effect, but nothing a cheaper, well-tuned first-order baseline does not already provide.

Akhilesh Gogikar

1Independent Researcher.

Neural Network OptimizationAdaptive Gradient MethodsNeural Network Training DynamicsImplicit Regularization
AskPDF
Jul 15, 2026

LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks

Physics-informed neural networks (PINNs) have had a broad research impact in modeling domains governed by partial differential equations (PDE). However, PINNs have been shown to perform poorly, sometimes even converging to trivial solutions, in challenging PDE domains, or when generalizing to unseen but related PDE domains. Previously proposed solutions detail hyperparameter tuning to reduce loss imbalance between data-driven and physics guided losses, curriculum learning based training strategies, or dynamic re-sampling of hard collocation points. These methods face certain pitfalls: hyperparameter tuning is expensive, designing a training curriculum is ambiguous in multi-parameter PDE settings, and dynamic resampling still fails in complex PDE settings. Complementary to this line of thinking, we believe the initial PINN network weights also play a crucial role in the emergence of catastrophic failures during training, yet the effect of PINN weight initialization has been surprisingly under-investigated. To this end, we propose a framework for Learned Initialization via Gated Layerwise Optimization (LIGO-PINN) to overcome PINN convergence failures. Through rigorous evaluation on 1D and 2D PDE domains, including a challenging 2D fluid dynamics setting, we demonstrate that our methodology outperforms state-of-the-art methods designed to alleviate PINN failures, achieving a 91.5% average performance improvement across six baselines and 81% over the strongest baseline. We also verify that LIGO-PINN generalizes to 3D unstructured domains. Finally, we analyze training dynamics across all three PDE domains to explain both LIGO-PINN's improvement and the convergence failure of traditional PINNs. Code: https://github.com/scailab/ligo-pinn Keywords: Machine Learning, Physics-Informed Neural Networks, Deep Learning, PDE Modeling

Nilay Anurag, Shital Adhikari, Taniya Kapoor +1

Department of Computer Science, Stevens Institute of Technology · Artificial Intelligence Group, Wageningen University & Research

Neural PDE SolversNeural Network OptimizationPhysics-Informed ML
AskPDF
Jul 15, 2026

How the Hessian-Spectrum of Neural Networks Depends on Data

The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc. Prior works have focused on empirical results or pursued a theoretical treatment under overly simplified settings. In this work, we derive the eigenvalues of the Hessian of linear networks with arbitrary widths and depths, and datasets with an arbitrary number of samples, features, and labels. Importantly, for classification tasks with MSE loss, we identify that the sharpness of the solution is directly related to the maximum proportion of samples belonging to any class. We empirically validate our predictions and systematically analyze the effects of shedding the impractical assumptions one at a time, as well as incorporating nonlinearities. We observe that our predictions are considerably robust in most cases, allowing us to extend our conclusions to more practical learning setups.

Jasraj Singh, Enea Monzio Compagnoni, Antonio Orvieto

ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany

Loss Landscape SharpnessNeural Network OptimizationDeep Linear NetworksNeural Network Training Dynamics
AskPDF
Jul 15, 2026

FastCentNN: Accelerating Centroid Neural Network with Entropy Proxy

Centroid neural network (CentNN) is an unsupervised competitive learning algorithm in which centroid splitting is triggered only after strict local stabilization, often leading to prolonged low-movement training phases before model expansion. This report proposes FastCentNN, an accelerated variant that addresses this inefficiency by introducing an early splitting strategy based on the total centroid movement per epoch, which serves as a training entropy proxy. As a result, FastCentNN reduces unnecessary reassignment epochs while preserving the original winner-loser learning dynamics. FastCentNN supports both absolute and stage-relative movement thresholds, allowing the splitting criterion to remain either fixed or adaptive throughout training. Experiments on some benchmark datasets show that FastCentNN consistently achieves clustering quality comparable to CentNN while reducing runtime by up to 16% on synthetic 2D datasets and about 5% on high-dimensional datasets. FastCentNN therefore provides a practical and efficient drop-in replacement for CentNN, retaining its online adaptive learning behavior while offering a simple and interpretable speed-stability trade-off through configurable splitting thresholds.

Le-Anh Tran

Neural Network OptimizationUnsupervised Clustering
AskPDF
Jul 15, 2026

Approximation of solutions of parameter-dependent problems by residual neural networks

We develop a convergent scheme to train neural networks involving analytic activation functions based on gradient flows. Convergence properties are guaranteed by Lojasiewicz theory. The main advantage of this approach is its simplicity of implementation. The coefficients of the network are approximated by solving a system of ordinary differential equations. We test the method by constructing residual neural network approximations of solutions of parametric problems. The dependence of the solutions of simple ordinary differential equations on a few parameters is correctly reproduced. The solutions of inverse problems involving wave constraints which depend on a few parameters can be reasonably approximated, even in regions in which the problem is severely ill posed.

Ana Carpio

Universidad Complutense de Madrid, Spain

Neural Surrogate ModelingNeural Network OptimizationNeural Network Approximation Theory
AskPDF
Jul 10, 2026

Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations. We introduce \method (\methodshort), a weight reparameterization for neural networks that combines a sign-aware symmetric-exponential pathway with an identity-like linear pathway. The symmetric-exponential pathway is near-linear for small raw weights but increasingly curved at larger magnitudes. Additive updates in logarithmic space map to magnitude-proportional changes in effective weight space. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway. These components create a curved parameter-space geometry that empirically improves speed of loss descent over standard linear parameterization. We also identify a useful mismatched initialization: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small-model ablations, this improves early optimization and may act as a form of symmetry breaking. We train transformers on OpenWebText over nine width×\timesdepth configurations, \methodshort reaches matched validation loss in 1.32--1.49×\times fewer training steps, with the largest widths seeing the biggest gains.

Ethan Smith

Canva Research

Neural Network OptimizationWeight-Space Learning
AskPDF
Jul 10, 2026

Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks

We introduce a novel method for both partial and full optimization of the connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs). Our training method utilizes a probability distribution over a set of connections per gate/lookup table (LUT) input pin, selecting the connection with highest merit, all whilst the optimal gate types or LUT-entries are learned in parallel. We show that the connection-optimized LGNs outperform standard fixed-connection LGNs on the Yin-Yang, MNIST Handwritten Digits and Fashion-MNIST benchmarks, while requiring only a fraction of the number of logic gates. We achieve 98.92% on the MNIST dataset with two layers of 8000 gates. With only one layer of 8000 gates, we obtain 98.45%, showing that our method requires almost 50 times fewer gates compared to fixed-connection LGNs. Training stability up to ten layers has been ensured by employing a high learning rate, straight-through estimators and trimming constant-output gate types. Additionally, we present a LUT neuron description that enables stable training with backpropagation, tested up to 6-layer deep networks. The model requires four times fewer trainable parameters and still achieves a higher accuracy compared to the fixed-connection LGN training algorithm. Our connection-training algorithm also works well for the LUTNs, achieving an accuracy of 98.88% for two layers of 2000 6-input LUTs.

Wout Mommen, Lars Keuninckx, Matthias Hartmann +2

imec, Kapeldreef 75, Leuven, 3001, , Belgium · Vrije Universiteit Brussel, Peinlaan 2, Elsene, 1050, , Belgium

Neural Network OptimizationDifferentiable Logic Gate NetworksDifferentiable Logic
AskPDF
Jul 9, 2026

Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and transformer families within the LEMUR neural network dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-dataset, provides a practical reference for principled scheduler selection.

Hafsa Mateen, Radu Timofte, Dmitry Ignatov

Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

Learning Rate SchedulingNeural Network OptimizationHyperparameter Optimization
AskPDF
Jul 8, 2026

Reliable mechanistic operator recovery with biologically-informed neural networks: principles for architecture and optimisation design

Many biological processes are governed by complex dynamical mechanisms that remain incompletely understood despite increasing volumes of experimental data. Biologically-informed neural networks (BINNs) seek to address this challenge by embedding mechanistic differential equations into neural network training, enabling interpretable constitutive operators to be recovered directly from sparse and noisy observations. However, reliable operator recovery depends sensitively on network architecture, optimisation strategy, and data informativeness. Here, we present a systematic empirical study of how these factors influence mechanistic inference using BINNs applied to canonical one-dimensional advection-diffusion-reaction partial differential equation models. Across a suite of benchmark problems, we investigate how network expressivity, learning rate, loss weighting, and batch size influence optimisation behaviour and operator recovery. We show that successful mechanistic inference depends on balancing competing objectives rather than maximising any single aspect of the model or optimisation. Moderately expressive architectures outperform overly complex networks, intermediate learning rates improve optimisation stability, balanced data and PDE losses are essential for accurate operator recovery, and intermediate batch sizes provide the best compromise between computational efficiency and reproducibility. We further identify practical diagnostics for recognising common failure modes, including over-fitting, unstable optimisation, and poor mechanistic recovery when the ground truth is unavailable. Together, these findings provide evidence-based guidelines for deploying BINNs as credible tools for biological model discovery.

Rebecca M. Crossley, Yuan Yin, Sarah L. Waters +1

Mathematical Institute, University of Oxford, Oxford, United Kingdom, OX2 6GG

Neural Network OptimizationMechanistic InterpretabilityPDE Operator Learning
AskPDF