Neural Network Optimization

Follow this topic Ask about this topic

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

When do complex-valued neural networks help? A study of representation, geometry, and optimization
May 26, 2026Ashutosh KumarNeural Network Optimization Complex-Valued Neural Networks
Owl Autonomous Imaging, Inc.
How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks
May 26, 2026Teodor-Mihai Stupariu, Andrei ManolacheNeural Representation Geometry Neural Network Optimization
1Bitdefender, Romania · 2Tudor Vianu High School of Com · University of Stuttgart, Germany +1
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
May 26, 2026Mohammed Adnan, Rohan Jain, Tom Jacobs +4Neural Network Optimization Dynamic Sparse Training
University of Calgary · 2Vector Institute · 3CISPA Helmholtz +2
Deep-layer limit and stability analysis of the basic forward-backward-splitting induced network (II): learning problems
May 26, 2026Xuan Lin, Chunlin WuNeural Network Stability Optimization Convergence Analysis
China Academy of Aerospace System and Innovation, 100088, Beijing, China · School of Mathematical Sciences, Nankai University, 94 Weijin Road, 300071, Tianjin, China
Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias
May 26, 2026James Town, Etienne Boursier, Ben Lewis +2ReLU Neural Networks Neural Network Optimization
University of Warwick · INRIA LMO, Université Paris-Saclay
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Orthogonal Optimization
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
May 26, 2026Wenjie Zhou, Bohan Wang, Hongtao Zhang +3Language Model Pretraining Neural Network Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2
PDEInvBench: A Comprehensive Dataset and Design Space Exploration of Neural Networks for PDE Inverse Problems
May 25, 2026Divyam Goel, Nithin Chalapathi, Sanjeev Raja +1Neural Network Optimization Synthetic Benchmark Generation
Department of Computer Science, UC Berkeley · Departments of Computer Science and Chemical EngineeringMay UC Berkeley; LBNL
Growing a Neural Network in Breadth, Depth, and Time
May 24, 2026Eivinas Butkus, Kedar Garzón Gupta, Nikolaus KriegeskorteRecurrent Neural Networks Dynamic Neural Networks
1Columbia University · 2NSF AI Institute for Artificial and Natural Intelligence
Feature Learning in Wide Neural Networks under $μ$ P: Identifiability and Sparse-Dictionary Decomposition of the Mean-Field Limit
May 23, 2026Akmal XodarevRepresentation Learning Parameter Identifiability
Independent Researcher · Tashkent, Uzbekistan
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuDeep Learning Optimization Transformer
Carnegie Mellon University
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
May 22, 2026Anherutowa CalvoNeural Network Optimization Second-Order Optimization
19D Labs
A lift for input-convex neural net training
May 22, 2026Ali SiahkoohiNeural Network Optimization Input Convex Neural Networks
Department of Computer Science University of Central Florida
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
May 22, 2026Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +3Transformer Neural Network Optimization
Adobe Research
Optimization of randomized neural networks for transfer operator approximation
May 22, 2026Mohammad Tabish, Stefan KlusDynamical Systems Neural Network Optimization
Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University, Edinburgh, UK · School of Mathematical & Computer Sciences, Heriot–Watt University, Edinburgh, UK
Non-normal spectral signatures of instability in neural network training dynamics
May 22, 2026Souvik GhoshDynamical Systems Neural Network Optimization
Department of Physics, National Sun Yat-sen University, Kaohsiung 80424, Taiwan
Coupling-Robust Accuracy in Multiphysics Physics Informed Neural Networks via Kronecker-Preconditioned Optimization
May 22, 2026Youngjae Park, Jaemin Kim, Junghwa HongGauss-Newton Optimization Neural Network Optimization
Dept. of Control and Instrumentation Engineering, Korea University, Sejong, South Korea · BK21 FOUR Smart Mobility Education and Research Team, Korea University, Sejong, South Korea
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzDeep Learning Optimization Transformer
CISPA Helmholtz Center for Information Security
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
May 20, 2026Xingyu Lyu, Qianqian Xu, Zhiyong Yang +2Neural Network Optimization Neural Network Training Dynamics
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China
DEL: Digit Entropy Loss for Numerical Learning of Large Language Models
May 19, 2026Zhaohui Zheng, Chenhang He, Shihao Wang +3Numerical Reasoning in Language Models Neural Network Optimization
The Hong Kong Polytechnic University · VCIP, College of Computer Science, Nankai University
Training Neural Networks with Optimal Double-Bayesian Learning
May 19, 2026Vy Bui, Hang Yu, Karthik Kantipudi +2Neural Network Optimization Bayesian Decision Theory
Lister Hill National Center for Biomedical Communications, National Library of Medicine National Institutes of Health, Bethesda, MD 20894, USA · Office of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases National Institutes of Health, Bethesda, MD 20892, USA
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
May 19, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Sign-Based Optimization
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
From SGD to Muon: Adaptive Optimization via Schatten-p Norms
May 19, 2026Thomas Massena, Corentin Friedrich, Mathieu SerrurierLMO Methods Neural Network Optimization
DTIPG - SNCF, IRIT · IRIT & SNCF · IRIT
Tight Complexity of Muon under Heavy-Tailed Noise
May 18, 2026Jiayu Zhang, Tianyi LinStochastic Optimization Neural Network Optimization
Department of Industrial Engineering and Operations Research, Columbia University
Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
May 18, 2026Tim Tsz-Kit Lau, Weijie SuNeural Network Optimization
University of Pennsylvania, Philadelphia, PA 19104, USA.
Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting
May 17, 2026Karthik Prakhya, Alp YurtseverNeural Network Optimization Deep Linear Networks
Umeå University, Sweden
WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
May 17, 2026Dongyue Li, Zechun Liu, Kai Yi +6LLM Quantization Quantization-Aware Training
1Northeastern University, MA · 2Meta AI, CA.
DynMuon: A Dynamic Spectral Shaping View of Muon
May 16, 2026Fangzhou Wu, Rikhav Shah, Sandeep Silwal +1Neural Network Optimization Muon Optimizer
University of Wisconsin–Madison · MIT · Elorian AI
Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking
May 15, 2026Vaidehi Bagaria, Nikshep Grampurohit, Pulkit VermaReinforcement Learning Neural Network Optimization
Indian Institute of Technology Madras
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
May 15, 2026Sihan Zeng, Sujay Bhatt, Sumitra GaneshDeep Learning Optimization Neural Network Optimization
JPMorgan AI Research, United States.

May 26, 2026J/K move · Enter open · S save

When do complex-valued neural networks help? A study of representation, geometry, and optimization

Ashutosh Kumar

Owl Autonomous Imaging, Inc.

Neural Network Optimization Complex-Valued Neural Networks Neural Network Training Dynamics

Open paper AskPDF arXiv

Complex-valued Neural Networks (CVNNs) are often motivated by domains where information is naturally encoded in magnitude and phase. Yet complex-valued inputs alone do not determine when complex arithmetic improves learning: the label signal may lie in amplitude, phase, their coupling, or a symmetry that real-valued models can also represent under suitable coordinates. We study this through a representation-first evaluation of CVNNs against Cartesian real, polar, phase-only, magnitude-only, parameter-matched real, and FLOP-matched real baselines. Across synthetic RF tasks, complex representations are useful but not universally superior. PSK-only tasks favor phase-aware and complex-valued models, QAM-only tasks favor magnitude-based models, mixed PSK+QAM gives only a small complex-valued advantage, and unseen carrier-phase rotations break coordinate-dependent models without augmentation. Similar patterns appear beyond RF: in quantum-wavefunction prediction, momentum is invisible to

|ψ|

but recoverable from phase, while EEG analytic-signal experiments show that phase locking, amplitude bursts, and phase-amplitude coupling each favor different coordinate views. We also identify a benchmarking artifact on RadioML 2018.01A. Under matched-shared-trial selection, a CReLU complex model exceeds the best real baseline by 22.94 PP; under independent per-family tuning on the same data and 16-trial search space, the gap collapses to 2.46 PP. Gradient analysis traces the inflated gap to high-learning-rate first-step instability in real baselines, while complex parameter coupling distributes the loss signal more robustly. A learning-rate

\times

activation factorial confirms the failure is primarily hyperparameter-driven. Overall, CVNNs are best viewed as structured inductive biases whose gains depend on representation, symmetry, and optimization, not as universally superior architectures.