Gradient-Based Optimization

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 49

All topics

Exact information accounting for SGD methods
Sep 30, 2026Akshay BalsubramaniStochastic Gradient Descent Gradient-Based Optimization
You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming
Sep 29, 2026Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su +3Visual In-Context Learning Text-Only Adaptation
The University of Melbourne · Fudan University
Is your uncertainty map wrong, or is its target? Exact diagnostics for the Tweedie diagonal, and a gradient-free alternative
Sep 27, 2026Vicent Ribas, Anna Oliveras TousDiagnosis Diffusion Models
Eurecat · University of Barcelona
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuBlock Sparse Flash Attention Gumbel-Softmax Relaxation
Optimization over covariance matrices with a parameterized metric
Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian Optimization Hessian
Nanyang Technological University, Singapore · Microsoft India · Indian Institute of Technology Bombay, India
Algorithmic Information Dynamics of Learning: A Certified, Differentiable Complexity Controller for Grokking
Aug 14, 2026Luan Ozelim, Abicumaran Uthamacumaran, Hector ZenilLearning Dynamics Gradient-Based Optimization
Oxford Immune Algorithmics, Oxford University Innovation & London Institute for Healthcare Engineering, U.K · Department of Biomedical Computing, School of Biomedical Engineering and Imaging Sciences & King’s Institute for AI, King’s College London, U.K
TANGCO: Learning Topology-Aware Capacity Allocation for Overload-driven Cascading Failures
Aug 13, 2026Orkun Irsoy, Leman Akoglu, Osman YaganCapacity Topology
Carnegie Mellon University · Electrical and Computer Engineering · Pittsburgh, Pennsylvania, USA +1
Can Bayesian Optimization Efficiently Find a Strong Single Expert in Neural Thickets?
Aug 11, 2026Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra +2Large Language Model Training Gradient-Based Optimization
University College London · Institut Polytechnique de Paris · University of Oxford
Flow-Corrected Shape Optimization: Taming Manifold Drift in High-Dimensional 3D Models
Aug 7, 2026Emilien Seiler, Nicolas Talabot, Yingxuan You +2Shape Optimization 3D Models
CVLab, EPFL
GROM: Gradient-Free Rapid One-Shot Machine Unlearning
Aug 6, 2026Paweł Batorski, Przemysław Spurek, Paul SwobodaGradient-Based Optimization Regularization
1Heinrich Heine University Düsseldorf · 2Jagiellonian University · 3IDEAS Research Institute
Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction
Jul 29, 2026Hiroki Hamaguchi, Yuya Hikima, Hiroshi Sawada +1Gradient-Based Optimization Loss Function
Graduate School of Information Science and Technology, The University of Tokyo · Department of Industrial Engineering and Economics, Institute of Science Tokyo · Communication Science Laboratories, NTT, Inc. +1
Greedy dynamical meta-learning
Jul 27, 2026Aria YomMeta-Learning Gradient-Based Optimization
Rhodora
Grad2Fair: A Gradient-driven Approach for Graph Fairness without Demographics
Jul 16, 2026Yuchang Zhu, Zezhong Xie, Huizhe Zhang +4Algorithmic Fairness Federated Graph Learning
School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou 510007, China · Institute of Artificial Intelligence, Xiamen University, Xiamen 361005, China · School of Software Engineering, Sun Yat-sen University, Zhuhai 519082, China
Gradient-free learning of a closed-loop wall controller for turbulent drag reduction
Jul 14, 2026Giorgio Maria Cavallazzi, Miguel Pérez Cuadrado, Alfredo PinelliTurbulent Flows Closed-Loop Feedback
Department of Engineering, City St. George’s, University of London, Northampton Square, EC1V 0HB, London, UK
Learning-enabled Parameter Synthesis for Nonlinear Systems from Signal Temporal Logic
Jul 9, 2026Alex Beaudin, Hanna Krasowski, Eric Palanques-Tost +2Signal Temporal Logic Nonlinear Dynamics
University of California, Berkeley · Boston University · University of Maryland, College Park
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Jul 9, 2026Hong ZhaoBackpropagation Gradient-Based Optimization
Department of Physics, Xiamen University, Xiamen 361005, China · Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China
Compositional Dynamics in Learning and Mechanics
Jun 27, 2026David I. SpivakGradient-Based Optimization Hamiltonian
Replica Symmetry Breaking and Algorithmic Thresholds in Empirical Risk Minimization under Multi-Index Model
Jun 26, 2026Andrea Montanari, Kangjie ZhouEmpirical Risk Minimization High-Dimensional
Department of Mathematics and Department of Statistics, Stanford University · Department of Statistics, Columbia University
GBC: Gradient-Based Connections for Optimizing Multi-Agent Systems
Jun 26, 2026Xiaocheng Yang, Abdulrahman Alrabah, Dilek Hakkani-Tür +1Multi-Agent Communication Credit Assignment
University of Illinois Urbana-Champaign
Gradient-Based Learning of Parametric Engine Sound Representations for Real-Time Resynthesis and Tuning on Embedded Systems
Jun 19, 2026Robin Doerfler, Matthieu Kuntz, Clemens ZimmerCombustion Control Tuning
Impulse Audio Lab GmbH Munich, Germany
Agentic Symbolic Search: Characterizing PDEs Beyond Hand-crafted Expressions, Meshes, and Neural Networks
Jun 18, 2026Zongmin Yu, Liu YangPartial Differential Equations Bifurcation
National University of Singapore
Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model
Jun 13, 2026Siyu Chen, Beining Wu, Miao Lu +2Optimal Sample Complexity Gradient-Based Optimization
Department of Statistics and Data Science, Yale University · Department of Statistics, University of Chicago · Department of Management Science and Engineering, Stanford University +1
Directing Open-Ended Evolution in Artificial Life via Multi-Scale Path Divergence
Jun 12, 2026Mikhail Akhtyrchenko, Mikhail I. Katsnelson, Andrey UstyuzhaninEvolutionary Optimization Methods Biological
1Constructor University, Bremen, Germany. · 2Constructor Knowledge Labs, Bremen, Germany. · Institute for Molecules and Materials, Radboud University, Nijmegen, Netherlands. +1
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Parameter-Efficient Fine-Tuning Methods Zeroth-Order Optimization
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University
Compile Once, Differentiate Everywhere: A Differentiable Meta-Circular Interpreter
Jun 7, 2026Lucas ShenemanDifferentiable Optimization Gradient-Based Optimization
Institute for Interdisciplinary Data Sciences University of Idaho
Need We Teach Foundation Models What is a Generative Image? Gradient-Free Generative Artifact Detection via Analytic Spectral Adaptation
Jun 3, 2026Qiaoyu Chen, Bing ZhangFace Forgery Detection Unsupervised Detection
Harbin University of Commerce
DAGGER: Gradient-Free Construction of Transiently Amplifying Networks under Hard Connectivity Constraints
May 31, 2026James C. FergusonSparsity Gradient-Based Optimization
The African Institute for Mathematical Sciences (Muizenberg, South Africa) · the Institute of Science and Technology Austria (Klosterneuburg, Austria)
Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks
May 29, 2026Ezekiel Williams, Alexandre Payeur, Guillaume LajoieRecurrent Neural Networks Learning Dynamics
Department of Mathematics and Statistics, Universit´e de Montr´eal, Canada · Mila Qu´ebec, Canada · Imperial College London, United Kingdom
Steered Generation via Gradient-Based Optimization on Sparse Query Features
May 21, 2026Sumanta Bhattacharyya, Pedram RooshenasLinear Activation Steering Steering
Department of Computer Science University of Illinois Chicago
Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing
May 15, 2026Zeji Yi, Chaoyi Pan, Guanya Shi +1Stochastic Convex Optimization Convergence
Carnegie Mellon University
Gradient-Free Training of Spiking Neural Networks via Low-Rank Evolution Strategies
May 14, 2026Dhruv Patankar, Sachit Ramesha GowdaSpiking Neural Networks Neural Network Training
Shunya Research
Inference-Time Machine Unlearning via Gated Activation Redirection
May 12, 2026Vinícius Conte Turani, Otávio Parraga, João Vitor Boer Abitante +7Machine Unlearning Inference-Time Steering
MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Harvard University · Kunumi Institute, Brazil
Multi-Timescale Conductance Spiking Networks: A Sparse, Gradient-Trainable Framework with Rich Firing Dynamics for Enhanced Temporal Processing
May 12, 2026Alex Fulleda-Garcia, Saray Soldado-Magraner, Josep Maria Margarit-TauléSpiking Neural Networks Time-To-First-Spike
Instituto de Microelectrónica de Barcelona (IMB-CNM), Consejo Superior de Investigaciones Cientíﬁcas (CSIC) Cerdanyola del Vallès, Spain · Department of Neurobiology, University of California Los Angeles (UCLA) Los Angeles, USA · Joint Senior authors
Gradient-Free Noise Optimization for Reward Alignment in Generative Models
May 12, 2026Jeongsol Kim, Hongeun Kim, Jian Wang +1Reward Gradients Diffusion Alignment
KAIST AI1 · Snap Inc.2
Exact Fixed-Point Constraints in Neural-ODEs with Provable Universality
May 11, 2026Feliciano Giuseppe Pacifico, Duccio Fanelli, Lorenzo Buffoni +3Neural Ordinary Differential Equations Fixed-Point Iteration
Department of Informatics and Computer Science, University of Pisa, Italy · Department of Physics and Astronomy and INFN, University of Florence, Sesto Fiorentino, Italy
Parameter-Efficient Neuroevolution for Diverse LLM Generation: Quality-Diversity Optimization via Prompt Embedding Evolution
May 10, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLarge Language Models(Llms Neuroevolution
The University of Hong Kong, China · Stellaris AI Limited, China
Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networks
May 7, 2026Patrick Elisii, Lucas Beauchemin, Dawer JamshedReplay-Based Continual Learning Plasticity
The Vanguard Group, Inc.
Training Non-Differentiable Networks via Optimal Transport
May 3, 2026An T. LeGradient-Based Optimization Backpropagation
Center for AI Research, VinUniversity, Vietnam · Intelligent Autonomous Systems, TU Darmstadt, Germany
Bi-Level Optimization for Contact and Motion Planning in Rope-Assisted Legged Robots
Apr 29, 2026Ruben Malacarne, Ioannis Tsikelis, Enrico Mingo Hoffman +1Locomotion Motion Planning
Dipartimento di Ingegneria e Scienza dell’Informazione (DISI), University of Trento · Inria, Universit´e de Lorraine, CNRS, 54000 Nancy, France
Understanding DNNs in Feature Interaction Models: A Dimensional Collapse Perspective
Apr 29, 2026Jiancheng Wang, Mingjia Yin, Hao Wang +1High-Order Correlations Interaction Prediction
University of Science and Technology of China & State Key Laboratory of Cognitive Intelligence Hefei, China
Machine learning moment closure models for the radiative transfer equation IV: enforcing symmetrizable hyperbolicity in two dimensions
Apr 22, 2026Juntao HuangSolar Irradiance Hyperbolic Space
Department of Mathematical Sciences, University of Delaware, Newark, DE, 19716, USA.
Intentional Updates for Streaming Reinforcement Learning
Apr 21, 2026Arsalan Sharifnassab, Mohamed Elsayed, Kris De Asis +2Gradient-Based Optimization Streaming
Openmind Research Institute · Department of Computing Science, University of Alberta.
Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation
Apr 17, 2026Yide Ran, Jianwen Xie, Minghui Wang +4Readout Gradient-Based Optimization
1Stevens Institute of Technology · 4Lambda Inc. · 2Columbia University Mailman School of Public Health +1
Stochastic-Dimension Frozen Sampled Neural Network for High-Dimensional Gross-Pitaevskii Equations on Unbounded Domains
Apr 10, 2026Zhangyong Liang, Huanhuan GaoHigh-Dimensional Gradient-Based Optimization
National Center for Applied Mathematics, Tianjin University, Tianjin, 300072, China. · School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130025, China.
Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
Mar 16, 2026Yuri Kinoshita, Naoki Nishikawa, Taro ToyoizumiDiffusion-Based Dataset Distillation Dataset Distillation
The University of Tokyo, Tokyo, Japan · Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain Science, Wako, Japan · RIKEN Center for Advanced Intelligence Project, Nihonbashi, Japan
Local Learning Rules for Out-of-Equilibrium Physical Generative Models
Jun 23, 2025Cyrill Bösch, Geoffrey Roeder, Marc Serra-Garcia +1Statistical Physics Generative Models
Department of Computer Science, Princeton University, Princeton, NJ 08540, USA · AMOLF, Science Park 104, 1098 XG Amsterdam, The Netherlands
Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning
May 13, 2025Ren ZhuangChain-of-Thought Reasoning Skip
School of Information Science and Technology, Hangzhou Normal University
Koopman-informed recurrent neural networks
Oct 30, 2024Erik Lien Bolager, Ana Čukarska, Iryna Burak +2Recurrent Neural Networks Koopman Operator
Munich Data Science Institute · School of Computation, Information and Technology, Technical University of Munich, Germany · Interdisciplinary Center for Scientific Computing (IWR) +1
Delta-AI: Local objectives for amortized inference in sparse graphical models
Oct 3, 2023Jean-Pierre Falet, Hae Beom Lee, Esmeralda S. Whitammer +6Probabilistic Inference Probabilistic Graphical Models
Mila – Qu´ebec AI Institute, Universit´e de Montr´eal · Mila – Québec AI Institute, Université de Montréal

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Exact information accounting for SGD methods

Akshay Balsubramani

Stochastic Gradient Descent Gradient-Based Optimization Generalization Bounds Information-Theoretic Limits

Open paper AskPDF arXiv

As an alternative to the standard geometric analyses, we give an exact, information-theoretic analysis of stochastic gradient descent (SGD) and its variants. We show that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information. The split extends to an identity for the objective itself. Convex convergence, strict-saddle-point escape, the link between flatness and generalization, the standard learning-rate schedules, adaptive optimizers, and the noisy, momentum, heavy-tailed, and gradient-free variants of SGD each correspond to a term or a special case of this identity. We measure its terms on synthetic and real training runs. On real networks it attributes the slack of classical convergence bounds to the terms their derivations drop and separates optimizers that reach the same training loss. That separation follows the number and consistency of their steps. Its relation to which of them generalizes better differs between networks. For gradient-free SGD the identity determines how a curvature preconditioner should enter the update. The sharpness-based generalization certificate it yields, with a data-independent isotropic prior, is vacuous at network scale unless the curvature spectrum is nearly flat across all parameters.

Momentum

Related topics

Latest papers 49