Adaptive Optimizers

Ask about this topic

Momentum

3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21
View on map

Latest papers 46

All topics
CardsList

  1. OptiCom : A Unified Framework for State-Conditioned Composition in LLM-Driven Optimization

    Sep 29, 2026Chenxing Wei, Sichen Liu, Lizhao Liu +6Optimization ModelingAdaptive Optimizers

    School of Computing and Data Science, The University of Hong Kong · Shenzhen Loop Area Institute · ByteDance +3

  2. Second-Moment Stochastic Approximation Methods

    Sep 29, 2026Tao Jiang, Lin XiaoStochastic ApproximationAdaptive Optimizers

    Fundamental AI Research (FAIR), Meta Superintelligence Labs

  3. Rotated Manifold Optimization for Low-Rank Adaptation

    Sep 28, 2026Yuhui Ding, Javier Zazo, James HensmanLow-Rank StructureAdaptive Optimizers

    ETH Zurich · Microsoft Research Cambridge

  4. Bridge of ΨΨ's: Quantum Circuit Optimization with Schrödinger Bridges

    Sep 22, 2026Lino S. Hofstetter, Lia Yeh, Prakash MuraliQuantum CircuitsSchrödinger Bridges

    ETH Zürich Zürich, Switzerland · University of Cambridge Cambridge, United Kingdom

  5. Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

    Sep 14, 2026Zihan Zhu, Zhehang Du, Xuyang Chen +5Low-Rank StructureAdaptive Optimizers

    University of Pennsylvania · DRW Associates LLC · University of Chicago

  6. Frontier LLMs are effective batch optimizers: Assessing reasoning models in continuous and discrete settings

    Sep 2, 2026Frank Hu, Shriram Chennakesavalu, David GraffAdaptive OptimizersOptimization Modeling

    Prescient Design, Genentech South San Francisco, CA, USA

  7. RLMOpt: Adaptive Prompt Optimization via Recursive Language Models

    Aug 11, 2026Subhash Bangalore Satheesha, Nirvik Pande, Deepthi Duddempudi +1Automatic Prompt OptimizationPrompt Engineering

    Autonomize AI · Carnegie Mellon University

  8. The Loss Does Not See the Basis, but Adam Does

    Aug 5, 2026Devender SinghLow-Rank StructureAdam

    Department of Computer Science Memorial University of Newfoundland

  9. AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

    Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Adaptive OptimizersNeural Network Optimization

  10. Guiding Posterior Exploration with Optimizer-Derived Geometry

    Jul 28, 2026Moritz Schlager, Emanuel Sommer, Thomas Möllenhoff +1Adaptive SamplingBayesian Neural Networks

    TUM, MCML · LMU Munich, MCML · RIKEN AIP

  11. Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization

    Jul 27, 2026Haoyue Liu, Xiaoyu Ma, Ye Chen +2Automatic Prompt OptimizationPrompt Engineering

    The Chinese University of Hong Kong, Shenzhen, China · Xi’an Jiaotong University, China · Peking University, China +1

  12. Hyperball May Not Be a Free Lunch

    Jul 24, 2026Yihao Xiao, Jialong Sun, Zitian Gao +5BatchAdaptive Optimizers

    1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2

  13. Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

    Jul 21, 2026Nuemaan MalikAdaptive OptimizersMixture-Of-Experts

    Independent Researcher

  14. Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

    Jul 15, 2026Wenxiao Wang, Priyatham Kattakinda, Soheil FeiziAgentic OptimizationReplay-Based Continual Learning

  15. Causal Optimizer Interaction Calculus: Hidden Geometric Relaxation and Identifiable Interventions

    Jul 8, 2026Zavier LiShape OptimizationAdaptive Optimizers

    Xidian University Xi’an, China

  16. Efficient Long-Horizon Learning for Learned Optimization

    Jul 7, 2026Xiaolong Huang, Benjamin Thérien, James Harrison +1Meta-LearningAdaptive Optimizers

    Mila - Quebec AI Institute · Concordia University · Université de Montréal +1

  17. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    Jul 7, 2026Noel ThomasLow-Rank StructureAdaptive Optimizers

    Mohamed bin Zayed University of Artificial Intelligence

  18. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    Jul 4, 2026Siyuan Li, Jiabao Pan, Yumou Liu +9Adaptive OptimizersTaxonomy

    Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4

  19. Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search

    Jun 25, 2026Ping Liu, Qianqi Shen, Jianqiang Shen +11JobsReinforcement Learning From Human Feedback

    LinkedIn Corporation, Mountain View, CA, USA

  20. Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

    Jun 24, 2026Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson +1Adaptive OptimizersDecoupling

    Machine Learning and Optimization Lab, EPFL

  21. ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

    Jun 23, 2026Taicheng Guo, Haomin Zhuang, Kehan Guo +4General Grid Search FrameworkAgentic Optimization

    University of Notre Dame United States

  22. Fast and Slow Variational Continual Learning

    Jun 22, 2026Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan +3Replay-Based Continual LearningAdaptive Optimizers

    Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany +3

  23. Muown Implicitly Performs Angular Step-size Decay

    Jun 22, 2026Florian Hübler, Kai Lion, Antonio Orvieto +1MuonAdaptive Optimizers

    Department of Computer Science · ETH Zurich, Switzerland · ELLIS Institute Tübingen, MPI-IS +1

  24. MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

    Jun 16, 2026Da Chang, Ganzhao YuanAdaptive OptimizersStochastic Optimization

    Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences +1

  25. Theoretical Grounding of Out-Of-Distribution Detection With Reinforcement Learning Optimizer

    Jun 16, 2026Salimeh Sekeh, Xin ZhangOut-Of-Distribution DetectionOff-Policy Learning

    San Diego State University

  26. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

    Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Adaptive OptimizersWeight Decay

    †Stanford University. · ‡Princeton University. · §Tsinghua University.

  27. Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

    Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Parameter-Efficient Fine-Tuning MethodsZeroth-Order Optimization

    1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University

  28. FOGO: Forgetting-aware Orthogonalization Optimizer

    Jun 9, 2026Toan Nguyen, Yang Liu, Trung Le +2Adaptive OptimizersContinual Learning

    School of Computer Science and Engineering University of New South Wales Sydney, Australia · Department of Data Science & AI Monash University Melbourne, Australia · DEVCOM Army Research Laboratory USA

  29. Large-scale empirical tuning and comparison of default optimizers for variational inference

    Jun 5, 2026Trevor Campbell, Jonathan H. Huggins, Kyurae Kim +1Variational InferenceVariational

    Department of Statistics, UBC · Department of Mathematics & Statistics, Boston University · Faculty of Computing & Statistics, Boston University +1

  30. Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling

    Jun 5, 2026Rohan ShravanMixture-Of-Experts ArchitecturesLarge Language Model Training

    The School of AI Bengaluru, India

  31. Towards Serverless Semi-Decentralized Federated Learning with Heterogeneous Optimizers

    Jun 4, 2026Su Wang, Mung Chiang, H. Vincent PoorFederated LearningHeterogeneity

    Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ, USA · Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA

  32. Beyond ℓ2\ell_2-norm and ℓ∞\ell_\infty-norm: A Curvature-Inspired ℓp\ell_p-Norm Scheme for Deep Neural Networks

    Jun 1, 2026Jianhao Xu, Zhuang YangNeural Network OptimizationAdaptive Optimizers

    School of Computer Science and Technology, Soochow University, Suzhou, 215006, China

  33. Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling

    May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Adaptive OptimizersStochastic Gradient Descent

    HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3

  34. A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm

    May 28, 2026Sakshi Kumari, Shyam Kumar M, Sushmitha PAdamAdaptive Optimizers

    #Department of Mathematics Indian Institute of Technology Patna Bihta 801 106 India · *Department of Mechanical Engineering Indian Institute of Technology Kharagpur2026 Kharagpur IndiaMay 28

  35. Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers

    May 20, 2026Nikhil Nayak, Julia White, Urchade Zaratiana +7Large Language Model PretrainingSpectral Preconditioning

    Fastino Labs

  36. Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

    May 18, 2026Tim Tsz-Kit Lau, Weijie SuAdaptive OptimizersOptimizer Design

    University of Pennsylvania, Philadelphia, PA 19104, USA.

  37. Learn2Splat: Extending the Horizon of Learned 3DGS Optimization

    May 15, 2026Naama Pearl, Stefano Esposito, Haofei Xu +6Recent 3D Gaussian SplattingNovel View Synthesis

    University of Tübingen, Tübingen AI Center · ETH Zurich · Meta Reality Labs

  38. Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

    May 12, 2026Kexuan Shi, Hanxuan Li, Zeju Qiu +3Adaptive OptimizersSpectral Preconditioning

    1The Chinese University of Hong Kong · 2Max Planck Institute for Intelligent Systems · 3Westlake University

  39. Optimistic Dual Averaging Unifies Modern Optimizers

    May 11, 2026Thomas Pethick, Wanyun Xie, Roman Machacek +1Adaptive OptimizersFederated Averaging

    Independent Researcher · EPFL (LIONS) · University of Bern

  40. AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

    May 9, 2026Ziyun Liu, Fengmiao Bian, Jian-Feng CaiSpectral PreconditioningLow-Rank Structure

    Department of Mathematics The Hong Kong University of Science and Technology

  41. Accelerating LMO-Based Optimization via Implicit Gradient Transport

    May 7, 2026Won-Jun Jang, Si-Hyeon LeeAdaptive OptimizersGradient

    School of Electrical Engineering · Korea Advanced Institute of Science and Technology (KAIST) · Daejeon, South Korea

  42. Direct From Darwin: Deriving Advanced Optimizers From Evolutionary First Principles

    May 6, 2026Daniel GrimmerEvolutionary OptimizationEvolutionary Optimization Methods

    Philosophy, Yale University, New Haven, 06511, USA

  43. Anon: Extrapolating Adaptivity Beyond SGD and Adam

    May 4, 2026Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5Adaptive OptimizersAdam

    Wuhan University · University of Macau · University of Calgary

  44. Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers

    Apr 24, 2026Kennon StewartMachine UnlearningZeroth-Order Optimization

    Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA

  45. Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

    Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3Replay-Based Continual LearningForgetting

    School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

  46. L-SR1: Learned Symmetric-Rank-One Preconditioning

    Aug 17, 2025Gal Lifshitz, Shahar Zuler, Ori Fouks +1Spectral PreconditioningAdaptive Optimizers

    Tel Aviv University, Tel Aviv, Israel.