Zeroth-Order Optimization

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 33

All topics

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
Oct 1, 2026Cristian McGee, El Houcine Bergou, Aritra DuttaZeroth-Order Optimization Zeroth-Order
University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco
HO-FL: Hybrid-Order Federated Learning for Heterogeneous Edge Devices
Sep 30, 2026Qiyuan Chen, Xian Wu, Yanan Ma +1Federated Learning Edge Devices
University of Hong Kong · City University of Hong Kong
TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Zeroth-Order Optimization Zeroth-Order
Institute for Intelligent Networked Systems, Northeastern University London, UK
Adaptively Incorporating Directional Hints into Zeroth-Order Optimization
Sep 8, 2026Alexander Ryabchenko, Jian Qian, Wenlong MouZeroth-Order Optimization Zeroth-Order
University of Toronto and Vector Institute · University of Hong Kong
Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
Aug 12, 2026Junming Zhang, Shuyu Yin, Peilin Liu +2Test-Time Adaptation Zeroth-Order Optimization
Shanghai Jiao Tong University, China
Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents
Aug 10, 2026Bingzhen Liu, Xiaomeng Fan, Yuwei Wu +4Self-Evolution Large Language Model Agents
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Alaya Lab
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models
Aug 4, 2026Lele Zheng, Weifeng Kong, Xinyi Zhang +3Zeroth-Order Optimization Continual Fine-Tuning
School of Computer Science and Technology, Xidian University
Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination
Jul 31, 2026Jingwen Fu, Zhen Liu, Yuhan Liu +2Zeroth-Order Optimization Feedback
1Zhongguancun Academy · 2Zhongguancun Institute of Artificial Intelligence · 3National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University +1
QScheduler: Adaptive Gradient Sampling for Zeroth-Order On-Device Training on INT8 NPUs
Jul 21, 2026Victor Felipe Domingues Do Amaral, Pierre Demaj, Erwan Libessart +3Zeroth-Order Optimization Neural Processing Units
GeePs · STMicroelectronics, France · GeePs, CNRS, CentraleSup´elec, Universit´e Paris-Saclay, Sorbonne Universit´e, France
Cross-device Collaborative Test-time Adaptation with Zeroth-order Optimization and Model Merging
Jul 3, 2026Yu Mitsuzumi, Akisato Kimura, Yasuhiro Fujiwara +1Test-Time Adaptation Zeroth-Order Optimization
NTT, Inc. · Kyoto University
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Jul 1, 2026Xun Dong, Yibo Xu, Naigang Wang +3Zeroth-Order Optimization Zeroth-Order
University at Albany, SUNY · IBM T. J. Watson Research Center
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Parameter-Efficient Fine-Tuning Methods Zeroth-Order Optimization
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Zeroth-Order Optimization Zeroth-Order
University of California, Santa Barbara
A Tight Theory of Error Feedback Algorithms in Distributed Optimization
May 29, 2026Daniel Berg Thomsen, Adrien Taylor, Aymeric DieuleveutDecentralized Optimization Convergence
Inria, D.I. ENS, CNRS, PSL Research University, Paris, France · CMAP, École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France
Wall-Clock Complexity for Zeroth-Order Optimization with Tunable Oracle Fidelity
May 29, 2026Alexandra Suvorikova, Igor Pavlov, Artem Vasin +4Zeroth-Order Optimization Fidelity
Weierstrass Institute for Applied Analysis and Stochastics, Berlin, Germany · Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · MSU Institute for Artificial Intelligence, Moscow, Russia +3
Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
May 28, 2026Zhihao Liu, Yifan Wu, Jian Lou +3Safety Alignment Large Language Model Safety
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Sun Yat-sen University +1
Self-Improving In-Context Learning
May 22, 2026Baturay Saglam, Dionysis KalogeriasIn-Context Learning Multiple Prompt Learning Frameworks
Department of Electrical and Computer Engineering Yale University
Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
May 15, 2026Sijia Liu, Yicheng Lang, Soumyadeep Pal +6Zeroth-Order Optimization Zeroth-Order
1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA
Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization
May 14, 2026Chen Liang, Xiatao Sun, Qian Wang +1Zeroth-Order Optimization Gradient
Department of Computer Science, Yale University, New Haven, USA.
Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
May 11, 2026Yao Shu, Zilin ZhuZeroth-Order Zeroth-Order Optimization
HKUST-GZ
Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
May 11, 2026Yao Shu, Jian Mu, Zhongxiang DaiZeroth-Order Zeroth-Order Optimization
HKUST(GZ)
Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization Zeroth-Order
Department of Computer Science City University of Hong Kong
On Adaptivity in Zeroth-Order Optimization
May 5, 2026Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1Zeroth-Order Optimization Stochastic Gradient Descent
Qualcomm AI Research
Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective
May 5, 2026Zhe Li, Bicheng Ying, Zidong Liu +1Zeroth-Order Optimization Zeroth-Order
Department of Computing and Information Sciences Ph.D., Rochester Institute of Technology, Rochester, NY, USA · Google Inc., Los Angeles, CA, USA · ComboCurve Inc., Houston, TX, USA
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
May 1, 2026Zhijie Cai, Haolong Chen, Guangxu ZhuAdam Zeroth-Order Optimization
Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong-Shenzhen · Shenzhen Loop Area Institute
Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
Apr 24, 2026Kennon StewartMachine Unlearning Zeroth-Order Optimization
Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
Apr 20, 2026Fei Wang, Li Shen, Liang Ding +3Zeroth-Order Optimization Zeroth-Order
South China University of Technology · Shenzhen Campus of Sun Yat-sen University · University of Sydney +2
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo
Apr 19, 2026S. Gratton, Ph. L. TointStochastic Convex Optimization Convergence
Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.
Zeroth-Order Optimization at the Edge of Stability
Apr 16, 2026Minhak Song, Liang Zhang, Bingcong Li +3Zeroth-Order Optimization Zeroth-Order
KAIST · KRAFTON · ETH Zurich +2
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Federated Learning Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.
Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis
Dec 31, 2025Seunghoon Paik, Kangjie Zhou, Matus Telgarsky +1Gradient Descent Zeroth-Order Optimization
University of California, Berkeley · Columbia University · New York University
Power Homotopy for Zeroth-Order Non-Convex Optimizations
Nov 17, 2025Chen XuStochastic Convex Optimization Zeroth-Order Optimization
Department of Engineering, Shenzhen MSU-BIT University, China.
Mirror Descent Linearized Augmented Lagrangian Methods for Nonconvex Constrained Stochastic Zeroth-Order Optimization
Apr 13, 2025Qiankun Shi, Han Yuan, Xiao Wang +1Stochastic Convex Optimization Mirror Descent
Sun Yat-sen University, Guangzhou, 510006, China · Pengcheng Laboratory, Shenzhen, 518066, China · Sun Yat-sen University, Guangzhou, 518066, China +1

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

Cristian McGee, El Houcine Bergou, Aritra Dutta

University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco

Zeroth-Order Optimization Zeroth-Order Large Language Model Fine-Tuning Neural Network Optimization Model Fine-Tuning

Open paper AskPDF arXiv

Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework.

Momentum

Related topics

Latest papers 33