Zeroth-Order

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 33

All topics

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
Oct 1, 2026Cristian McGee, El Houcine Bergou, Aritra DuttaZeroth-Order Optimization Zeroth-Order
University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco
A Parameter-Free Zeroth-Order Method with Covariance Matrix Adaptation and Effective Dimension
Sep 29, 2026Alexander Sholokhov, Alexander RogozinZeroth-Order Low-Rank Structure
Moscow Institute of Physics and Technology, Russia
AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning
Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Large Language Model Fine-Tuning Continual Fine-Tuning
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1
TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Zeroth-Order Optimization Zeroth-Order
Institute for Intelligent Networked Systems, Northeastern University London, UK
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Preference Alignment Sequential Preference Optimization
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
Adaptively Incorporating Directional Hints into Zeroth-Order Optimization
Sep 8, 2026Alexander Ryabchenko, Jian Qian, Wenlong MouZeroth-Order Optimization Zeroth-Order
University of Toronto and Vector Institute · University of Hong Kong
SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces
Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Zeroth-Order Large Language Model Fine-Tuning
Beijing Normal University · Singapore Management University · Xiaomi Inc. +2
Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
Aug 12, 2026Junming Zhang, Shuyu Yin, Peilin Liu +2Test-Time Adaptation Zeroth-Order Optimization
Shanghai Jiao Tong University, China
A lower bound for stepsize-based acceleration of gradient descent
Aug 11, 2026Jianhao Ma, Yuxin ChenStep Accuracy Gradient Descent
Department of Industrial Engineering, Tsinghua University · Department of Statistics and Data Science, Wharton School, University of Pennsylvania
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Jul 1, 2026Xun Dong, Yibo Xu, Naigang Wang +3Zeroth-Order Optimization Zeroth-Order
University at Albany, SUNY · IBM T. J. Watson Research Center
Non-Linear Strategic Classification Made Practical
Jun 26, 2026Jack Geary, Boyan Gao, Henry GoukStrategic Reasoning Classifier
School of Informatics, University of Edinburgh, Edinburgh, United Kingdom · Department of Engineering Science, University of Oxford, Oxford, United Kingdom
MMAO: A Metabolic Multi-Agent Optimizer with Endogenous Resource Allocation for Continuous and Discrete Optimization
Jun 26, 2026Jinliang Xu, Liping MaMetaheuristics Agentic Optimization
Department of Disease Control and Prevention, The Seventh Medical Center of Chinese PLA General Hospital, Beijing, China
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Parameter-Efficient Fine-Tuning Methods Zeroth-Order Optimization
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University
Flatland: The Adventures of Gradient Descent with Large Step Sizes
Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Gradient Descent Step Accuracy
LMU Munich · Munich Center for Machine Learning · University of British Columbia +1
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Jun 3, 2026Wanhao Yu, Ziyan Wang, Zheng Wang +7Continual Fine-Tuning Zeroth-Order
University of North Carolina at Charlotte · University of Houston · State University of New York at Binghamton +2
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Zeroth-Order Optimization Zeroth-Order
University of California, Santa Barbara
Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens
May 29, 2026Junbin Qiu, Zhaowei Hong, Renzhe Xu +1Hessian Zeroth-Order
Hong Kong University of Science and Technology (Guangzhou) · Shanghai University of Finance and Economics
Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLarge Language Model Fine-Tuning Model Fine-Tuning
University of Minnesota
AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
May 19, 2026Ziye Chen, Hongbin Lin, Chenyu Zhang +3Zeroth-Order Model Fine-Tuning
Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback
May 15, 2026Pengcheng Dai, He Wang, Dongming Wang +2Multi-Agent Reinforcement Learning Reinforcement Learning From Human Feedback
Singapore University of Technology and Design, Singapore 487372, Singapore · Southeast University, Nanjing 211189, China · University of California, Riverside, CA 92521, USA +2
Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
May 15, 2026Sijia Liu, Yicheng Lang, Soumyadeep Pal +6Zeroth-Order Optimization Zeroth-Order
1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA
Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
May 11, 2026Yao Shu, Zilin ZhuZeroth-Order Zeroth-Order Optimization
HKUST-GZ
Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
May 11, 2026Yao Shu, Jian Mu, Zhongxiang DaiZeroth-Order Zeroth-Order Optimization
HKUST(GZ)
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Learning-Augmented Algorithms Zeroth-Order
Computer Science and Automation, Indian Institute of Science, Bengaluru · Edwardson School of Industrial Engineering, Purdue University · Department of Computer Science, IMT Atlantique
Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization Zeroth-Order
Department of Computer Science City University of Hong Kong
PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization
May 7, 2026Murat Bilgehan Ertan, Xiaochen Zhu, Phuong Ha Nguyen +2Zeroth-Order Membership Inference Attacks
CWI Amsterdam · ∗Affiliated with Vrije Universiteit Amsterdam. · MIT +1
Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective
May 5, 2026Zhe Li, Bicheng Ying, Zidong Liu +1Zeroth-Order Optimization Zeroth-Order
Department of Computing and Information Sciences Ph.D., Rochester Institute of Technology, Rochester, NY, USA · Google Inc., Los Angeles, CA, USA · ComboCurve Inc., Houston, TX, USA
From Cursed to Competitive: Closing the ZO-FO Gap via Input-to-State Stability
Apr 28, 2026Amir Ali Farzin, Philipp Braun, Iman ShamesZeroth-Order Lyapunov Function
School of Engineering · Australian National University · Canberra, ACT 2601 +3
Beyond ZOH: Advanced Discretization Strategies for Vision Mamba
Apr 22, 2026Fady Ibrahim, Guangjun Liu, Guanghui WangVision State Space Models Vision Foundation Models
Toronto Metropolitan University, 350 Victoria St, Toronto, ON M5B 2K3
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
Apr 20, 2026Fei Wang, Li Shen, Liang Ding +3Zeroth-Order Optimization Zeroth-Order
South China University of Technology · Shenzhen Campus of Sun Yat-sen University · University of Sydney +2
Zeroth-Order Optimization at the Edge of Stability
Apr 16, 2026Minhak Song, Liang Zhang, Bingcong Li +3Zeroth-Order Optimization Zeroth-Order
KAIST · KRAFTON · ETH Zurich +2
Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs
Mar 25, 2026Zhangyong Liang, Huanhuan GaoZeroth-Order Partial Differential Equations
National Center for Applied Mathematics, Tianjin University, Tianjin 300072, China. · School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130025, China
Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles
May 4, 2025Amir Ali Farzin, Yuen-Man Pun, Philipp Braun +1Nonconvex Zeroth-Order
School of Engineering Australian National University · Department of Electrical and Electronic Engineering University of Melbourne · Department of Electrical and Electronic Engineering22 University of Melbourne

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

Cristian McGee, El Houcine Bergou, Aritra Dutta

University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco

Zeroth-Order Optimization Zeroth-Order Large Language Model Fine-Tuning Neural Network Optimization Model Fine-Tuning

Open paper AskPDF arXiv

Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework.

Momentum

Related topics

Latest papers 33