Zeroth-Order Optimization

Ask about this topic

Overview

Zeroth-order optimization studies how to improve an objective using only its evaluations, making it useful when gradients are unavailable, expensive, or incompatible with black-box systems. Common approaches estimate directions with finite differences or random perturbations, then reduce cost and noise through variance reduction, adaptive sampling, low-dimensional subspaces, or structured updates. The methods are used in settings such as language-model fine-tuning, robotics, test-time adaptation, and optimization under privacy or communication limits, alongside theoretical work on convergence and query complexity. A central tension is that avoiding backpropagation can save memory and enable broader access, but noisy estimates and many required evaluations can make optimization slower or less reliable than first-order methods.

Latest papers

CardsList

86 papers15 papers in the last four weeks, up 275% on the four weeks before. 0.1% of all new papers.

  1. Lower Bounds for Smooth Nonconvex Zeroth-Order Optimization: The Price of Determinism under Low Curvature Trace

    Oct 20, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaZeroth-Order OptimizationNonconvex Optimization

    Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China

  2. Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization

    Oct 19, 2026Maria-Eleni Sfyraki, Jun-Kun WangLearning with Noisy LabelsZeroth-Order Optimization

    University of California San Diego

  3. On Global Joint Lower Bounds in Zeroth-Order Nonconvex Optimization

    Oct 14, 2026Yuxuan Xia, Xiao-Tong YuanZeroth-Order OptimizationNonconvex Stochastic Optimization

  4. StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

    Oct 12, 2026Xuanlin Yang, Jia Li, Zhouchen LinZeroth-Order OptimizationMemory-Efficient Fine-Tuning

    Peking University · Beijing Normal University

  5. Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

    Oct 10, 2026Chuang Tong, Ling ChenLLM Fine-TuningZeroth-Order Optimization

    State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University

  6. Partially Informed Stochastic Optimization via Zeroth-Order Residual Tracking

    Oct 9, 2026El Mahdi Chayti, Alireza Mirrokni, Martin JaggiStochastic OptimizationNonconvex Stochastic Optimization

    Machine Learning and Optimization Laboratory (MLO), EPFL

  7. Optimal Algorithms for Nonsmooth Nonconvex Stochastic Optimization with Constraints

    Oct 9, 2026Gon Buzaglo, Guy KornowskiNonconvex Stochastic OptimizationConstrained Optimization

    Princeton University · Institute for Advanced Study

  8. Sample-Adaptive Zeroth-Order Post-Training

    Oct 8, 2026Haiyuan Wan, Yanyi Li, Weicheng Lin +1Zeroth-Order OptimizationLLM Fine-Tuning

    School of Physics, Peking University · School of Intelligence Science and Technology, Peking University

  9. VLA-ZO: Fast Zeroth-Order Adaptation for Vision-Language-Action Models

    Oct 5, 2026Jaemin Kim, Jiahn Kim, Taesik GongZeroth-Order OptimizationEfficient VLA Model Inference

    UNIST

  10. Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

    Oct 1, 2026Cristian McGee, El Houcine Bergou, Aritra DuttaZeroth-Order OptimizationLLM Fine-Tuning

    University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco

  11. Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment

    Sep 30, 2026Jinho Chang, Jong Chul YeDiffusion Model AlignmentZeroth-Order Optimization

    Kim Jaechul Graduate School of AI KAIST Seoul, South Korea

  12. Parameter-Free Zeroth-Order Optimization with Ellipsoidal Sampling

    Sep 29, 2026Alexander Sholokhov, Alexander RogozinZeroth-Order OptimizationCovariance Matrix Adaptation Evolution Strategy

    Moscow Institute of Physics and Technology, Russia

  13. Probe-Space Preconditioning for Fast and Stable Zero-Order Training

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationMemory-Efficient Fine-Tuning

    Stanford University, Stanford, CA, USA.

  14. Scaling Zero-Order Pretraining through Model Sharding

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationLanguage Model Pretraining

    Stanford University

  15. AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

    Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace OptimizationMemory-Efficient Fine-Tuning

    Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1

  16. TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge

    Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-TuningZeroth-Order Optimization

    Institute for Intelligent Networked Systems, Northeastern University London, UK

  17. PolyStepOR: Learning to Decide Without Optimal Decisions

    Sep 26, 2026Viet The Nguyen, Gunther Gust, An Thai LeDecision-Focused LearningPredict-Then-Optimize

    University of Würzburg, Germany · Center for Artificial Intelligence and Data Science (CAIDAS), Germany · Center for AI Research, VinUniversity, Vietnam +1

  18. Derivative-Free Structured Updates for Muon

    Sep 15, 2026Pengcheng XieBlack-Box OptimizationZeroth-Order Optimization

    Lawrence Berkeley National Laboratory & University of California Berkeley, CA 94720, USA

  19. Adaptively Incorporating Directional Hints into Zeroth-Order Optimization

    Sep 8, 2026Alexander Ryabchenko, Jian Qian, Wenlong MouZeroth-Order OptimizationNonconvex Optimization

    University of Toronto and Vector Institute · University of Hong Kong

  20. MpSub: A Momentum pp-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models

    Sep 7, 2026Yuyang Wang, Haoyu Yao, Pengcheng XieMemory-Efficient Fine-TuningZeroth-Order Optimization

    Xi’an Jiaotong University, Xi’an, China · Lawrence Berkeley National Laboratory, Berkeley, CA, USA

  21. Coverage, Not Credit: Failure-Credit Routing of Zeroth-Order Perturbation Budgets Does Not Improve On-Pool Sample Efficiency for LLM Agents

    Aug 28, 2026Yuxu GeLLM Agent TrainingFine-Grained Credit Assignment

    University of York

  22. Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

    Aug 21, 2026Tengteng Lei, Prabodh Katti, Rashi Dutt +5Zeroth-Order OptimizationSpiking Neural Networks

    Institute for Intelligent Networked Systems (INSI), Northeastern University London, London E1 8PH, U.K. · Centre for Intelligent Information Processing Systems (CIIPS), Department of Engineering, King’s College London, London WC2R 2LS, U.K.

  23. ZoAQ: Adaptive Zeroth-Order Querying via Query-Reuse Coupling

    Aug 19, 2026Yangyang Feng, Yao ShuZeroth-Order OptimizationBlack-Box Optimization

    The Hong Kong University of Science and Technology (Guangzhou)

  24. SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

    Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Memory-Efficient Fine-TuningZeroth-Order Optimization

    Beijing Normal University · Singapore Management University · Xiaomi Inc. +2

  25. Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation

    Aug 12, 2026Junming Zhang, Shuyu Yin, Peilin Liu +2On-Device TrainingTest-Time Adaptation

    Shanghai Jiao Tong University, China

  26. Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents

    Aug 10, 2026Bingzhen Liu, Xiaomeng Fan, Yuwei Wu +4Zeroth-Order OptimizationLLM Agent Training

    Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Alaya Lab

  27. ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

    Aug 8, 2026Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1VLM AdaptationZeroth-Order Optimization

    Electrical and Computer Engineering Department, UC Santa Barbara · Computer Science Department, UC Santa Barbara

  28. Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models

    Aug 4, 2026Lele Zheng, Weifeng Kong, Xinyi Zhang +3Memory-Efficient Fine-TuningPrivacy-Preserving Language Models

    School of Computer Science and Technology, Xidian University

  29. Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models

    Aug 3, 2026Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim +1Zeroth-Order OptimizationLLM Fine-Tuning

    Chosun University · Korea University Sejong Campus

  30. Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination

    Jul 31, 2026Jingwen Fu, Zhen Liu, Yuhan Liu +2Zeroth-Order OptimizationLarge Language Model-Guided Optimization

    1Zhongguancun Academy · 2Zhongguancun Institute of Artificial Intelligence · 3National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University +1