Overview

Zeroth-order optimization studies how to improve an objective using only its evaluations, making it useful when gradients are unavailable, expensive, or incompatible with black-box systems. Common approaches estimate directions with finite differences or random perturbations, then reduce cost and noise through variance reduction, adaptive sampling, low-dimensional subspaces, or structured updates. The methods are used in settings such as language-model fine-tuning, robotics, test-time adaptation, and optimization under privacy or communication limits, alongside theoretical work on convergence and query complexity. A central tension is that avoiding backpropagation can save memory and enable broader access, but noisy estimates and many required evaluations can make optimization slower or less reliable than first-order methods.

Latest papers

CardsList

99 papers13 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

    Nov 12, 2026Ziqing Wen, Xiaohui Song, Jiahuan Wang +4Zeroth-Order OptimizationLLM Fine-Tuning

  2. Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning

    Nov 7, 2026Xinjue Wang, Xiuheng Wang, Yejun Zhang +2Zeroth-Order OptimizationRiemannian Optimization

  3. Zeroth-Order Langevin Dynamics for Stochastic Nonconvex Optimization: Curvature Regularization and Non-Asymptotic Guarantees

    Nov 7, 2026Stefano Bruno, Dong-Young LimZeroth-Order OptimizationNonconvex Stochastic Optimization

  4. An Educated Guess: Deriving Statistically Aligned Gradient Estimates for Zeroth-Order Optimisation

    Nov 5, 2026Justus Huebotter, Serge Thill, Marcel van Gerven +1Zeroth-Order OptimizationNeural Network Optimization

  5. Randomized Parallel Complexity of Weakly Convex Optimization

    Nov 5, 2026Kang LiuQuery ComplexityNonconvex Optimization

  6. Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

    Nov 3, 2026Liyan Tan, Yequan Zhao, Ziyue Liu +3On-Device TrainingLLM Training

  7. ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning

    Nov 2, 2026Zhuoli Ouyang, Changxi Chi, Siyuan Li +1Zeroth-Order OptimizationLLM Fine-Tuning

  8. The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning

    Nov 2, 2026Lianmin Chen, Junbin Qiu, Chenxing Wei +2Zeroth-Order OptimizationRL Post-Training

  9. Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute

    Nov 2, 2026Mingchuan Ma, Dongqi Cai, Daqian Yang +8Language Model PretrainingZeroth-Order Optimization

  10. Lower Bounds for Smooth Nonconvex Zeroth-Order Optimization: The Price of Determinism under Low Curvature Trace

    Oct 20, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaZeroth-Order OptimizationNonconvex Optimization

  11. StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

    Oct 12, 2026Xuanlin Yang, Jia Li, Zhouchen LinZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  12. Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

    Oct 10, 2026Chuang Tong, Ling ChenLLM Fine-TuningZeroth-Order Optimization

  13. Partially Informed Stochastic Optimization via Zeroth-Order Residual Tracking

    Oct 9, 2026El Mahdi Chayti, Alireza Mirrokni, Martin JaggiStochastic OptimizationNonconvex Stochastic Optimization

  14. Sample-Adaptive Zeroth-Order Post-Training

    Oct 8, 2026Haiyuan Wan, Yanyi Li, Weicheng Lin +1Zeroth-Order OptimizationLLM Fine-Tuning

  15. Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

    Oct 1, 2026Cristian McGee, El Houcine Bergou, Aritra DuttaZeroth-Order OptimizationLLM Fine-Tuning

  16. Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment

    Sep 30, 2026Jinho Chang, Jong Chul YeDiffusion Model AlignmentZeroth-Order Optimization

  17. Parameter-Free Zeroth-Order Optimization with Ellipsoidal Sampling

    Sep 29, 2026Alexander Sholokhov, Alexander RogozinZeroth-Order OptimizationCovariance Matrix Adaptation Evolution Strategy

  18. Probe-Space Preconditioning for Fast and Stable Zero-Order Training

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  19. Scaling Zero-Order Pretraining through Model Sharding

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationLanguage Model Pretraining

  20. AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

    Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace OptimizationMemory-Efficient Fine-Tuning

  21. TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge

    Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-TuningZeroth-Order Optimization

  22. PolyStepOR: Learning to Decide Without Optimal Decisions

    Sep 26, 2026Viet The Nguyen, Gunther Gust, An Thai LeDecision-Focused LearningPredict-Then-Optimize

  23. Derivative-Free Structured Updates for Muon

    Sep 15, 2026Pengcheng XieBlack-Box OptimizationZeroth-Order Optimization

  24. Adaptively Incorporating Directional Hints into Zeroth-Order Optimization

    Sep 8, 2026Alexander Ryabchenko, Jian Qian, Wenlong MouZeroth-Order OptimizationNonconvex Optimization

  25. MpSub: A Momentum pp-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models

    Sep 7, 2026Yuyang Wang, Haoyu Yao, Pengcheng XieMemory-Efficient Fine-TuningZeroth-Order Optimization