Zeroth-Order Optimization

Follow this topic Ask about this topic

Recent momentum

-57%

3 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

2 new papers

A weekly snapshot of new work published in Zeroth-Order Optimization.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Zeroth-Order Optimization.

69 papers

Latest in Zeroth-Order Optimization

Open your feed →

AdamX: Cosine similarity meets gradient descent
Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresAdam Zeroth-Order Optimization
Adaptively Incorporating Directional Hints into Zeroth-Order Optimization
Sep 8, 2026Alexander Ryabchenko, Jian Qian, Wenlong MouZeroth-Order Optimization Optimization Landscape
A Study of Hidden-State Optimization Order in Predictive Coding Networks
Sep 1, 2026Xueyuan Li, Danilo Vasconcellos VargasPredictive Coding Singular Learning Theory
Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
Aug 12, 2026Junming Zhang, Shuyu Yin, Peilin Liu +2Test-Time Adaptation Zeroth-Order Optimization
A lower bound for stepsize-based acceleration of gradient descent
Aug 11, 2026Jianhao Ma, Yuxin ChenLarge Step Sizes Gradient Descent
Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents
Aug 10, 2026Bingzhen Liu, Xiaomeng Fan, Yuwei Wu +4Self-Evolution Large Language Model Agents
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models
Aug 4, 2026Lele Zheng, Weifeng Kong, Xinyi Zhang +3Differential Privacy Dp-Sgd
Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination
Jul 31, 2026Jingwen Fu, Zhen Liu, Yuhan Liu +2Zeroth-Order Optimization Natural Language Feedback
Variational Boosting for Physics-Informed Neural Networks
Jul 27, 2026Pavlos Protopapas, Kaylee VoConventional Physics-Informed Neural Networks Neural Network Optimization
QScheduler: Adaptive Gradient Sampling for Zeroth-Order On-Device Training on INT8 NPUs
Jul 21, 2026Victor Felipe Domingues Do Amaral, Pierre Demaj, Erwan Libessart +3Neural Processing Units Zeroth-Order Optimization
Differentially Private Natural Gradient Descent
Jul 7, 2026Pan Li, Kai Chen, Shuai Chang +3Differential Privacy Dp-Sgd
Cross-device Collaborative Test-time Adaptation with Zeroth-order Optimization and Model Merging
Jul 3, 2026Yu Mitsuzumi, Akisato Kimura, Yasuhiro Fujiwara +1Test-Time Adaptation Continual Model Merging
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Jul 1, 2026Xun Dong, Yibo Xu, Naigang Wang +3Finetuning Large Language Model Fine-Tuning
Non-Linear Strategic Classification Made Practical
Jun 26, 2026Jack Geary, Boyan Gao, Henry GoukMulticlass Classification Zeroth-Order Optimization
MMAO: A Metabolic Multi-Agent Optimizer with Endogenous Resource Allocation for Continuous and Discrete Optimization
Jun 26, 2026Jinliang Xu, Liping MaMetaheuristics Metabolic
A Zeroth-Order Deep Learning Method for Fully Nonlinear Parabolic Partial Differential Equations with Unknown Coefficients
Jun 23, 2026Yanwei Jia, Du Ouyang, Huyên Pham +1Partial Differential Equations Differentiable Programming
On the Oracle Complexity of Interpolation-Based Gradient Descent
Jun 18, 2026Dongmin Lee, William Lu, Anuran MakurGradient Descent Empirical Risk Minimization
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Parameter-Efficient Fine-Tuning Methods Finetuning
Parameter Tuning with Generalization Guarantees for GPU-Accelerated Linear Programming
Jun 7, 2026Siddharth Prasad, Dravyansh SharmaHyperparameter Mixed-Integer Linear Programming
Flatland: The Adventures of Gradient Descent with Large Step Sizes
Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Gradient Descent Large Step Sizes
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Jun 3, 2026Wanhao Yu, Ziyan Wang, Zheng Wang +7Finetuning Zeroth-Order Optimization
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Zeroth-Order Optimization Large Language Model Fine-Tuning
Stochastic convergence of parallel asynchronous adaptive first-order methods
Jun 1, 2026Serge Gratton, Philippe L. TointZeroth-Order Optimization Nonconvex
A Tight Theory of Error Feedback Algorithms in Distributed Optimization
May 29, 2026Daniel Berg Thomsen, Adrien Taylor, Aymeric DieuleveutOptimization Convergence Analysis Optimal Learner
Wall-Clock Complexity for Zeroth-Order Optimization with Tunable Oracle Fidelity
May 29, 2026Alexandra Suvorikova, Igor Pavlov, Artem Vasin +4Zeroth-Order Optimization Structural Fidelity
Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens
May 29, 2026Junbin Qiu, Zhaowei Hong, Renzhe Xu +1Hessian Matrix Zeroth-Order Optimization
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadGradient Descent Adam
Efficient Test-Time Finetuning of LLMs via Convex Reconstruction and Gradient Caching
May 28, 2026Alaa Khamis, Alaa MaaloufFinetuning Test Time
Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
May 28, 2026Zhihao Liu, Yifan Wu, Jian Lou +3Safety Alignment Large Language Model Safety
Inference-Native Zeroth-Order Optimization
May 27, 2026Zelin Li, Caiwen DingLarge Language Model Fine-Tuning LLM Inference Efficiency
Self-Improving In-Context Learning
May 22, 2026Baturay Saglam, Dionysis KalogeriasIn-Context Learning Automatic Prompt Optimization
AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
May 19, 2026Ziye Chen, Hongbin Lin, Chenyu Zhang +3Adaptive-Rank Low-Rank Adaptation Baselines Zeroth-Order Optimization
Stochastic Penalty-Barrier Methods for Constrained Machine Learning
May 18, 2026Adam Bosák, Andrii Kliachkin, Jana Lepšová +2Quadratic Slack Penalty Fairness-Aware Learning
New Insight of Variance reduce in Zero-Order Hard-Thresholding: Mitigating Gradient Error and Expansivity Contradictions
May 18, 2026Xinzhe Yuan, William de Vazelhes, Bin Gu +1Variance Reduction Zeroth-Order Optimization
Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training
May 15, 2026Yishun Lu, Junhao Zhang, Zeyu Yang +1Large Language Model Training Low-Precision Training
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourMultimodal Alignment Text Modality
Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback
May 15, 2026Pengcheng Dai, He Wang, Dongming Wang +2Multi-Agent Reinforcement Learning Reinforcement Learning From Human Feedback
Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
May 15, 2026Sijia Liu, Yicheng Lang, Soumyadeep Pal +6Zeroth-Order Optimization Low-Data Regimes
Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfe
May 14, 2026El Mahdi ChaytiZeroth-Order Optimization Stochastic
Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization
May 14, 2026Chen Liang, Xiatao Sun, Qian Wang +1Zeroth-Order Optimization Spectral Descent
Gradient-Free Noise Optimization for Reward Alignment in Generative Models
May 12, 2026Jeongsol Kim, Hongeun Kim, Jian Wang +1Reward Gradients Gradient-Based Optimization
Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
May 11, 2026Yao Shu, Zilin ZhuLow-Bit Quantization Zeroth-Order Optimization
Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
May 11, 2026Yao Shu, Jian Mu, Zhongxiang DaiZeroth-Order Optimization Catastrophic Forgetting
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Zeroth-Order Optimization Projection
Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization Muon
Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling
May 8, 2026Manojlo Vukovic, Dusan JakoveticStochastic Gradient Descent Dp-Sgd
Accelerated Relax-and-Round for Concave Coverage Problems
May 7, 2026Matthew Fahrbach, Mehraneh Liaee, Morteza ZadimoghaddamApproximation Algorithms Maximum Independent Set
Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
May 7, 2026Sayantan Choudhury, Xiaoran Cheng, Martin Takáč +2Nesterov Muon
PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization
May 7, 2026Murat Bilgehan Ertan, Xiaochen Zhu, Phuong Ha Nguyen +2Finetuning Membership Inference Attacks
Order Matters: Improving Domain Adaptation by Reordering Data
May 6, 2026Andrea Napoli, Paul WhiteUnsupervised Domain Adaptation Domain Adaptation
On Adaptivity in Zeroth-Order Optimization
May 5, 2026Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1Zeroth-Order Optimization Dp-Sgd
Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective
May 5, 2026Zhe Li, Bicheng Ying, Zidong Liu +1Zeroth-Order Optimization Neural Tangent Kernel
A Parameter-Free First-Order Algorithm for Non-Convex Optimization with $\tilde{\mkern1mu O}(ε^{-5/3})$ Global RateA Parameter-Free First-Order Algorithm for Non-Convex Optimization with \tilde{\mkern1mu O}(ε^{-5/3}) Global Rate
May 4, 2026Sichao Xiong, Sadok Jerad, Coralia CartisConvex Optimization Zeroth-Order Optimization
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
May 1, 2026Zhijie Cai, Haolong Chen, Guangxu ZhuAdam Zeroth-Order Optimization
From Cursed to Competitive: Closing the ZO-FO Gap via Input-to-State Stability
Apr 28, 2026Amir Ali Farzin, Philipp Braun, Iman ShamesZeroth-Order Optimization Lyapunov Function
Inference of Online Newton Methods with Nesterov's Accelerated Sketching
Apr 25, 2026Haoxuan Wang, Xinchen Du, Sen NaNesterov Newton
Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
Apr 24, 2026Kennon StewartMachine Unlearning Zeroth-Order Optimization
Beyond ZOH: Advanced Discretization Strategies for Vision Mamba
Apr 22, 2026Fady Ibrahim, Guangjun Liu, Guanghui WangVision Foundation Models State Space Models
Generalization at the Edge of Stability
Apr 21, 2026Mario Tuci, Caner Korkmaz, Umut Şimşekli +1Generalization Bounds Hessian Eigenvalue
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
Apr 20, 2026Fei Wang, Li Shen, Liang Ding +3Zeroth-Order Optimization Universal

Sep 10, 2026 · cs.LGJ/K move · Enter open · S save

AdamX: Cosine similarity meets gradient descent

Francisco Caldas, Ruben Belo, Cláudia Soares

We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.

Adam Zeroth-Order Optimization Adopd Parameter-Efficient Adaptation Gradient Descent Cosine Similarity Benchmark Datasets

Open paper Ask about it PDF