cs.LGNov 12, 2026

Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

Authors: Ziqing Wen, Xiaohui Song, Jiahuan Wang, Ping Luo, Junlin Zeng, Dongsheng Li, Tao Sun

Organizations: National University of Defense Technology. · OPPO AI center

Abstract

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight--activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

    Mar 23, 2026Shuo Wang, Ziyu Chen, Ming TangZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  2. Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

    Apr 20, 2026Fei Wang, Li Shen, Liang Ding +3Zeroth-Order OptimizationLLM Fine-Tuning

  3. GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

    Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Zeroth-Order OptimizationMemory-Efficient Training