Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization

Maria-Eleni Sfyraki; Jun-Kun Wang

cs.LGOct 19, 2026

Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization

Authors: Maria-Eleni Sfyraki, Jun-Kun Wang

Organizations: University of California San Diego

Abstract

Training a machine learning (ML) model is an optimization process that typically requires a set of examples with ground-truth labels beforehand. Obtaining these ground truths usually requires laborious human effort and can be costly. The scalability of human labeling becomes a concern when there are a large number of data points that need to be labeled. To address this practical issue, we consider an optimization setting where a subset of samples has ground-truth labels, while the remaining data points were labeled by an ML predictor, which could potentially assign incorrect labels to those points. We propose both first-order and zeroth-order stochastic optimization methods for training on datasets consisting of a mix of true-labeled and ML-labeled samples. We further conduct comprehensive empirical studies from training a linear regression model to fine-tuning a large language model (LLM) to showcase the effectiveness of the proposed algorithms for training with ML-labeled data.

Figures & tables

Figure 2: Test accuracy (%) over training epochs for the first-order methods on CIFAR-10 with

n=64,N=512

(left),

n=64,N=1024

(middle), and

n=128,N=1024

(right). Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation. For visualization, test-accuracy curves are smoothed using a five-epoch moving average.

Figure 3: Test accuracy (%) over training steps for the zeroth-order methods on SST-2 (left) and SST-5 (right). Test accuracy is evaluated every

250

optimization steps. Solid curves denote the mean across five random seeds, and shaded regions indicate

\pm 1

standard deviation.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Test accuracy (%) over training steps for the first-order methods on SST-2 (left) and SST-5 (right). Test accuracy is evaluated every

25

optimization steps. Solid curves denote the mean across five random seeds, and shaded regions indicate

\pm 1

standard deviation.

Figure 5: Tuning parameter trajectories for the first-order methods on SST-2 (left) and SST-5 (right). Tuning parameters are recorded every

25

optimization steps. Top row : Power-tuning parameter trajectories over optimization steps. Bottom row : Power-tuning parameter trajectories together with the across-seed means of the minimum and maximum coordinatewise Vector FTL parameters. Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation.

Method	SST-2	SST-5
SGD	$1e-4$	$1e-4$
PP-SSL	$5e-5$	$1e-4$
Scalar FTL	$1e-4$	$1e-4$
Vector FTL	$1e-4$	$1e-4$

Appendix

Table 1: Learning rates selected using mean validation accuracy for each first-order method on SST-2 and SST-5.

Figure 6: Test accuracy (%) over training steps for the zeroth-order methods on SST-2 (left) and SST-5 (right). Test accuracy is evaluated every

250

optimization steps. Solid curves denote the mean across five random seeds, and shaded regions indicate

\pm 1

standard deviation.

Figure 7: Tuning parameter trajectories for the zeroth-order methods on SST-2 (left) and SST-5 (right). Tuning parameters are recorded every

250

optimization steps. Top row : Power-tuning parameter trajectories over optimization steps. Bottom row : Power-tuning parameter trajectories together with the across-seed means of the minimum and maximum coordinatewise ZO Vector FTL parameters. Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation.

Method	SST-2	SST-5
ZO	$1e-6$	$1e-6$
ZO Scalar FTL	$1e-6$	$1e-6$
ZO Vector FTL	$1e-6$	$1e-6$

Appendix

Table 2: Learning rates selected using mean validation accuracy for each zeroth-order method on SST-2 and SST-5.

n	N	SGD	PP-SSL	Scalar FTL	Vector FTL
64	128	$86.29\pm 0.08$	$86.38\pm 0.10$	$\mathbf{86.59\pm 0.05}$	$85.78\pm 0.47$
64	256	$86.29\pm 0.08$	$86.44\pm 0.05$	$\mathbf{86.63\pm 0.08}$	$86.57\pm 0.04$
64	512	$86.29\pm 0.08$	$86.43\pm 0.05$	$\mathbf{86.63\pm 0.06}$	$86.48\pm 0.17$
64	1024	$86.29\pm 0.08$	$86.43\pm 0.05$	$\mathbf{86.63\pm 0.06}$	$\mathbf{86.63\pm 0.12}$
128	256	$86.26\pm 0.07$	$86.39\pm 0.05$	$\mathbf{86.47\pm 0.06}$	$85.97\pm 0.20$
128	512	$86.26\pm 0.07$	$86.40\pm 0.04$	$\mathbf{86.53\pm 0.06}$	$86.32\pm 0.09$

Appendix

Table 3: Test accuracy (%) on CIFAR-10. For each method and batch configuration, the learning rate and training checkpoint are selected based on mean validation accuracy, and test accuracy is reported at the selected checkpoint. Results report the mean

\pm

standard error over five random seeds. For each batch configuration, the highest test accuracy is shown in bold.

Figure 8: Test accuracy (%) over training epochs for the first-order methods on CIFAR-10 with

n=64,N=512

(left),

n=64,N=1024

(middle), and

n=128,N=1024

(right). Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation. For visualization, test-accuracy curves are smoothed using a five-epoch moving average.

Figure 9: Tuning parameter trajectories for the first-order methods on CIFAR-10 with

n=64,N=512

(left),

n=64,N=1024

(middle), and

n=128,N=1024

(right). Top row : Tuning parameter trajectories over training epochs. Bottom row : Tuning parameter trajectories together with the across-seed mean of the minimum and maximum coordinatewise Vector FTL parameters at each epoch. Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation.

n	N	SGD	PP-SSL	Scalar FTL	Vector FTL
$64$	$128$	$5e-2$	$5e-2$	$5e-2$	$5e-2$
$64$	$256$	$5e-2$	$5e-2$	$5e-2$	$5e-2$
$64$	$512$	$5e-2$	$5e-2$	$5e-2$	$5e-2$
$64$	$1024$	$5e-2$	$5e-2$	$5e-2$	$5e-2$
$128$	$256$	$5e-2$	$5e-2$	$5e-2$	$5e-2$
$128$	$512$	$5e-2$	$5e-2$	$5e-2$	$5e-2$

Appendix

Table 4: Learning rates selected using mean validation accuracy for each first-order method and batch-size configuration on CIFAR-10.

Figure 10: Test accuracy (%) over training epochs for the zeroth-order methods on CIFAR-10 with

n=256,N=512

(left),

n=256,N=1024

(middle), and

n=512,N=1024

(right). Values are reported every 25 epochs. Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation.

Figure 11: Tuning parameter trajectories for the zeroth-order methods on CIFAR-10 with

n=256,N=512

(left),

n=256,N=1024

(middle), and

n=512,N=1024

(right). Top row : Tuning parameter trajectories over training epochs. Bottom row : Tuning parameter trajectories together with the across-seed mean of the minimum and maximum coordinatewise ZO Vector FTL parameters at each epoch. Values are reported every 25 epochs. Curves show the mean across five random seeds, with shaded regions indicating

\pm 1

standard deviation.

n	N	ZO	ZO Scalar FTL	ZO Vector FTL
$256$	$512$	$5e-4$	$5e-4$	$5e-4$
$256$	$1024$	$5e-4$	$5e-4$	$5e-4$
$512$	$1024$	$1e-3$	$1e-3$	$1e-3$

Appendix

Table 5: Learning rates selected using mean validation accuracy for each zeroth-order method and batch-size configuration on CIFAR-10.

Figure 12: Test MSE over training steps for the first-order methods on California Housing with

n=16,N=64

(left),

n=16,N=128

(middle), and

n=16,N=256

(right). Values are reported every 25 epochs. Curves show the mean across ten random seeds, with shaded regions indicating

\pm 1

standard deviation. For visualization, test-MSE curves are smoothed using a five-step moving average.

Figure 13: Tuning parameter trajectories for the first-order methods on California Housing with

n=16,N=64

(left),

n=16,N=128

(middle), and

n=16,N=256

\pm 1

standard deviation. For visualization, the tuning parameter curves are smoothed using a five-step moving average.

Figure 14: Test MSE over training steps for the zeroth-order methods on California Housing with

n=32,N=128

(left),

n=32,N=256

(middle), and

n=32,N=512

(right). Values are reported every 25 epochs. Curves show the mean across ten random seeds, with shaded regions indicating

\pm 1

standard deviation. For visualization, test-MSE curves are smoothed using a five-step moving average.

Figure 15: Tuning parameter trajectories for the first-order methods on California Housing with

n=32,N=128

(left),

n=32,N=256

(middle), and

n=32,N=512

\pm 1

standard deviation. For visualization, the tuning parameter curves are smoothed using a five-step moving average.

n	N	ZO	ZO Scalar FTL	ZO Vector FTL
$32$	$128$	$1e-2$	$1e-2$	$5e-3$
$32$	$256$	$1e-2$	$1e-2$	$1e-2$
$32$	$1024$	$1e-2$	$1e-2$	$1e-2$

Appendix

Table 6: Learning rates selected using mean validation accuracy for each zeroth-order method and batch-size configuration on California Housing.

Figure 16: Test Accuracy (%) over training steps for the first-order methods on Satimage with

n=16,N=64

(left),

n=16,N=128

(middle), and

n=16,N=256

(right). Values are reported every 25 epochs. Curves show the mean across ten random seeds, with shaded regions indicating

\pm 1

standard deviation. For visualization, test-MSE curves are smoothed using a five-step moving average.

Figure 17: Tuning parameter trajectories for the first-order methods on Satimage with

n=16,N=64

(left),

n=16,N=128

(middle), and

n=16,N=256

\pm 1

standard deviation. For visualization, the tuning parameter curves are smoothed using a five-step moving average.

Figure 18: Test Accuracy (%) over training steps for the zeroth-order methods on Satimage with

n=32,N=128

(left),

n=16,N=256

(middle), and

n=16,N=512

(right). Values are reported every 25 epochs. Curves show the mean across ten random seeds, with shaded regions indicating

\pm 1

standard deviation. For visualization, test-MSE curves are smoothed using a ten-step moving average.

Figure 19: Tuning parameter trajectories for the first-order methods on Satimage with $n=32,N=128$ (left), $n=16,N=256$ (middle), and $n=16,N=512$ (right). Top row : Tuning parameter trajectories over training epochs. Bottom row : Tuning parameter trajectories together with the across-seed mean of the minimum and maximum coordinatewise ZO Vector FTL parameters at each epoch. Curves show the mean across ten random seeds, with shaded regions indicating $\pm 1$ standard deviation. For visualization, the tuning parameter curves are smoothed using a ten-step moving average.

Learning with Noisy Labels

Learning with noisy labels studies how to train reliable models when annotations are incorrect, incomplete, subjective, or weakly supervised.

See all 150 papers

Sample-Adaptive Zeroth-Order Post-Training
Oct 8, 2026Haiyuan Wan, Yanyi Li, Weicheng Lin +1Zeroth-Order Optimization LLM Fine-Tuning
School of Physics, Peking University · School of Intelligence Science and Technology, Peking University
Learning to Optimize through Solver-Grounded Self-Play
Sep 28, 2026Xia Jiang, Yaoxin Wu, Chenyu Zhou +3Self-Play RL Natural Language to Optimization Modeling
Eindhoven University of Technology · Shanghai Jiaotong University
ProML: Progressive Multi-label Learning
Oct 11, 2026Yi Kou, Jieting Wang, Yuhua QianMulti-Label Classification
Institute of Big Data Science and Industry, the School of Artificial Intelligence, and the Key Laboratory of Evolutionary Science Intelligence of Shanxi Province, Shanxi University, Taiyuan 030006, Shanxi, China

Oct 8, 2026J/K move · Enter open · S save

Sample-Adaptive Zeroth-Order Post-Training

Haiyuan Wan, Yanyi Li, Weicheng Lin, Cong Fang

School of Physics, Peking University · School of Intelligence Science and Technology, Peking University

Zeroth-Order Optimization LLM Fine-Tuning Adaptive Sampling

Open paper AskPDF arXiv

Zeroth-order (ZO) optimization offers a memory-efficient alternative to backpropagation for fine-tuning large language models. However, existing ZO methods typically sample training examples uniformly, overlooking their evolving importance and leaving avoidable estimation variance. To address this limitation, we propose Sample-Adaptive Zeroth-Order Optimization (SAZO), which learns sample importance online and adaptively allocates queries to accelerate convergence. SAZO extracts gradient-norm information from existing directional differences to estimate sample importance without access to true gradients. Furthermore, to track evolving importance from sparse minibatch feedback, SAZO maintains per-example exponential moving averages that aggregate observations across updates. The method requires no additional model evaluations and only

O(N)

additional scalar state compared with MeZO. We prove that SAZO asymptotically matches the leading convergence term of the minimum-variance sampling oracle, and achieves a strictly faster convergence rate than standard methods. Extensive experiments across four model families demonstrate improved average downstream performance and up to

6\times

faster convergence to the same training loss in terms of query count.

Abstract

Learning with Noisy Labels

Explore similar work

Sample-Adaptive Zeroth-Order Post-Training

Learning to Optimize through Solver-Grounded Self-Play

ProML: Progressive Multi-label Learning