cs.CVApr 8, 2026

Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment

Authors: Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Kai Wang, Zheng Wang, Peng Hu, Xi Peng, Hongyuan Zhu

Organizations: Institute for Infocomm Research (I2R), A*STAR, Singapore · National University of Singapore · Wuhan University · Sichuan University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Existing dynamic data pruning methods often fail under noisy-label settings, as they typically rely on per-sample loss as the ranking criterion. This could mistakenly lead to preserving noisy samples due to their high loss values, resulting in significant performance drop. To address this, we propose AlignPrune, a noise-robust module designed to enhance the reliability of dynamic pruning under label noise. Specifically, AlignPrune introduces the Dynamic Alignment Score (DAS), which is a loss-trajectory-based criterion that enables more accurate identification of noisy samples, thereby improving pruning effectiveness. As a simple yet effective plug-and-play module, AlignPrune can be seamlessly integrated into state-of-the-art dynamic pruning frameworks, consistently outperforming them without modifying either the model architecture or the training pipeline. Extensive experiments on five widely-used benchmarks across various noise types and pruning ratios demonstrate the effectiveness of AlignPrune, boosting accuracy by up to 6.3% over state-of-the-art baselines. Our results offer a generalizable solution for pruning under noisy data, encouraging further exploration of learning in real-world scenarios. Code is available at: https://github.com/leonqin430/AlignPrune.

Figures & tables

Figure 1: Illustrations of: (a) Our Motivation: Performance comparison of static vs . dynamic pruning methods under label noise. ( upper ) Dynamic pruning is inherently more robust to noise than static pruning. Random dynamic pruning consistently outperforms random static pruning across all noise types. ( bottom ) This motivates our work to further improve dynamic methods, where our AlignPrune achieves a new state-of-the-art, consistently outperforming dynamic (InfoBatch [ 42 ] ) and static-robust (Prune4ReL [ 38 ] , labeled as ‘Static SOTA’) baselines. (b) Our Observation: Failure of loss-based pruning methods. Existing methods use a loss threshold (dashed line) that is unreliable, which could mistakenly prune many low-loss clean samples while retain high-loss noisy ones. As pruning progresses, the accumulation of noisy examples in the training loop degrades the overall model performance. (c) Our Insight: Divergent Loss Trajectories of Clean vs . Noisy Samples. We find that clean samples typically follow a smooth, monotonic decrease in loss, while noisy samples exhibit non-monotonic loss patterns, highlighting the potential of trajectory-based reference signals in separating clean from noisy data.
Figure 2: Illustration of the proposed plug-and-play AlignPrune module. Existing dynamic pruning methods [ 42 , 67 ] rank samples based on their loss values, which could mistakenly result in a final subset S(t)\mathcal{S}^{(t)} retained with noisy samples. Our method replaces the ranking score with DAS, where noisy samples tend to have low scores to be correctly pruned, resulting in a robust subset S(t)\mathcal{S}^{(t)} composed primarily of clean samples. Best viewed in color.
Noisy Type →\rightarrowCleanRealSymmetricAsymmetricMean
Pruning Method ↓\downarrow0.20.50.80.20.4Δ\Delta
Full-training78.256.171.458.639.872.463.3–
Prune Ratio ∼\sim 30%
Static Random73.8 −{-} 4.453.3 −{-} 2.867.9 −{-} 3.451.1 −{-} 7.530.3 −{-} 9.568.4 −{-} 4.057.8 −{-} 5.5-5.3
Dynamic Random [ 36 ]77.3 −{-} 0.954.7 −{-} 1.469.9 −{-} 1.458.8 +{+} 0.240.1 +{+} 0.371.5 −{-} 0.963.2 −{-} 0.1-0.6
InfoBatch [ 42 ]79.0 +{+} 0.856.1 +{+} 0.071.4 +{+} 0.059.7 +{+} 1.141.8 +{+} 2.071.9 −{-} 0.564.2 +{+} 0.9+0.6
Table 1: Classification results on CIFAR-100N with ResNet-18. Performance gaps relative to the full-training setting are indicated as superscripts. The mean Δ\Delta is computed across all noise types.
Noisy Type →\rightarrowCleanRealSymmetricAsymmetricMean
Pruning Method ↓\downarrowReal-AReal-W0.20.50.80.20.4Δ\Delta
Full-training95.690.778.391.385.465.690.685.8–
Prune Ratio ∼\sim 30%
Static Random94.6 −{-} 1.089.9 −{-} 0.775.0 −{-} 3.389.0 −{-} 2.381.2 −{-} 4.257.6 −{-} 8.087.9 −{-} 2.882.5 −{-} 3.3-3.2
Dynamic Random [ 36 ]94.8 −{-} 0.890.3 −{-} 0.377.3 −{-} 1.090.8 −{-} 0.584.2 −{-} 1.264.8 −{-} 0.891.5 +{+} 0.984.5 −{-} 1.3-0.6
InfoBatch [ 42 ]95.3 −{-} 0.391.0 +{+} 0.378.1 −{-} 0.392.0 +{+} 0.785.6 +{+} 0.167.5 +{+} 1.991.3 +{+} 0.787.0 +{+} 1.3+0.5
Table 2: Classification results on CIFAR-10N with ResNet-18. Performance gaps relative to the full-training setting are indicated as superscripts. The mean Δ\Delta is computed across all noise types.
Figure 3: Data pruning performance comparison: (a) WebVision; (b) Clothing-1M. Best viewed in color.
 Model Architecture →\rightarrowConvNeXt [ 31 ]DeiT [ 53 ]Swin [ 30 ]Mean
 Pruning Method ↓\downarrowTinyBaseSmallBaseTinyBaseΔ\Delta
CleanFull-training73.375.671.977.873.576.9–
 InfoBatch73.2 −{-} 0.175.4 −{-} 0.271.5 −{-} 0.477.8 +{+} 0.073.6 +{+} 0.177.1 +{+} 0.2-0.1
 InfoBatch + Ours73.4 +{+} 0.175.7 +{+} 0.172.1 +{+} 0.277.8 +{+} 0.073.6 +{+} 0.177.2 +{+} 0.3+0.1
NoisyFull-training66.467.364.768.965.067.8–
 InfoBatch65.8 −{-} 0.666.8 −{-} 0.564.1 −{-} 0.668.1 −{-} 0.864.3 −{-} 0.766.9 −{-} 0.9-0.7
Table 3: Classification results on ImageNet-1K across modern CNN and ViT architectures. Performance gaps relative to the full-training setting are indicated as superscripts.
Figure 4: Ablation on hyper-parameter selection on CIFAR-100N. Best viewed in color.
 StaticDynamicSeTaInfoBatchOursFull Data
Acc (%) ↑\uparrow51.854.155.756.060.756.1
Time (mins) ↓\downarrow24.724.425.823.922.546.7
Table 4: Efficiency comparison on CIFAR-100N. Results are reported with ResNet-18 under 50% prune ratio with Real noise for 200 epochs on two RTX-A6000-GPUs. “Ours” refers to AlignPrune with InfoBatch.
Reference-set Fraction →\rightarrow100%100\%10%10\%1%1\%
Reference-set Type ↓\downarrowCleanRealCleanRealCleanReal
Noisy Train-set78.953.779.054.179.053.8
Clean Validation-set79.056.878.956.379.156.0
Pseudo Clean-set (estimated by [ 19 ] )79.156.079.156.879.056.4
Pseudo Clean-set (estimated by [ 56 ] )78.656.579.256.478.755.9
Table 5: Ablation on AlignPrune with estimated pseudo clean set. We evaluate AlignPrune using pseudo clean subsets selected by SmallL [ 19 ] and Moderate [ 56 ] as reference sets. A random subset from noisy training set is also included for comparison.
Figure 5: Ablation on AlignPrune with varying reference set sizes. Reference set size is reduced from 100%100\% to 0.1%0.1\% of the original one. AlignPrune remains effective even with minimal clean supervision. Best viewed in color.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parametersCIFAR-100NCIFAR-10NWebVisionClothing-1M
Training SetuparchitectureResNet-18ResNet-18InceptionResNetV2ResNet-50 (Pre-trained)
training epoch20020010010
batch size1281283232
optimizerLARSLARSSGDSGD
learning rate5.25.620.020.002
lr schedulerOneCycleOneCycleMultiStep-50thMultiStep-5th
Appendix
Table A : Summary of the experimental setups and hyper-parameters on CIFAR-100N, CIFAR-10N, WebVision and Clothing-1M datasets.
Noisy Type →\rightarrowCleanRealSymmetricAsymmetricMean
Pruning Method ↓\downarrow0.20.50.80.20.4Δ\Delta
Full-training78.256.171.458.639.872.463.3–
Prune Ratio ∼\sim 30%
Static Random73.8 −{-} 4.453.3 −{-} 2.867.9 −{-} 3.451.1 −{-} 7.530.3 −{-} 9.568.4 −{-} 4.057.8 −{-} 5.5-5.3
SmallL [ 19 ]71.3 −{-} 6.958.0 +{+} 1.969.2 −{-} 2.251.9 −{-} 6.713.7 −{-} 26.168.2 −{-} 4.257.0 −{-} 6.2-7.2
Margin [ 7 ]75.4 −{-} 2.848.0 −{-} 8.159.2 −{-} 12.215.9 −{-} 42.76.3 −{-} 33.557.3 −{-} 15.142.1 −{-} 21.2-19.4
Appendix
Table B : Extended classification results on CIFAR-100N with ResNet-18. Performance gaps relative to the full-training setting are indicated as superscripts. Static pruning baselines are highlighted with gray. The mean Δ\Delta is computed across all noise types. † indicates Prune4ReL with standard CE loss to ensure fair comparison on pure pruning methods.
Noisy Type →\rightarrowCleanRealSymmetricAsymmetricMean
Pruning Method ↓\downarrowReal-AReal-W0.20.50.80.20.4Δ\Delta
Full-training95.690.778.391.385.465.690.685.8–
Prune Ratio ∼\sim 30%
Static Random94.6 −{-} 1.089.9 −{-} 0.775.0 −{-} 3.389.0 −{-} 2.381.2 −{-} 4.257.6 −{-} 8.087.9 −{-} 2.882.5 −{-} 3.3-3.2
SmallL [ 19 ]88.9 −{-} 6.787.2 −{-} 3.578.2 −{-} 0.189.3 −{-} 2.072.1 −{-} 13.324.0 −{-} 41.687.4 −{-} 3.368.9 −{-} 16.9-10.9
Margin [ 7 ]94.5 −{-} 1.188.9 −{-} 1.864.6 −{-} 13.785.4 −{-} 5.915.0 −{-} 70.512.0 −{-} 53.679.8 −{-} 10.847.6 −{-} 38.2-24.4
Appendix
Table C : Extended classification results on CIFAR-10N with ResNet-18. Performance gaps relative to the full-training setting are indicated as superscripts. Static pruning methods are highlighted with gray. The mean Δ\Delta is computed across all noise types. † indicates Prune4ReL with standard CE loss to ensure fair comparison on pure pruning methods.
Noisy Type →\rightarrowRealSymmetricAsymmetricMean
Pruning Method ↓\downarrow0.20.50.80.20.4Δ\Delta
Full-training56.171.458.639.872.463.3–
Prune Ratio ∼\sim 30%
Prune4ReL [ 38 ]64.4 +{+} 8.373.0 +{+} 1.768.2 +{+} 9.622.3 −{-} 17.572.5 +{+} 0.165.9 +{+} 2.6+0.8
InfoBatch [ 42 ]66.5 +{+} 10.474.0 +{+} 2.668.9 +{+} 10.321.5 −{-} 18.374.6 +{+} 2.271.7 +{+} 8.4+2.6
InfoBatch + Ours67.5 +{+} 11.476.3 +{+} 5.070.4 +{+} 11.832.5 −{-} 7.375.8 +{+} 3.572.9 +{+} 9.6+5.7
Appendix
Table D : Classification results of re-labeling integration on CIFAR-100N with ResNet-18. We combine AlignPrune with SOP+ [ 29 ] and compare it against re-labeling-augmented pruning baselines. Performance gaps relative to the full-training setting are indicated as superscripts. The mean Δ\Delta is computed across all noise types.
Noisy Type →\rightarrowRealSymmetricAsymmetricMean
Pruning Method ↓\downarrowReal-AReal-W0.20.50.80.20.4Δ\Delta
Full-training90.778.391.385.465.690.685.8–
Prune Ratio ∼\sim 30%
Prune4ReL [ 38 ]94.0 +{+} 3.489.7 +{+} 11.494.2 +{+} 2.991.1 +{+} 5.756.5 −{-} 9.294.0 +{+} 3.492.0 +{+} 6.2+3.4
InfoBatch [ 42 ]94.8 +{+} 4.191.1 +{+} 12.895.2 +{+} 3.994.0 +{+} 8.666.8 +{+} 1.294.7 +{+} 4.193.2 +{+} 7.4+6.0
InfoBatch + Ours95.1 +{+} 4.491.7 +{+} 13.495.8 +{+} 4.594.6 +{+} 9.170.7 +{+} 5.195.5 +{+} 4.993.5 +{+} 7.8+7.0
Appendix
Table E : Classification results of re-labeling integration on CIFAR-10N with ResNet-18. We combine AlignPrune with SOP+ [ 29 ] and compare it against re-labeling-augmented pruning baselines. Performance gaps relative to the full-training setting are indicated as superscripts. The mean Δ\Delta is computed across all noise types.
 InfoBatchOurs
Avg. pruning ratio per epoch28.77%32.94%
Appendix
Table F : Average ratio of pruned samples per epoch. Results are reported with ResNet-18 under same prune probability.
Noisy Type →\rightarrowCleanRealSymmetricAsymmetric
Window Size NN ↓\downarrow0.20.50.80.20.4
277.355.270.358.840.371.463.2
378.055.970.759.240.871.263.1
4 - 50 (avg ± std)78.9 ± 0.256.5 ± 0.371.3 ± 0.360.4 ± 0.341.4 ± 0.372.2 ± 0.264.3 ± 0.3
Appendix
Table G : Ablation on AlignPrune with varying window sizes. We evaluate AlignPrune using window sizes ranging from 2 to 50, aiming to validate the necessity of such trajectory window design.
Figure A : Ablation on ρ\rho : Accuracy (Left); Time Cost (Right).
Figure B : Comparison between calibrated InfoBatch and Ours.
Figure C : Loss and DAS curves of hard vs . noisy samples. Hard clean samples are identified as correctly labeled instances with the highest per-sample average loss across training, where Top x% Loss denotes the top x% of clean samples by average loss.
Reference-set Fraction →\rightarrow100%100\%10%10\%1%1\%
Reference-set Type ↓\downarrowCleanRealCleanRealCleanReal
Clean79.056.878.956.379.156.0
Pseudo by ELFS [ 66 ]78.856.679.156.378.956.1
Appendix
Table H : More results with estimated pseudo clean set.
Figure D : Noise ratio in the retained subset on CIFAR-100N with Real noise-label. Best viewed in color.
Noisy Type →\rightarrowCleanRealSymmetricAsymmetric
Pruning Method ↓\downarrow0.20.50.80.20.4
InfoBatch79.0 ± 0.156.1 ± 0.271.4 ± 0.259.7 ± 0.241.8 ± 0.171.9 ± 0.264.2 ± 0.2
InfoBatch + Ours79.3 ± 0.159.4 ± 0.271.8 ± 0.166.0 ± 0.241.8 ± 0.172.6 ± 0.168.0 ± 0.2
SeTa79.0 ± 0.155.6 ± 0.170.2 ± 0.159.0 ± 0.241.6 ± 0.171.4 ± 0.263.2 ± 0.2
SeTa + Ours79.3 ± 0.156.3 ± 0.170.8 ± 0.260.5 ± 0.241.6 ± 0.171.9 ± 0.164.3 ± 0.2
Appendix
Table J : Statistical significance analysis on CIFAR-100N with ResNet-18.
Noisy Type →\rightarrowCleanSymmetricPairflip
Pruning Method ↓\downarrow0.20.50.45
Full-training42.537.126.726.5
InfoBatch42.8 +{+} 0.336.7 −{-} 0.426.2 −{-} 0.627.4 +{+} 0.9
InfoBatch + Ours42.9 +{+} 0.437.9 +{+} 0.828.4 +{+} 1.727.7 +{+} 1.2
Appendix
Table K : Classification results on NEWS with 3-layer MLP.
More in this topic

Dataset Pruning

Dataset pruning selects compact, informative training subsets to reduce computation while preserving or improving model performance.

See all 22 papers

Explore similar work

CardsList

  1. Batch Loss Score for Dynamic Data Pruning

    Apr 6, 2026Qing Zhou, Bingxuan Zhao, Tao Yang +3Dataset PruningData-Efficient Learning

    Northwestern Polytechnical University · The University of Hong Kong

  2. Data Pruning: Redundant, Problematic, and Interdependent Samples

    Jun 20, 2026Leon Freese, Marthinus W. TheunissenDataset PruningTraining Data Selection

    Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa

  3. RCAP: Robust, Class-Aware, Probabilistic Dynamic Dataset Pruning

    Jun 10, 2026Atif Hassan, Swanand Khare, Jiaul H. PaikDataset PruningTraining Data Selection

    Department of Artificial Intelligence, IIT Kharagpur, Kharagpur, West Bengal, India · Department of Mathematics, IIT Kharagpur, Kharagpur, West Bengal, India