Batch Loss Score for Dynamic Data Pruning

Qing Zhou; Bingxuan Zhao; Tao Yang; Hongyuan Zhang; Junyu Gao; Qi Wang

cs.LGApr 6, 2026

Batch Loss Score for Dynamic Data Pruning

Authors: Qing Zhou, Bingxuan Zhao, Tao Yang, Hongyuan Zhang, Junyu Gao, Qi Wang

Organizations: Northwestern Polytechnical University · The University of Hong Kong

Abstract

Dynamic data pruning accelerates deep learning by selectively omitting less informative samples during training. While per-sample loss is a common importance metric, obtaining it can be challenging or infeasible for complex models or loss functions, often requiring significant implementation effort. This work proposes the Batch Loss Score (BLS), a computationally efficient alternative using an Exponential Moving Average (EMA) of readily available batch losses to assign scores to individual samples. We frame the batch loss, from the perspective of a single sample, as a noisy measurement of its scaled individual loss, with noise originating from stochastic batch composition. It is formally shown that the EMA mechanism functions as a first-order low-pass filter, attenuating high-frequency batch composition noise. This yields a score approximating the smoothed and persistent contribution of the individual sample to the loss, providing a theoretical grounding for BLS as a proxy for sample importance. BLS demonstrates remarkable code integration simplicity (three-line injection) and readily adapts existing per-sample loss-based methods (one-line proxy). Its effectiveness is demonstrated by enhancing two such methods to losslessly prune 20%-50% of samples across 14 datasets, 11 tasks and 18 models, highlighting its utility and broad applicability, especially for complex scenarios where per-sample loss is difficult to access. Code is available at https://github.com/mrazhou/BLS.

Figures & tables

Figure 1: Score acquisition complexity (excluding scheduling logic). BLS: 3 line | InfoBatch: 33+ lines + intrusive modifications.

Method	Pruned %	NoCaps Val (CIDEr)				COCO
Method	Pruned %	In	Near	Out	Overall	Test
Full	-	63.2	68.8	70.2	70.5	95.2
InfoBatch	23.6	63.4	68.2	70.4	70.2	94.4
SeTa	31.7	63.4	69.7	71.4	71.5	95.3
BLS-InfoBatch	23.8	65.1	68.9	69.8	70.6	94.7
BLS-SeTa	32.0	64.3	69.4	71.0	71.2	95.1

Table 1: BLS effectively proxies per-sample loss methods (InfoBatch, SeTa) on large-scale datasets. Results show comparable or improved performance for BLS-variants across diverse vision-language and scene text recognition tasks. (a) ToCa dataset (zero-shot captioning, 3M samples). (b) MJ+ST dataset (scene text recognition, 15M samples). (c) SS1M dataset (cross-domain captioning, 3M samples). Pruned % indicates data reduction.

Method	CIFAR10			CIFAR100
Method	30%	50%	70%	30%	50%	70%
ResNet18	95.6			78.2
Random	94.6 ↓1.0	93.3 ↓2.3	90.2 ↓5.4	73.8 ↓4.4	72.1 ↓6.1	69.7 ↓8.5
GraNd-4 [ 34 ]	95.3 ↓0.3	94.6 ↓1.0	91.2 ↓4.4	74.6 ↓3.6	71.4 ↓6.8	68.8 ↓9.4
EL2N-20 [ 45 ]	95.3 ↓0.3	95.1 ↓0.5	91.9 ↓3.7	77.2 ↓1.0	72.1 ↓6.1	-
DP [ 50 ]	94.9 ↓0.7	93.8 ↓1.8	90.8 ↓4.8	77.2 ↓1.0	73.1 ↓5.1	-

Table 2: Comparison of state-of-the-art methods on CIFAR classification (ResNet18, Accuracy %) at different pruning rates. BLS effectively proxies per-sample loss methods to achieve comparable accuracy to original methods under the same pruning rate. † : adjusted epochs in original paper. ‡ : same epochs as other methods. : unattainable ratio. Random*: dynamic random. Performance relative to Full training is indicated (

\downarrow

: drop,

\uparrow

: gain).

Method	ImageNet-1K						CIFAR100
	CNN			Transformer		Mamba	CIFAR100
	R18	R50	EfficientNet	ViT	Swin	Vim	R18	R50
Full	69.5	78.6	76.1	73.3	80.0	75.7	78.2	80.6
BLS-InfoBatch	69.4 / 32.8	78.2 / 31.5	75.7 / 32.8	73.2 / 26.2	80.0 / 30.8	75.6 / 22.9	78.3 / 33.9	80.5 / 38.6
BLS-SeTa	69.5 / 31.3	78.4 / 32.3	76.0 / 31.1	73.3 / 25.2	80.0 / 44.5	75.6 / 29.4	78.4 / 40.1	80.7 / 40.7

Table 3: BLS demonstrates strong cross-architecture generalization on ImageNet-1K and CIFAR100 with CNNs, Transformers, and Mambas. Format: Accuracy / Pruned %.

Method	Pruned %	NoCaps Val (CIDEr)				COCO
Method	Pruned %	In	Near	Out	Overall	B@4	M	C	S
Full	-	58.4	63.1	65.3	65.2	27.1	24.6	91.5	18.0
BLS-InfoBatch	20.9	58.8	63.1	65.8	65.3	27.1	24.6	91.9	17.9
BLS-SeTa	28.1	58.9	63.1	65.6	65.3	27.2	24.7	92.0	18.0

Table 4: Consolidated results for various methods on different datasets and tasks. (a) ViECap on COCO dataset: Image captioning performance with 556K samples. (b) ViECap on MSR-VTT dataset: Video captioning performance with 180K samples. (c) Ada-MVS on WHU-MVS dataset: Multi-view stereo performance with 28K image-depth map samples.

Method	VAE		DDPM		DDPM-CFG
Method	Pruned %	FID	Pruned %	FID	Pruned %	FID
Full	-	35.34	-	16.38	-	14.89
BLS-InfoBatch	28.9	35.55	28.7	17.01	22.0	14.99
BLS-SeTa	34.0	35.34	30.8	16.35	22.3	14.72

Table 5: Image generation with different architectures (VAE on MNIST, DDPM on CIFAR10, DDPM with Classifier Guidance (CFG) on CIFAR10). FID: Frechet Inception Distance.

Method	Image		Text		Audio
Method	Pruned %	Acc	Pruned %	Acc	Pruned %	Acc
Full	-	61.9	-	53.7	-	64.5
BLS-InfoBatch	31.3	61.9	38.1	54.4	25.0	64.5
BLS-SeTa	47.4	61.9	49.4	54.7	28.7	64.6

Table 6: Semi-supervised learning including image classification (FixMatch on CIFAR100), text sentiment classification (FlexMatch on Yelp Review) and audio classification (Dash on ESC-50).

Method	Image Classification			Object Detection			Instance Segmentation
Method	Pruned %	Acc@1	Acc@5	Pruned %	mAP .5	mAP .5:.95	Pruned %	mAP .5	mAP .5:.95
YOLOv5n	-	58.0	84.6	-	38.2	22.1	-	29.5	16.3
BLS-InfoBatch	20.4	57.6	84.0	21.6	38.5	22.2	29.5	31.0	17.2
BLS-SeTa	35.4	58.0	84.6	25.2	38.3	22.1	36.1	31.7	17.6

Table 7: Enhancing computational efficiency of a unified YOLOv5 framework for image classification on CIFAR100, alongside object detection and instance segmentation on COCO, through BLS.

Figure 2: Empirical validation of frequency separation. Top: Average PSD for scaled signal ( $\text{Mean }\mathcal{S}_{i}$ ) and batch noise ( $\text{Mean }\mathcal{N}_{i}$ ). Bottom: Time series for a representative sample. $\mathcal{S}_{i}$ magnitude is much smaller (see inset).

Method	R18			R50
Method	Acc	Time	P %	Acc	Time	P %
Full	78.2	2.46h	-	80.6	4.10h	-
InfoBatch	78.2	1.63h	33.8	80.6	2.74h	33.0
+ w/o EMA BLS	77.9	1.65h	32.8	79.9	2.75h	32.8
+ BLS	78.3	1.63h	33.9	80.6	2.74h	33.1
SeTa	78.3	1.53h	40.0	80.7	2.53h	39.8

Table 8: Ablation study of BLS components on CIFAR100 (ResNet18/50). "P %" denotes Pruning Percentage. "Time" includes pruning cost and training time. "+ BLS" refers to the full BLS integrated method. "+ w/o EMA BLS" uses only the last batch loss for scoring active samples.

Figure 3: Effect of EMA decay factor $\alpha$ . Accuracy and Pruning Ratio are shown for ResNet18 and ResNet50.

Figure 4: Workflow Comparison: BLS Black-Box Simplicity vs. Per-Sample Loss Complexity.

Method	Overhead	R18 P/B 734.1s	R50 P/B 2122.4s
InfoBatch	0.236s	0.03%	0.01%
BLS-InfoBatch	0.253s	0.03%	0.01%
SeTa	10.001s	1.4%	0.5%
BLS-SeTa	10.021s	1.4%	0.5%

Table 9: Computational overhead of BLS with InfoBatch and SeTa (1M samples, NVIDIA RTX 3090 GPU).

Dataset Pruning

Dataset pruning selects compact, informative training subsets to reduce computation while preserving or improving model performance.

See all 22 papers

Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment
Apr 8, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +5Dataset Pruning Learning with Noisy Labels
Institute for Infocomm Research (I2R), A*STAR, Singapore · National University of Singapore · Wuhan University +1
OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework
Jun 7, 2026Chenhan Jin, Shengze Xu, Qingsong Wang +3Dataset Pruning Data-Efficient Learning
The Chinese University of Hong Kong · Xiangtan University · University of Utah +3
Data Pruning: Redundant, Problematic, and Interdependent Samples
Jun 20, 2026Leon Freese, Marthinus W. TheunissenDataset Pruning Training Data Selection
Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa

Apr 8, 2026J/K move · Enter open · S save

Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment

Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Kai Wang+4

Institute for Infocomm Research (I2R), A*STAR, Singapore · National University of Singapore · Wuhan University+1

Dataset Pruning Learning with Noisy Labels

Open paper AskPDF arXiv

Existing dynamic data pruning methods often fail under noisy-label settings, as they typically rely on per-sample loss as the ranking criterion. This could mistakenly lead to preserving noisy samples due to their high loss values, resulting in significant performance drop. To address this, we propose AlignPrune, a noise-robust module designed to enhance the reliability of dynamic pruning under label noise. Specifically, AlignPrune introduces the Dynamic Alignment Score (DAS), which is a loss-trajectory-based criterion that enables more accurate identification of noisy samples, thereby improving pruning effectiveness. As a simple yet effective plug-and-play module, AlignPrune can be seamlessly integrated into state-of-the-art dynamic pruning frameworks, consistently outperforming them without modifying either the model architecture or the training pipeline. Extensive experiments on five widely-used benchmarks across various noise types and pruning ratios demonstrate the effectiveness of AlignPrune, boosting accuracy by up to 6.3% over state-of-the-art baselines. Our results offer a generalizable solution for pruning under noisy data, encouraging further exploration of learning in real-world scenarios. Code is available at: https://github.com/leonqin430/AlignPrune.

Abstract

Dataset Pruning

Explore similar work

Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment

OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework

Data Pruning: Redundant, Problematic, and Interdependent Samples