PruneFuse: Efficient Data Selection via Weight Pruning and Network Fusion

Humaira Kousar; Hasnain Irshad Bhatti; Jaekyun Moon

cs.LGMar 27, 2026

PruneFuse: Efficient Data Selection via Weight Pruning and Network Fusion

Authors: Humaira Kousar, Hasnain Irshad Bhatti, Jaekyun Moon

Organizations: School of Electrical Engineering, KAIST · School of Electrical Engineering and Kim Jaechul Graduate School of AI, KAIST

Abstract

Efficient data selection is crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use. We introduce PruneFuse, a novel strategy that leverages pruned networks for data selection and later fuses them with the original network to optimize training. PruneFuse operates in two stages: First, it applies structured pruning to create a smaller pruned network that, due to its structural coherence with the original network, is well-suited for the data selection task. This small network is then trained and selects the most informative samples from the dataset. Second, the trained pruned network is seamlessly fused with the original network. This integration leverages the insights gained during the training of the pruned network to facilitate the learning process of the fused network while leaving room for the network to discover more robust solutions. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.

Figures & tables

Figure 1: Overview of the PruneFuse method : (1) An untrained neural network is initially pruned to form a structured, pruned network $\theta_{p}$ . (2) This pruned network $\theta_{p}$ queries the dataset to select prime candidates for annotation, similar to active learning techniques. (3) $\theta_{p}$ is then trained on these labeled samples to form the trained pruned network $\theta_{p}^{*}$ . (4) The trained pruned network $\theta_{p}^{*}$ is fused with the base model $\theta$ , resulting in a fused model. (5) The fused model is further trained on a selected subset of the data, incorporating knowledge distillation from $\theta_{p}^{*}$ . At regular intervals $T_{\text{sync}}$ , the fused model is utilized to dynamically update the pruned model for subsequent data selection.

Panel of Figure 2 — Figure 2: Evolution of training trajectories . Conceptual illustration of how Pruning $\theta$ to $\theta_{p}$ tailors the loss landscape from 2(b) to 2(b) , allowing $\theta_{p}$ to converge on an effective configuration, denoted as $\theta^{*}_{p}$ . This model, $\theta^{*}_{p}$ , is later fused with the original $\theta$ , which provides a better initialization and leads to an improved trajectory for $\theta_{F}$ to follow, as depicted in 2(c) .

Table 1: Components of PruneFuse.

		$\uparrow$ Accuracy (%)					$\downarrow$ Computation ( $\times 10^{16}$ )
	Budget ( $b$ )	10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
Method	Params	CIFAR-10
Baseline (AL)	0.85 M	80.53 $\pm$ 0.20	87.74 $\pm$ 0.15	90.85 $\pm$ 0.11	92.24 $\pm$ 0.16	93.00 $\pm$ 0.11	0.31	1.76	4.64	8.94	14.66
PruneFuse $p{=}0.5$	0.21 M	80.92 $\pm$ 0.41	88.35 $\pm$ 0.33	91.44 $\pm$ 0.15	92.77 $\pm$ 0.03	93.65 $\pm$ 0.14	0.08	0.44	1.16	2.24	3.67
PruneFuse $p{=}0.6$	0.14 M	80.58 $\pm$ 0.33	87.79 $\pm$ 0.20	90.94 $\pm$ 0.13	92.58 $\pm$ 0.31	93.08 $\pm$ 0.42	0.05	0.28	0.74	1.43	2.35
PruneFuse $p{=}0.7$	0.08 M	80.19 $\pm$ 0.45	87.88 $\pm$ 0.05	90.70 $\pm$ 0.21	92.44 $\pm$ 0.24	93.40 $\pm$ 0.11	0.03	0.16	0.42	0.80	1.32

Table 2: Performance comparison of Baseline and PruneFuse on CIFAR-10, CIFAR-100, Tiny ImageNet-200 and ImageNet-1K. This table summarizes the top-1 test accuracy of the final model (original in case of AL and Fused in PruneFuse ) and computational cost of the data selector (in terms of FLOPs) for various pruning ratios (

p

) and labeling budgets(

b

). Params corresponds to the number of parameters of the data selector model. All results use Least-Confidence sampling with

T_{\text{sync}}\!=\!0

. ResNet-56 is utilized for CIFAR-10/100, while ResNet-50 is used for Tiny-ImageNet and ImageNet-1K. Results better than the Baseline are highlighted in Bold .

Figure 3: Accuracy-cost trade-off for PruneFuse. This figure illustrates the total number of FLOPs utilized by PruneFuse for data selection, compared to the baseline Active Learning method, for

T_{\text{sync}}{=}0,1

with labeling budgets

b=10\%,30\%,50\%

. The experiments are conducted on the CIFAR-10 dataset using the ResNet-56 architecture. Subfigures (a), (b), (c), and (d) correspond to different pruning ratios of 0.5, 0.6, 0.7, and 0.8, respectively.

Method	Params	$\uparrow$ Accuracy (%)					$\downarrow$ Computation ( $\times 10^{16}$ FLOPs)
Method	Budget $(b)$	10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
Baseline (AL)	0.85 M	80.53 $\pm$ 0.20	87.74 $\pm$ 0.15	90.85 $\pm$ 0.11	92.24 $\pm$ 0.16	93.00 $\pm$ 0.11	0.31	1.76	4.64	8.94	14.66
SVP	0.26 M	80.76 $\pm$ 0.70	87.31 $\pm$ 0.56	90.77 $\pm$ 0.45	92.59 $\pm$ 0.25	92.95 $\pm$ 0.33	0.10	0.56	1.47	2.84	4.66
PruneFuse ( $T_{\text{sync}}{=}0$ )	0.21 M	80.92 $\pm$ 0.41	88.35 $\pm$ 0.33	91.44 $\pm$ 0.15	92.77 $\pm$ 0.03	93.65 $\pm$ 0.14	0.08	0.44	1.16	2.24	3.67
PruneFuse ( $T_{\text{sync}}{=}2$ )	0.21 M	80.90 $\pm$ 0.21	88.40 $\pm$ 0.46	91.55 $\pm$ 0.63	93.05 $\pm$ 0.36	93.74 $\pm$ 0.44	0.08	1.17	1.89	5.14	6.58
PruneFuse ( $T_{\text{sync}}{=}1$ )	0.21 M	81.02 $\pm$ 0.46	88.52 $\pm$ 0.37	91.76 $\pm$ 0.08	93.15 $\pm$ 0.18	93.78 $\pm$ 0.45	0.08	1.17	3.34	6.60	10.94

Table 3: Comparison with baselines: Final top-1 test accuracy and cumulative selector cost for different label budgets compared with baselines for ResNet-56 on CIFAR-10.

Figure 4: Impact of Model Fusion on PruneFuse performance: This figure compares the accuracy over epochs for different training variants within the PruneFuse framework on CIFAR-10 with ResNet-56. We compare fusion only, knowledge distillation (KD) only, fusion with KD, and training without fusion and KD. Subfigures (a), (b), and (c) correspond to

p=0.5

0.6

, and

0.7

, respectively, for

b=30\%

Method	Model Params			Selection Metric
	Data	Target	Budget	Entropy	Least	Forgetting	Moderate	CSS
	Selector	Model	$(b)$	Entropy	Conf.	Events	Moderate	CSS
Baseline	0.85M	0.85M	25%	86.13 $\pm$ 0.41	86.50 $\pm$ 0.21	86.01 $\pm$ 0.71	86.27 $\pm$ 0.65	87.21 $\pm$ 0.68
PruneFuse	0.21M	0.85M	25%	86.71 $\pm$ 0.44	86.68 $\pm$ 0.42	87.84 $\pm$ 0.09	87.63 $\pm$ 0.31	88.85 $\pm$ 0.29
Baseline	0.85M	0.85M	50%	91.41 $\pm$ 0.21	91.28 $\pm$ 0.35	93.31 $\pm$ 0.76	90.97 $\pm$ 0.55	90.68 $\pm$ 0.35
PruneFuse	0.21M	0.85M	50%	92.24 $\pm$ 0.45	92.75 $\pm$ 0.65	93.40 $\pm$ 0.19	91.08 $\pm$ 0.49	90.79 $\pm$ 0.51

Table 4: Evaluation on Coreset Selection task: Baseline vs. PruneFuse (

p=0.5

and

T_{sync}=0

) with Various Selection Metrics including Forgetting Events ( Toneva et al., 2019 ) , Moderate ( Xia et al., 2023 ) , and CSS ( Zheng et al., 2023 ) on CIFAR-10 dataset using ResNet-56 architecture.

	CIFAR-10					CIFAR-100
Method	Label Budget ( $b$ )					Label Budget ( $b$ )
	10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
Baseline $(AL)$	53.63	65.62	71.09	76.86	80.59	36.59	52.19	58.59	63.8	64.02
PruneFuse $(p=0.5)$	59.32	71.63	75.61	81.50	83.64	46.84	57.16	62.56	65.8	67.75
PruneFuse $(p=0.6)$	57.80	70.45	75.22	80.22	82.77	45.60	56.04	61.09	65.01	67.24
PruneFuse $(p=0.7)$	56.17	69.25	73.87	79.47	82.15	44.46	55.36	60.99	64.47	66.85

Table 5: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with Vision Transformers (ViT) . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

	Amazon Review Polarity					Amazon Review Full
Method	Label Budget ( $b$ )					Label Budget ( $b$ )
	10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
Baseline $(AL)$	94.13	95.06	95.54	95.73	95.71	58.46	60.65	61.50	62.20	62.43
PruneFuse $(p=0.5)$	94.66	95.45	95.71	95.87	95.87	59.45	61.28	62.02	62.66	62.84
PruneFuse $(p=0.6)$	94.62	95.38	95.69	95.82	95.88	59.28	61.14	62.08	62.62	62.81
PruneFuse $(p=0.7)$	94.47	95.43	95.71	95.83	95.84	59.42	61.05	61.98	62.48	62.85

Table 6: Performance comparison of Baseline and PruneFuse on Amazon Review Polarity Dataset and Amazon Review Full Dataset with VDCNN Architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

Method	Pruning Criteria	Label Budget ( $b$ )
Method	Pruning Criteria	10%	20%	30%	40%	50%
Baseline (AL)	-	80.53	87.74	90.85	92.24	93.00
PruneFuse $(T_{sync}=0)$	Magnitude Imp.	79.73	87.16	91.08	92.29	93.19
PruneFuse $(T_{sync}=0)$	GroupNorm Imp.	80.10	88.25	91.01	92.25	93.74
(Dynamic Pruning)	LAMP Imp.	81.51	87.45	90.64	92.41	93.25
PruneFuse $(T_{sync}=0)$	Magnitude Imp.	80.92	88.35	91.44	92.77	93.65

Table 7: Effect of Pruning techniques and Pruning criteria on PruneFuse (

p=0.5

) on CIFAR-10 with ResNet-56.

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Data Selector $(\tilde{\theta}_{P})$	78.09	84.62	88.08	89.60	90.44
Data Selector $(\theta_{P})$	79.90	87.35	90.08	91.28	91.55
Target Model $(\theta)$	81.00	88.54	91.77	93.12	93.77
$\Delta_{t}^{\mathrm{fresh}}$	2.61	3.27	3.34	3.24	3.04
$\Delta_{t}^{\mathrm{sync}}$	1.10	1.19	1.69	1.84	2.22

Table 8: Empirical Proxy–Target Alignment This table summarizes performance comparison between the pruned selector and the fused dense model across label budgets, with and without synchronization on CIFAR-10 with ResNet-56 for pruning ratio

p=0.5

. The rest of experimental setup is held same for fair comparison.

Table 9: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with ResNet-56 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

), labeling budgets (

b

), and data selection metrics.

Table 10: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with ResNet-110 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

), labeling budgets (

b

), and data selection metrics.

Table 11: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with ResNet-164 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

), labeling budgets (

b

), and data selection metrics.

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Baseline $(AL)$	14.86 $\pm$ 0.11	33.62 $\pm$ 0.52	43.96 $\pm$ 0.22	49.86 $\pm$ 0.56	54.65 $\pm$ 0.38
PruneFuse $(p=0.5)$	18.71 $\pm$ 0.21	39.70 $\pm$ 0.31	47.41 $\pm$ 0.20	51.84 $\pm$ 0.10	55.89 $\pm$ 1.21
PruneFuse $(p=0.6)$	19.25 $\pm$ 0.72	38.84 $\pm$ 0.70	47.02 $\pm$ 0.30	52.09 $\pm$ 0.29	55.29 $\pm$ 0.28
PruneFuse $(p=0.7)$	18.32 $\pm$ 0.95	39.24 $\pm$ 0.75	46.45 $\pm$ 0.58	52.02 $\pm$ 0.65	55.63 $\pm$ 0.55
PruneFuse $(p=0.8)$	18.34 $\pm$ 0.93	37.86 $\pm$ 0.42	47.15 $\pm$ 0.31	51.77 $\pm$ 0.40	55.18 $\pm$ 0.50

Table 12: Performance Comparison of Baseline and PruneFuse on Tiny ImageNet-200 with ResNet-50 architecture , including test accuracy and corresponding standard deviations. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Baseline $(AL)$	81.63	89.47	91.25	91.54	91.85
PruneFuse $(p=0.5)$	84.49	90.07	92.63	93.49	93.55
PruneFuse $(p=0.6)$	84.16	90.22	92.56	93.34	93.43
PruneFuse $(p=0.7)$	84.10	90.21	92.46	93.29	93.22

Table 13: Performance Comparison of Baseline and PruneFuse on CIFAR-10 with MobileNetV2 Architecture . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Baseline $(AL)$	53.63	65.62	71.09	76.86	80.59
PruneFuse $(p=0.5)$	59.32	71.63	75.61	81.50	83.64
PruneFuse $(p=0.6)$	57.80	70.45	75.22	80.22	82.77
PruneFuse $(p=0.7)$	56.17	69.25	73.87	79.47	82.15

Table 14: Performance Comparison of Baseline and PruneFuse on CIFAR-10 with Vision Transformers (ViT) . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Baseline $(AL)$	94.13	95.06	95.54	95.73	95.71
PruneFuse $(p=0.5)$	94.66	95.45	95.71	95.87	95.87
PruneFuse $(p=0.6)$	94.62	95.38	95.69	95.82	95.88
PruneFuse $(p=0.7)$	94.47	95.43	95.71	95.83	95.84
PruneFuse $(p=0.8)$	94.33	95.37	95.63	95.79	95.85

Table 15: Performance Comparison of Baseline and PruneFuse on Amazon Review Polarity Dataset with VDCNN Architecture . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Baseline $(AL)$	58.46	60.65	61.50	62.20	62.43
PruneFuse $(p=0.5)$	59.45	61.28	62.02	62.66	62.84
PruneFuse $(p=0.6)$	59.28	61.14	62.08	62.62	62.81
PruneFuse $(p=0.7)$	59.42	61.05	61.98	62.48	62.85
PruneFuse $(p=0.8)$	59.24	61.05	61.94	62.45	62.77

Table 16: Performance Comparison of Baseline and PruneFuse on Amazon Review Full Dataset with VDCNN Architecture . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (

p

) and labeling budgets (

b

Method	Label Budget ( $b$ )
Method	10%	20%	30%	40%	50%
Baseline $(AL)$ (ID)	48.67	58.95	65.23	72.30	73.23
PruneFuse $(p=0.5)$ (ID)	51.20	64.68	67.52	73.28	77.71
Baseline $(AL)$ (OOD)	42.95	48.56	53.62	57.27	58.32
PruneFuse $(p=0.5)$ (OOD)	44.58	52.44	54.60	58.36	63.37

Table 17: Results of CIFAR-10 (in-distribution, ID) and CIFAR-10-C (OOD corruptions) using a ResNet-56 backbone.

PruneFuse Method	Label Budget ( $b$ )
PruneFuse Method	10%	20%	30%	40%	50%
Zero initialization	79.96	87.01	90.32	91.80	92.52
Retained Initial Weights	80.61	88.22	91.19	92.75	93.66
Random re-Initialized Weights	80.92	88.35	91.44	92.77	93.65

Table 18: Initialization schemes for Fused Model. Performance comparison of initializing the remaining weights of the Fused model after fusion with pruned model via 1) retaining weights from first initialization during pruned model initialization, 2) zero weights and 3) random re-initialization (PruneFuse) on ResNet-56 with CIFAR-10 dataset.

Figure 5: Ablation Study of Fusion on PruneFuse (

p=0.5

). Experiments are performed on ResNet-56 architecture with CIFAR-10.

Figure 6: Ablation Study of Fusion on PruneFuse (

p=0.6

). Experiments are performed on ResNet-56 architecture with CIFAR-10.

Figure 7: Ablation Study of Fusion on PruneFuse (

p=0.7

). Experiments are performed on ResNet-56 architecture with CIFAR-10.

Table 19: Ablation Study of Knowledge Distillation on PruneFuse presented in 19(a) , 19(b) , and 19(c) with different architectures and datasets.

Techniques	Model	Architecture	Params	Label Budget ( $b$ )
Techniques	Model	Architecture	(Million)	10%	20%	30%	40%	50%
SVP	Data Selector	ResNet-8	0.074	77.85	83.35	85.43	86.83	86.90
SVP	Target	ResNet-20	0.26	80.18	86.34	89.22	90.75	91.88
PruneFuse	Data Selector	ResNet-20 ( $p=0.5$ )	0.066	76.58	83.41	85.83	87.07	88.06
PruneFuse	Target	ResNet-20	0.26	80.25	87.57	90.20	91.70	92.29

Table 20: Comparison of SVP and PruneFuse on Small Models.

Figure 8: Comparison of PruneFuse with SVP. Scatter plot shows final accuracy on target model against the model size for different ResNet models on CIFAR-10,

b=

50%. (a) shows ResNet-14 (with

p=0.5

and

p=0.6

) and ResNet-8 models are used as data selectors for PruneFuse and SVP, respectively. While in (b), PruneFuse utilizes ResNet20 (i.e.

p=0.5

and

p=0.6

) and SVP utilizes ResNet-8 models.

Method	Model	Architecture	Params	Label Budget ( $b$ )
Method	Model	Architecture	(Million)	10%	20%	30%	40%	50%
SVP	Data Selector	ResNet-20	0.26	81.07	86.51	89.77	91.08	91.61
SVP	Target	ResNet-56	0.85	80.76	87.31	90.77	92.59	92.95
PruneFuse	Data Selector	ResNet-56 ( $p=0.5$ )	0.21	78.62	84.92	88.17	89.93	90.31
PruneFuse	Target	ResNet-56	0.85	82.68	88.97	91.63	93.24	93.69

Table 21: Comparison with SVP.

Table 22: Ablation study of

k

on CIFAR-10 using ResNet-56 architecture and least confidence as a selection metric.

Method	Selection	Label Budget ( $b$ )			Selection	Label Budget ( $b$ )
Method	Size (k)	20%	40%	60%	Size (k)	20%	40%	60%
Baseline $(AL)$	5,000	88.51	93.04	93.83	10,000	86.92	92.51	93.81
PruneFuse	5,000	88.52	93.15	93.90	10,000	87.49	93.11	94.04

Table 23: Ablation study of

k

on Cifar-10 using ResNet-56 with

p=0.5

and

T_{sync}=1

Method	Epochs	Label Budget ( $b$ )
Method	Epochs	10%	20%	30%	40%	50%
Least Conf.	181	80.92±0.409	88.35±0.327	91.44±0.148	92.77±0.026	93.65±0.141
Least Conf.	110	80.51±0.375	87.64±0.222	90.79±0.052	92.11±0.154	93.00±0.005
Entropy	181	81.08±0.155	88.74±0.103	91.33±0.045	92.78±0.045	93.48±0.042
Entropy	110	80.51±0.401	87.46±0.416	90.97±0.116	92.2±0.108	92.88±0.264
Random	181	80.43±0.273	86.28±0.367	88.75±0.17	90.36±0.022	91.42±0.125

Table 24: Performance Comparison when Early Stopping strategy is utilized alongside PruneFuse (

p=0.5

). Experiments are performed with Resnet-56 on CIFAR-10.

Table 25: Performance Comparison of Baseline and PruneFuse presented in 25(a) , 25(b) , and 25(c) with different architectures and datasets.

Table 26: Performance Comparison of Baseline and PruneFuse (

p=0.4

) on Cifar-10 and Cifar-100 using ResNet-56 architecture.

Method	Selection Metric	Data Selector’s	Target Model’s	Accuracy
Method	Selection Metric	Params	Params	( $b=25\%$ )
Baseline	Entropy			86.13
	Least Confidence			86.50
	Forgetting Events	0.85 Million	0.85 Million	86.01
	Moderate			86.27
	CSS			87.21

Table 27: Performance Comparison of Baseline (Coreset) and PruneFuse (

p=0.5

) for Various selection metrics including Forgetting Events ( Toneva et al., 2019 ) , Moderate ( Xia et al., 2023 ) , and CSS ( Zheng et al., 2023 ) on Cifar-10 dataset using ResNet-56 architecture.

Method	Pruning Technique	Pruning Criteria	Label Budget ( $b$ )
Method	Pruning Technique	Pruning Criteria	10%	20%	30%	40%	50%
Baseline (AL)	-	-	80.53	87.74	90.85	92.24	93.00
PruneFuse		Magnitude Imp.	79.73	87.16	91.08	92.29	93.19
PruneFuse	Dynamic Pruning	GroupNorm Imp.	80.10	88.25	91.01	92.25	93.74
$(T_{sync}=0)$		LAMP Imp.	81.51	87.45	90.64	92.41	93.25
PruneFuse		Magnitude Imp.	80.92	88.35	91.44	92.77	93.65

Table 28: Effect of Pruning techniques and Pruning criteria on PruneFuse (

p=0.5

) on CIFAR-10 dataset with ResNet-56.

Datasets	Data Selectors	Training Runime
Datasets	(Selection Models)	(Minutes)
CIFAR-10	ResNet-56 (Baseline)	127.67
CIFAR-10	ResNet-56 (PruneFuse ( $p=0.5$ ))	72.55
	ResNet-56 (PruneFuse ( $p=0.8$ ))	67.23
	ResNet-18 (Baseline)	85.68
	ResNet-18 (PruneFuse ( $p=0.5$ ))	61.15

Table 29: Training Runtime of data selector network i.e. pruned network in the case of PruneFuse and dense network for baseline, for various network architectures. The reported time is the training time when the network is trained before selecting final subset of the data (

b=50\%

Datasets	Label Budget	Data Selectors	Data Selection Time	Target Model
	( $b$ )	(Training Time )	(Minutes)	(Training Time)
		(Minutes)		(Minutes)
Baseline (AL)	10%	48.80	4.43	48.80
	20%	99.23	3.50	99.23
	30%	145.32	3.15	145.32
	40%	195.38	2.72	195.38

Table 30: Detailed Training time of Baseline and PruneFuse(

p=0.5

) for TinyImageNet-200 for Resnet-50 using Least Confidence as selection metric.

Model / Selection Metric	Method with ALSE	10%	20%	30%	40%	50%
ResNet20 / Least Conf.	Baseline	80.24	86.28	89.07	90.27	91.09
	PruneFuse ( $p=0.4$ )	81.05	86.82	90.02	90.89	91.54
	PruneFuse ( $p=0.5$ )	80.72	86.65	89.79	90.79	91.51
	PruneFuse ( $p=0.6$ )	80.53	86.68	89.56	90.81	91.32
ResNet56 / Least Conf.	Baseline	80.73	88.13	90.99	92.58	93.13
	PruneFuse ( $p=0.4$ )	80.86	88.28	91.36	93.15	93.44

Table 31: Results with ResNet20 and ResNet56 on CIFAR-10 (Least Confidence & Entropy) using ALSE Jung et al. (2023) .

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 182 papers

Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks
Apr 14, 2026Amar Gahir, Varshil Patel, Shreyank N GowdaTraining Data Selection Adaptive Sampling
School of Computer Science, University of Nottingham, Nottingham, United Kingdom
Data Agent: Learning to Select Data via End-to-End Dynamic Optimization
Mar 8, 2026Suorong Yang, Fangjian Su, Hai Gan +5Data Selection Training Data Selection
National University of Singapore · Nanjing University · Nanyang Technological University
Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration
Jun 11, 2026Dongyue Wu, Zilin Guo, Xiaoyu Li +4Dataset Pruning Training Data Selection
State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Ant Group, Hangzhou, China

Apr 14, 2026J/K move · Enter open · S save

Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

Amar Gahir, Varshil Patel, Shreyank N Gowda

School of Computer Science, University of Nottingham, Nottingham, United Kingdom

Training Data Selection Adaptive Sampling Data-Efficient Learning Neural Network Optimization

Open paper AskPDF arXiv

Deep neural networks are typically trained by uniformly sampling large datasets across epochs, despite evidence that not all samples contribute equally throughout learning. Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization, but existing methods rely on fixed schedules that do not adapt during training. In this work, we propose Adaptive Data Dropout, a simple framework that dynamically adjusts the subset of training data based on performance feedback. Inspired by self-regulated learning, our approach treats data selection as an adaptive process, increasing or decreasing data exposure in response to changes in training accuracy. We introduce a lightweight stochastic update mechanism that modulates the dropout schedule online, allowing the model to balance exploration and consolidation over time. Experiments on standard image classification benchmarks show that our method reduces effective training steps while maintaining competitive accuracy compared to static data dropout strategies. These results highlight adaptive data selection as a promising direction for efficient and robust training. Code will be released.

Abstract

Training Data Selection

Explore similar work

Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

Data Agent: Learning to Select Data via End-to-End Dynamic Optimization

Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration