Efficient data selection is crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use. We introduce PruneFuse, a novel strategy that leverages pruned networks for data selection and later fuses them with the original network to optimize training. PruneFuse operates in two stages: First, it applies structured pruning to create a smaller pruned network that, due to its structural coherence with the original network, is well-suited for the data selection task. This small network is then trained and selects the most informative samples from the dataset. Second, the trained pruned network is seamlessly fused with the original network. This integration leverages the insights gained during the training of the pruned network to facilitate the learning process of the fused network while leaving room for the network to discover more robust solutions. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.
Figures & tables
Figure 1: Overview of the PruneFuse method : (1) An untrained neural network is initially pruned to form a structured, pruned network θp . (2) This pruned network θp queries the dataset to select prime candidates for annotation, similar to active learning techniques. (3) θp is then trained on these labeled samples to form the trained pruned network θp∗ . (4) The trained pruned network θp∗ is fused with the base model θ , resulting in a fused model. (5) The fused model is further trained on a selected subset of the data, incorporating knowledge distillation from θp∗ . At regular intervals Tsync , the fused model is utilized to dynamically update the pruned model for subsequent data selection.
Figure 2: Evolution of training trajectories . Conceptual illustration of how Pruning θ to θp tailors the loss landscape from 2(b) to 2(b) , allowing θp to converge on an effective configuration, denoted as θp∗ . This model, θp∗ , is later fused with the original θ , which provides a better initialization and leads to an improved trajectory for θF to follow, as depicted in 2(c) .
Table 1: Components of PruneFuse.
↑ Accuracy (%)
↓ Computation ( ×1016 )
Budget ( b )
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Method
Params
CIFAR-10
Baseline (AL)
0.85 M
80.53 ± 0.20
87.74 ± 0.15
90.85 ± 0.11
92.24 ± 0.16
93.00 ± 0.11
0.31
1.76
4.64
8.94
14.66
PruneFuse p=0.5
0.21 M
80.92 ± 0.41
88.35 ± 0.33
91.44 ± 0.15
92.77 ± 0.03
93.65 ± 0.14
0.08
0.44
1.16
2.24
3.67
PruneFuse p=0.6
0.14 M
80.58 ± 0.33
87.79 ± 0.20
90.94 ± 0.13
92.58 ± 0.31
93.08 ± 0.42
0.05
0.28
0.74
1.43
2.35
PruneFuse p=0.7
0.08 M
80.19 ± 0.45
87.88 ± 0.05
90.70 ± 0.21
92.44 ± 0.24
93.40 ± 0.11
0.03
0.16
0.42
0.80
1.32
Table 2: Performance comparison of Baseline and PruneFuse on CIFAR-10, CIFAR-100, Tiny ImageNet-200 and ImageNet-1K. This table summarizes the top-1 test accuracy of the final model (original in case of AL and Fused in PruneFuse ) and computational cost of the data selector (in terms of FLOPs) for various pruning ratios ( p ) and labeling budgets( b ). Params corresponds to the number of parameters of the data selector model. All results use Least-Confidence sampling with Tsync=0 . ResNet-56 is utilized for CIFAR-10/100, while ResNet-50 is used for Tiny-ImageNet and ImageNet-1K. Results better than the Baseline are highlighted in Bold .
Figure 3: Accuracy-cost trade-off for PruneFuse. This figure illustrates the total number of FLOPs utilized by PruneFuse for data selection, compared to the baseline Active Learning method, for Tsync=0,1 with labeling budgets b=10%,30%,50% . The experiments are conducted on the CIFAR-10 dataset using the ResNet-56 architecture. Subfigures (a), (b), (c), and (d) correspond to different pruning ratios of 0.5, 0.6, 0.7, and 0.8, respectively.
Method
Params
↑ Accuracy (%)
↓ Computation ( ×1016 FLOPs)
Budget (b)
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Baseline (AL)
0.85 M
80.53 ± 0.20
87.74 ± 0.15
90.85 ± 0.11
92.24 ± 0.16
93.00 ± 0.11
0.31
1.76
4.64
8.94
14.66
SVP
0.26 M
80.76 ± 0.70
87.31 ± 0.56
90.77 ± 0.45
92.59 ± 0.25
92.95 ± 0.33
0.10
0.56
1.47
2.84
4.66
PruneFuse ( Tsync=0 )
0.21 M
80.92 ± 0.41
88.35 ± 0.33
91.44 ± 0.15
92.77 ± 0.03
93.65 ± 0.14
0.08
0.44
1.16
2.24
3.67
PruneFuse ( Tsync=2 )
0.21 M
80.90 ± 0.21
88.40 ± 0.46
91.55 ± 0.63
93.05 ± 0.36
93.74 ± 0.44
0.08
1.17
1.89
5.14
6.58
PruneFuse ( Tsync=1 )
0.21 M
81.02 ± 0.46
88.52 ± 0.37
91.76 ± 0.08
93.15 ± 0.18
93.78 ± 0.45
0.08
1.17
3.34
6.60
10.94
Table 3: Comparison with baselines: Final top-1 test accuracy and cumulative selector cost for different label budgets compared with baselines for ResNet-56 on CIFAR-10.
Figure 4: Impact of Model Fusion on PruneFuse performance: This figure compares the accuracy over epochs for different training variants within the PruneFuse framework on CIFAR-10 with ResNet-56. We compare fusion only, knowledge distillation (KD) only, fusion with KD, and training without fusion and KD. Subfigures (a), (b), and (c) correspond to p=0.5 , 0.6 , and 0.7 , respectively, for b=30% .
Method
Model Params
Selection Metric
Data
Target
Budget
Entropy
Least
Forgetting
Moderate
CSS
Selector
Model
(b)
Conf.
Events
Baseline
0.85M
0.85M
25%
86.13 ± 0.41
86.50 ± 0.21
86.01 ± 0.71
86.27 ± 0.65
87.21 ± 0.68
PruneFuse
0.21M
0.85M
86.71 ± 0.44
86.68 ± 0.42
87.84 ± 0.09
87.63 ± 0.31
88.85 ± 0.29
Baseline
0.85M
0.85M
50%
91.41 ± 0.21
91.28 ± 0.35
93.31 ± 0.76
90.97 ± 0.55
90.68 ± 0.35
PruneFuse
0.21M
0.85M
92.24 ± 0.45
92.75 ± 0.65
93.40 ± 0.19
91.08 ± 0.49
90.79 ± 0.51
Table 4: Evaluation on Coreset Selection task: Baseline vs. PruneFuse ( p=0.5 and Tsync=0 ) with Various Selection Metrics including Forgetting Events ( Toneva et al., 2019 ) , Moderate ( Xia et al., 2023 ) , and CSS ( Zheng et al., 2023 ) on CIFAR-10 dataset using ResNet-56 architecture.
CIFAR-10
CIFAR-100
Method
Label Budget ( b )
Label Budget ( b )
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Baseline (AL)
53.63
65.62
71.09
76.86
80.59
36.59
52.19
58.59
63.8
64.02
PruneFuse (p=0.5)
59.32
71.63
75.61
81.50
83.64
46.84
57.16
62.56
65.8
67.75
PruneFuse (p=0.6)
57.80
70.45
75.22
80.22
82.77
45.60
56.04
61.09
65.01
67.24
PruneFuse (p=0.7)
56.17
69.25
73.87
79.47
82.15
44.46
55.36
60.99
64.47
66.85
Table 5: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with Vision Transformers (ViT) . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Amazon Review Polarity
Amazon Review Full
Method
Label Budget ( b )
Label Budget ( b )
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Baseline (AL)
94.13
95.06
95.54
95.73
95.71
58.46
60.65
61.50
62.20
62.43
PruneFuse (p=0.5)
94.66
95.45
95.71
95.87
95.87
59.45
61.28
62.02
62.66
62.84
PruneFuse (p=0.6)
94.62
95.38
95.69
95.82
95.88
59.28
61.14
62.08
62.62
62.81
PruneFuse (p=0.7)
94.47
95.43
95.71
95.83
95.84
59.42
61.05
61.98
62.48
62.85
Table 6: Performance comparison of Baseline and PruneFuse on Amazon Review Polarity Dataset and Amazon Review Full Dataset with VDCNN Architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Method
Pruning Criteria
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
-
80.53
87.74
90.85
92.24
93.00
PruneFuse (Tsync=0)
Magnitude Imp.
79.73
87.16
91.08
92.29
93.19
GroupNorm Imp.
80.10
88.25
91.01
92.25
93.74
(Dynamic Pruning)
LAMP Imp.
81.51
87.45
90.64
92.41
93.25
PruneFuse (Tsync=0)
Magnitude Imp.
80.92
88.35
91.44
92.77
93.65
Table 7: Effect of Pruning techniques and Pruning criteria on PruneFuse ( p=0.5 ) on CIFAR-10 with ResNet-56.
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Data Selector (θ~P)
78.09
84.62
88.08
89.60
90.44
Data Selector (θP)
79.90
87.35
90.08
91.28
91.55
Target Model (θ)
81.00
88.54
91.77
93.12
93.77
Δtfresh
2.61
3.27
3.34
3.24
3.04
Δtsync
1.10
1.19
1.69
1.84
2.22
Table 8: Empirical Proxy–Target Alignment This table summarizes performance comparison between the pruned selector and the fused dense model across label budgets, with and without synchronization on CIFAR-10 with ResNet-56 for pruning ratio p=0.5 . The rest of experimental setup is held same for fair comparison.
Table 9: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with ResNet-56 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ), labeling budgets ( b ), and data selection metrics.
Table 10: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with ResNet-110 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ), labeling budgets ( b ), and data selection metrics.
Table 11: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with ResNet-164 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ), labeling budgets ( b ), and data selection metrics.
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
14.86 ± 0.11
33.62 ± 0.52
43.96 ± 0.22
49.86 ± 0.56
54.65 ± 0.38
PruneFuse (p=0.5)
18.71 ± 0.21
39.70 ± 0.31
47.41 ± 0.20
51.84 ± 0.10
55.89 ± 1.21
PruneFuse (p=0.6)
19.25 ± 0.72
38.84 ± 0.70
47.02 ± 0.30
52.09 ± 0.29
55.29 ± 0.28
PruneFuse (p=0.7)
18.32 ± 0.95
39.24 ± 0.75
46.45 ± 0.58
52.02 ± 0.65
55.63 ± 0.55
PruneFuse (p=0.8)
18.34 ± 0.93
37.86 ± 0.42
47.15 ± 0.31
51.77 ± 0.40
55.18 ± 0.50
Table 12: Performance Comparison of Baseline and PruneFuse on Tiny ImageNet-200 with ResNet-50 architecture , including test accuracy and corresponding standard deviations. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
81.63
89.47
91.25
91.54
91.85
PruneFuse (p=0.5)
84.49
90.07
92.63
93.49
93.55
PruneFuse (p=0.6)
84.16
90.22
92.56
93.34
93.43
PruneFuse (p=0.7)
84.10
90.21
92.46
93.29
93.22
Table 13: Performance Comparison of Baseline and PruneFuse on CIFAR-10 with MobileNetV2 Architecture . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
53.63
65.62
71.09
76.86
80.59
PruneFuse (p=0.5)
59.32
71.63
75.61
81.50
83.64
PruneFuse (p=0.6)
57.80
70.45
75.22
80.22
82.77
PruneFuse (p=0.7)
56.17
69.25
73.87
79.47
82.15
Table 14: Performance Comparison of Baseline and PruneFuse on CIFAR-10 with Vision Transformers (ViT) . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
94.13
95.06
95.54
95.73
95.71
PruneFuse (p=0.5)
94.66
95.45
95.71
95.87
95.87
PruneFuse (p=0.6)
94.62
95.38
95.69
95.82
95.88
PruneFuse (p=0.7)
94.47
95.43
95.71
95.83
95.84
PruneFuse (p=0.8)
94.33
95.37
95.63
95.79
95.85
Table 15: Performance Comparison of Baseline and PruneFuse on Amazon Review Polarity Dataset with VDCNN Architecture . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
58.46
60.65
61.50
62.20
62.43
PruneFuse (p=0.5)
59.45
61.28
62.02
62.66
62.84
PruneFuse (p=0.6)
59.28
61.14
62.08
62.62
62.81
PruneFuse (p=0.7)
59.42
61.05
61.98
62.48
62.85
PruneFuse (p=0.8)
59.24
61.05
61.94
62.45
62.77
Table 16: Performance Comparison of Baseline and PruneFuse on Amazon Review Full Dataset with VDCNN Architecture . This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios ( p ) and labeling budgets ( b ).
Method
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL) (ID)
48.67
58.95
65.23
72.30
73.23
PruneFuse (p=0.5) (ID)
51.20
64.68
67.52
73.28
77.71
Baseline (AL) (OOD)
42.95
48.56
53.62
57.27
58.32
PruneFuse (p=0.5) (OOD)
44.58
52.44
54.60
58.36
63.37
Table 17: Results of CIFAR-10 (in-distribution, ID) and CIFAR-10-C (OOD corruptions) using a ResNet-56 backbone.
PruneFuse Method
Label Budget ( b )
10%
20%
30%
40%
50%
Zero initialization
79.96
87.01
90.32
91.80
92.52
Retained Initial Weights
80.61
88.22
91.19
92.75
93.66
Random re-Initialized Weights
80.92
88.35
91.44
92.77
93.65
Table 18: Initialization schemes for Fused Model. Performance comparison of initializing the remaining weights of the Fused model after fusion with pruned model via 1) retaining weights from first initialization during pruned model initialization, 2) zero weights and 3) random re-initialization (PruneFuse) on ResNet-56 with CIFAR-10 dataset.
Figure 5: Ablation Study of Fusion on PruneFuse ( p=0.5 ). Experiments are performed on ResNet-56 architecture with CIFAR-10.
Figure 6: Ablation Study of Fusion on PruneFuse ( p=0.6 ). Experiments are performed on ResNet-56 architecture with CIFAR-10.
Figure 7: Ablation Study of Fusion on PruneFuse ( p=0.7 ). Experiments are performed on ResNet-56 architecture with CIFAR-10.
Table 19: Ablation Study of Knowledge Distillation on PruneFuse presented in 19(a) , 19(b) , and 19(c) with different architectures and datasets.
Techniques
Model
Architecture
Params
Label Budget ( b )
(Million)
10%
20%
30%
40%
50%
SVP
Data Selector
ResNet-8
0.074
77.85
83.35
85.43
86.83
86.90
Target
ResNet-20
0.26
80.18
86.34
89.22
90.75
91.88
PruneFuse
Data Selector
ResNet-20 ( p=0.5 )
0.066
76.58
83.41
85.83
87.07
88.06
Target
ResNet-20
0.26
80.25
87.57
90.20
91.70
92.29
Table 20: Comparison of SVP and PruneFuse on Small Models.
Figure 8: Comparison of PruneFuse with SVP. Scatter plot shows final accuracy on target model against the model size for different ResNet models on CIFAR-10, b= 50%. (a) shows ResNet-14 (with p=0.5 and p=0.6 ) and ResNet-8 models are used as data selectors for PruneFuse and SVP, respectively. While in (b), PruneFuse utilizes ResNet20 (i.e. p=0.5 and p=0.6 ) and SVP utilizes ResNet-8 models.
Method
Model
Architecture
Params
Label Budget ( b )
(Million)
10%
20%
30%
40%
50%
SVP
Data Selector
ResNet-20
0.26
81.07
86.51
89.77
91.08
91.61
Target
ResNet-56
0.85
80.76
87.31
90.77
92.59
92.95
PruneFuse
Data Selector
ResNet-56 ( p=0.5 )
0.21
78.62
84.92
88.17
89.93
90.31
Target
ResNet-56
0.85
82.68
88.97
91.63
93.24
93.69
Table 21: Comparison with SVP.
Table 22: Ablation study of k on CIFAR-10 using ResNet-56 architecture and least confidence as a selection metric.
Method
Selection
Label Budget ( b )
Selection
Label Budget ( b )
Size (k)
20%
40%
60%
Size (k)
20%
40%
60%
Baseline (AL)
5,000
88.51
93.04
93.83
10,000
86.92
92.51
93.81
PruneFuse
5,000
88.52
93.15
93.90
10,000
87.49
93.11
94.04
Table 23: Ablation study of k on Cifar-10 using ResNet-56 with p=0.5 and Tsync=1 .
Method
Epochs
Label Budget ( b )
10%
20%
30%
40%
50%
Least Conf.
181
80.92±0.409
88.35±0.327
91.44±0.148
92.77±0.026
93.65±0.141
110
80.51±0.375
87.64±0.222
90.79±0.052
92.11±0.154
93.00±0.005
Entropy
181
81.08±0.155
88.74±0.103
91.33±0.045
92.78±0.045
93.48±0.042
110
80.51±0.401
87.46±0.416
90.97±0.116
92.2±0.108
92.88±0.264
Random
181
80.43±0.273
86.28±0.367
88.75±0.17
90.36±0.022
91.42±0.125
Table 24: Performance Comparison when Early Stopping strategy is utilized alongside PruneFuse ( p=0.5 ). Experiments are performed with Resnet-56 on CIFAR-10.
Table 25: Performance Comparison of Baseline and PruneFuse presented in 25(a) , 25(b) , and 25(c) with different architectures and datasets.
Table 26: Performance Comparison of Baseline and PruneFuse ( p=0.4 ) on Cifar-10 and Cifar-100 using ResNet-56 architecture.
Method
Selection Metric
Data Selector’s
Target Model’s
Accuracy
Params
Params
( b=25% )
Baseline
Entropy
86.13
Least Confidence
86.50
Forgetting Events
0.85 Million
0.85 Million
86.01
Moderate
86.27
CSS
87.21
Table 27: Performance Comparison of Baseline (Coreset) and PruneFuse ( p=0.5 ) for Various selection metrics including Forgetting Events ( Toneva et al., 2019 ) , Moderate ( Xia et al., 2023 ) , and CSS ( Zheng et al., 2023 ) on Cifar-10 dataset using ResNet-56 architecture.
Method
Pruning Technique
Pruning Criteria
Label Budget ( b )
10%
20%
30%
40%
50%
Baseline (AL)
-
-
80.53
87.74
90.85
92.24
93.00
PruneFuse
Magnitude Imp.
79.73
87.16
91.08
92.29
93.19
Dynamic Pruning
GroupNorm Imp.
80.10
88.25
91.01
92.25
93.74
(Tsync=0)
LAMP Imp.
81.51
87.45
90.64
92.41
93.25
PruneFuse
Magnitude Imp.
80.92
88.35
91.44
92.77
93.65
Table 28: Effect of Pruning techniques and Pruning criteria on PruneFuse ( p=0.5 ) on CIFAR-10 dataset with ResNet-56.
Datasets
Data Selectors
Training Runime
(Selection Models)
(Minutes)
CIFAR-10
ResNet-56 (Baseline)
127.67
ResNet-56 (PruneFuse ( p=0.5 ))
72.55
ResNet-56 (PruneFuse ( p=0.8 ))
67.23
ResNet-18 (Baseline)
85.68
ResNet-18 (PruneFuse ( p=0.5 ))
61.15
Table 29: Training Runtime of data selector network i.e. pruned network in the case of PruneFuse and dense network for baseline, for various network architectures. The reported time is the training time when the network is trained before selecting final subset of the data ( b=50% ).
Datasets
Label Budget
Data Selectors
Data Selection Time
Target Model
( b )
(Training Time )
(Minutes)
(Training Time)
(Minutes)
(Minutes)
Baseline (AL)
10%
48.80
4.43
48.80
20%
99.23
3.50
99.23
30%
145.32
3.15
145.32
40%
195.38
2.72
195.38
Table 30: Detailed Training time of Baseline and PruneFuse( p=0.5 ) for TinyImageNet-200 for Resnet-50 using Least Confidence as selection metric.
Model / Selection Metric
Method with ALSE
10%
20%
30%
40%
50%
ResNet20 / Least Conf.
Baseline
80.24
86.28
89.07
90.27
91.09
PruneFuse ( p=0.4 )
81.05
86.82
90.02
90.89
91.54
PruneFuse ( p=0.5 )
80.72
86.65
89.79
90.79
91.51
PruneFuse ( p=0.6 )
80.53
86.68
89.56
90.81
91.32
ResNet56 / Least Conf.
Baseline
80.73
88.13
90.99
92.58
93.13
PruneFuse ( p=0.4 )
80.86
88.28
91.36
93.15
93.44
Table 31: Results with ResNet20 and ResNet56 on CIFAR-10 (Least Confidence & Entropy) using ALSE Jung et al. (2023) .
Deep neural networks are typically trained by uniformly sampling large datasets across epochs, despite evidence that not all samples contribute equally throughout learning. Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization, but existing methods rely on fixed schedules that do not adapt during training. In this work, we propose Adaptive Data Dropout, a simple framework that dynamically adjusts the subset of training data based on performance feedback. Inspired by self-regulated learning, our approach treats data selection as an adaptive process, increasing or decreasing data exposure in response to changes in training accuracy. We introduce a lightweight stochastic update mechanism that modulates the dropout schedule online, allowing the model to balance exploration and consolidation over time. Experiments on standard image classification benchmarks show that our method reduces effective training steps while maintaining competitive accuracy compared to static data dropout strategies. These results highlight adaptive data selection as a promising direction for efficient and robust training. Code will be released.
Amar Gahir, Varshil Patel, Shreyank N Gowda
School of Computer Science, University of Nottingham, Nottingham, United Kingdom
Dynamic Data selection aims to accelerate training by prioritizing informative samples during online training. However, existing methods typically rely on task-specific handcrafted metrics or static/snapshot-based criteria to estimate sample importance, limiting scalability across learning paradigms and making it difficult to capture the evolving utility of data throughout training. To address this challenge, we propose Data Agent, an end-to-end dynamic data selection framework that formulates data selection as a training-aware sequential decision-making problem. The agent learns a sample-wise selection policy that co-evolves with model optimization, guided by a composite reward that integrates loss-based difficulty and confidence-based uncertainty signals. The reward signals capture complementary objectives of optimization impact and information gain, together with a tuning-free adaptive weighting mechanism that balances these signals over training. Extensive experiments across a wide range of datasets and architectures demonstrate that Data Agent consistently accelerates training while preserving or improving performance, e.g., reducing costs by over 50% on ImageNet-1k and MMLU with lossless performance. Moreover, its dataset-agnostic formulation and modular reward make it plug-and-play across tasks and scenarios, e.g., robustness to noisy datasets, highlighting its potential in real-world scenarios. Code is available at https://github.com/Jackbrocp/Data-Agent.
Suorong Yang, Fangjian Su, Hai Gan +5
National University of Singapore · Nanjing University · Nanyang Technological University
The rapid growth of modern training datasets has significantly increased computational cost, motivating dataset pruning~(DP) methods which retain only a subset of informative samples to reduce training cost. Existing pruning criteria typically rely on either intrinsic signals that assess samples independently or extrinsic signals that promote diversity via pairwise relations. While effective in their own specific regimes, each captures only one aspect of sample utility and lacks robustness across different pruning ratios or data distribution. In this work, we present a unified graph-based DP framework. By modeling the dataset as a weighted graph, where node weights encode intrinsic value and edge weights encode extrinsic value, DP can be cast as a Maximum Weight Clique Problem (MWCP). Although MWCP is NP-hard, its structure admits a principled greedy solution based on sample-wise marginal gains. Under a few mild conditions, we further prove that this unified objective enjoys a formal approximation guarantee, which applies to a broad family of importance metrics and provides practical design guidelines. Extensive experiments show that our method outperforms existing DP methods while substantially reducing training cost, reducing training time by over 40% without sacrificing accuracy on ImageNet-1k with ResNet-50.
Dongyue Wu, Zilin Guo, Xiaoyu Li +4
State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Ant Group, Hangzhou, China