Rethinking Representativeness and Diversity in Dynamic Data Selection

Yuzhe Zhou; Zhenglin Hua; Haiyun Guo; Yuheng Jia

cs.AIMar 5, 2026

Rethinking Representativeness and Diversity in Dynamic Data Selection

Authors: Yuzhe Zhou, Zhenglin Hua, Haiyun Guo, Yuheng Jia

Organizations: Southeast University, Nanjing, China · Chinese Academy of Sciences, Beijing, China

Abstract

Dynamic data selection accelerates training by sampling a changing subset of the dataset while preserving accuracy. We rethink two core notions underlying sample evaluation: representativeness and diversity. Instead of local geometric centrality, we define representativeness as coverage of dataset-level common or high-frequency feature factors. Instead of within-subset dispersion, we define diversity at the process level, requiring the selection trajectory to gradually include complementary rare factors over training. Based on this view, we propose a dynamic selection framework with three components. First, we score representativeness in a plug-in feature space to prioritize samples covering frequent factors. We instantiate this with a sparse autoencoder trained on the target dataset, using sparse unit activations to summarize both individual samples and dataset-wide factor statistics. Second, we realize process-level diversity by combining rare-factor sampling with a Usage-Frequency Penalty that promotes sample rotation, provably discourages monopoly, and reduces gradient bias. Third, we couple the two-dimensional scoring with a smooth scheduler that transitions selection from core-pattern consolidation to rare-factor exploration, without extra gradients, influence estimates, or second-order computations on the training model. Extensive experiments on five benchmarks across vision and text tasks demonstrate improved accuracy-efficiency trade-offs across models. Our method matches or exceeds full-data accuracy with over 2x training acceleration. Code will be released.

Figures & tables

Figure 1: Conceptual comparison with prior data selection methods. Previous methods relying on geometry-based metrics may overemphasize local centrality and miss implicit feature factors. Our framework measures representativeness as dataset-level coverage of common or high-frequency factors, and enforces process-level diversity by promoting sample rotation across epochs rather than optimizing a single static subset.

Figure 2: Our framework for dynamic data selection. A plug-in feature encoder (CLIP by default) maps inputs to a fixed feature space, where an SAE yields sparse unit activations and dataset-wide statistics. At epoch $t$ , each example is scored by representativeness (weighted coverage of high-frequency factors), diversity (factor rarity), and a usage-frequency penalty that discourages repeated selection and enforces rotation. A scheduler $\alpha(t)$ smoothly balances representativeness and diversity, transitioning from core-pattern consolidation to rare-feature exploration.

Figure 3: MMD comparison between subsets selected by our representativeness score (Rep-TopK) and the geometric baseline K-Center under different selection ratios. Lower is better.

Backbone	Method	Type	CIFAR-10 (Selection Ratio)		CIFAR-100 (Selection Ratio)
Backbone	Method	Type	70%	30%	70%	30%
ResNet-18	Random	✗	94.6 ↓1.5	90.2 ↓5.9	73.8 ↓4.9	69.7 ↓9.0
	GraNd-4 ( Paul et al., 2021 )	✗	95.3 ↓0.8	91.2 ↓4.9	74.6 ↓3.6	68.8 ↓9.4
	MoDS ( Xia et al., 2023 )	✗	93.9 ↓2.2	90.6 ↓5.5	74.6 ↓3.6	65.3 ↓12.9
	MoSo ( Tan et al., 2023 )	✗	95.3 ↓0.8	91.1 ↓6.0	77.5 ↓0.7	70.9 ↓7.3
	$\mathbb{D}^{2}$ ( Maharana et al., 2024 )	✗	95.7 ↓0.4	93.3 ↓2.8	78.2 ↑0.0	70.5 ↓7.7

Table 1: Accuracy (%) comparison on CIFAR-10/100. Random ∗ denotes dynamic random selection. The reported ratios denote the selection ratios . Ours is highlighted in grey. ✓ = Dynamic, ✗ = Static.

Dataset	Method	100%	70%	50%	30%
ImageNet-1K (ResNet-34)	Random	73.1	72.2	70.3	66.7
	Entropy	73.1	72.3	70.8	64.1
	$\mathbb{D}^{2}$	73.1	73.0	71.4	64.8
	Moderate-DS	73.1	72.0	70.3	65.9
	K-center	73.1	72.2	67.2	48.8
	Forgetting	73.1	72.6	70.9	66.5

Table 2: Top-1 accuracy (%) on ImageNet-1K under different selection ratios . Random ∗ denotes dynamic random selection per epoch.

Dataset	CIFAR10		CIFAR100
Model	R-18	R-50	R-18	R-50	VGG-16
Full	96.1	95.8	78.2	79.1	73.9
Ours	96.1	95.6	78.4	79.1	74.0

Table 3: The accuracy(%) of different model architectures on CIFAR10/100 datasets. The model was trained using our method with 70% samples selected per epoch. Our proposed method outperforms the full dataset on various occasions.

Training Protocol	Selection Ratio per Epoch (%)
Training Protocol	70	50	30	Full
I-1K $\to$ C-10	96.1	95.6	95.7	96.1
C-10 $\to$ C-10	96.1	95.6	95.3	96.1

Table 4: Top-1 accuracy (%) on CIFAR-10 using ResNet-18. Models are trained with varying pruning ratios per epoch. I-1K

\to

C-10 denotes ImageNet-1K trained SAE on CIFAR-10 scoring, then train ResNet-18 by our framework; C-10

\to

C-10 denotes our proposed method.

Figure 4: Convergence speed comparison on CIFAR-10 (ResNet-18). Ours denotes our method, RCAP denotes the previous SOTA method, and Full uses the full dataset.

Dataset	Method	Acc (%)	Overhead	Speedup
Dataset	Method	Acc (%)	(GPU h)	Speedup
C-10	Full	96.1 ±0.1	–	1.0×
C-10	Ours	95.7 ±0.1	0.0028	2.51×
T-Img	Full	58.2 ±0.1	–	1.0×
T-Img	Ours	58.1 ±0.2	0.12	1.34×

Table 5: Training efficiency and accuracy comparison. Overhead includes CLIP feature extraction, SAE training, and samples scoring. Speedup = Full / Ours training time.

Variant	$R$	$D$	$P$	$\alpha$	50 epochs	Final
Full Method	✓	✓	✓	✓	57.7	79.1
w/o $Rep$	✗	✓	✓	✓	57.1	78.1
w/o $Div$	✓	✗	✓	✓	57.5	77.2
w/o $Pen$	✓	✓	✗	✓	54.7	76.4
w/o $\alpha$	✓	✓	✓	✗	54.5	78.2
Static	✓	✓	✗	✗	53.8	76.3

Table 6: Ablation on CIFAR-100 (ResNet-50, 200 epochs).

R

D

, and

P

denote the representativeness score, process-level diversity score, and usage-frequency penalty, respectively;

\alpha

is the curriculum scheduler. w/o Full-data removes the final full-data refinement stage. only

Pen

selects samples by rotation only, i.e., prioritizing those with lower historical usage.

Figure 5: Illustrations of comparing our method with various data selection baselines on different datasets and models. We trained VGG-16 on Tiny-ImageNet in Fig (a), ViT-small on CIFAR-10 in Fig (b).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: SAE analysis of the class ’cat’. The left panel shows the activation intensity of each neuron; selecting nearly the top 10% of neurons as high-frequency neurons covers all neurons with high activation intensity. The right panel displays the number of samples that activate each neuron, where a maximum value of 500 indicates that all images in this class activate at least one neuron.

Figure 7: Visualization of two types of highly activated neurons and the nine images (on the left column) that most strongly activate them. The top row shows a class-specific neuron, which is strongly activated only in the class of street car. The bottom row shows a universal neuron, which is strongly activated across multiple classes and images with a plain background.

Probe Feature Space	30%	50%	70%
CLIP embeddings	96.1	95.6	95.3
ResNet-18 embeddings	95.9	95.4	95.1

Appendix

Table 7: Effect of sparse-unit probe instantiation on ResNet-18. We compare probes trained on CLIP features and downstream ResNet-18 representations under different selection ratios.

Figure 8: Per-sample usage statistics under 20% symmetric label noise on CIFAR-100 (ResNet-50): samples are sorted by sampled times in descending order. Loss-driven selection exhibits a more concentrated usage distribution, while our method yields a flatter curve, indicating more even sample participation and reduced long-term selection bias.

Setting	Selection ratio = 20%			Selection ratio = 30%
	Clean	Noisy	$\boldsymbol{\Delta}$	Clean	Noisy	$\boldsymbol{\Delta}$
CLIP	54.10	46.05	8.05	61.90	58.34	3.56
MoSo	-	31.01	-	58.27	43.73	14.54
MoDS	51.89	40.25	11.64	56.00	48.53	7.47
InfoBatch	56.90	54.49	2.41	63.81	55.67	8.14
Ours	64.29	60.33	3.96	75.70	60.86	14.84

Appendix

Table 8: Noise robustness on CIFAR-100 with ResNet-50 under 20% label noise. We report top-1 accuracy (%) for clean and noisy training, and the drop

\Delta

(Clean – Noisy).

Panel of Figure 9 — Figure 9: Visualization of the SAE latent space. (a) presents the UMAP dimensionality reduction analysis of partial vehicle and plant classes. We observe that classes with similar semantic information are closely distributed in the SAE latent space, while those with high semantic discriminability are far apart. The heatmap in (b) further validates the successful training of our SAE.

Figure 10: The score distribution of our method.

Backbone	Method	Acc. (%)	Speedup
ResNet-50	Full-data	63.5	1.00 $\times$
	InfoBatch	63.4	1.71 $\times$
	Ours	63.7	1.26 $\times$
VGG-16	Full-data	58.21	1.00 $\times$
	CLIP	56.34	1.10 $\times$
	Ours	58.89	1.34 $\times$

Appendix

Table 9: Tiny-ImageNet: comparison against backbone-specific SOTA baselines. For each backbone (ResNet-50 / VGG-16), we compare with the strongest prior method we reproduce. Cost is the normalized training cost (Full-data

=1.00\times

), and Speedup

=1/\text{Cost}

Figure 11: Visualization of the images of class 8 with the highest representativeness and diversity score. The top ones are of the highest representativeness score, while the bottom images display high-diversity samples.

LR Scheduler	Method	Acc	$\Delta$ Acc	Time
MultiStep	Full	77.3	–	0.7
MultiStep	Ours	77.6	+0.3	0.4
OneCycle	Full	78.2	–	0.7
OneCycle	Ours	78.2	+0.0	0.4
CosAnneal	Full	78.2	–	0.7
CosAnneal	Ours	78.4	+0.2	0.4

Appendix

Table 10: Top-1 accuracy (%) and training time (GPU hours) on CIFAR-100 (ResNet-18) under different learning-rate schedulers.

\Delta

Acc denotes the accuracy difference relative to full-data training under the same scheduler. Lower time is better.

Figure 12: In the early stages of training, samples with high representativeness scores are primarily selected, while in the middle and late stages, samples are chosen by comprehensively considering both representativeness and diversity scores. The continuous epoch sampling visualization in the figure also intuitively demonstrates how our usage-frequency penalty functions.

Diversity-Aware Sampling

See all 20 papers

Data Agent: Learning to Select Data via End-to-End Dynamic Optimization
Mar 8, 2026Suorong Yang, Fangjian Su, Hai Gan +5Data Selection Training Data Selection
National University of Singapore · Nanjing University · Nanyang Technological University
Spokes: Optimizing for Diverse Pretraining Data Selection
Jun 13, 2026Clarence Lee, Yejin Choi, Luke Zettlemoyer +2Language Model Pretraining Diversity-Aware Sampling
DSO National Laboratories · Stanford University · University of Washington
Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training
May 14, 2026Suorong Yang, Hanqi Zhu, Hai Gan +4Deep Learning Optimization Data Selection
1National University of Singapore · 2Nanjing University · 3Hokkaido University +1

Mar 8, 2026J/K move · Enter open · S save

Data Agent: Learning to Select Data via End-to-End Dynamic Optimization

Suorong Yang, Fangjian Su, Hai Gan, Ziqi Ye+4

National University of Singapore · Nanjing University · Nanyang Technological University

Data Selection Training Data Selection Data-Efficient Learning Adaptive Sampling

Open paper AskPDF arXiv

Dynamic Data selection aims to accelerate training by prioritizing informative samples during online training. However, existing methods typically rely on task-specific handcrafted metrics or static/snapshot-based criteria to estimate sample importance, limiting scalability across learning paradigms and making it difficult to capture the evolving utility of data throughout training. To address this challenge, we propose Data Agent, an end-to-end dynamic data selection framework that formulates data selection as a training-aware sequential decision-making problem. The agent learns a sample-wise selection policy that co-evolves with model optimization, guided by a composite reward that integrates loss-based difficulty and confidence-based uncertainty signals. The reward signals capture complementary objectives of optimization impact and information gain, together with a tuning-free adaptive weighting mechanism that balances these signals over training. Extensive experiments across a wide range of datasets and architectures demonstrate that Data Agent consistently accelerates training while preserving or improving performance, e.g., reducing costs by over 50% on ImageNet-1k and MMLU with lossless performance. Moreover, its dataset-agnostic formulation and modular reward make it plug-and-play across tasks and scenarios, e.g., robustness to noisy datasets, highlighting its potential in real-world scenarios. Code is available at https://github.com/Jackbrocp/Data-Agent.

Abstract

Diversity-Aware Sampling

Explore similar work

Data Agent: Learning to Select Data via End-to-End Dynamic Optimization

Spokes: Optimizing for Diverse Pretraining Data Selection

Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training