Mind the Tail: Data Selection Through Sparse Concepts

Ziheng Zhang; Neha Hulkund; Colin Lee; Zheda Mai; Hui Shen; Ying Ding; Tanya Berger-Wolf; Sara Beery

cs.CVOct 14, 2026

Mind the Tail: Data Selection Through Sparse Concepts

Authors: Ziheng Zhang, Neha Hulkund, Colin Lee, Zheda Mai, Hui Shen, Ying Ding, Tanya Berger-Wolf, Sara Beery,

Organizations: University of Texas at Austin · The Ohio State University · Massachusetts Institute of Technology · University of Michigan ·

Abstract

Data selection reallocates training support across visual concepts, including objects, attributes, color, texture, and backgrounds. Different allocation strategies may produce varied downstream model behavior, especially for tail concepts supported by only a few samples. Discarding them can hurt test-time robustness, while overemphasizing samples lying on category boundaries may introduce unstable training signals. This work explicitly investigates this relationship across fine-grained, spurious-correlation, and general image classification, as well as multimodal pretraining. Specifically, we use sparse autoencoders (SAEs) to extract concept activations from individual images. During selection, we accumulate the dataset-level concept support for each added sample. Applying diminishing returns to this distribution down-weights well-represented concepts and shifts the focus to the distribution tail. We then incorporate local support to find samples that reliably strengthen under-supported concepts. Empirical results show consistent improvements over strong data selection baselines across all benchmarks. We also reveal the connection between concepts our method enhances and the induced behavioral changes, including improved robustness against out-of-distribution and spurious contexts. This work establishes concept support, beyond individual sample importance, as a useful principle for data-efficient and robust learning.

Figures & tables

Figure 1: Concept-support distribution produced by different selectors. Data selection reallocates the concept distribution from the original dataset. The top bar compares three previous methods and our approach on the relative change in concept support in a $10$ % subset. Our method substantially amplifies tail concepts (insets show the absolute change, where red is suppression and blue is amplification). In contrast, K-Center, Forgetting, and D 2 Pruning largely preserve the original concept distribution. Bottom: representative SAE concepts across the same prevalence spectrum, from broadly supported generic concepts, through specific patterns shared by a subset of images, to rare concepts supported by only a few images. Highlighted patches indicate high-activation regions.

Figure 2: Overview of concept-based subset selection. We use SAEs to decompose foundation-model patch features into sparse concept vectors and track the accumulated support of each concept in the selected subset. Concept-level diminishing returns reduce the marginal value of already well-covered evidence, steering the selection toward tail concepts. We then use local support to locate samples that contribute to filling the distributional gap together with their neighbors. The resulting objective favors candidates representing under-supported evidence with local support over redundant head concepts or isolated rare regions.

Methods	NABirds						Stanford Cars						Waterbirds						Spawrious
Selection Rate	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%
Random	$35.6$	$55.3$	$64.9$	$73.3$	$77.1$	$80.3$	$21.9$	$45.2$	$63.1$	$\underline{78.0}$	$83.7$	$87.5$	$52.3$	$66.2$	$67.5$	$68.8$	$\underline{79.1}$	$80.1$	$63.8$	$67.4$	$72.6$	$68.8$	$69.5$	$76.3$
K-Center	$31.6$	$54.4$	$64.9$	$74.1$	$77.8$	-	$16.6$	$39.1$	$56.3$	$77.5$	$83.9$	-	$57.5$	$61.3$	$64.2$	$\mathbf{79.4}$	$69.1$	-	$71.3$	$71.7$	$71.9$	$69.9$	$69.1$	-
Forgetting	$22.4$	$49.8$	$63.3$	$74.3$	$78.1$	-	$11.8$	$37.0$	$56.7$	$72.2$	$81.1$	-	$31.6$	$43.6$	$60.1$	$73.4$	$77.0$	-	$61.1$	$68.5$	$67.9$	$\underline{74.3}$	$\underline{74.3}$	-
EL2N	$16.8$	$44.4$	$61.2$	$74.4$	$\mathbf{78.5}$	-	$8.9$	$28.4$	$49.2$	$71.0$	$79.3$	-	$31.0$	$52.3$	$64.2$	$72.9$	$76.5$	-	$\mathbf{74.6}$	$\mathbf{74.3}$	$\underline{75.6}$	$61.0$	$68.7$	-
Moderate	$34.6$	$56.0$	$64.9$	$73.7$	$77.5$	-	$19.7$	$45.9$	$59.8$	$77.3$	$83.8$	-	$60.5$	$\underline{68.3}$	$68.0$	$71.1$	$71.9$	-	$70.4$	$72.0$	$72.7$	$68.4$	$67.3$	-

Table 1: Results on image classification datasets. Comparison of subset selection performance with existing coreset selection methods using ResNet-50. We report classification accuracy for NABirds and Stanford Cars, and worst-group accuracy (WGA) for Waterbirds and Spawrious. All results are averaged over three runs. Bold and underlined indicate the best and second-best results, respectively.

Methods	Zero-Shot ImageNet-1K						Linear Probe ImageNet-1K						I2T Flickr30K						T2I Flickr30K
Selection Rate	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%
Random	$3.6$	$7.7$	$12.9$	$20.9$	$26.4$	$32.4$	$21.6$	$30.1$	$39.1$	$48.1$	$54.2$	$58.7$	$\underline{3.8}$	$8.5$	$17.2$	$31.1$	$42.5$	$51.1$	$\mathbf{3.1}$	$6.8$	$11.4$	$22.4$	$30.5$	$38.1$
CLIP Score	$3.2$	$9.9$	$15.2$	$23.9$	$28.6$	-	$23.0$	$36.1$	$40.6$	$49.4$	$55.0$	-	$\mathbf{3.9}$	$9.3$	$14.7$	$28.7$	$42.0$	-	$2.7$	$7.0$	$10.6$	$20.1$	$31.0$	-
K-center	$1.7$	$6.8$	$10.7$	$21.5$	$29.3$	-	$18.5$	$31.8$	$38.5$	$49.6$	$55.8$	-	$1.8$	$12.6$	$21.3$	$\mathbf{38.4}$	$\underline{46.4}$	-	$1.6$	$8.3$	$13.9$	$\underline{26.7}$	$\underline{33.8}$	-
Moderate	$3.8$	$10.1$	$14.6$	$22.5$	$29.0$	-	$22.3$	$34.9$	$41.7$	$50.1$	$55.8$	-	$3.1$	$11.0$	$19.1$	$32.8$	$43.3$	-	$2.4$	$8.3$	$13.0$	$22.9$	$32.4$	-
CCS	$\underline{4.0}$	$\underline{10.3}$	$\underline{15.4}$	$22.5$	$25.2$	-	$\underline{23.6}$	$\underline{36.3}$	$42.5$	$50.0$	$53.5$	-	$3.7$	$\underline{13.5}$	$\underline{22.3}$	$35.4$	$41.7$	-	$\underline{2.8}$	$\underline{10.0}$	$\underline{15.4}$	$24.7$	$29.3$	-

Table 2: Results on multimodal pretraining. Comparison of subset selection performance on CC12M using CLIP ViT-B/16. We report zero-shot and linear-probe classification accuracy on ImageNet-1K, and Recall@1 for image-to-text (I2T) and text-to-image (T2I) retrieval on Flickr30K. Bold and underlined entries indicate the best and second-best results, respectively.

Table 3: (a) Robustness to different pretrained features used for SAE training and subset selection. (b) Effect of normalization over neighbor similarity. We report accuracy for NABirds and worst-group accuracy (WGA) for Waterbirds averaged across selection ratios. Best results are bolded .

Figure 7: Qualitative examples of enhanced visual concepts. Our selection enhances concepts of both fine-grained attributes and contextual evidence. Finer-CAM visualizations compare the regions that models trained on our, random, and RAM-APL subsets rely on for prediction, showing that broader concept exposure during training improves test-time robustness.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Run	$\mathbf{L}_{0}$	Norm. MSE $\downarrow$	$\mathbf{n}_{\mathrm{dead}}$ $\downarrow$	Mean KL $\downarrow$	Top-1 Drop $\downarrow$	CKNNA $\uparrow$	CS $\uparrow$
ReLU $\lambda=0.1$	$6.75$	$0.60$	$6{,}331$	$2.87$	$0.51$	$0.28$	$0.91$
ReLU $\lambda=0.01$	$150.69$	$0.22$	$2{,}773$	$0.24$	$0.06$	$0.46$	$0.97$
TopK-16	$16.00$	$0.32$	$529$	$0.34$	$0.09$	$0.59$	$0.96$
TopK-32	$32.00$	$0.26$	$472$	$0.21$	$0.06$	$0.69$	$0.96$
BatchTopK-16	$16.04$	$0.32$	$230$	$0.35$	$0.09$	$0.64$	$0.96$
BatchTopK-32	$32.12$	$0.26$	$191$	$0.22$	$0.06$	$0.52$	$0.96$

Appendix

Table 8: Quantitative comparison of SAE variants on ImageNet-1K. We compare ReLU SAEs with

\ell_{1}

sparsity coefficients

\lambda\in\{0.1,0.01\}

, TopK SAEs with

k\in\{16,32\}

active latents per token, and BatchTopK SAEs with an average batch-wide activation budget of

k\in\{16,32\}

latents per token. Arrows indicate the preferred direction.

Table 12

$\bm{\beta}$	NABird Acc			Stanford Cars Acc			Waterbirds Acc			Waterbirds WGA			Spawrious Acc			Spawrious WGA			Tiny-ImageNet Acc
Selection Rate	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%
$0$	$37.9$	$66.6$	$\underline{75.0}$	$17.8$	$62.2$	$78.6$	$91.9$	$\mathbf{94.6}$	$\mathbf{94.5}$	$65.7$	$75.9$	$79.1$	$\underline{87.8}$	$\mathbf{87.3}$	$\underline{87.1}$	$\mathbf{76.1}$	$72.8$	$71.9$	$12.0$	$32.5$	$45.1$
$0.05$	$\mathbf{42.9}$	$\mathbf{67.4}$	$\mathbf{75.2}$	$23.9$	$65.3$	$\mathbf{78.8}$	$92.3$	$94.0$	$\underline{94.4}$	$\underline{68.6}$	$\mathbf{77.9}$	$\mathbf{79.3}$	$87.5$	$87.1$	$86.7$	$74.3$	$74.7$	$72.3$	$15.4$	$37.0$	$46.5$
$0.1$	$\underline{42.8}$	$67.2$	$\underline{75.0}$	$24.2$	$65.1$	$\underline{78.7}$	$92.4$	$93.9$	$93.8$	$66.6$	$71.7$	$78.4$	$\mathbf{88.1}$	$\underline{87.2}$	$\mathbf{87.4}$	$74.8$	$\mathbf{76.9}$	$\underline{74.6}$	$15.7$	$36.9$	$46.6$
$0.25$	$42.7$	$\underline{67.3}$	$74.5$	$24.4$	$\mathbf{65.8}$	$\underline{78.7}$	$\underline{92.5}$	$94.2$	$94.2$	$65.7$	$\underline{76.5}$	$\mathbf{79.3}$	$87.5$	$87.0$	$86.2$	$75.6$	$75.9$	$\mathbf{74.9}$	$16.8$	$34.7$	$45.1$
$0.5$	$42.6$	$67.0$	$74.1$	$24.2$	$\underline{65.6}$	$78.2$	$\mathbf{92.8}$	$\underline{94.3}$	$94.3$	$\mathbf{69.8}$	$75.4$	$75.9$	$87.0$	$87.1$	$86.4$	$75.9$	$75.1$	$71.9$	$21.1$	$38.3$	$45.9$

Appendix

Table 12: Sensitivity to the local support weight

\beta

. Comparison of performance across different local support weights and selection rates. We report classification accuracy (Acc.) for all classification datasets and worst-group accuracy (WGA) for Waterbirds and Spawrious. Bold and underlined entries indicate the best and second-best results, respectively.

$\bm{\beta}$	Utility	NABirds Acc			Waterbirds Acc			Waterbirds WGA
$\bm{\beta}$	Utility	10%	30%	50%	10%	30%	50%	10%	30%	50%
$0$	Linear	$37.1$	$61.6$	$70.9$	$89.7$	$91.1$	$92.7$	$56.7$	$61.1$	$61.2$
	Power	$\mathbf{39.9}$	$\underline{66.7}$	$\underline{74.1}$	$\underline{92.2}$	$93.7$	$\mathbf{94.6}$	$\mathbf{68.3}$	$\underline{74.4}$	$\underline{77.3}$
	Log	$\underline{37.9}$	$66.6$	$\mathbf{75.0}$	$91.9$	$\mathbf{94.6}$	$\underline{94.5}$	$65.7$	$\mathbf{75.9}$	$\mathbf{79.1}$
	Bounded exp.	$\underline{37.9}$	$\mathbf{66.8}$	$\mathbf{75.0}$	$\mathbf{93.4}$	$\underline{94.0}$	$\underline{94.5}$	$\underline{67.1}$	$74.1$	$\mathbf{79.1}$
$0.05$	Linear	$37.8$	$61.1$	$69.9$	$90.4$	$90.9$	$91.8$	$50.6$	$59.4$	$61.2$

Appendix

Table 13: Ablation study on the utility function. We compare different utility functions used in our selection objective. Within each

\beta

group, the best and second-best results in each column are bolded and underlined, respectively.

Selection Rate	NABirds Acc		Waterbirds WGA
Selection Rate	K-Center (DINO)	K-Center (SAE)	K-Center (DINO)	K-Center (SAE)
10%	$31.6$	$\mathbf{32.7}$	$57.5$	$\mathbf{61.2}$
30%	$64.9$	$\mathbf{66.5}$	$64.2$	$\mathbf{70.3}$
50%	$74.1$	$\mathbf{75.0}$	$\mathbf{79.4}$	$77.0$

Appendix

Table 14: Geometric selection in DINO and SAE feature spaces. We compare K-Center using dense DINO features and sparse SAE features.The better result is bolded .

Selection Rate	NABirds Acc		Waterbirds WGA		Waterbirds Acc
Selection Rate	Target SAE	IN1K SAE	Target SAE	IN1K SAE	Target SAE	IN1K SAE
10%	$\mathbf{43.0}$	$\underline{42.9}$	$\underline{63.4}$	$\mathbf{68.6}$	$\underline{91.7}$	$\mathbf{92.3}$
30%	$\mathbf{67.4}$	$\mathbf{67.4}$	$\underline{73.2}$	$\mathbf{77.9}$	$\underline{93.8}$	$\mathbf{94.0}$
50%	$\underline{74.4}$	$\mathbf{75.2}$	$\underline{78.7}$	$\mathbf{79.3}$	$\mathbf{95.1}$	$\underline{94.4}$

Appendix

Table 15: Robustness to SAE pretraining data. We compare SAEs trained on the target dataset with an SAE trained on ImageNet-1K. Bold and underlined entries indicate the best and second-best results, respectively.

Method	NABirds Acc			Waterbirds WGA			Waterbirds Acc
Selection Rate	10%	30%	50%	10%	30%	50%	10%	30%	50%
Random	$35.6$ / $40.2$	$64.9$ / $65.9$	$73.3$ / $73.8$	$52.3$ / $54.3$	$67.5$ / $\underline{66.0}$	$68.8$ / $\underline{68.6}$	$84.1$ / $76.7$	$89.6$ / $85.8$	$91.6$ / $85.9$
K-Center	$31.6$ / $33.7$	$64.9$ / $65.0$	$74.1$ / $73.5$	$57.5$ / $21.7$	$64.2$ / $49.5$	$\mathbf{79.4}$ / $62.3$	$89.4$ / $85.5$	$89.9$ / $84.9$	$91.7$ / $88.7$
Forgetting	$22.4$ / $26.9$	$63.3$ / $63.6$	$74.3$ / $73.7$	$31.6$ / $3.9$	$60.1$ / $25.6$	$73.4$ / $54.4$	$73.8$ / $75.7$	$85.1$ / $81.7$	$91.0$ / $89.8$
EL2N	$16.8$ / $21.0$	$61.2$ / $61.0$	$74.4$ / $73.4$	$31.0$ / $30.4$	$64.2$ / $38.2$	$72.9$ / $63.2$	$78.4$ / $79.9$	$87.0$ / $85.0$	$92.7$ / $89.7$
D2 Pruning	$25.8$ / $31.0$	$\underline{66.6}$ / $66.6$	$\underline{74.7}$ / $\underline{74.1}$	$47.0$ / $50.5$	$56.4$ / $37.5$	$60.8$ / $47.2$	$89.0$ / $\underline{88.6}$	$\underline{92.2}$ / $87.9$	$\underline{93.3}$ / $\underline{91.2}$

Appendix

Table 16: Cross-architecture generalization. Comparison of subset selection performance using the same selected subsets with ResNet-50 and ViT-B/16. We report classification accuracy (Acc.) for NABirds, and worst-group accuracy (WGA) and overall accuracy (Acc.) for Waterbirds; each cell shows ResNet-50 / ViT-B/16. Bold and underlined entries indicate the best and second-best results for each architecture, respectively.

Method

ImageNet-A

ImageNet-R

ImageNet-D

ImageNet-V2

ImageNet-1K Val

10\%

30\%

50\%

10\%

30\%

50\%

10\%

30\%

50\%

10\%

30\%

50\%

10\%

30\%

50\%

Random

\underline{1.4}

1.2

1.1

15.8

24.1

\mathbf{25.9}

\underline{2.9}

\underline{6.3}

6.8

36.3

\underline{49.3}

\underline{50.9}

48.6

64.2

67.2

EL2N

1.2

1.0

1.1

9.3

19.4

23.2

1.4

3.6

6.1

13.4

23.4

45.2

11.2

45.4

64.1

CCS

1.3

\underline{1.6}

\underline{1.4}

\underline{16.0}

\underline{24.5}

\underline{25.1}

2.7

6.2

\underline{6.9}

\mathbf{37.4}

49.1

\mathbf{51.2}

\mathbf{50.1}

\underline{64.8}

\mathbf{68.4}

Ours

\mathbf{1.7}

\mathbf{2.0}

\mathbf{1.5}

\mathbf{16.4}

\mathbf{25.8}

\mathbf{25.9}

\mathbf{3.3}

\mathbf{6.8}

\mathbf{7.6}

\underline{37.2}

\mathbf{50.1}

50.7

\underline{49.2}

\mathbf{65.2}

\underline{67.7}

Appendix

Table 17: ImageNet-1K classification and robustness results. We report top-1 accuracy (%) on four robustness benchmarks and the standard ImageNet-1K validation set at different selection rates. The best and second-best results are bolded and underlined, respectively.

Table 18: Concept-support profiles of 10% subsets for all baselines on (a) NABirds and (b) Stanford Cars. Every baseline stays close to the random profile on all three measures, whereas our selection covers more concepts, spreads its support more evenly, and shifts support toward less prevalent concepts.

Figure 10: Additional examples of enhanced and reduced SAE concepts. We show representative images from concepts whose support changes most under our selection relative to random selection. Highlighted patches indicate high-activation regions.

Figure 11: Additional Finer-CAM visualizations on NABirds and Stanford Cars. We show attribution maps for models trained on subsets selected by our method, Random, RAM-APL, and CCS. The Concept column highlights image patches with high activation for the corresponding SAE latent.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 186 papers

Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning
Aug 24, 2026Dongyue Wu, Tao MaDataset Pruning Training Data Curation
State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Ant Group · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Hierarchical Scaffolding Enables Human-Like Cognitive Selectivity under Data Scarcity
Jul 6, 2026Juhyoung Park, Jaehyuk Bae, Hyeonbo Yang +1Hierarchical Representation Learning Data-Efficient Learning
School of Computing 2Department of Brain and Cognitive Sciences Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Department of Brain and Cognitive Sciences Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning
Jun 12, 2026Shilong Xiang, Zirui Zhang, Chengzhi MaoVision-Language Models Self-Supervised Learning
Rutgers University

Aug 24, 2026J/K move · Enter open · S save

Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning

Dongyue Wu, Tao Ma

State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Ant Group · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

Dataset Pruning Training Data Curation

Open paper AskPDF arXiv

Existing data pruning methods predominantly rely on high-dimensional feature embeddings to measure sample importance. However, these compressed vectors often obscure fine-grained semantic interactions, leading to suboptimal coverage of rare semantic concepts in the pruned subsets. In this paper, we propose Mapping the Concept Landscape (MCL), a novel structural perception framework for transparent data pruning. Instead of abstract embeddings, we represent each image-caption pair as an explicit sample-level graph comprising entities, events, and attributes. By integrating these individual graphs into a comprehensive dataset-level graph, we characterize the global distribution of semantic concepts and quantify their rarity across the entire corpus. Based on this structured perception, we develop a greedy concept-coverage maximization algorithm that iteratively selects samples to maximize the marginal gain of high-value, under-represented concepts. Experimental results on various benchmarks demonstrate that our method not only achieves superior pruning efficiency compared to state-of-the-art methods but also provides a transparent and interpretable audit trail for the selection process.

Abstract

Training Data Selection

Explore similar work

Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning

Hierarchical Scaffolding Enables Human-Like Cognitive Selectivity under Data Scarcity

S2^22COPE: Self-Supervised Concept Discovery via Preference Learning

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning