Mind the Tail: Data Selection Through Sparse Concepts

Ziheng Zhang; Neha Hulkund; Colin Lee; Zheda Mai; Hui Shen; Ying Ding; Tanya Berger-Wolf; Sara Beery

cs.CVOct 14, 2026

Mind the Tail: Data Selection Through Sparse Concepts

Authors: Ziheng Zhang, Neha Hulkund, Colin Lee, Zheda Mai, Hui Shen, Ying Ding, Tanya Berger-Wolf, Sara Beery,

Organizations: University of Texas at Austin · The Ohio State University · Massachusetts Institute of Technology · University of Michigan ·

Abstract

Data selection reallocates training support across visual concepts, including objects, attributes, color, texture, and backgrounds. Different allocation strategies may produce varied downstream model behavior, especially for tail concepts supported by only a few samples. Discarding them can hurt test-time robustness, while overemphasizing samples lying on category boundaries may introduce unstable training signals. This work explicitly investigates this relationship across fine-grained, spurious-correlation, and general image classification, as well as multimodal pretraining. Specifically, we use sparse autoencoders (SAEs) to extract concept activations from individual images. During selection, we accumulate the dataset-level concept support for each added sample. Applying diminishing returns to this distribution down-weights well-represented concepts and shifts the focus to the distribution tail. We then incorporate local support to find samples that reliably strengthen under-supported concepts. Empirical results show consistent improvements over strong data selection baselines across all benchmarks. We also reveal the connection between concepts our method enhances and the induced behavioral changes, including improved robustness against out-of-distribution and spurious contexts. This work establishes concept support, beyond individual sample importance, as a useful principle for data-efficient and robust learning.

Figures & tables

Figure 1: Concept-support distribution produced by different selectors. Data selection reallocates the concept distribution from the original dataset. The top bar compares three previous methods and our approach on the relative change in concept support in a $10$ % subset. Our method substantially amplifies tail concepts (insets show the absolute change, where red is suppression and blue is amplification). In contrast, K-Center, Forgetting, and D 2 Pruning largely preserve the original concept distribution. Bottom: representative SAE concepts across the same prevalence spectrum, from broadly supported generic concepts, through specific patterns shared by a subset of images, to rare concepts supported by only a few images. Highlighted patches indicate high-activation regions.

Figure 2: Overview of concept-based subset selection. We use SAEs to decompose foundation-model patch features into sparse concept vectors and track the accumulated support of each concept in the selected subset. Concept-level diminishing returns reduce the marginal value of already well-covered evidence, steering the selection toward tail concepts. We then use local support to locate samples that contribute to filling the distributional gap together with their neighbors. The resulting objective favors candidates representing under-supported evidence with local support over redundant head concepts or isolated rare regions.

Methods	NABirds						Stanford Cars						Waterbirds						Spawrious
Selection Rate	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%
Random	$35.6$	$55.3$	$64.9$	$73.3$	$77.1$	$80.3$	$21.9$	$45.2$	$63.1$	$\underline{78.0}$	$83.7$	$87.5$	$52.3$	$66.2$	$67.5$	$68.8$	$\underline{79.1}$	$80.1$	$63.8$	$67.4$	$72.6$	$68.8$	$69.5$	$76.3$
K-Center	$31.6$	$54.4$	$64.9$	$74.1$	$77.8$	-	$16.6$	$39.1$	$56.3$	$77.5$	$83.9$	-	$57.5$	$61.3$	$64.2$	$\mathbf{79.4}$	$69.1$	-	$71.3$	$71.7$	$71.9$	$69.9$	$69.1$	-
Forgetting	$22.4$	$49.8$	$63.3$	$74.3$	$78.1$	-	$11.8$	$37.0$	$56.7$	$72.2$	$81.1$	-	$31.6$	$43.6$	$60.1$	$73.4$	$77.0$	-	$61.1$	$68.5$	$67.9$	$\underline{74.3}$	$\underline{74.3}$	-
EL2N	$16.8$	$44.4$	$61.2$	$74.4$	$\mathbf{78.5}$	-	$8.9$	$28.4$	$49.2$	$71.0$	$79.3$	-	$31.0$	$52.3$	$64.2$	$72.9$	$76.5$	-	$\mathbf{74.6}$	$\mathbf{74.3}$	$\underline{75.6}$	$61.0$	$68.7$	-
Moderate	$34.6$	$56.0$	$64.9$	$73.7$	$77.5$	-	$19.7$	$45.9$	$59.8$	$77.3$	$83.8$	-	$60.5$	$\underline{68.3}$	$68.0$	$71.1$	$71.9$	-	$70.4$	$72.0$	$72.7$	$68.4$	$67.3$	-

Table 1: Results on image classification datasets. Comparison of subset selection performance with existing coreset selection methods using ResNet-50. We report classification accuracy for NABirds and Stanford Cars, and worst-group accuracy (WGA) for Waterbirds and Spawrious. All results are averaged over three runs. Bold and underlined indicate the best and second-best results, respectively.

Methods	Zero-Shot ImageNet-1K						Linear Probe ImageNet-1K						I2T Flickr30K						T2I Flickr30K
Selection Rate	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%	10%	20%	30%	50%	70%	100%
Random	$3.6$	$7.7$	$12.9$	$20.9$	$26.4$	$32.4$	$21.6$	$30.1$	$39.1$	$48.1$	$54.2$	$58.7$	$\underline{3.8}$	$8.5$	$17.2$	$31.1$	$42.5$	$51.1$	$\mathbf{3.1}$	$6.8$	$11.4$	$22.4$	$30.5$	$38.1$
CLIP Score	$3.2$	$9.9$	$15.2$	$23.9$	$28.6$	-	$23.0$	$36.1$	$40.6$	$49.4$	$55.0$	-	$\mathbf{3.9}$	$9.3$	$14.7$	$28.7$	$42.0$	-	$2.7$	$7.0$	$10.6$	$20.1$	$31.0$	-
K-center	$1.7$	$6.8$	$10.7$	$21.5$	$29.3$	-	$18.5$	$31.8$	$38.5$	$49.6$	$55.8$	-	$1.8$	$12.6$	$21.3$	$\mathbf{38.4}$	$\underline{46.4}$	-	$1.6$	$8.3$	$13.9$	$\underline{26.7}$	$\underline{33.8}$	-
Moderate	$3.8$	$10.1$	$14.6$	$22.5$	$29.0$	-	$22.3$	$34.9$	$41.7$	$50.1$	$55.8$	-	$3.1$	$11.0$	$19.1$	$32.8$	$43.3$	-	$2.4$	$8.3$	$13.0$	$22.9$	$32.4$	-
CCS	$\underline{4.0}$	$\underline{10.3}$	$\underline{15.4}$	$22.5$	$25.2$	-	$\underline{23.6}$	$\underline{36.3}$	$42.5$	$50.0$	$53.5$	-	$3.7$	$\underline{13.5}$	$\underline{22.3}$	$35.4$	$41.7$	-	$\underline{2.8}$	$\underline{10.0}$	$\underline{15.4}$	$24.7$	$29.3$	-

Table 2: Results on multimodal pretraining. Comparison of subset selection performance on CC12M using CLIP ViT-B/16. We report zero-shot and linear-probe classification accuracy on ImageNet-1K, and Recall@1 for image-to-text (I2T) and text-to-image (T2I) retrieval on Flickr30K. Bold and underlined entries indicate the best and second-best results, respectively.

Table 3: (a) Robustness to different pretrained features used for SAE training and subset selection. (b) Effect of normalization over neighbor similarity. We report accuracy for NABirds and worst-group accuracy (WGA) for Waterbirds averaged across selection ratios. Best results are bolded .

Figure 7: Qualitative examples of enhanced visual concepts. Our selection enhances concepts of both fine-grained attributes and contextual evidence. Finer-CAM visualizations compare the regions that models trained on our, random, and RAM-APL subsets rely on for prediction, showing that broader concept exposure during training improves test-time robustness.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Run	$\mathbf{L}_{0}$	Norm. MSE $\downarrow$	$\mathbf{n}_{\mathrm{dead}}$ $\downarrow$	Mean KL $\downarrow$	Top-1 Drop $\downarrow$	CKNNA $\uparrow$	CS $\uparrow$
ReLU $\lambda=0.1$	$6.75$	$0.60$	$6{,}331$	$2.87$	$0.51$	$0.28$	$0.91$
ReLU $\lambda=0.01$	$150.69$	$0.22$	$2{,}773$	$0.24$	$0.06$	$0.46$	$0.97$
TopK-16	$16.00$	$0.32$	$529$	$0.34$	$0.09$	$0.59$	$0.96$
TopK-32	$32.00$	$0.26$	$472$	$0.21$	$0.06$	$0.69$	$0.96$
BatchTopK-16	$16.04$	$0.32$	$230$	$0.35$	$0.09$	$0.64$	$0.96$
BatchTopK-32	$32.12$	$0.26$	$191$	$0.22$	$0.06$	$0.52$	$0.96$

Appendix

Table 8: Quantitative comparison of SAE variants on ImageNet-1K. We compare ReLU SAEs with

\ell_{1}

sparsity coefficients

\lambda\in\{0.1,0.01\}

, TopK SAEs with

k\in\{16,32\}

active latents per token, and BatchTopK SAEs with an average batch-wide activation budget of

k\in\{16,32\}

latents per token. Arrows indicate the preferred direction.

Table 12

$\bm{\beta}$	NABird Acc			Stanford Cars Acc			Waterbirds Acc			Waterbirds WGA			Spawrious Acc			Spawrious WGA			Tiny-ImageNet Acc
Selection Rate	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%	10%	30%	50%
$0$	$37.9$	$66.6$	$\underline{75.0}$	$17.8$	$62.2$	$78.6$	$91.9$	$\mathbf{94.6}$	$\mathbf{94.5}$	$65.7$	$75.9$	$79.1$	$\underline{87.8}$	$\mathbf{87.3}$	$\underline{87.1}$	$\mathbf{76.1}$	$72.8$	$71.9$	$12.0$	$32.5$	$45.1$
$0.05$	$\mathbf{42.9}$	$\mathbf{67.4}$	$\mathbf{75.2}$	$23.9$	$65.3$	$\mathbf{78.8}$	$92.3$	$94.0$	$\underline{94.4}$	$\underline{68.6}$	$\mathbf{77.9}$	$\mathbf{79.3}$	$87.5$	$87.1$	$86.7$	$74.3$	$74.7$	$72.3$	$15.4$	$37.0$	$46.5$
$0.1$	$\underline{42.8}$	$67.2$	$\underline{75.0}$	$24.2$	$65.1$	$\underline{78.7}$	$92.4$	$93.9$	$93.8$	$66.6$	$71.7$	$78.4$	$\mathbf{88.1}$	$\underline{87.2}$	$\mathbf{87.4}$	$74.8$	$\mathbf{76.9}$	$\underline{74.6}$	$15.7$	$36.9$	$46.6$
$0.25$	$42.7$	$\underline{67.3}$	$74.5$	$24.4$	$\mathbf{65.8}$	$\underline{78.7}$	$\underline{92.5}$	$94.2$	$94.2$	$65.7$	$\underline{76.5}$	$\mathbf{79.3}$	$87.5$	$87.0$	$86.2$	$75.6$	$75.9$	$\mathbf{74.9}$	$16.8$	$34.7$	$45.1$
$0.5$	$42.6$	$67.0$	$74.1$	$24.2$	$\underline{65.6}$	$78.2$	$\mathbf{92.8}$	$\underline{94.3}$	$94.3$	$\mathbf{69.8}$	$75.4$	$75.9$	$87.0$	$87.1$	$86.4$	$75.9$	$75.1$	$71.9$	$21.1$	$38.3$	$45.9$

Appendix

Table 12: Sensitivity to the local support weight

\beta

. Comparison of performance across different local support weights and selection rates. We report classification accuracy (Acc.) for all classification datasets and worst-group accuracy (WGA) for Waterbirds and Spawrious. Bold and underlined entries indicate the best and second-best results, respectively.

$\bm{\beta}$	Utility	NABirds Acc			Waterbirds Acc			Waterbirds WGA
$\bm{\beta}$	Utility	10%	30%	50%	10%	30%	50%	10%	30%	50%
$0$	Linear	$37.1$	$61.6$	$70.9$	$89.7$	$91.1$	$92.7$	$56.7$	$61.1$	$61.2$
	Power	$\mathbf{39.9}$	$\underline{66.7}$	$\underline{74.1}$	$\underline{92.2}$	$93.7$	$\mathbf{94.6}$	$\mathbf{68.3}$	$\underline{74.4}$	$\underline{77.3}$
	Log	$\underline{37.9}$	$66.6$	$\mathbf{75.0}$	$91.9$	$\mathbf{94.6}$	$\underline{94.5}$	$65.7$	$\mathbf{75.9}$	$\mathbf{79.1}$
	Bounded exp.	$\underline{37.9}$	$\mathbf{66.8}$	$\mathbf{75.0}$	$\mathbf{93.4}$	$\underline{94.0}$	$\underline{94.5}$	$\underline{67.1}$	$74.1$	$\mathbf{79.1}$
$0.05$	Linear	$37.8$	$61.1$	$69.9$	$90.4$	$90.9$	$91.8$	$50.6$	$59.4$	$61.2$

Appendix

Table 13: Ablation study on the utility function. We compare different utility functions used in our selection objective. Within each

\beta

group, the best and second-best results in each column are bolded and underlined, respectively.

Selection Rate	NABirds Acc		Waterbirds WGA
Selection Rate	K-Center (DINO)	K-Center (SAE)	K-Center (DINO)	K-Center (SAE)
10%	$31.6$	$\mathbf{32.7}$	$57.5$	$\mathbf{61.2}$
30%	$64.9$	$\mathbf{66.5}$	$64.2$	$\mathbf{70.3}$
50%	$74.1$	$\mathbf{75.0}$	$\mathbf{79.4}$	$77.0$

Appendix

Table 14: Geometric selection in DINO and SAE feature spaces. We compare K-Center using dense DINO features and sparse SAE features.The better result is bolded .

Selection Rate	NABirds Acc		Waterbirds WGA		Waterbirds Acc
Selection Rate	Target SAE	IN1K SAE	Target SAE	IN1K SAE	Target SAE	IN1K SAE
10%	$\mathbf{43.0}$	$\underline{42.9}$	$\underline{63.4}$	$\mathbf{68.6}$	$\underline{91.7}$	$\mathbf{92.3}$
30%	$\mathbf{67.4}$	$\mathbf{67.4}$	$\underline{73.2}$	$\mathbf{77.9}$	$\underline{93.8}$	$\mathbf{94.0}$
50%	$\underline{74.4}$	$\mathbf{75.2}$	$\underline{78.7}$	$\mathbf{79.3}$	$\mathbf{95.1}$	$\underline{94.4}$

Appendix

Table 15: Robustness to SAE pretraining data. We compare SAEs trained on the target dataset with an SAE trained on ImageNet-1K. Bold and underlined entries indicate the best and second-best results, respectively.

Method	NABirds Acc			Waterbirds WGA			Waterbirds Acc
Selection Rate	10%	30%	50%	10%	30%	50%	10%	30%	50%
Random	$35.6$ / $40.2$	$64.9$ / $65.9$	$73.3$ / $73.8$	$52.3$ / $54.3$	$67.5$ / $\underline{66.0}$	$68.8$ / $\underline{68.6}$	$84.1$ / $76.7$	$89.6$ / $85.8$	$91.6$ / $85.9$
K-Center	$31.6$ / $33.7$	$64.9$ / $65.0$	$74.1$ / $73.5$	$57.5$ / $21.7$	$64.2$ / $49.5$	$\mathbf{79.4}$ / $62.3$	$89.4$ / $85.5$	$89.9$ / $84.9$	$91.7$ / $88.7$
Forgetting	$22.4$ / $26.9$	$63.3$ / $63.6$	$74.3$ / $73.7$	$31.6$ / $3.9$	$60.1$ / $25.6$	$73.4$ / $54.4$	$73.8$ / $75.7$	$85.1$ / $81.7$	$91.0$ / $89.8$
EL2N	$16.8$ / $21.0$	$61.2$ / $61.0$	$74.4$ / $73.4$	$31.0$ / $30.4$	$64.2$ / $38.2$	$72.9$ / $63.2$	$78.4$ / $79.9$	$87.0$ / $85.0$	$92.7$ / $89.7$
D2 Pruning	$25.8$ / $31.0$	$\underline{66.6}$ / $66.6$	$\underline{74.7}$ / $\underline{74.1}$	$47.0$ / $50.5$	$56.4$ / $37.5$	$60.8$ / $47.2$	$89.0$ / $\underline{88.6}$	$\underline{92.2}$ / $87.9$	$\underline{93.3}$ / $\underline{91.2}$

Appendix

Table 16: Cross-architecture generalization. Comparison of subset selection performance using the same selected subsets with ResNet-50 and ViT-B/16. We report classification accuracy (Acc.) for NABirds, and worst-group accuracy (WGA) and overall accuracy (Acc.) for Waterbirds; each cell shows ResNet-50 / ViT-B/16. Bold and underlined entries indicate the best and second-best results for each architecture, respectively.

Method

ImageNet-A

ImageNet-R

ImageNet-D

ImageNet-V2

ImageNet-1K Val

10\%

30\%

50\%

10\%

30\%

50\%

10\%

30\%

50\%

10\%

30\%

50\%

10\%

30\%

50\%

Random

\underline{1.4}

1.2

1.1

15.8

24.1

\mathbf{25.9}

\underline{2.9}

\underline{6.3}

6.8

36.3

\underline{49.3}

\underline{50.9}

48.6

64.2

67.2

EL2N

1.2

1.0

1.1

9.3

19.4

23.2

1.4

3.6

6.1

13.4

23.4

45.2

11.2

45.4

64.1

CCS

1.3

\underline{1.6}

\underline{1.4}

\underline{16.0}

\underline{24.5}

\underline{25.1}

2.7

6.2

\underline{6.9}

\mathbf{37.4}

49.1

\mathbf{51.2}

\mathbf{50.1}

\underline{64.8}

\mathbf{68.4}

Ours

\mathbf{1.7}

\mathbf{2.0}

\mathbf{1.5}

\mathbf{16.4}

\mathbf{25.8}

\mathbf{25.9}

\mathbf{3.3}

\mathbf{6.8}

\mathbf{7.6}

\underline{37.2}

\mathbf{50.1}

50.7

\underline{49.2}

\mathbf{65.2}

\underline{67.7}

Appendix

Table 17: ImageNet-1K classification and robustness results. We report top-1 accuracy (%) on four robustness benchmarks and the standard ImageNet-1K validation set at different selection rates. The best and second-best results are bolded and underlined, respectively.

Table 18: Concept-support profiles of 10% subsets for all baselines on (a) NABirds and (b) Stanford Cars. Every baseline stays close to the random profile on all three measures, whereas our selection covers more concepts, spreads its support more evenly, and shifts support toward less prevalent concepts.

Figure 10: Additional examples of enhanced and reduced SAE concepts. We show representative images from concepts whose support changes most under our selection relative to random selection. Highlighted patches indicate high-activation regions.

Figure 11: Additional Finer-CAM visualizations on NABirds and Stanford Cars. We show attribution maps for models trained on subsets selected by our method, Random, RAM-APL, and CCS. The Concept column highlights image patches with high activation for the corresponding SAE latent.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 186 papers

Aug 24, 2026

Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning

Existing data pruning methods predominantly rely on high-dimensional feature embeddings to measure sample importance. However, these compressed vectors often obscure fine-grained semantic interactions, leading to suboptimal coverage of rare semantic concepts in the pruned subsets. In this paper, we propose Mapping the Concept Landscape (MCL), a novel structural perception framework for transparent data pruning. Instead of abstract embeddings, we represent each image-caption pair as an explicit sample-level graph comprising entities, events, and attributes. By integrating these individual graphs into a comprehensive dataset-level graph, we characterize the global distribution of semantic concepts and quantify their rarity across the entire corpus. Based on this structured perception, we develop a greedy concept-coverage maximization algorithm that iteratively selects samples to maximize the marginal gain of high-value, under-represented concepts. Experimental results on various benchmarks demonstrate that our method not only achieves superior pruning efficiency compared to state-of-the-art methods but also provides a transparent and interpretable audit trail for the selection process.

Dongyue Wu, Tao Ma

State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Ant Group · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

Dataset Pruning Training Data Curation

Ask PDF

Jul 6, 2026

Hierarchical Scaffolding Enables Human-Like Cognitive Selectivity under Data Scarcity

Modern machine learning systems demand extensive datasets for visual recognition. Conversely, humans learn with high efficiency despite severe data limitations, often by acquiring broad categorical structures before refining finer distinctions. Inspired by this contrast, we introduce SCALA (Scaffolded Cognitive Architecture for Learning under limited dAta), a hierarchical learning framework grounded in cognitive psychology that guides models from coarse conceptual structures to fine-grained recognition. Our model exhibits human-like cognitive selectivity by effectively prioritizing task-relevant features while suppressing background distractors, a mechanism that induces a fundamental shift in representation learning. This shift is characterized by accelerated cluster formation, reduced intra-class dispersion, and enhanced semantic separability. Empirically, SCALA achieves significant accuracy improvements under severe data scarcity. Furthermore, this hierarchical scaffolding promotes robust generalization to unseen classes and accelerates the acquisition of novel categories. Collectively, our results establish SCALA as a powerful framework for achieving human-level sample efficiency and resilient category generalization in data-constrained environments.

Juhyoung Park, Jaehyuk Bae, Hyeonbo Yang +1

School of Computing 2Department of Brain and Cognitive Sciences Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Department of Brain and Cognitive Sciences Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea

Hierarchical Representation Learning Data-Efficient Learning

Ask PDF

Jun 12, 2026

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning

Current representation learning paradigms force a fundamental compromise: self-supervised methods scale to massive datasets but yield opaque features, whereas interpretable models remain bottlenecked by the need for dense human annotation. We introduce Self-Supervised Concept discOvery via Preference lEarning (\model), a label-free framework that resolves this dilemma. Instead of treating Vision-Large-Language Models (VLLMs) as static feature extractors, \model leverages them as active participants in a self-supervised preference optimization loop. By autonomously hypothesizing, validating, and reinforcing candidate visual attributes directly from raw imagery, our framework discovers novel, structured concepts without a single label. Extensive experiments across natural, medical, and physics domains demonstrate that \model successfully extracts domain-specific concepts where standard VLLMs often fail to generate. By amortizing concept discovery directly into the VLLM backbone through our self-supervised preference objective -- rather than relying on static generation and disjoint filtering -- we achieve up to a 24-point absolute improvement in downstream top-1 classification accuracy on unseen data. Our work suggest that interpretability can emerge through a model's autonomous interaction with incidental visual structures, without any human supervision.

Shilong Xiang, Zirui Zhang, Chengzhi Mao

Rutgers University

Vision-Language Models Self-Supervised Learning VLM Interpretability Preference Learning Unsupervised Visual Concept Discovery

Ask PDF

Abstract

Training Data Selection

Explore similar work

Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruning

Hierarchical Scaffolding Enables Human-Like Cognitive Selectivity under Data Scarcity

S2^22COPE: Self-Supervised Concept Discovery via Preference Learning

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning