Diversity Without Labels: Task-Agnostic Coreset Selection Across Perception and Action

Julia Hindel; Benjamin Ramtoula; Kyuhwan Yeon; Nick Heppert; Paul Newman; Daniele De Martini; Abhinav Valada

cs.CVOct 16, 2026

Diversity Without Labels: Task-Agnostic Coreset Selection Across Perception and Action

Authors: Julia Hindel, Benjamin Ramtoula, Kyuhwan Yeon, Nick Heppert, Paul Newman, Daniele De Martini, Abhinav Valada

Organizations: University of Freiburg · University of Oxford

Abstract

While data collection for model training is relatively easy, curating high-quality datasets from large collections is inherently challenging. Coreset estimation methods address this problem by identifying the best data subsets to train models on. However, existing approaches often require proxy model training, extensive annotations, or task-specific tailoring. Unsupervised alternatives typically exploit geometric properties in foundation model feature spaces, but are limited to pairwise image-to-image comparisons or comparisons against a fixed subset. We introduce CoreDNA, an unsupervised, task-agnostic coreset estimation approach that pools neuron activations from vision foundation models into per-image and per-dataset distributions. By comparing each image's distribution to the dataset-level distribution, CoreDNA can capture both low-level visual structure and high-level semantic information across layers. We find that high-quality samples concentrate in the tails of the last layer's activation statistics, which we validate by relating them to training-based coreset methods. Building on these findings, we develop a computationally efficient selection mechanism to construct task-agnostic, high-quality coresets for selective annotation and model training. Across image classification, semantic segmentation, depth estimation, and imitation learning for end-to-end driving and mobile manipulation, CoreDNA matches or outperforms the state of the art, demonstrating unprecedented generalization across perception and action tasks. These results highlight the effectiveness of compact activation-based representations for data selection.

Figures & tables

Figure 1: CoreDNA extracts neuron activation histograms from a vision foundation model, pooling them into per-layer mean $\mu$ , standard deviation $\sigma$ , and maximum $m$ . We use the last layer’s $\mu$ , $\sigma$ , and cross-layer dispersion $\delta$ to select a coreset via sliced Wasserstein minimization (“sliced-EMD”) or farthest-point sampling (“FPS”).

Figure 2: Example Cityscapes-DVPS images ranked by $\mu_{12}$ into low ( $L$ ), medium ( $M$ ), and high ( $H$ ) buckets. High- $\mu_{12}$ images tend to contain rare classes, while near-duplicates appear have similar scores in both $L$ and $H$ .

Figure 3: Correlation analysis of image patterns averaged across all datasets. Same-layer statistics correlate strongly; cross-layer correlation decays.

Selection space	Coverage $\uparrow$	Fidelity $\uparrow$
$\mu_{12}$	0.273	0.300
$\mu_{12}$ + $\delta_{\max}$	0.276	0.340
$\mu_{12}$ + $\sigma_{12}$	0.280	0.341
$\mu_{12}$ + $\delta_{\max}$ + $\sigma_{12}$	0.308	0.357

Table 1: Axes of the selection space (base:

\mu_{12}

, the mean of layer

\ell_{12}

). Correlation of run performance with the coverage and fidelity metrics of the selected set, averaged over the metrics of each family and over seven dataset-task combinations (5,265 historical runs). Best in bold.

Figure 4: 3D scatter of

z(x)

, colored by EL2N-measured hardness, for CIFAR10, PascalVOC, and Cityscapes.

		CIFAR10 $\uparrow$ (Full: 95.05)							Cityscapes $\uparrow$ (Full: 76.22)					PascalVOC $\uparrow$ (Full: 79.01)
	Method	0.01	0.05	0.1	0.2	0.3	0.5	0.7	0.01	0.1	0.3	0.5	0.7	0.01	0.1	0.3	0.5	0.7
Ref.	Random	41.41	71.21	80.90	88.17	90.69	93.11	94.43	48.81	65.33	71.87	73.68	74.87	40.02	55.56	74.81	76.65	78.06
Training w/ labels	AUM	16.10	22.80	31.93	60.54	92.53	95.24	95.28	50.96	69.38	71.41	75.42	74.00	26.51	29.03	71.73	76.84	78.14
	Forgetting	25.88	30.95	39.50	68.78	90.35	95.08	95.32	50.42	69.33	70.61	74.75	74.68	25.44	31.22	75.21	77.30	78.26
	EL2N	13.94	18.16	26.31	68.37	90.34	94.90	95.34	51.51	69.88	71.33	75.21	73.89	16.38	29.12	72.93	77.12	78.26
	CCS	41.68	74.88	86.49	88.85	91.03	91.16	91.40	47.58	61.78	72.04	71.50	74.63	44.01	43.15	73.94	75.47	78.05

Table 2: Downstream accuracy (%) across coreset ratios for CIFAR10, Cityscapes, and PascalVOC. Full: training on all data. WS: Sliced Wasserstein sampling. FPS: farthest-point-sampling.

		Cityscapes-DVPS Segm. $\uparrow$ (Full: 66.43)					Cityscapes-DVPS Depth $\downarrow$ (Full: 3.91)			SemanticKITTI-DVPS Segm. $\uparrow$ (Full: 55.58)					SemanticKITTI-DVPS Depth $\downarrow$ (Full: 3.39)
	Method	0.01	0.1	0.3	0.5	0.7	0.01	0.1	0.3	0.01	0.1	0.3	0.5	0.7	0.01	0.1	0.3
Ref.	Temporal Stride	38.28	56.71	63.83	64.47	65.02	6.17	4.28	3.97	38.74	53.14	55.28	55.80	56.32	3.67	3.34	3.35
Ref.	Random	37.88	55.77	61.23	64.03	64.83	5.70	4.36	4.02	36.81	51.17	55.08	55.98	55.90	3.70	3.36	3.36
Training w/ labels	AUM	35.54	51.28	57.52	60.56	63.21	-	-	-	27.76	42.85	51.03	53.45	55.06	-	-	-
	Forgetting	32.86	45.70	56.63	60.15	62.63	5.56	4.44	4.12	26.44	40.35	46.26	52.84	54.46	3.95	3.45	3.34
	EL2N	32.84	52.01	58.99	62.00	64.65	7.40	5.74	5.15	28.07	43.58	51.12	53.55	55.04	3.74	3.52	3.40

Table 3: Downstream accuracy across coreset ratios for Cityscapes-DVPS and SemanticKITTI-DVPS. Segmentation is evaluated in mIoU [%] and monocular metric depth estimation in RMSE. Full: training on all data. WS: Sliced Wasserstein sampling. FPS: farthest-point-sampling.

				EPDMS $\uparrow$
Method	Img	State	Pos.	0.1	0.5
Full				80.51
Random				65.44	78.88
CoreDNA (WS)	✓			66.31	78.17
CoreDNA (WS)	✓	✓		65.29	78.90
CoreDNA (WS)	✓	✓	✓	67.03	78.62

Table 4: Downstream driving performance on NAVSIM. Img, State, and Pos. indicate the coreset selection axes.

LIBERO-10 Frame-level sampling
	External	Wrist	Success
Method	Img	Img	Rate $\uparrow$
Full (5 demos)			16.9
CoreDNA (WS)	✓		17.3
CoreDNA (WS)	✓	✓	16.8
CoreDNA (FPS)	✓	✓	17.1

Table 5: Mean success rate across all tasks in LIBERO-10.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Demos
Book–Caddy	{7, 8, 19, 27, 41}
Bowl–Cabinet	{11, 26, 32, 38, 43}
Mug–Microwave	{23, 27, 37, 44, 48}
Stove–Moka	{19, 27, 30, 40, 43}
Soup–Sauce	{3, 11, 25, 44, 45}
Cream–Butter	{8, 13, 29, 36, 42}

Appendix

Table 6: Selected demonstration indices for each LIBERO-10 task.

Method	Img	State	Pos.	Ratio	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Full				–	96.99	89.20	99.07	99.74	87.81	95.86	95.50	98.34	85.95	80.51
Random				0.1	93.23	78.27	94.31	99.65	86.27	91.22	88.61	98.27	72.67	65.44
CoreDNA (WS)	✓			0.1	93.64	78.88	94.69	99.64	86.20	92.01	89.95	98.28	67.53	66.31
CoreDNA (WS)	✓	✓		0.1	93.03	77.97	94.01	99.60	86.81	91.38	88.58	98.28	73.32	65.29
CoreDNA (WS)	✓	✓	✓	0.1	93.69	79.31	94.14	99.65	86.01	92.10	89.24	98.27	77.09	67.03
CoreDNA (FPS)	✓			0.1	93.60	79.09	94.91	99.65	86.22	92.03	90.35	98.28	74.04	66.98

Appendix

Table 7: Per-metric end-to-end driving performance on NAVSIM.

Ext

Wrist

Joint

Soup–

Cream–

Stove–

Bowl–

Mug–

Book–

Mug–

Soup–

Moka–

Mug–

Success

Method

Img

State

Sauce

Butter

Moka

Cabinet

Mug

Caddy

Pudd.

Cheese

Moka

Micr.

Rate

\uparrow

LIBERO-10 Frame-level sampling

Full (5 demos)

9.3

\pm

5.0

25.3

\pm

11.6

30.7

\pm

9.0

36.7

\pm

16.7

6.0

\pm

7.1

17.3

\pm

3.4

1.3

\pm

1.9

14.0

\pm

2.8

10.0

\pm

6.5

18.7

\pm

6.2

16.9

\pm

7.0

CoreDNA (WS)

✓

16.0

\pm

11.8

21.3

\pm

5.2

32.0

\pm

2.8

40.0

\pm

5.9

2.7

\pm

3.8

19.3

\pm

4.1

2.7

\pm

0.9

10.7

\pm

0.9

6.0

\pm

5.7

22.0

\pm

11.8

17.3

\pm

5.3

CoreDNA (WS)

✓

19.3

\pm

7.7

17.3

\pm

10.5

39.3

\pm

1.9

44.7

\pm

7.4

1.3

\pm

1.9

22.0

\pm

8.6

0.7

\pm

0.9

7.3

\pm

2.5

5.3

\pm

0.9

10.7

\pm

0.9

16.8

\pm

4.3

CoreDNA (FPS)

✓

18.0

\pm

2.8

23.3

\pm

2.5

34.0

\pm

4.9

51.3

\pm

12.7

0.7

\pm

0.9

20.0

\pm

1.6

0.0

\pm

0.0

11.3

\pm

4.1

4.7

\pm

1.9

8.0

\pm

4.3

17.1

\pm

3.6

Appendix

Table 8: Mean success rate across all tasks in LIBERO-10, with per-task results, split across all experiments. We report different variations of CoreDNA’s modality input, the external camera image, the agent’s wrist camera image and its joint state.

	Coverage $\uparrow$				Fidelity $\uparrow$
Dataset	$\mu_{1}$	$\mu_{6}$	$\mu_{12}$	$\mu_{1:12}$	$\mu_{1}$	$\mu_{6}$	$\mu_{12}$	$\mu_{1:12}$
CIFAR10	0.17	0.19	0.18	0.20	0.26	0.27	0.29	0.28
PascalVOC	0.28	0.33	0.32	0.34	0.32	0.38	0.40	0.40
Cityscapes	0.26	0.26	0.26	0.23	0.01	0.05	0.03	0.04
Cityscapes-DVPS (Depth)	0.27	0.27	0.29	0.29	0.48	0.53	0.52	0.52
Cityscapes-DVPS (Segm.)	0.39	0.37	0.36	0.36	0.53	0.54	0.54	0.51

Appendix

Table 9: Layer

\ell

of the mean axis (second axis fixed to

\delta_{\max}

);

\mu_{1:12}

is the mean over all layers. Pearson correlation between run performance and the metrics of its selected set averaged over the metrics of each family.

	Coverage $\uparrow$					Fidelity $\uparrow$
Dataset	none	$m_{12}$	$\sigma_{12}$	$\delta_{\mathrm{avg}}$	$\delta_{\max}$	none	$m_{12}$	$\sigma_{12}$	$\delta_{\mathrm{avg}}$	$\delta_{\max}$
CIFAR10	0.18	0.19	0.19	0.16	0.18	0.27	0.28	0.29	0.24	0.29
PascalVOC	0.32	0.32	0.32	0.28	0.32	0.36	0.39	0.41	0.33	0.40
Cityscapes	0.38	0.29	0.27	0.19	0.26	0.02	0.04	0.04	0.01	0.03
Cityscapes-DVPS (Depth)	0.19	0.24	0.23	0.30	0.29	0.47	0.47	0.45	0.53	0.52
Cityscapes-DVPS (Segm.)	0.40	0.36	0.36	0.37	0.36	0.47	0.51	0.54	0.52	0.54

Appendix

Table 10: Second axis (mean axis fixed to layer

\ell_{12}

; none = the mean axis alone). Pearson correlation between run performance and the metrics of its selected set averaged over the metrics of each family.

	Coverage $\uparrow$			Fidelity $\uparrow$
Dataset	$\mu_{12}$ + $\sigma_{12}$	$\mu_{12}$ + $\delta_{\max}$	$\mu_{12}$ + $\sigma_{12}$ + $\delta_{\max}$	$\mu_{12}$ + $\sigma_{12}$	$\mu_{12}$ + $\delta_{\max}$	$\mu_{12}$ + $\sigma_{12}$ + $\delta_{\max}$
CIFAR10	0.19	0.18	0.20	0.29	0.29	0.30
PascalVOC	0.32	0.32	0.35	0.41	0.40	0.42
Cityscapes	0.27	0.26	0.28	0.04	0.03	0.04
Cityscapes-DVPS (Depth)	0.23	0.29	0.29	0.45	0.52	0.50
Cityscapes-DVPS (Segm.)	0.36	0.36	0.42	0.54	0.54	0.57

Appendix

Table 11: Correlation between run performance and the metric of its selected set for the three-axis space and for each two-axis subspace. Coverage and fidelity are the same metric families as in the other tables.

Panel of Figure 5 — Figure 5: We compare different histogram bin sizes $B$ in (a) and different backbones for neuron-pooling feature extraction in (b, c).

Mean	Std	Third axis	0.1	0.5
			Accuracy $\uparrow$
$\mu_{1:12}$	$\sigma_{12}$	$\delta_{\max}$	82.06	92.88
$\mu_{1}$	$\sigma_{12}$	$\delta_{\max}$	80.67	93.05
$\mu_{1:12}$	$\sigma_{1:12}$	$\delta_{\max}$	79.21	93.22
$\mu_{1}$	$\sigma_{1}$	$\delta_{\max}$	79.39	93.05
$\mu_{12}$	$\sigma_{12}$	$\delta_{\mathrm{avg}}$	82.63	93.14

Appendix

Table 12: Ablation over axis choices for coreset selection (CIFAR10, sliced Wasserstein selection). Performance is downstream accuracy (%).

		Accuracy $\uparrow$
$E$	$g$	0.1	0.5
3	0.01	81.35	93.34
5	0.01	80.16	93.40
10	0.001	82.06	93.56
10	0.01	82.40	93.49
10	0.1	80.88	93.51

Appendix

Table 13: Sensitivity to the sliced Wasserstein hyperparameters

E

(number of projections) and

g

(near-duplicate gap) on CIFAR10. Performance is downstream accuracy (%).

	Normal $\uparrow$		With Duplicates $\uparrow$
Method	50000 ( $\alpha=0.1$ )	250000 ( $\alpha=0.5$ )	50000 ( $\alpha=0.01$ )	250000 ( $\alpha=0.05$ )
Full	95.05	95.05	95.33 (+0.28)	95.33 (+0.28)
Random	80.90	93.11	81.46 (+0.56)	92.57 ( $-$ 0.54)
AUM	31.93	95.24	18.87 ( $-$ 13.06)	31.88 ( $-$ 63.36)
Forgetting	39.50	95.08	36.42 ( $-$ 3.08)	62.58 ( $-$ 32.50)
EL2N	26.31	94.90	17.64 ( $-$ 8.67)	31.25 ( $-$ 63.65)

Appendix

Table 14: Downstream accuracy (%) on CIFAR10, comparing the normal dataset to a version with duplicated samples CIFAR10-Dupl. Values in parentheses show the difference (duplicated

-

normal).

Data Selection

Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.

See all 84 papers

A Coreset Selection Framework with Ensemble Aggregation for Image Classification
Jul 10, 2026Pedro Rocha Dantas, Lucas Pascotti ValemTraining Data Selection Coreset Selection
Institute of Mathematics and Computer Science (ICMC) University of S˜ao Paulo (USP) S˜ao Carlos – SP – Brazil
Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?
Jun 16, 2026Trisha Mittal, Akshay Mehra, Joshua KimballTraining Data Selection Dataset Distillation
1Dolby Laboratory
Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
Jul 7, 2026Cemil-Andrei Dilmac, Florinel-Alin Croitoru, Radu Tudor IonescuTraining Data Selection Coreset Selection
Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania

Jul 10, 2026J/K move · Enter open · S save

A Coreset Selection Framework with Ensemble Aggregation for Image Classification

Pedro Rocha Dantas, Lucas Pascotti Valem

Institute of Mathematics and Computer Science (ICMC) University of S˜ao Paulo (USP) S˜ao Carlos – SP – Brazil

Training Data Selection Coreset Selection Image Classification

Open paper AskPDF arXiv

The rapid growth of image data has produced large-scale datasets, raising concerns about the time and memory costs of model training. Selecting representative training subsets, however, remains challenging: individual sample contributions are unclear, and model behavior varies across datasets and runs. We address these challenges with a framework that combines coreset selection with an ensemble aggregation over multiple runs. For coreset selection, we propose SCOre-Stratified Selection (SCOSS), which partitions the training data into intervals based on a chosen score and samples from each interval. The ensemble combines predictions from multiple runs, each performed on an independently sampled training subset. As baselines, we use moderate and random selection, each in original and class-balanced versions. We assess the framework with Simple Graph Convolution (SGC) and Support Vector Machine (SVM) classifiers under different sampling ratios. Experiments show that SCOSS is competitive with baselines, often the best choice for SGC, and enables favorable trade-offs between accuracy and efficiency. On the fine-grained dataset, SGC with SCOSS outperforms SVMs when using fewer labeled samples. The code and supplementary materials are publicly available at http://scoss.lucasvalem.com.

Abstract

Data Selection

Explore similar work

A Coreset Selection Framework with Ensemble Aggregation for Image Classification

Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation