Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget

Xiyang Zhang; Hongzhi Wang; Yuanhe Tian; Yan Song

cs.LGOct 24, 2025

Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget

Authors: Xiyang Zhang, Hongzhi Wang, Yuanhe Tian, Yan Song

Organizations: Faculty of Computing, Harbin Institute of Technology

Abstract

Selecting compact training sets from heterogeneous text sources requires balancing target relevance, label reliability, and coverage. For small classifiers, discarding unfamiliar but useful texts can be as harmful as retaining annotation errors. Gradient-based selection estimates immediate target utility, whereas confidence and coverage scores capture complementary properties. Combining them with fixed weights leaves the utility sacrificed for this evidence uncontrolled across candidate pools. We propose Data Value Contribution (\DVC), which gates lexical, hidden-layer, and prediction-level evidence by neighborhood label support and calibrates its influence under a local utility budget; exact MLP gradient contractions and a monotone scoring path make the budget directly verifiable. At 20% source retention, DVC achieves 75.4 mean macro F1 across eight classification and domain-transfer tasks, exceeding the strongest baseline under shared final training by 0.7 points and random selection by 4.3; an equal-budget tuning study confirms a 0.6-point gain over calibrated quality-only selection.

Figures & tables

Figure 1: DVC combines layer-wise and global evidence while limiting the reference utility lost to its auxiliary correction. Source queries and subset rankings use the same current model; a fresh classifier trains on the retained source examples.

Method	TREC	Polarity	20 News	AG News	Amazon avg.	8-task avg.
Random	68.8	62.6	49.4	86.4	75.5	71.1
Balanced random	69.8	62.3	49.7	86.4	75.6	71.3
Entropy	70.1	62.1	47.9	87.2	75.3	71.1
$k$ -center	70.5	61.8	48.2	86.5	76.3	71.5
BADGE-style	67.2	62.4	48.9	86.8	75.3	70.8
EL2N	46.7	59.2	39.1	36.5	73.7	59.5

Table 1: Comparison at 20% source retention over ten matched seeds. All methods share the final optimizer, learning rate, class weighting, and update count within each task. Amazon averages four target domains; the last column weights all eight tasks equally. Full-pool and reference-only controls use different training resources and do not enter subset rankings. Bold marks the best subset means, including ties at the displayed precision.

Views	TREC	Amazon avg.	8-task avg.
Two views	81.1	78.5	75.4
Input only	80.7	77.9	74.9
Hidden only	73.5	78.2	74.3

Table 2: Complementary representation controls over ten paired seeds. Only auxiliary views change; full-parameter utility and other settings remain fixed. Full task results appear in Appendix C.10 .

Correction	Fixed	Budgeted
Full hierarchy	75.2	75.1
Quality	74.8	74.5
Compact hierarchy	–	75.1

Table 3: Independently tuned scoring corrections: eight-task mean macro F1 over ten matched seeds. Each cell receives 36 configurations on three development seeds; final training is shared. Compact scoring removes uncertainty and stability. Complete task results and paired intervals appear in Appendix C.9 .

Figure 2: Utility loss (a), synthetic-noise label purity (b), and target-reference neighborhood support (c) of newly selected batches. Candidates, parameters, and class quotas are held fixed. Lines average three selection states within each of five seeds; shading gives the standard deviation across seed averages. The dashed line is the permitted utility loss.

Figure 3: TREC examples exchanged under the same observed-label quota at two fixed selector states. The original corpus label is shown only for analysis. Gradient-only selection favors the higher-utility text in each pair; DVC favors the better-supported label. Scores come from development seed 101 and the primary settings.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Classes	Pool	Ref.	Dev.	Test
TREC	6	4,601	512	256	500
Polarity	2	7,778	512	256	2,139
20 News	20	10,189	512	256	7,317
AG News	4	119,071	512	256	7,600
Books	2	5,898	256	256	1,444
DVD	2	5,900	256	256	1,442

Appendix

Table 4: Dataset sizes after text cleaning and duplicate removal. The Amazon row names the held-out target domain; its pool consists of the other three domains.

Method	TREC	Polarity	20 News	AG News	Books	DVD	Electronics	Kitchen
Random	68.8 $\pm$ 3.7	62.6 $\pm$ 1.2	49.4 $\pm$ 0.4	86.4 $\pm$ 0.3	71.3 $\pm$ 2.8	73.6 $\pm$ 1.1	77.3 $\pm$ 2.0	79.8 $\pm$ 1.2
Balanced random	69.8 $\pm$ 4.4	62.3 $\pm$ 1.5	49.7 $\pm$ 0.5	86.4 $\pm$ 0.2	71.9 $\pm$ 1.2	73.7 $\pm$ 1.1	76.6 $\pm$ 1.5	80.2 $\pm$ 1.5
Entropy	70.1 $\pm$ 2.0	62.1 $\pm$ 1.4	47.9 $\pm$ 0.9	87.2 $\pm$ 0.2	69.8 $\pm$ 1.7	73.0 $\pm$ 1.3	78.0 $\pm$ 1.4	80.5 $\pm$ 1.4
$k$ -center	70.5 $\pm$ 2.1	61.8 $\pm$ 1.8	48.2 $\pm$ 1.2	86.5 $\pm$ 0.3	71.0 $\pm$ 1.7	74.7 $\pm$ 0.7	77.9 $\pm$ 1.3	81.6 $\pm$ 0.5
BADGE-style	67.2 $\pm$ 3.3	62.4 $\pm$ 1.5	48.9 $\pm$ 0.8	86.8 $\pm$ 0.2	71.3 $\pm$ 2.2	72.8 $\pm$ 1.6	76.9 $\pm$ 1.1	80.4 $\pm$ 1.8
EL2N	46.7 $\pm$ 6.6	59.2 $\pm$ 1.3	39.1 $\pm$ 0.7	36.5 $\pm$ 1.1	70.7 $\pm$ 1.6	70.8 $\pm$ 2.3	73.6 $\pm$ 1.8	79.7 $\pm$ 1.3

Appendix

Table 5: Comparison at 20% source retention over ten matched seeds. All methods share the final optimizer, learning rate, class weighting, and update count within each task. Full-pool and reference-only controls use different training resources and do not enter subset rankings. Entries give per-task macro F1 mean

\pm

sample standard deviation; bold indicates the best subset mean at the displayed precision.

Task	Width	Steps	LR	Decay	Power
TREC	64	400	0.001	0.1	0.5
Polarity	64	400	0.001	0.1	1
20 News	64	800	0.001	0.1	0.5
AG News	64	800	0.001	0.1	0.5
Books	64	400	0.003	0.1	0
DVD	64	200	0.001	0.1	0

Appendix

Table 6: Common final-training protocol for the primary comparison. AdamW, minibatch size 64, and these settings apply to every subset, full-pool, and reference-only method. Power denotes inverse-class-frequency weighting; GradMatch also preserves its example weights.

Task	$\rho$	Active (%)	Mean $t$	Max. loss (%)
TREC	0.5	0.0	1.000	39.2
Polarity	0.5	0.0	1.000	39.5
20 News	0.5	0.0	1.000	18.4
AG News	0.025	100.0	0.126	2.5
Books	0.025	100.0	0.141	2.5
DVD	0.5	1.6	0.999	49.9

Appendix

Table 7: Calibration in the primary comparison over ten matched seeds, using the selection trajectories underlying Table 1 . Active is the percentage of rounds with

t<1

, averaged across runs; mean

t

first averages rounds within each run. Max. loss is the largest computed utility gap across all rounds and seeds, as a percentage of the feasible utility range.

Comparison	Subset and final-training protocol	Seeds
Primary MLP	Source-only subsets; shared task-specific final-training protocol; seventeen selectors.	10
Method-specific MLP	Source-only subsets; separately chosen final learning rate and weighting power; 20% retention / other budgets.	10 / 5
Primary CNN transfer	The primary MLP subsets held fixed; shared CNN settings, without reference examples.	10
Independent scoring	Full/quality evidence with/without calibration and compact evidence; equal development search; shared final-training protocol and sample-ID training order.	10
Nested reference	Fixed source pool; three 128-example reference compositions within a 512-example reference set.	5
Components and tolerance	Fixed selection settings; shared final MLP training; complete selection trajectories.	5

Appendix

Table 8: Experimental protocols. Seed counts refer to evaluation, not development. All comparisons use disjoint development seeds. The independently tuned scoring controls retain common final training and use a separate paired evaluation.

Benchmark weighting	vs. Alignment + quality	vs. Calibrated quality
Eight tasks	0.7 [0.3, 1.1]	0.8 [0.5, 1.1]
Seven tasks, excluding TREC	0.3 [-0.1, 0.7]	0.4 [0.1, 0.7]
Five corpora	0.9 [0.5, 1.3]	1.0 [0.6, 1.3]
Four corpora, excluding TREC	0.3 [0.0, 0.7]	0.4 [0.2, 0.7]

Appendix

Table 9: Descriptive sensitivity of DVC gains in the primary comparison. Entries are macro-F1 point differences and paired-seed 95% intervals. Each interval aggregates tasks within each of ten seeds before estimating variation. Corpus weighting first averages the four Amazon domains, then weights that mean equally with each other corpus. These exploratory summaries concern the fixed benchmarks and do not estimate uncertainty over unseen tasks.

Task	$k$	$\lambda_{\max}$	$\rho$	Ref. init.	Quotas	Initial set
TREC	4	2	0.5	50	yes	release
Polarity	32	2	0.5	50	yes	keep
20 News	32	2	0.5	50	yes	keep
AG News	32	4	0.025	50	yes	keep
Books	8	4	0.025	0	yes	release
DVD	8	4	0.5	20	yes	release

Appendix

Table 10: DVC selection settings, fixed across retention budgets. Reference initialization is measured in gradient updates.

Method	Macro F1	Accuracy
Random	69.6 $\pm$ 0.6	70.0 $\pm$ 0.6
Balanced random	70.0 $\pm$ 1.0	70.4 $\pm$ 1.0
Entropy	65.8 $\pm$ 1.7	66.5 $\pm$ 1.5
$k$ -center	71.5 $\pm$ 0.8	71.8 $\pm$ 0.8
BADGE-style	68.7 $\pm$ 1.1	69.1 $\pm$ 0.9
EL2N	58.7 $\pm$ 1.5	59.9 $\pm$ 1.3

Appendix

Table 11: Prospective BANKING77 evaluation at a 20% source budget (10 seeds per setting). Entries give mean and sample standard deviation in percent. Each method uses its 20 Newsgroups settings without BANKING77 tuning. Full-pool and reference-only controls have different training resources.

Figure 4: MLP macro F1 as the retained source budget increases. Lines show seed means; shaded bands show one standard error computed with the observed seed count. The 20% points use 10 seeds per setting; other budgets use 5 seeds per setting. AG News starts at 5%.

Selector	10% / 5%	20%	30% / 20%	40%
Random	66.1 $\pm$ 0.6	71.6 $\pm$ 0.7	74.1 $\pm$ 0.2	75.4 $\pm$ 0.3
Balanced random	65.3 $\pm$ 0.9	71.1 $\pm$ 0.4	73.3 $\pm$ 0.3	74.4 $\pm$ 0.7
Entropy	66.2 $\pm$ 1.1	71.5 $\pm$ 0.4	74.3 $\pm$ 0.7	75.8 $\pm$ 0.4
$k$ -center	66.3 $\pm$ 1.3	71.7 $\pm$ 0.6	73.5 $\pm$ 0.5	74.6 $\pm$ 1.0
BADGE-style	65.9 $\pm$ 0.8	71.0 $\pm$ 0.7	74.3 $\pm$ 0.6	75.8 $\pm$ 0.3
EL2N	57.0 $\pm$ 1.2	59.7 $\pm$ 0.8	59.6 $\pm$ 0.4	65.9 $\pm$ 1.3

Appendix

Table 12: Eight-task macro F1 under method-specific training settings. Means and sample standard deviations first average tasks within each matched seed. The 20% column uses ten seeds; other columns use the five seeds common to every task. In split headers, the first budget applies to seven tasks and the second to AG News; unsplit headers apply to all eight. Per-task curves for five selectors appear in Figure 4 ; the horizontal axes give each task’s actual retention budget.

Task	DVC	Full, fixed	Quality, budgeted	Quality, fixed	Compact, budgeted
TREC	80.5 $\pm$ 1.5	80.4 $\pm$ 1.0	77.0 $\pm$ 1.4	78.9 $\pm$ 1.9	78.9 $\pm$ 2.3
Polarity	65.5 $\pm$ 0.6	65.9 $\pm$ 0.9	65.7 $\pm$ 0.7	65.8 $\pm$ 0.8	65.3 $\pm$ 1.0
20 News	57.0 $\pm$ 0.4	57.3 $\pm$ 0.2	56.4 $\pm$ 0.5	56.7 $\pm$ 0.2	57.2 $\pm$ 0.3
AG News	86.2 $\pm$ 0.3	85.8 $\pm$ 0.3	84.9 $\pm$ 0.7	85.5 $\pm$ 0.5	86.0 $\pm$ 0.3
Books	74.2 $\pm$ 1.2	74.0 $\pm$ 1.1	74.6 $\pm$ 1.0	73.9 $\pm$ 1.6	74.9 $\pm$ 1.2
DVD	75.8 $\pm$ 1.2	75.8 $\pm$ 1.2	77.1 $\pm$ 1.1	75.5 $\pm$ 1.6	76.2 $\pm$ 1.0

Appendix

Table 13: Independently developed structure and calibration controls at 20% retention over ten matched evaluation seeds. All families share the data resources and final-training protocol within each task. Entries are macro F1 mean

\pm

sample standard deviation; bold indicates the best mean. The last row compares DVC with each family using paired seed-wise eight-task means. Intervals describe variation on these fixed tasks. Identical selected configurations share evaluations (DVD: DVC, Full, fixed).

Task	$\rho$ (%)	Active (%)	Changed (%)	Max. loss (%)	Fixed viol. (%)	Select. (s)
TREC	50	1.4	1.4	50.0	1.4	1.0
Polarity	25	67.1	67.1	25.0	90.4	1.6
20 News	25	0.0	0.0	15.2	0.0	4.3
AG News	25	91.4	91.4	25.0	32.9	66.1
Books	25	38.3	38.3	25.0	62.8	0.9
DVD	100	0.0	0.0	43.0	0.0	1.0

Appendix

Table 14: Actual calibration on the evaluation trajectories. Active and changed are the mean percentages of rounds with a reduced coefficient and a changed batch membership, respectively. Max. loss is the largest computed gap over every round and seed, as a percentage of the feasible utility range. Viol. counts all budget violations. Fixed viol. is the percentage of fixed-full rounds that exceed the corresponding DVC tolerance, measured on that method’s own trajectory. Selection time is the median elapsed seconds across seeds and includes selector training.

Task	Shared (%)	$\Delta$ noise (pp)	$\Delta$ macro F1 [95% CI]	Corrected	Introduced
20 News	52.9	+0.2	+0.6 [+0.3, +0.8]	528.5	442.7
AG News	29.2	-14.5	+1.3 [+0.9, +1.7]	338.3	243.6
Books	54.0	–	-0.5 [-1.5, +0.5]	131.6	138.8
DVD	39.1	–	-1.3 [-2.5, 0.0]	101.7	119.1
Electronics	40.6	–	-0.1 [-1.5, +1.2]	134.9	136.7
Kitchen	35.3	–	+1.4 [+0.6, +2.2]	146.7	126.0

Appendix

Table 15: Endpoint comparison of independently tuned DVC and calibrated quality under the same final-training protocol. Shared is the proportion of retained examples common to both subsets. Differences are DVC minus calibrated quality; noise is the known synthetic corruption rate. Amazon has no synthetic corruption and its annotation accuracy is not known. Corrected and Introduced count test errors respectively repaired and introduced by DVC. Means and paired intervals use all matched evaluation seeds.

Figure 5: TREC endpoint differences between independently tuned DVC and calibrated quality. Bars show matched-seed means; intervals are paired 95% intervals over ten seeds. All six observed class counts are identical between the two subsets. Uncorrupted counts use synthetic corruption only for analysis.

Method	TREC	Polarity	20 News	AG News	Books	DVD	Electronics	Kitchen
Two views	81.1 $\pm$ 1.6	65.9 $\pm$ 1.2	56.3 $\pm$ 0.3	86.1 $\pm$ 0.4	74.5 $\pm$ 1.1	76.2 $\pm$ 0.8	80.8 $\pm$ 1.4	82.4 $\pm$ 0.6
Input only	80.7 $\pm$ 1.1	65.0 $\pm$ 0.8	56.0 $\pm$ 0.5	85.7 $\pm$ 0.4	74.0 $\pm$ 1.0	74.3 $\pm$ 1.2	80.5 $\pm$ 1.0	83.0 $\pm$ 1.1
Hidden only	73.5 $\pm$ 3.6	65.0 $\pm$ 0.8	56.6 $\pm$ 0.5	86.4 $\pm$ 0.5	74.1 $\pm$ 0.9	76.0 $\pm$ 1.3	80.9 $\pm$ 1.2	81.8 $\pm$ 1.0

Appendix

Table 16: Complementary representation controls over ten paired seeds. Quality, relevance, and diversity use both views, the input view, or the hidden view; full-parameter utility and all other settings remain fixed. Entries give per-task macro F1 mean

\pm

sample standard deviation; bold indicates the best subset mean at the displayed precision.

Figure 6: Complete-trajectory tolerance sweep under a shared final-training protocol. Top: macro F1 after fresh training. Bottom: the largest realized utility loss divided by its feasible range across each run’s selection rounds; the dashed curve is the allowance

\rho

. Points show means and sample standard deviations over five seeds. All other selection and training settings are fixed.

$\rho$	TREC F1	Polarity F1	Books F1	Electronics F1	TREC gap	Polarity gap	Books gap	Electronics gap
0	70.3 $\pm$ 3.7	65.4 $\pm$ 0.7	74.3 $\pm$ 1.1	77.2 $\pm$ 0.9	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0
0.025	76.3 $\pm$ 3.3	65.1 $\pm$ 0.8	73.8 $\pm$ 0.6	79.1 $\pm$ 1.0	2.5 $\pm$ 0.0	2.5 $\pm$ 0.0	2.5 $\pm$ 0.0	2.5 $\pm$ 0.0
0.1	79.2 $\pm$ 1.5	66.2 $\pm$ 1.1	75.4 $\pm$ 0.5	80.0 $\pm$ 1.3	10.0 $\pm$ 0.0	10.0 $\pm$ 0.0	10.0 $\pm$ 0.0	10.0 $\pm$ 0.0
0.25	81.3 $\pm$ 1.2	66.1 $\pm$ 0.9	75.1 $\pm$ 0.9	81.3 $\pm$ 1.7	24.7 $\pm$ 0.3	24.9 $\pm$ 0.2	25.0 $\pm$ 0.0	25.0 $\pm$ 0.0
0.5	81.6 $\pm$ 1.4	66.4 $\pm$ 1.5	75.3 $\pm$ 0.5	81.0 $\pm$ 0.7	31.0 $\pm$ 4.6	34.5 $\pm$ 3.4	47.7 $\pm$ 2.1	48.5 $\pm$ 1.0
1	81.6 $\pm$ 1.4	66.4 $\pm$ 1.5	75.7 $\pm$ 0.3	80.9 $\pm$ 0.7	31.0 $\pm$ 4.6	34.5 $\pm$ 3.4	50.4 $\pm$ 4.9	49.8 $\pm$ 3.7

Appendix

Table 17: Complete-trajectory utility-tolerance analysis over five matched seeds. F1 is in percent. Gap is the maximum normalized utility loss across selection rounds within a run, in percent of the feasible utility range; entries give means and sample standard deviations across seeds. Only the tolerance changes. These are final-classifier results, rather than fixed-state batch measurements.

Variant	TREC	Polarity	20 News	AG News	Books	DVD	Electronics	Kitchen
DVC	81.6 $\pm$ 1.4	66.4 $\pm$ 1.5	56.3 $\pm$ 0.3	86.0 $\pm$ 0.4	73.8 $\pm$ 0.6	76.4 $\pm$ 0.8	81.3 $\pm$ 1.7	82.5 $\pm$ 0.5
No auxiliary terms	68.0 $\pm$ 2.5	64.9 $\pm$ 1.0	55.9 $\pm$ 0.7	86.2 $\pm$ 0.3	73.9 $\pm$ 1.5	73.3 $\pm$ 1.5	78.2 $\pm$ 2.2	81.6 $\pm$ 1.0
No quality gate	74.0 $\pm$ 7.0	65.8 $\pm$ 1.1	56.3 $\pm$ 0.6	86.6 $\pm$ 0.6	73.1 $\pm$ 0.8	74.9 $\pm$ 0.9	79.8 $\pm$ 0.9	83.0 $\pm$ 0.6
No relevance	80.5 $\pm$ 1.2	65.9 $\pm$ 0.9	56.3 $\pm$ 0.5	86.3 $\pm$ 0.4	73.9 $\pm$ 0.8	76.1 $\pm$ 1.1	80.7 $\pm$ 1.1	82.4 $\pm$ 1.1
No diversity	80.3 $\pm$ 0.9	66.0 $\pm$ 0.8	56.3 $\pm$ 0.4	85.9 $\pm$ 0.5	73.8 $\pm$ 1.1	75.4 $\pm$ 0.5	80.7 $\pm$ 1.0	82.3 $\pm$ 0.9
No uncertainty	75.8 $\pm$ 2.0	65.9 $\pm$ 1.3	56.4 $\pm$ 0.7	85.8 $\pm$ 0.6	73.7 $\pm$ 1.4	75.7 $\pm$ 0.9	80.7 $\pm$ 0.6	82.0 $\pm$ 1.1

Appendix

Table 18: Complete component comparison: macro F1 mean

\pm

sample standard deviation over five paired seeds. Each control retains DVC initialization, quotas, coefficients, tolerances, and final-training protocol. The no-impact control removes the ranking utility and its calibration. Single-signal deletions average the remaining auxiliary signals; compact drops uncertainty and stability jointly.

Variant	TREC	Polarity	20 News	AG News	Books	DVD	Electronics	Kitchen
DVC	79.1 $\pm$ 3.7	66.3 $\pm$ 1.1	56.4 $\pm$ 0.4	86.2 $\pm$ 0.1	75.3 $\pm$ 0.9	76.2 $\pm$ 0.7	81.7 $\pm$ 1.5	82.5 $\pm$ 1.3
No utility constraint	79.1 $\pm$ 3.7	66.3 $\pm$ 1.1	56.4 $\pm$ 0.4	86.3 $\pm$ 0.3	74.3 $\pm$ 1.2	76.2 $\pm$ 0.7	81.0 $\pm$ 0.8	82.0 $\pm$ 0.8
Keep initial subset	71.1 $\pm$ 4.4	–	–	–	–	–	–	–
No class quotas	70.6 $\pm$ 7.8	66.4 $\pm$ 1.1	56.5 $\pm$ 0.6	86.8 $\pm$ 0.2	74.5 $\pm$ 1.7	76.9 $\pm$ 0.4	81.1 $\pm$ 1.0	82.6 $\pm$ 0.8
No prior calibration	70.5 $\pm$ 8.3	66.3 $\pm$ 1.1	56.2 $\pm$ 0.4	86.0 $\pm$ 0.2	74.7 $\pm$ 1.2	76.2 $\pm$ 1.3	81.1 $\pm$ 0.7	82.3 $\pm$ 0.7

Appendix

Table 19: Structural controls in the supplementary method-specific comparison: macro F1 mean and sample standard deviation over 5 seeds per setting. Each control keeps its matched DVC settings except the named change; removing the utility constraint retains the same maximum correction. Initial-subset retention is evaluated on TREC only; dashes denote unreported combinations.

Selector	Observed	Uncorrupted	Noise (%)
DVC	14.0	11.8	2.0
Random	53.2	10.5	22.6
GLISTER-style	21.4	9.3	5.0
Flat six-signal	4.3	4.2	1.8
Alignment	15.3	2.9	11.2

Appendix

Table 20: TREC abbreviation-class support in the primary comparison at 20% retention, averaged over ten seeds. Observed counts examples labeled ABBR; uncorrupted counts retained ABBR examples whose observed label equals the corpus label. Noise is the corruption percentage of the entire retained subset.

Task	Sources	Reference	Fixed DVC	Tuned DVC	Calibrated quality	Random
TREC	clean	128	62.4 $\pm$ 5.5	63.7 $\pm$ 3.1	72.3 $\pm$ 1.0	80.0 $\pm$ 2.7
TREC	clean	512	76.9 $\pm$ 7.8	–	68.9 $\pm$ 9.2	80.0 $\pm$ 2.7
TREC	mixed	128	56.5 $\pm$ 3.0	64.7 $\pm$ 4.3	61.9 $\pm$ 4.3	70.2 $\pm$ 2.9
TREC	mixed	512	71.0 $\pm$ 7.6	–	62.9 $\pm$ 7.2	70.2 $\pm$ 2.9
Polarity	clean	128	66.8 $\pm$ 0.3	66.3 $\pm$ 0.6	65.4 $\pm$ 0.6	69.5 $\pm$ 0.7
Polarity	clean	512	68.6 $\pm$ 0.6	–	68.2 $\pm$ 0.3	69.5 $\pm$ 0.7

Appendix

Table 21: Controlled reference-size diagnosis at 20% source retention. Source, development, test, and feature partitions are fixed. At 128 references, means and standard deviations use five training-seed means, each averaged over three reference compositions. At 512 references there is one shared reference set and five training seeds. Calibrated quality is independently tuned at 128 and inherits fixed DVC settings at 512; tuned DVC is evaluated only at 128.

Task	$\lambda_{Q}$	RHO init.	Initial set
TREC	2	0	keep
Polarity	1	0	keep
20 News	2	0	release
AG News	0.25	50	keep
Books	1	0	keep
DVD	1	0	keep

Appendix

Table 22: Comparator selection settings. Alignment plus quality inherits all other DVC selection settings. RHO-LOSS uses a frozen loss model fitted for 400 updates; final training follows Table 6 .

$\rho$	$\Delta$ purity	$\Delta$ coverage	$\Delta$ loss gain
0	0.0	0.0	29.0
0.025	+0.8	0.4	-15.0
0.1	+0.6	-1.5	-292.0
0.25	-0.5	-4.0	-175.0
0.5	-0.3	-6.4	-274.0
1	-0.3	-6.4	-274.0

Appendix

Table 23: Full hierarchy minus quality correction at a matched utility loss. Purity is in percentage points over TREC and Polarity; coverage and online loss gains average all four tasks. Coverage and loss gains are multiplied by

10^{3}

and

10^{6}

, respectively. Each mean averages three states within five seeds. Primary uses each task’s selected tolerance.

Selector	TREC	Polarity	20 News	AG News	Mean
Macro F1
Random	$62.4\pm 6.0$	$58.0\pm 1.2$	$24.1\pm 0.8$	$80.1\pm 0.8$	$56.2\pm 1.7$
GLISTER-style	$73.9\pm 3.4$	$61.0\pm 0.8$	$37.1\pm 1.0$	$83.1\pm 0.4$	$63.8\pm 0.8$
Flat six-signal	$65.5\pm 2.6$	$62.3\pm 1.2$	$38.2\pm 0.5$	$85.5\pm 0.4$	$62.9\pm 0.9$
Calibrated quality	$75.8\pm 3.4$	$62.8\pm 0.7$	$37.5\pm 0.6$	$81.2\pm 0.9$	$64.3\pm 0.8$
DVC	$76.3\pm 3.2$	$62.4\pm 1.1$	$37.8\pm 0.9$	$81.5\pm 1.2$	$64.5\pm 0.9$

Appendix

Table 24: Fresh CNN training on subsets selected in the primary comparison. Macro F1 entries are mean

\pm

sample standard deviation over ten matched seeds, in percent. All five selectors retain 20% of source examples and share the same CNN settings within each task. The final column averages tasks within each seed. The CNN learns embeddings from scratch; reference examples are excluded from its training.

Task	Method	Select	Train	Scored
TREC	Random	0.0	64.0	0
TREC	Alignment	46.0	62.0	5,376
TREC	GLISTER-style	265.0	63.0	7,424
TREC	DVC	179.0	63.0	7,424
Polarity	Random	0.0	61.0	0
Polarity	Alignment	92.0	62.0	10,496

Appendix

Table 25: Median CPU timings at 20% budget (3 seeds per setting), measured sequentially with one PyTorch thread on an Apple M5. Selection includes the initialization, reference pretraining, source queries, scoring, and online updates required by each method. Scored counts candidate evaluations, including repeat queries. Select and Train are milliseconds. Loading, lexical preprocessing, and final evaluation are excluded.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 184 papers

When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation
Oct 26, 2026Yingjie Bai, Weiming ZhiData Selection Training Data Selection
School of Computer Science, The University of Sydney
HARP: Efficient Data Selection for Finetuning Large Language Models
Jun 5, 2026Ning Wang, Zhengxin Zhang, Maosen Tang +3LLM Fine-Tuning Active Learning
Cornell University · The Hong Kong University of Science and Technology
MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
May 28, 2026Haowen Wang, Yaxin Du, Jian Yang +9Data Selection Efficient Language Model Training
1Beihang University · 2IQuest Research · 3Shanghai Jiao Tong University +2

Oct 26, 2026J/K move · Enter open · S save

When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation

Yingjie Bai, Weiming Zhi

School of Computer Science, The University of Sydney

Data Selection Training Data Selection Data Valuation Statistical Learning Theory

Open paper AskPDF arXiv

Per-point data-valuation scores -- Data Shapley, influence, leverage -- are routinely used to select training data: keep the points with the top

k

scores. We argue that top-

k

by a fixed, mode-blind per-point score is the wrong selection rule precisely in the regime where data selection matters. The reason is that a point's value is not a fixed number but depends on what else has been selected -- a near-duplicate of an already-chosen point adds almost nothing, however high its score -- and a per-point scalar cannot express this interaction. The theory quantifies this failure exactly: in an equal latent-mode model, fixed top-

k

selection incurs a closed-form excess risk over adaptive coverage because it repeatedly spends budget on already-covered modes. The gap is largest when the budget is tight, the learner representation is redundant, and missing a mode is costly. The practical rule is simple: use balance when the labels available at selection identify the learner's modes, and use explicit coverage when they do not. On standard classification tasks with strong pretrained features, class-balancing the same per-point scores recovers the missing coverage and matches or outperforms generic set coverage across our datasets. Without labels, label-free coverage improves accuracy by up to +0.20 while raw top-

k

can fall below random; when labels are coarser than the modes, coverage within coarse strata performs best. We turn this distinction into a pre-specified diagnostic and validate it across both favorable and stress-test settings. The limits are part of the claim: the advantage is representation-relative. Coverage fails when the relevant modes are not expressed in the learner geometry, as with spurious subgroups, or when data quality varies independently of coverage -- exactly as the theory predicts.

Abstract

Training Data Selection

Explore similar work

When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation

HARP: Efficient Data Selection for Finetuning Large Language Models

MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection