LACS: Distribution-Sensitive Label-Agnostic Confidence Selection for Dynamic Data Pruning
Organizations: Yonsei University · Sooil Development · NCreate
Abstract
Training on large datasets motivates selecting which examples receive expensive training computation. Fixed quotas determine the amount of pruning in advance, while absolute confidence cutoffs depend on the score scale. We study a population-relative alternative that adapts the threshold and retained count together. We introduce LACS (Label-Agnostic Confidence Selector), a training-time rule based on maximum predictive confidence, requiring neither labels at selection time nor per-example gradients. LACS removes examples at or above a log-mean-exp reference plus a margin. We characterize its sensitivity to score gaps and mass, and derive an exact no-pruning condition. For , the removed fraction is at most and at least one example remains. The unsigned score permits both confidently correct and confidently wrong examples to be removed. Seven evaluation settings include neural training, language modeling, and boosting. Neural execution studies reduce backward-active rows by 10-35%; on CIFAR-10, 29% fewer backward rows accompany accuracy means of 89.77% versus 89.82% for full CE. At a matched backward budget and number of optimizer updates, LACS is 1.0 pp above small-batch training and 1.2 pp above random selection over five seeds. Across 14 tabular datasets, quadratic boosting reduces mean test error from 23.36% to 22.96%. Its full training replays use about 30% fewer fitting rows; these counters span a different horizon from validation-selected accuracy. In one RBF-SVM case study, elapsed time falls by 37% with slightly lower error. These findings separate selective-work reduction from realized acceleration: the current exact-current-score neural implementation uses an additional scoring forward, so its backward sparsity does not yet translate into wall-clock acceleration. This overhead is an execution choice rather than a requirement of the selection rule.
Figures & tables
| Setting | Work retained / saved | Reference selected method | Statistical scope |
|---|---|---|---|
| Quadratic boosting, R2 | replay rows of full | error | pp, CI , post-hoc Holm |
| Tree boosting (LACS-T), depth 2 | same per-round as GOSS | error | pp, CI , new-block Holm ; LACS-T variant |
| RBF-SVM boosting | fit rows M | error ; wall s | one common-endpoint replay; no multi-seed significance claim |
| CIFAR-10 neural | backward rows M (28.9% fewer) | accuracy | three seeds; CI crosses zero; current two-pass wall is slower |
| Method | Test acc. (%) | Bwd. (M) | Extra fwd. (M) | Steps |
|---|---|---|---|---|
| Full CE | 2.496 | 0.000 | 19,500 | |
| LACS | 1.776 | 2.496 | 19,500 | |
| InfoBatch (V) | 2.034 | 0.000 | 15,893 | |
| InfoBatch (B) | 1.776 | 0.000 | 13,876 | |
| RePB (N) | 1.827 | 0.000 | 14,274 | |
| Selective-Backprop | 1.795 | 2.446 | 14,047 |
| Setting | Task metric: full LACS | Expensive work: full LACS | Wall: full LACS |
|---|---|---|---|
| CIFAR-10 / ResNet-32 GN | acc. | bwd. M | s |
| Credit / RBF-SVM boosting | err. | fit rows M | s |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Confidence population | LACS removed | Top-10% | ||
|---|---|---|---|---|
| 100 at .95 | 100.00 | 0 | 10 | 100 |
| 90 at .70; 10 at .71 | 83.69 | 0 | 10 | 0 |
| 90 at .70; 10 at .75 | 43.11 | 10 | 10 | 10 |
| 40 at .70; 60 at .75 | 74.72 | 0 | 10 | 60 |
| 40 at .70; 60 at .80 | 65.41 | 60 | 10 | 60 |
| 20 at .70; 80 at .90 | 80.37 | 0 | 10 | 80 |
| High-group fraction | Minimum | LACS removal above boundary | Quantile recall within high group |
|---|---|---|---|
| .01 | .361 | 1% | 100% |
| .10 | .405 | 10% | 100% |
| .40 | .693 | 40% | 25% |
| .60 | 1.386 | 60% | 16.7% |
| .70 or more | no finite boundary | 0% (two-level case) |
| Setting | Support rule | Retained weights / domain |
| midrule Neural selective training | LACS, minibatch scores | Continuous detached weights; |
| Compact weak learners | LACS, ensemble confidence | Ordinary logistic fitting weights; searched |
| Tree/GOSS comparisons | LACS-T, fixed cutoff | Unit fitting weights; not the normalized cutoff |
| RBF-SVM case | LACS, ensemble confidence | Native LACS weights; fixed |
| Semi-supervised learning | LACS, cached unlabeled pool | FlexMatch loss; admissible HPO |
| bottomrule |
| Method | Test accuracy (%) | Recorded wall (s) |
|---|---|---|
| Full CE | 78.77 | 2608.30 |
| Native InfoBatch-CE | 78.30 | 2121.12 |
| Older LACS subset-BN | 78.06 | 3280.53 |
| Selective-Backprop | 78.22 | 3358.99 |
| Arm | Test acc. (%) | Bwd. frac. | Steps | Wall (s) |
| Full CE | .998 | 19,500 | 289 | |
| LACS | .709 | 19,500 | 280 | |
| Small-batch CE (91) | .719 | 19,764 | 288 | |
| Random drop | .693 | 19,500 | 381 | |
| LACS small-batch CE | per seed | |||
| LACS random drop | per seed | |||
| Arm | Test acc. (%) | Bwd. frac. | Wall (s) | LACS arm |
|---|---|---|---|---|
| Full CE | .998 | 203 | ||
| LACS | .870 | 264 | ||
| InfoBatch, work-matched | .868 | 190 | ||
| RePB, near budget | – | – | – | |
| Selective-Backprop, work-matched | .863 | 254 | ||
| RS2 | .863 | 179 |
| Arm | Test acc. (%) | Bwd. frac. | Wall (s) |
|---|---|---|---|
| C-LACS | .721 | 226 | |
| LACS | .710 | 221 | |
| InfoBatch, work-matched | .722 | 139 | |
| C-LACS InfoBatch | |||
| LACS InfoBatch | |||
| Selected-prefix rows | Replay rows | Replay fit time | ||||
|---|---|---|---|---|---|---|
| Learner | Mean | Pooled | Mean | Pooled | Mean | Pooled |
| MLP H1 | 4.827 | .963 | .887 | .865 | 1.094 | .937 |
| MLP H2 | 1.097 | .799 | .821 | .816 | 1.002 | .807 |
| MLP H4 | 1.372 | .832 | .788 | .785 | .817 | .723 |
| Quad R1 | 4.601 | .776 | .760 | .750 | 1.086 | .892 |
| Quad R2 | .936 | .641 | .700 | .695 | .941 | .747 |
| Depth | Matched | Pooled 95% CI | New-only | Own-HP ( ) | |
|---|---|---|---|---|---|
| 1 | .0049 | .1250 | (.0665) | ||
| 2 | .0049 | (.0314) | |||
| 3 | .0068 | .1250 | (.6238) |
| Epoch | Mean | Full- rel. residual | Cosine range | Wrong drops | Cancellation factor |
|---|---|---|---|---|---|
| 1 | .61 | .015–.036 | .9994–.9999 | 0 | .26–.33 |
| 10 | .61 | .004–.014 | .9999–1.0000 | 0 | .30–.36 |
| 20 | 1.00 | 0 | 1 | none | not applicable |
| Epoch | Method | Relative | Cosine | Norm ratio | |
|---|---|---|---|---|---|
| 1 | LACS hard, survivor mean | 77.63 | .6488 | .9997 | 1.6480 |
| LACS continuous | 77.63 | 1.2447 | .9912 | 2.2288 | |
| Random | 77.63 | .6526 | .8370 | 1.1951 | |
| OMP fitted scale | 44.13 | .0925 | .9956 | .9826 | |
| OMP normalized weights | 44.13 | 1.8165 | .9956 | 2.8097 | |
| 10 | LACS hard, survivor mean | 77.50 | .6560 | 1.0000 | 1.6559 |
| Variant | Selected setting | Test error | Active fraction | Fit rows (M) |
|---|---|---|---|---|
| Full | — | 1.000 | 3.40 | |
| Native hard, max probability | .677 | 1.92 | ||
| Native hard, true-label probability | .682 | 2.78 | ||
| Native hard, negative CE | 1.000 | 3.40 | ||
| Native continuous | .988 | 1.82 | ||
| Fixed keep, max probability | keep .3 | .300 | .67 |
| Dataset | Full | U | C | W |
|---|---|---|---|---|
| bank-marketing | ||||
| Diabetes130US | ||||
| electricity | ||||
| california | ||||
| MagicTelescope | ||||
| credit |
| Dataset | Full | LACS-U | IB-style | Pruned-set Jaccard U/IB |
|---|---|---|---|---|
| Diabetes130US | ||||
| MagicTelescope | ||||
| electricity |
| Warm-up | Recorded val. error (%) | Mean | Bwd. (M) | ||
|---|---|---|---|---|---|
| .5 | 0 | 49.609 | .9967 | 9.569 | |
| .2 | 0 | 49.609 | .9984 | 9.585 | |
| .2 | 0 | 50.391 | .9804 | 9.423 | |
| .1 | 0 | 51.172 | .9365 | 9.025 | |
| .05 | 0 | 54.688 | .6567 | 6.539 | |
| .5 | 5,000 | 48.438 | 1.0000 | 9.600 |
| Method | score | boundary / selection | cardinality / no-prune | retained-example treatment |
|---|---|---|---|---|
| LACS | max predictive confidence | deterministic upper tail above log-mean-exp + margin | endogenous; no prune iff | hard 0/1 support, or declared continuous variant |
| InfoBatch | per-sample loss | loss below arithmetic mean is eligible; each eligible row is thinned with a configured stochastic pruning probability | realized count depends on both the below-mean population and random thinning | retained low-loss rows are rescaled to correct gradient expectation; final full-data phase |
| SeTa | per-sample loss | configured random reduction, 1D -means difficulty groups, then an easy-to-hard sliding window over groups | controlled by keep ratio, window scale, schedule, and cluster sizes; no analogous closed-form distribution condition for nontrivial settings | selected rows are used as a hard subset |
| Fixed quantile | chosen scalar score | order statistic at a prescribed quota | quota fixed in advance (modulo ties) | hard subset |
| Absolute cutoff | chosen scalar score | fixed score value | data-dependent count; no prune if all scores stay below cutoff | hard subset |
Data Selection
Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.
See all 91 papers