Large Classification-Risk-Optional Label Acquisition

F. Setoudehtanzangi; Geoffrey J. McLachlan

stat.MLSep 6, 2026

Large Classification-Risk-Optional Label Acquisition

Authors: F. Setoudehtanzangi, Geoffrey J. McLachlan

Organizations: Dipartimento di Scienze Statistiche, Universit`a di Padova, Padova, Italy · School of Mathematics and Physics, The University of Queensland, Australia

Abstract

We study how a limited labeling budget should be allocated to minimize multiclass zero-one classification risk. We consider parametric classification problems in which features are observed for all sampling units while class labels can be acquired selectively. By combining the Fisher information supplied by an acquired label with the local geometry of multiclass excess risk, we derive an acquisition criterion that minimizes the leading asymptotic coefficient of expected multiclass excess risk. The resulting rule values a label according to how strongly its information is aligned with parameter directions that perturb the active Bayes decision boundary, rather than according to posterior uncertainty or global parameter information alone. We characterize the oracle acquisition design, establish its threshold structure, and derive face-specific and cost-sensitive extensions. An analytic example shows that posterior uncertainty and classification value can produce different, and even reversed, acquisition rankings. We further develop a two-stage adaptive procedure that attains the oracle leading-risk criterion under regularity conditions and provide explicit results for Gaussian discriminant analysis. Three-class QDA experiments illustrate the resulting acquisition geometry, while an application to the six-class Statlog Landsat Satellite data shows that classification-risk acquisition can differ materially from both uncertainty-based acquisition and the complete-classification-information comparator. The adaptive classification-risk design attains lower mean error than this Fisher comparator across the labeling budgets considered, although it does not uniformly outperform entropy or margin sampling and differences among the targeted strategies become small as the labeling budget increases.

Figures & tables

Figure 1: Classification geometry in Scenario 1. Panel (a) shows the Bayes classification regions and active pairwise boundaries. Panel (b) shows normalized posterior entropy and panel (c) the classification-value function $\psi_{a}(y)$ under a uniform $20\%$ acquisition design. Panel (d) compares the locations selected by classification-risk-optimal and entropy acquisition under the same labeling budget; gray points show the candidate feature population.

Figure 2: Classification geometry in Scenario 2 with weak covariance heterogeneity. The panels have the same interpretation as in Figure 1 . The active Bayes boundaries are now substantially closer to linear, providing a control regime in which posterior uncertainty and classification value are more closely aligned.

Acquisition rule	$10\%$	$20\%$	$30\%$
Scenario 1: heterogeneous QDA
Random	0.15209 (0.000258)	0.15109 (0.000175)	0.15043 (0.000128)
Entropy	0.15175 (0.000250)	0.15054 (0.000172)	0.14992 (0.000087)
Margin	0.15159 (0.000183)	0.15037 (0.000120)	0.14987 (0.000084)
Fisher	0.15156 (0.000270)	0.15031 (0.000125)	0.14989 (0.000085)
Adaptive risk-optimal	0.15138 (0.000232)	0.15020 (0.000103)	0.14980 (0.000082)

Table 1: Mean test classification error over 100 Monte Carlo replications. Monte Carlo standard errors are shown in parentheses. The smallest mean error among the implementable acquisition rules within each labeling budget is shown in bold; the oracle risk-optimal rule is included only as a theoretical benchmark.

Figure 3: Mean test classification error as a function of the total proportion of acquired labels for (a) Scenario 1 with heterogeneous QDA geometry and (b) Scenario 2 with weak covariance heterogeneity. Error bars are $95\%$ Monte Carlo intervals based on 100 replications. The horizontal dotted lines indicate the corresponding estimated Bayes errors, $0.14845$ in Scenario 1 and $0.15882$ in Scenario 2.

Figure 4: Acquisition geometry for the Statlog Landsat Satellite data in representative replication 15. Panel (a) displays the six land-cover classes in the first two principal components of the four standardized central-pixel spectral measurements. Panel (b) shows posterior classification uncertainty measured by entropy, whereas panel (c) shows the estimated relative classification value $\widehat{\psi}_{a}(y)$ . Panel (d) compares the additional observations selected by classification-risk-optimal and entropy acquisition under a 20% total labeling budget; black points denote acquired labels and gray points the remaining training pool. The principal components are used only for visualization: posterior probabilities, classification values, and acquisition decisions are computed from the full four-dimensional QDA model. The replication was selected as the one whose paired AdaptiveRisk-versus-Entropy difference was closest to the median across the 50 replications.

	10%		20%		30%
Method	Error	Balanced	Error	Balanced	Error	Balanced
Random	0.18511 (0.00296)	0.22786 (0.00495)	0.16159 (0.00129)	0.19480 (0.00218)	0.15639 (0.00090)	0.19245 (0.00144)
Entropy	0.17098 (0.00264)	0.20396 (0.00328)	0.15709 (0.00055)	0.19310 (0.00085)	0.15682 (0.00026)	0.19549 (0.00042)
Margin	0.17250 (0.00248)	0.21120 (0.00316)	0.15626 (0.00050)	0.19256 (0.00084)	0.15638 (0.00024)	0.19483 (0.00044)
Fisher	0.17822 (0.00255)	0.22160 (0.00444)	0.15835 (0.00041)	0.19474 (0.00076)	0.15716 (0.00029)	0.19573 (0.00052)
Adaptive risk	0.17470 (0.00255)	0.21620 (0.00438)	0.15675 (0.00042)	0.19271 (0.00072)	0.15657 (0.00030)	0.19511 (0.00048)

Table 2: Mean benchmark test classification error and balanced error for the Statlog Landsat Satellite data over 50 repeated pilot/acquisition replications. Monte Carlo standard errors are reported in parentheses. The smallest mean error within each labeling budget is shown in bold.

Figure 5: Mean benchmark test classification error for the Statlog Landsat Satellite data over 50 repeated pilot/acquisition replications. Error bars represent $\pm 1.96$ Monte Carlo standard errors. The benchmark training and test sets are fixed across replications; the random pilot labels and resulting acquisition decisions vary between replications.

Figure S1 : Acquisition geometry for the Dry Bean data in representative replication 26. Panel (a) displays the seven observed classes in the first two principal components. Panel (b) shows posterior classification uncertainty measured by entropy, whereas panel (c) shows the estimated relative classification value $\widehat{\psi}_{a}(y)$ . Panel (d) compares the additional observations selected by classification-risk-optimal and entropy acquisition under a 10% total labeling budget; black points denote acquired labels and gray points the remaining training pool. The displayed principal components are used only for visualization: posterior uncertainty, classification value, and acquisition decisions are calculated from the full five-dimensional model. The replication was selected as the one whose paired AdaptiveRisk-versus-Entropy improvement was closest to the median paired improvement across the 50 replications.

	10%		20%		30%
Method	Error	Balanced	Error	Balanced	Error	Balanced
Random	0.11735 (0.00179)	0.09654 (0.00145)	0.09852 (0.00065)	0.08194 (0.00058)	0.09178 (0.00066)	0.07672 (0.00054)
Entropy	0.10981 (0.00126)	0.09047 (0.00113)	0.10089 (0.00087)	0.08318 (0.00079)	0.09690 (0.00074)	0.08009 (0.00067)
Margin	0.10803 (0.00084)	0.08864 (0.00081)	0.10064 (0.00092)	0.08308 (0.00085)	0.09668 (0.00072)	0.07994 (0.00067)
Fisher	0.10771 (0.00107)	0.08940 (0.00094)	0.09588 (0.00078)	0.08015 (0.00073)	0.09390 (0.00068)	0.07823 (0.00066)
Adaptive risk	0.10679 (0.00100)	0.08829 (0.00093)	0.09655 (0.00079)	0.08022 (0.00073)	0.09437 (0.00071)	0.07843 (0.00066)

Table S1 : Mean test classification error and balanced error for the Dry Bean data over 50 repeated stratified train–test splits. Monte Carlo standard errors are reported in parentheses. The smallest mean error within each labeling budget is shown in bold.

Figure S2 : Mean test classification error for the Dry Bean data over 50 repeated stratified train–test splits. Error bars represent $\pm 1.96$ Monte Carlo standard errors. Within each replication, all acquisition rules use the same train–test split and initial pilot sample.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 277 papers

Sep 16, 2026

How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction

Classifiers can make identical predictions yet require labels to compare their selective performance: confidence ranks weight the same errors differently. We quantify this requirement for the area under the generalized risk-coverage curve (AUGRC). A prelabel lower bound rules out insufficient budgets. With all labels known, a covering linear program bounds the minimum number of labels sufficient to fix the winner (the certificate size) within

K-1

labels for

K

candidates. For fixed

K

, independent uniform orders and identical predictions, the prelabel bound approaches one quarter of the pool. With iid Bernoulli errors independent of the orders, every exact acquisition policy reads almost all labels asymptotically, although a two-candidate certificate needs only half. Across 108 feature-panel comparisons on nine datasets, disagreement labels settle every accuracy choice but no AUGRC choice. A 20% budget is ruled out in 96 conditions; certificates need 56-57% on average. On ten conditions with pretrained image classifiers, confidence-score choice reads 68-91% of 10,000 labels for exact selection and 50-67% with AUGRC tolerance

5\times10^{-4}

. An exact stopping test works with any acquisition order. Together, these results link confidence ranks to label budgets and certified model comparison.

Tetsuji Kuboyama

Computer Centre, Gakushuin University

Model Selection Selective Prediction Classification Statistical Learning Theory

Ask PDF

Aug 31, 2026

Informative Label Missingness in Multiclass Classification Information Geometry and Excess Risk

Informative label missingness can change the usual efficiency ordering between completely and partially labelled classifiers because the pattern of missing labels may itself carry information about the classification model. We develop a general likelihood-based theory for this phenomenon in parametric multiclass classification. An efficient-information decomposition separates information lost through unavailable class memberships from information contributed by the missing-label mechanism. We then derive a quadratic expansion of plug-in excess risk over the active pairwise faces of the multiclass Bayes boundary, showing that classification efficiency depends on how information gains and losses align with directions that perturb the decision boundary. This yields a classification-weighted generalized-eigenvalue criterion under which informative partial classification may have smaller asymptotic classification risk without globally dominating complete classification in Fisher information. Near missing completely at random, with the marginal missing-label proportion fixed, redistribution of missing labels changes lost class-label information at first order, whereas efficient information from the missingness pattern appears only at second order. Three-class quadratic discriminant calculations, finite-sample experiments, and a semi-synthetic multiclass application illustrate the resulting regime-dependent behaviour.

Fariborz Setoudehtazang, Geoffrey J. McLachlan

Dipartimento di Scienze Statistiche, Universit`a di Padova, Padova, Italy · School of Mathematics and Physics, The University of Queensland, Australia

Information Geometry Learning with Missing Data Multiclass Classification Classification Statistical Learning Theory

Ask PDF

Oct 16, 2026

Towards optimal selection schedules for online active learning

We consider the problem of online linear classification, where a stream of unlabeled samples is available, and annotation is costly. We investigate the optimal query policy, that prescribes whether and when a sample should be labeled and used for a training step, or instead discarded, so as to minimize a given objective. We focus on the realizable case, under Gaussian design, in the high-dimensional regime where data dimension and number of unlabeled samples grow proportionally. We consider two objectives: 1) the generalization error under a labeling budget, and 2) a cumulative cost function where mistakes have unit cost and queries cost

Δ

. We first show that the myopic policies that greedily minimize at all times the increment in the objective correspond to either small or intermediate-margin policies, that prescribe the selection of samples whose margins fall within certain intervals that shrink with time. Within this class, we numerically ascertain the optimal schedules for the selection thresholds that minimize the final values of the objectives. Taking inspiration from the myopic policy for the cumulative cost, we design a very simple query policy schedule that we compare to other proposed heuristics, on synthetic data and real medical image datasets. Finally, we discuss myopic policies for two-layer neural networks, and exemplify how naive margin-based policies can cease to be optimal.

Hugo Cui, Francesca Mignacco, Francesco Mori

Université Paris-Saclay, CNRS, Laboratoire de mathématiques d’Orsay, 91405, Orsay, France · Joseph Henry Laboratories of Physics and Lewis–Sigler Institute, Princeton University, Princeton NJ 08544, USA · Initiative for the Theoretical Sciences, The Graduate Center, City University of New York, 365 Fifth Ave, New York NY 10016, USA +1

Active Learning Linear Classification Margin-Based Learning Cost-Sensitive Learning

Ask PDF

Abstract

Active Learning

Explore similar work

How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction

Informative Label Missingness in Multiclass Classification Information Geometry and Excess Risk

Towards optimal selection schedules for online active learning