Accurate classifiers can fail when background cues change without altering the correct labels. Predictive performance alone does not quantify how much task information remains after accounting for such nuisance variation. Here we extend the Fisher-glass framework to connect nuisance-adjusted information with local classification difficulty and subpopulation auditing. We derive lower bounds on local minimax excess classification risk under explicit distinguishability and decision-separation conditions, alongside statistical audits accounting for estimation error and regularization. On Waterbirds, a feature-based diagnostic of task-background alignment selects an accurately classified group, with selection frozen before held-out evaluation. Across five training seeds, background replacements preserving the extracted bird foreground and labels yield 93.2% accuracy when background type is preserved and 40.6% when it changes, a controlled difference of 52.6 percentage points. Separately, search using task labels and a fixed nuisance direction identifies an independently evaluated population with 98.8% membership-weighted accuracy and low estimated projected predictive Fisher information. Training and acquisition experiments show that information gains and predictive gains can diverge. These findings support joint assessment of predictive performance, information under declared nuisance models and controlled sensitivity when diagnosing vulnerability and evaluating model improvements.
Figures & tables
Figure 1: Background sensitivity despite high control accuracy. (A,C) A matched Waterbirds image pair preserves the extracted bird foreground and label while changing background type. For the selected waterbird/water group, mean accuracy across five training seeds falls from 93.2% with same-type backgrounds to 40.6% with opposite-type backgrounds, a drop of 52.6 percentage points. (B) Schematic score geometry: projection removes the nuisance-aligned component of the task score, leaving a small residual.
Figure 2: A 52.6-point controlled accuracy drop in the selected group. (A) The uncentered feature–Schur fraction selects waterbird/water from two eligible groups; matched accuracy and NLL selectors agree. (B) Accuracy falls from 93.2% under same-type background recomposition to 40.6% under opposite-type recomposition, with bird foregrounds and labels preserved. Points show means and bars nominal 95% normal-approximation intervals across five training seeds.
Figure 3: Discovery finds accurate slices with low projected predictive information. (A) CLIP-guided search finds low projected predictive Fisher information and 98.8% membership-weighted accuracy; the random-direction control also attains low information. (B) Both searches yield low uncalibrated NLL (log scale). Selectors target 10% within-class mass. Points show means and bars 95% Student- t intervals across five paired seeds, with gates frozen before testing.
Objective
Average accuracy (%)
Worst-group accuracy (%)
Fisher-Glass certificate
ERM
77.2±0.1
55.7±0.3
0.198±0.001
ERM + Fisher-Glass
80.4±0.2
56.3±0.3
0.399±0.006
Group DRO
80.9±0.3
67.3±0.3
0.233±0.003
Group DRO + Fisher-Glass
80.6±0.9
69.4±1.3
0.614±0.059
Table 1: Frozen-feature Waterbirds training. Means with nominal 95% normal-approximation intervals ( ±1.96 standard errors) across five seeds; uncentered feature–Schur ridge certificates use α=0.95 . Validation selection follows Section 6 .
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Near-alignment produces a heavy inverse-information tail. (A) Projection leaves a residual score of norm δ , giving information λ=δ2 . (B) Raw information 1+δ2 remains near one as λ approaches zero. (C) Empirical survival probabilities (blue) follow the x−1/2 reference (dashed). Panels B–C use 300,000 environments with δ∼Uniform(0,0.35) and L=(δ2+10−12)−1 .
Conditioning and ridge gap
Certificates
Scope
Audits
Minimum JNN
Maximum group gap
Maximum inflation
Minimum exact
Minimum Wald lower
CivilComments diagnostics
5
1.456 (1.417)
0.00116 (0.00133)
0.00018
0.4970
0.4618
Waterbirds diagnostics
5
0.211 (0.178)
0.00125 (0.00166)
0.00071
0.1639
0.1504
Frozen-feature training
30
0.227 (0.192)
0.00144 (0.00199)
0.00062
0.1972
0.1814
End-to-end training
35
0.349 (0.349)
0.00252 (0.00300)
0.00140
0.2185
0.2004
All audits
75
0.211 (0.178)
0.00252 (0.00300)
0.00140
0.1639
0.1504
Appendix
Table B.1: Same-cohort summaries of 75 default-ridge held-out audits. Conditioning/gap parentheses are within-run nominal Wald lower/upper values; entries are extrema across runs, not joint confidence statements across seeds or methods. Inflation is ridge minus exact plug-in certificate.
Training setting
Ridge plug-in
Exact plug-in
Wald lower
Frozen FG, λFG=0.03
0.4002
0.3999
0.3663
Frozen Group DRO + FG, selected from {0.1,0.3}
0.6254
0.6252
0.5664
End-to-end Group DRO + FG, λFG=10−4
0.5447
0.5442
0.4890
Appendix
Table B.2: Separate fresh training audits: five-seed means at the stated coefficients. Table B.1 instead includes frozen models selected over the full grid {0.01,0.03,0.1,0.3} .
Component
Specification
Representation/nuisance
Gate: frozen ImageNet MobileNetV3-small; nuisance: CLIP ViT-B/32 image projection onto a fixed water-minus-land text direction.
Head and calibration
Binary linear head: 1,200 full-batch AdamW steps, learning rate 0.01 , weight decay 10−4 ; temperature minimizes calibration NLL over 400 log-spaced values in [0.05,20] .
Gate optimization
Ten restarts; 1,000 AdamW steps, learning rate 0.03 , weight decay 10−4 ; Gaussian initial weights with standard deviation 0.02 .
Matrix and stabilization
Mi=pi(1−pi)vivi⊤ on two task/nuisance coordinates; η=0.10 , λrid=10−3 , ε=10−8 .
Replication/sensitivity
Seeds 17–21; five validation folds; mass and shrinkage grids each {0.05,0.10,0.20} , headline 0.10 .
Appendix
Table C.1: Fixed hidden-discovery settings; sensitivity grids are declared before metadata reveal.
Stage
Fitted or computed
Carried forward
Training/calibration
Split official training data 85/15 by task label: fit head and coordinate normalizers on 85%; temperature on 15%. CLIP scores use official-training, then fit-subset normalization.
Five shared task-stratified assignments across selectors; discover on complementary folds, evaluate on held-out folds; assess stability on the 15% training-calibration subset.
Protocol assessment; fold gates are not ensembled.
Full validation
Refit the gate under the fixed protocol; select class, gate, and restart by discovery utility.
Freeze gate weights, calibrated intercept, selected class, support settings.
Test
Apply frozen memberships; recompute admissible weights, same-class pool, and metrics before metadata reveal.
Held-out results, then post-hoc interpretation.
Appendix
Table C.2: Fitting and evaluation sequence under the frozen protocol.
Frozen MobileNetV3-small and logistic fitting; feature–Schur certificate
17–21
Appendix
Table D.1: Diagnostic targets and replication units.
Group
n
Accuracy
NLL
Eligible
Landbird/land
467
0.990±0.005
0.038±0.016
Yes
Landbird/water
466
0.703±0.019
1.670±0.156
No
Waterbird/land
133
0.390±0.021
7.718±0.800
No
Waterbird/water
133
0.920±0.013
0.638±0.038
Yes
Appendix
Table D.2: Validation selection. Means ± nominal 95% normal-approximation half-widths across seeds 22–26; accuracy is a fraction. Information uses uncentered feature moments.
Group
Birds
Original
Same type
Opposite type
Controlled drop
Landbird/land
2,255
98.9
99.3
72.6
26.7±1.8
Landbird/water
2,255
71.3
72.3
99.2
−26.9±2.0
Waterbird/land
642
42.3
40.6
92.3
−51.7±2.0
Waterbird/water
642
92.0
93.2
40.6
52.6±1.4
Appendix
Table D.3: Held-out interventions. Accuracies and flip rates are seed means in percent; controlled drops are same-type minus opposite-type percentage points with nominal 95% normal-approximation half-widths. Panel (b) counts are mean confident-half counts.
Statistic
Candidates
Selected group, seeds
Drop
Minimum raw task information
Eligible
Landbird/land, 5/5
26.7
Minimum absolute projected information
Eligible
Landbird/land, 5/5
26.7
Minimum surviving fraction
Eligible
Waterbird/water, 5/5
52.6
Minimum validation accuracy or maximum validation NLL
Eligible
Waterbird/water, 5/5
52.6
Minimum normalized-weight fraction
Eligible
Waterbird/water, 5/5
52.6
Maximum NLL or minimum accuracy
All four
Waterbird/land, 5/5
−51.7
Appendix
Table D.4: Selection and matrix sensitivity, evaluated within each seed. Panel (a) distinguishes original eligible and all-group sets; drops are mean percentage points. Panel (b) fixes the original eligible LL/WW set and interventions; fractions/drops are means ± nominal 95% normal-approximation half-widths.
Seed
Uncentered rρ(Jg)
Centered rρ(Σg)
Predictive rρ(Ig)
22
0.5570 / 0.2291
0.7430 / 0.8542
0.9817 / 0.2587
23
0.6048 / 0.2494
0.8098 / 0.8481
0.9920 / 0.2729
24
0.6356 / 0.2755
0.8132 / 0.8359
0.9620 / 0.3032
25
0.5803 / 0.2392
0.7514 / 0.9156
0.9998 / 0.2571
26
0.5716 / 0.2460
0.7779 / 0.8189
0.9998 / 0.2732
Appendix
Table D.5: Per-seed fractions on the original global scales. Each cell is LL / WW; the smaller value determines selection. Centering reverses the ranking in every seed.
Cohort
Model/change
A : raw
P : projected
H : harmonic
C : inverse-CVaR
Primary
ERM
1.142±0.012
0.500±0.010
0.442±0.011
0.289±0.014
Selected FG
0.730±0.086
0.484±0.047
0.435±0.043
0.330±0.043
FG − ERM
−0.412±0.079
−0.016±0.046
−0.007±0.041
+0.041±0.046
Development
ERM
1.152±0.005
0.489±0.011
0.413±0.014
0.255±0.018
Selected FG
0.754±0.118
0.519±0.065
0.433±0.057
0.303±0.082
FG − ERM
−0.398±0.118
+0.030±0.063
+0.020±0.066
+0.048±0.095
Appendix
Table D.6: Four validation summaries and their intervention associations. Panels (a)–(b): means ± nominal 95% normal-approximation half-widths, 1.96s/5 ; changes are paired within seed and drops are signed percentage points. Selected FG includes accuracy-budget fallbacks (primary seeds 22, 24; development seed 20). Panel (c): descriptive Spearman coefficients for ten models arranged in five ERM/FG seed pairs per cohort, with average ranks for tied outcomes.
Analysis
Rows
Flip rate
Log-likelihood gain
Δ AUPRC
Δ AUROC
All groups, no indicators
9,398
0.199
+0.016
+0.035 (5/5)
+0.096 (5/5)
All groups, with indicators
9,398
0.199
+0.001
−0.007 (0/5)
+0.003 (5/5)
Selected slice
1,181
0.564
+0.013
+0.026 (5/5)
+0.015 (5/5)
Appendix
Table D.7: Incremental flip prediction. Counts and rates are seed means; parentheses count seeds with improvement. Changes are paired within seed.
Quantity
Estimate
Mean accuracy
94.54±0.00%
Fifth-percentile environment accuracy
88.67±0.00%
CVaR0.95 of environment-mean NLL
0.262±0.000
I0.95FG
(1.22±0.53)×10−8
Inverse-information exponent β
0.498±0.003
Appendix
Table D.8: Synthetic checks. Panel (a) and the branch rows are five-seed means ± nominal 95% normal-approximation half-widths; zero widths reflect rounding. Mass-sweep rows are reported single-seed values.
Selector
Predictive projected information
Uncalibrated slice NLL
Fisher search, fixed CLIP direction
0.005±0.001
0.031±0.017
High loss, same target within-class mass
0.019±0.006
5.779±0.496
Random, same target within-class mass
0.013±0.001
1.025±0.059
Constant class-conditional gate
0.013±0.001
1.027±0.049
Fisher search, random direction
0.006±0.001
0.031±0.015
Appendix
Table D.9: Hidden-slice comparisons. Panel (a): means ± 95% Student- t half-widths across five paired seeds; predictive information retains calibration and shrinkage η=0.10 . Panel (b): CLIP-search minus comparator, with paired 95% Student- t intervals; negative differences indicate lower information.
Dataset
Average accuracy (%)
Worst-group accuracy (%)
Loss- CVaR0.95
I0.95FG
CivilComments
86.10±0.22
43.75±0.64
0.998±0.012
0.500±0.001
Waterbirds
77.11±0.00
56.54±0.00
1.741±0.000
0.165±0.000
Appendix
Table D.10: Original predefined-group diagnostics, distinct from fresh audits in Appendix B.4 . Means ± nominal 95% normal-approximation half-widths; Waterbirds zeros reflect fixed-model fitting.
Specification
CivilComments
Waterbirds
Role
Metadata
0.587
0.266
Semantic-family member
Learned probe
0.499
0.165
Semantic-family minimum
Random
0.588
0.482
Negative control
Under-projected
0.588
0.482
Under-projection control
Over-projected
0.001
0.012
Task-erasure control
Hidden-loss
0.060
0.177
Loss-coordinate stress control
Appendix
Table D.11: Nuisance sensitivity of fixed-model diagnostics. Entries are I0.95FG ; worst-group accuracy remains 44.0% / 56.5% for CivilComments/Waterbirds.
Method
Avg. acc. (%)
Worst acc. (%)
Loss-CVaR .95
ERM
77.2±0.1
55.7±0.3
1.969±0.007
Group DRO
80.9±0.3
67.3±0.3
1.143±0.021
JTT
77.4±0.1
54.1±0.2
2.167±0.004
Loss-CVaR
78.2±0.1
52.1±0.3
2.427±0.041
ERM + Fisher-Glass
80.4±0.2
56.3±0.3
1.866±0.025
Group DRO + FG
80.6±0.9
69.4±1.3
0.855±0.082
Appendix
Table E.1: Original frozen-feature training cohort, seeds 17–21. Both panels give means ± nominal 95% normal-approximation half-widths; accuracy is percent and the uncentered feature–Schur ridge certificate uses α=.95 .
Objective / λFG
Avg. acc. (%)
Worst acc. (%)
Loss-CVaR .95
Group DRO reference
85.4
53.2
0.925
5×10−5
79.7±6.2
56.6±5.8
0.733±0.061
10−4
79.5±10.2
57.7±8.3
0.733±0.110
3×10−4
69.9±9.7
58.8±8.7
0.653±0.064
10−3
62.7±7.9
49.9±1.8
0.710±0.012
3×10−3
57.7±4.5
48.6±6.6
0.717±0.036
Appendix
Table E.2: End-to-end Waterbirds Group DRO + Fisher-Glass coefficient sweep. FG entries are means ± sample standard deviations over five seeds, not confidence intervals. The Group DRO reference is the baseline from the λFG=10−4 sweep; each sweep reran its baseline, so this is not a shared paired reference for every coefficient.
Arm
Avg. acc. (%)
Worst acc. (%)
Unchanged
Matched ERM continuation
76.3±0.7
57.3±1.3
—
Loss repair
76.3±0.6
56.9±1.3
3/5
Fisher repair
76.3±0.7
57.3±1.3
5/5
Combined repair
76.4±0.9
56.7±1.6
4/5
Oracle Group DRO
80.0±0.3
64.8±0.9
—
Arm
Slice NLL
Projected info.
LL mass (%)
Appendix
Table E.3: Constrained repair on the primary hidden-slice cohort, seeds 17–21. Accuracy, NLL and information are means ± 95% Student- t half-widths; composition is a mean percentage. Panel two audits each arm’s rediscovered CLIP-defined slice. The Group DRO oracle uses group annotations.
Policy
Average accuracy (%)
Worst accuracy (%)
Frozen ERM / matched zero
75.5[74.3,76.6]
55.7[53.3,58.2]
Fixed λ=.03
76.6[76.1,77.2]
51.5[47.1,55.8]
Risk only / group-free
77.5[76.5,78.6]
43.1[33.1,53.1]
Group-supervised comparator
77.7[76.6,78.8]
44.2[34.2,54.3]
Policy
Coefficient counts
Slice information
Frozen ERM / matched zero
— / 0:5
0.007[0.005,0.009]
Appendix
Table E.4: Post hoc coefficient selection, five paired seeds. Accuracies are test results; information is projected predictive information in test slices defined by validation-rediscovered, frozen gates. Brackets are 95% Student- t intervals. Policies selecting identical models across these seeds share rows; the text explains both coincidences. Coefficient counts are λ:number of seeds .
(a) Strength and budget: implemented geometric score × loss versus high loss
TTIα
High loss
γcf
Budget
Δa
ΔI
Δa
ΔI
0.25
1%
+4.05
+0.0308
+5.76
+0.0411
2%
+7.17
+0.0366
+7.63
+0.0477
5%
+8.88
+0.0521
+10.12
+0.0590
0.5
1%
+6.39
+0.0205
+4.36
+0.0328
Appendix
Table E.5: Feature-space acquisition, means over seeds 17–21. Δa is worst-group accuracy gain in percentage points and ΔI the absolute ridge-certificate gain. Panel (a) gives the complete strength-by-budget comparison; panel (b) retains pure-tail and alternative-product controls at unit strength. Only aggregate reports survive, so historical seed-level intervals are unavailable.
We study how a limited labeling budget should be allocated to minimize multiclass zero-one classification risk. We consider parametric classification problems in which features are observed for all sampling units while class labels can be acquired selectively. By combining the Fisher information supplied by an acquired label with the local geometry of multiclass excess risk, we derive an acquisition criterion that minimizes the leading asymptotic coefficient of expected multiclass excess risk. The resulting rule values a label according to how strongly its information is aligned with parameter directions that perturb the active Bayes decision boundary, rather than according to posterior uncertainty or global parameter information alone. We characterize the oracle acquisition design, establish its threshold structure, and derive face-specific and cost-sensitive extensions. An analytic example shows that posterior uncertainty and classification value can produce different, and even reversed, acquisition rankings. We further develop a two-stage adaptive procedure that attains the oracle leading-risk criterion under regularity conditions and provide explicit results for Gaussian discriminant analysis. Three-class QDA experiments illustrate the resulting acquisition geometry, while an application to the six-class Statlog Landsat Satellite data shows that classification-risk acquisition can differ materially from both uncertainty-based acquisition and the complete-classification-information comparator. The adaptive classification-risk design attains lower mean error than this Fisher comparator across the labeling budgets considered, although it does not uniformly outperform entropy or margin sampling and differences among the targeted strategies become small as the labeling budget increases.
F. Setoudehtanzangi, Geoffrey J. McLachlan
Dipartimento di Scienze Statistiche, Universit`a di Padova, Padova, Italy · School of Mathematics and Physics, The University of Queensland, Australia
We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) are more sensitive to errors on certain labels than on others. Our proposed ATLAS framework couples anytime-valid inference with a task-aware adaptive auditing policy. Representing label corrections through a confusion matrix, we prioritize which data points to inspect next based on both label uncertainty and the influence of potential label corrections on the downstream score function. For linear scores such as the mislabel rate, our priority-aware approach certifies the overall label error rate to ±2 percentage points with 24% and 11% fewer audits than random sampling on biodiversity data from Kenya and a widely used CIFAR-10N benchmark, respectively. For nonlinear scores such as the Hill-Shannon diversity, we exploit a linear-envelope representation that reduces inference to a family of linear-score audits; priority-aware sampling reaches the target precision with 36% fewer audits. When distribution-free guarantees are prohibitively costly, we develop a model-based Bayesian alternative that trades assumption-free validity for substantially fewer audits. Our biodiversity case study highlights a gap between statistical validity and practical feasibility: generic methods such as Serfling's bound and existing confidence-sequence procedures are rigorous but can require prohibitively many expert reviews in practice. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees.
We revisit the foundations of fairness and its interplay with utility and efficiency in settings where the training data contain richer labels, such as individual types, rankings, or risk estimates, rather than just binary outcomes. In this context, we propose algorithms that achieve stronger notions of evidence-based fairness than are possible in standard supervised learning. Our methods support classification and ranking techniques that preserve accurate subpopulation classification rates, as suggested by the underlying data distributions, across a broad class of classification rules and downstream applications. Furthermore, our predictors enable loss minimization, whether aimed at maximizing utility or in the service of fair treatment. Complementing our algorithmic contributions, we present impossibility results demonstrating that simultaneously achieving accurate classification rates and optimal loss minimization is, in some cases, computationally infeasible. Unlike prior impossibility results, our notions are not inherently in conflict and are simultaneously satisfied by the Bayes-optimal predictor. Furthermore, we show that each notion can be satisfied individually via efficient learning. Our separation thus stems from the computational hardness of learning a sufficiently good approximation of the Bayes-optimal predictor. These computational impossibilities present a choice between two natural and attainable notions of accuracy that could both be motivated by fairness.