Classification heads built on conic functions give each class a bounded region of the feature space, and such regions are used to reject samples of unseen classes. We ask whether these regions also keep the features bounded when the head is trained with softmax cross-entropy. For a head whose class scores combine a linear term with a Euclidean norm, we first show that the region of each class is bounded by a quadric of revolution, whose eccentricity is the reciprocal of one learned ratio. We then attach to each class an asymptotic margin, which compares the growth of its score with the growth of all the other scores and can be computed by a second-order cone program. The loss of a class has infimum zero exactly when its margin is positive, and it then approaches zero only along diverging features. When the margin is negative, the loss is coercive and its minimizers lie in an explicit ball. Since softmax depends only on the differences between scores, a class with a bounded region can still have a positive margin, so bounded regions do not confine the features. The result extends to other norms, including the polyhedral conic functions. On CIFAR-10 and CIFAR-100, all 1210 cells of 67 trained conic heads have positive margins, whereas a prototype head keeps nine of its ten cells bounded, as the theory predicts.
Figures & tables
Figure 1: The panels illustrate Theorems 3.1 and 3.2 with four cones of unequal apertures and non-parallel axes. The left panel shows the score surface minkhk , the lower envelope of the cones, whose red curves mark the creases along which the lowest cone changes from one class to another. The right panel shows the same configuration from above, where the creases appear as curved decision boundaries.
setting
acc., affine
acc., conic
p
∥z∥ , affine
∥z∥ , conic
batch norm, lr 0.1
95.03±0.14
94.90±0.17
0.25
6.01±0.01
5.27±0.24
no batch norm, lr 0.01
90.48±0.17
91.20±0.28
0.008
39.9±1.7
24.7±0.7
no batch norm, lr 0.08
93.31±0.19
93.26±0.20
0.68
26.5±0.4
19.9±0.6
Table 1: Final test accuracy (%) and mean norm of the test features, mean ± sd over five seeds, with permutation p -values for the accuracy.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
batch norm, lr 0.1
no batch norm, lr 0.01
no batch norm, lr 0.08
head
acc.
∥z∥
AUC
acc.
∥z∥
AUC
acc.
∥z∥
AUC
affine
94.92
6.04
0.712
90.51
39.4
0.767
93.21
26.2
0.847
conic
94.88
5.26
0.806
91.09
24.8
0.816
93.32
20.1
0.875
conic, ηk without decay
94.90
6.09
0.705
91.22
11.3
0.832
93.33
7.57
0.888
prototype
95.13
3.86
0.868
—
—
—
93.47
6.69
0.897
cosine
94.99
1.08
0.914
91.79
0.38
0.867
93.44
0.25
0.897
Appendix
Table 2: Additional runs: final test accuracy (%), mean norm of the test features, and area under the ROC curve of the feature norm for CIFAR-10 against CIFAR-100 test images, each the mean over five seeds.
batch norm
no batch norm
lr
affine
conic
τ
lr
affine
conic
τ
0.05
0.9212
0.8979
0.489
0.005
0.7071
0.7377
1.456
0.1
0.9205
0.8970
0.475
0.01
0.7740
0.8055
1.417
0.2
0.9137
0.9101
0.588
0.02
0.8429
0.8617
1.299
0.4
0.9017
0.9061
0.609
0.04
0.8828
0.8859
1.062
0.5
—
0.8953
0.617
0.08
0.8861
0.8881
0.680
Appendix
Table 3: Final test accuracy after 20 epochs as a function of the learning rate, with the final τ of the conic head.
configuration
final accuracy
final τ
affine, batch norm
0.9495
—
conic ε=0.1 , τ free
0.9505
0.941
conic ε=0.5 , τ≥1
0.9474
1.884
conic ε=0.5 , τ frozen 5 ep
0.9482
0.929
affine, no batch norm, lr 0.01
0.9045
—
conic ε=0.5 , no batch norm, lr 0.01
0.9126
1.209
Appendix
Table 4: Exploratory single-seed runs, on CIFAR-10 unless otherwise stated.
setting
τ , mean ± sd
range
decay path
batch norm, lr 0.1
0.914±0.072
0.809 – 0.990
0.693
batch norm, lr 0.01
0.756±0.007
0.747 – 0.765
0.953
no batch norm, lr 0.01
1.203±0.011
1.190 – 1.212
0.953
no batch norm, lr 0.08
0.844±0.012
0.828 – 0.856
0.693
Appendix
Table 5: Final τ of the conic head over five seeds per setting, with the value at the end of the decay path from τ=1.5 .
Figure 2: The panels show where training places the ratio τ , averaged over the classes. Panel (a) shows τ during 100 epochs in the four settings of Table 5 , five seeds each, together with the decay paths from τ=1.5 , drawn dotted. At the matched rate 0.01 , τ ends below its decay path with batch normalization and above it without. At the larger rates, both settings fall early and then recover above the path. Panel (b) shows τ after 20 epochs as a function of the learning rate (filled markers), together with the decay-only values (open markers).
initial rate
epochs 0–24
25–49
50–74
75–99
0.01
0.353
0.382
0.225
0.041
0.08
0.015
0.217
0.582
0.186
0.1
0.005
0.156
0.612
0.227
Appendix
Table 6: Share of the weights ΓT,s of Lemma H.1 in each quarter of training.
Optimization analyses for cross-entropy training rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective. These surrogates are reliable only inside the Taylor convergence radius of the true loss along the update direction. That radius is set not by real-line curvature alone but by the nearest complex singularity. For cross-entropy, the softmax partition function F=∑jexp(zj) has complex zeros -- ``ghosts of softmax'' -- that induce logarithmic singularities in the loss and cap this radius. To make this geometry usable, we derive closed-form expressions under logit linearization along the proposed update direction. In the binary case, the exact radius is ρ∗=δ2+π2/Δa. In the multiclass case, we obtain the lower bound ρa=π/Δa, where Δa=maxkak−minkak is the spread of directional logit derivatives ak=∇zk⋅v. This bound costs one Jacobian-vector product and reveals what makes a step fragile: samples that are both near a decision flip and highly sensitive to the proposed direction tighten the radius. The normalized step size r=τ/ρa separates safe from dangerous updates. Across six tested architectures and multiple step directions, no model fails for r<1, yet collapse appears once r≥1. Temperature scaling confirms the mechanism: normalizing by ρa shrinks the onset-threshold spread from standard deviation 0.992 to 0.164. A controller that enforces τ≤ρa survives learning-rate spikes up to 10,000× in our tests, where gradient clipping still collapses. Together, these results identify a geometric constraint on cross-entropy optimization that operates through Taylor convergence rather than Hessian curvature.
Classification and feature learning are two closely interrelated fundamental tasks. Feature learning in supervised settings is typically achieved by training classification models, under the common assumption that higher classification accuracy implies better feature quality. In this paper, we first point out that classification accuracy reflects the sample-wise local properties of a model on a dataset, whereas feature learning aims to learn features with good dataset-level global properties, such as intra-class compactness and inter-class separability. From the first principle of feature learning, we redesign the binary cross-entropy (BCE) loss, a counterpart of the cross-entropy (CE) loss used in classification. Then, in normalized feature space, we analyze the minima of the BCE and CE losses and show that both can lead to neural collapse, where sample features from different classes become maximally compact and maximally separated. More importantly, by investigating the convergence behaviors of the two losses as the scale factor γ varies in the normalized space, we reveal how classification accuracy and feature quality change differently across spaces of different sizes. Under a fixed training strategy, a very large γ achieves perfect classification accuracy but slows convergence toward the ideal feature geometry, whereas a very small γ makes classification highly sensitive to small perturbations. A moderate γ is therefore necessary for simultaneously achieving reliable classification and well-structured features. Extensive experiments validate these findings and demonstrate that BCE loss exhibits advantages in long-tail and open-set recognitions, as it better aligns with the global requirements of feature learning.
Qiufu Li, Huibin Xiao, Linlin Shen +3
School of Artificial Intelligence, Shenzhen University, Shenzhen, 518060, China. · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, Shenzhen, 518060, China. · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China. +3
Loss functions play a central role in supervised classification. Cross-entropy (CE) is widely used, whereas the mean absolute error (MAE) loss can offer robustness but is difficult to optimize. Interpolating between the CE and MAE losses, generalized cross-entropy (GCE) has recently been introduced to provide a trade-off between optimization difficulty and robustness. Existing formulations of GCE result in a non-convex optimization over classification margins that is prone to underfitting, leading to poor performances with complex datasets. In this paper, we propose a minimax formulation of generalized cross-entropy (MGCE) that results in a convex optimization over classification margins. Moreover, we show that MGCEs can provide an upper bound on the classification error. The proposed bilevel convex optimization can be efficiently implemented using stochastic gradient computed via implicit differentiation. Using benchmark datasets, we show that MGCE achieves strong accuracy, faster convergence, and better calibration, especially in the presence of label noise.
Kartheek Bondugula, Santiago Mazuelas, Aritz Pérez +1
Basque Center for Applied Mathematics, Bilbao, Spain · Ikerbasque, Basque Foundation for Science · Johns Hopkins University, Baltimore, USA