Classification heads built on conic functions give each class a bounded region of the feature space, and such regions are used to reject samples of unseen classes. We ask whether these regions also keep the features bounded when the head is trained with softmax cross-entropy. For a head whose class scores combine a linear term with a Euclidean norm, we first show that the region of each class is bounded by a quadric of revolution, whose eccentricity is the reciprocal of one learned ratio. We then attach to each class an asymptotic margin, which compares the growth of its score with the growth of all the other scores and can be computed by a second-order cone program. The loss of a class has infimum zero exactly when its margin is positive, and it then approaches zero only along diverging features. When the margin is negative, the loss is coercive and its minimizers lie in an explicit ball. Since softmax depends only on the differences between scores, a class with a bounded region can still have a positive margin, so bounded regions do not confine the features. The result extends to other norms, including the polyhedral conic functions. On CIFAR-10 and CIFAR-100, all 1210 cells of 67 trained conic heads have positive margins, whereas a prototype head keeps nine of its ten cells bounded, as the theory predicts.
Figures & tables
Figure 1: The panels illustrate Theorems 3.1 and 3.2 with four cones of unequal apertures and non-parallel axes. The left panel shows the score surface minkhk , the lower envelope of the cones, whose red curves mark the creases along which the lowest cone changes from one class to another. The right panel shows the same configuration from above, where the creases appear as curved decision boundaries.
setting
acc., affine
acc., conic
p
∥z∥ , affine
∥z∥ , conic
batch norm, lr 0.1
95.03±0.14
94.90±0.17
0.25
6.01±0.01
5.27±0.24
no batch norm, lr 0.01
90.48±0.17
91.20±0.28
0.008
39.9±1.7
24.7±0.7
no batch norm, lr 0.08
93.31±0.19
93.26±0.20
0.68
26.5±0.4
19.9±0.6
Table 1: Final test accuracy (%) and mean norm of the test features, mean ± sd over five seeds, with permutation p -values for the accuracy.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
batch norm, lr 0.1
no batch norm, lr 0.01
no batch norm, lr 0.08
head
acc.
∥z∥
AUC
acc.
∥z∥
AUC
acc.
∥z∥
AUC
affine
94.92
6.04
0.712
90.51
39.4
0.767
93.21
26.2
0.847
conic
94.88
5.26
0.806
91.09
24.8
0.816
93.32
20.1
0.875
conic, ηk without decay
94.90
6.09
0.705
91.22
11.3
0.832
93.33
7.57
0.888
prototype
95.13
3.86
0.868
—
—
—
93.47
6.69
0.897
cosine
94.99
1.08
0.914
91.79
0.38
0.867
93.44
0.25
0.897
Appendix
Table 2: Additional runs: final test accuracy (%), mean norm of the test features, and area under the ROC curve of the feature norm for CIFAR-10 against CIFAR-100 test images, each the mean over five seeds.
batch norm
no batch norm
lr
affine
conic
τ
lr
affine
conic
τ
0.05
0.9212
0.8979
0.489
0.005
0.7071
0.7377
1.456
0.1
0.9205
0.8970
0.475
0.01
0.7740
0.8055
1.417
0.2
0.9137
0.9101
0.588
0.02
0.8429
0.8617
1.299
0.4
0.9017
0.9061
0.609
0.04
0.8828
0.8859
1.062
0.5
—
0.8953
0.617
0.08
0.8861
0.8881
0.680
Appendix
Table 3: Final test accuracy after 20 epochs as a function of the learning rate, with the final τ of the conic head.
configuration
final accuracy
final τ
affine, batch norm
0.9495
—
conic ε=0.1 , τ free
0.9505
0.941
conic ε=0.5 , τ≥1
0.9474
1.884
conic ε=0.5 , τ frozen 5 ep
0.9482
0.929
affine, no batch norm, lr 0.01
0.9045
—
conic ε=0.5 , no batch norm, lr 0.01
0.9126
1.209
Appendix
Table 4: Exploratory single-seed runs, on CIFAR-10 unless otherwise stated.
setting
τ , mean ± sd
range
decay path
batch norm, lr 0.1
0.914±0.072
0.809 – 0.990
0.693
batch norm, lr 0.01
0.756±0.007
0.747 – 0.765
0.953
no batch norm, lr 0.01
1.203±0.011
1.190 – 1.212
0.953
no batch norm, lr 0.08
0.844±0.012
0.828 – 0.856
0.693
Appendix
Table 5: Final τ of the conic head over five seeds per setting, with the value at the end of the decay path from τ=1.5 .
Figure 2: The panels show where training places the ratio τ , averaged over the classes. Panel (a) shows τ during 100 epochs in the four settings of Table 5 , five seeds each, together with the decay paths from τ=1.5 , drawn dotted. At the matched rate 0.01 , τ ends below its decay path with batch normalization and above it without. At the larger rates, both settings fall early and then recover above the path. Panel (b) shows τ after 20 epochs as a function of the learning rate (filled markers), together with the decay-only values (open markers).
initial rate
epochs 0–24
25–49
50–74
75–99
0.01
0.353
0.382
0.225
0.041
0.08
0.015
0.217
0.582
0.186
0.1
0.005
0.156
0.612
0.227
Appendix
Table 6: Share of the weights ΓT,s of Lemma H.1 in each quarter of training.
School of Artificial Intelligence, Shenzhen University, Shenzhen, 518060, China. · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, Shenzhen, 518060, China. · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China. +3