cs.LGNov 5, 2026

Comparing Classification and Feature Learning by Analyzing Their Loss Functions in Normalized Spaces with Different Scale Factors

Authors: Qiufu Li, Huibin Xiao, Linlin Shen, Weijia Fan, Xi Jia, Yongdong Huang

Organizations: School of Artificial Intelligence, Shenzhen University, Shenzhen, 518060, China. · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, Shenzhen, 518060, China. · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China. · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China. · 

Abstract

Classification and feature learning are two closely interrelated fundamental tasks. Feature learning in supervised settings is typically achieved by training classification models, under the common assumption that higher classification accuracy implies better feature quality. In this paper, we first point out that classification accuracy reflects the sample-wise local properties of a model on a dataset, whereas feature learning aims to learn features with good dataset-level global properties, such as intra-class compactness and inter-class separability. From the first principle of feature learning, we redesign the binary cross-entropy (BCE) loss, a counterpart of the cross-entropy (CE) loss used in classification. Then, in normalized feature space, we analyze the minima of the BCE and CE losses and show that both can lead to neural collapse, where sample features from different classes become maximally compact and maximally separated. More importantly, by investigating the convergence behaviors of the two losses as the scale factor γγ varies in the normalized space, we reveal how classification accuracy and feature quality change differently across spaces of different sizes. Under a fixed training strategy, a very large γγ achieves perfect classification accuracy but slows convergence toward the ideal feature geometry, whereas a very small γγ makes classification highly sensitive to small perturbations. A moderate γγ is therefore necessary for simultaneously achieving reliable classification and well-structured features. Extensive experiments validate these findings and demonstrate that BCE loss exhibits advantages in long-tail and open-set recognitions, as it better aligns with the global requirements of feature learning.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Neural Collapse on Imbalanced Datasets and Open-Set Scenario

    Nov 5, 2026Qiufu Li, Huibing Xiao, Linlin Shen +1Neural CollapseClass-Imbalanced Learning

  2. Neural Collapse by Design: Learning Class Prototypes on the Hypersphere

    May 19, 2026Panagiotis Koromilas, Theodoros Giannakopoulos, Mihalis A. Nicolaou +1Prototype-Based ClassificationHyperspherical Representation Learning

  3. Asymptotic Margins and Feature Confinement in Conic Classification Heads under Softmax Cross-Entropy

    Nov 13, 2026Omer Nezih GerekClassificationMargin-Based Learning