Cross Entropy Loss

Cross-entropy loss is a widely used objective function in machine learning, primarily for training classification models by minimizing the difference between predicted and true probability distributions. Current research focuses on addressing its limitations, particularly in large-scale applications like recommender systems and large language models, where modifications like scalable or reduced cross-entropy are being developed to improve efficiency and memory usage. Furthermore, research explores alternative loss functions or combinations with other methods (e.g., contrastive learning, Wasserstein loss) to enhance model performance, calibration, and robustness, especially in scenarios with limited data or imbalanced classes. These advancements have significant implications for improving the accuracy, efficiency, and reliability of various machine learning applications.

Papers

January 21, 2023

Improving Deep Regression with Ordinal Entropy
Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, Angela Yao
Cross Entropy Loss Regression Task Deep Regression Feature Vector Contrastive Ordinal

January 8, 2023

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning
Jidong Ge, Yuxiang Liu, Jie Gui, Lanting Fang, Ming Lin, James Tin-Yau Kwok, LiGuo Huang, Bin Luo
Contrastive Learning Self Supervised Learning Cross Entropy Loss Relational Information Clustering Loss Similarity Loss

December 2, 2022

Avoiding spurious correlations via logit correction
Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, Carlos Fernandez-Granda
Cross Entropy Loss Empirical Risk Minimization Spurious Correlation Spurious Attribute

November 27, 2022

Cross-domain Few-shot Segmentation with Transductive Fine-tuning
Yuhang Lu, Xinyi Wu, Zhenyao Wu, Song Wang
Cross Entropy Loss Unseen Domain Shot Segmentation Cross Domain Few Shot

November 23, 2022

Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systems
Jirong Yi, Qiaosheng Zhang, Zhen Chen, Qiao Liu, Wei Shao, Yusen He, Yaohua Wang
Stochastic Gradient Descent Mutual Information Cross Entropy Loss Information Theoretic Entropy Minimization Machine Learning Regression

November 22, 2022

VBLC: Visibility Boosting and Logit-Constraint Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions
Mingjia Li, Binhui Xie, Shuang Li, Chi Harold Liu, Xinjing Cheng
LeArning Abstract Cross Entropy Loss Adverse Condition Domain Adaptive Semantic Segmentation Brand Visibility Robust Adaptation Domain Adaptation Benchmark Adverse Condition Image

November 21, 2022

Blind Knowledge Distillation for Robust Image Classification
Timo Kaiser, Lukas Ehmann, Christoph Reinders, Bodo Rosenhahn
Knowledge Distillation Image Classification Noisy Label Cross Entropy Loss Noisy Sample

November 18, 2022

November 15, 2022

Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction
Leilei Gan, Baokui Li, Kun Kuang, Yating Zhang, Lei Wang, Luu Anh Tuan, Yi Yang, Fei Wu
Contrastive Learning Cross Entropy Loss Supervised Contrastive Learning Legal Judgment Prediction Empirical Evidence Legal Article

November 10, 2022

Regression as Classification: Influence of Task Formulation on Neural Network Features
Lawrence Stewart, Francis Bach, Quentin Berthet, Jean-Philippe Vert
Neural Network Classification Code Novel Regression Cross Entropy Loss External Influence Gradient Method Gradient Based Optimization Two Layer ReLU Task Formulation

November 7, 2022

Contrastive Classification and Representation Learning with Probabilistic Interpretation
Rahaf Aljundi, Yash Patel, Milan Sulc, Daniel Olmeda, Nikolay Chumerin
Contrastive Learning Representation Learning Cross Entropy Loss Probabilistic Interpretation

October 20, 2022

Multi-Granularity Optimization for Non-Autoregressive Translation
Yafu Li, Leyang Cui, Yongjing Yin, Yue Zhang
Back Propagation Cross Entropy Loss Non Autoregressive Translation Non Autoregressive Machine Translation

October 16, 2022

Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Condition
Yitzchak Shmalo
Deep Neural Network Training Data Core Stability Cross Entropy Loss DNN Framework

October 11, 2022

Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization
Ziquan Liu, Antoni B. Chan
Adversarial Attack Adversarial Training Adversarial Robustness Visual Perspective Cross Entropy Loss Adversarial Vulnerability

October 3, 2022

Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems
Jirong Yi, Qiaosheng Zhang, Zhen Chen, Qiao Liu, Wei Shao
Deep Neural Network Mutual Information Cross Entropy Loss Information Theoretic Cross Entropy Training Deep Deep Neural Network Classifier

October 1, 2022

Federated Representation Learning via Maximal Coding Rate Reduction
Juan Cervino, Navid NaderiAlizadeh, Alejandro Ribeiro
Meaningful Representation Cross Entropy Loss Low Dimensional Representation Rate Reduction Federated Representation Learning

September 22, 2022

CAMRI Loss: Improving Recall of a Specific Class without Sacrificing Accuracy
Daiki Nishiyama, Kazuto Fukuchi, Youhei Akimoto, Jun Sakuma
Loss Function Cross Entropy Loss Multi Class Classification Class Specific Recall Initiator Angular Margin Loss Penalty Based

September 14, 2022

Meta Pattern Concern Score: A Novel Evaluation Measure with Human Values for Multi-classifiers
Yanyun Wang, Dehui Du, Yuanhao Liu
Cross Entropy Loss Human Value Novel Evaluation Meta Feature Multi Classification

September 13, 2022

On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness
Adrian Galdran, Gustavo Carneiro, Miguel Ángel González Ballester
Loss Function Cross Entropy Loss Lesion Segmentation Item Combination Dice Loss Distribution Robustness

Cross Entropy Loss

Papers

Improving Deep Regression with Ordinal Entropy

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning

Avoiding spurious correlations via logit correction

Cross-domain Few-shot Segmentation with Transductive Fine-tuning

Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systems

VBLC: Visibility Boosting and Logit-Constraint Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions

Blind Knowledge Distillation for Robust Image Classification

A Fair Loss Function for Network Pruning

Weighted Ensemble Self-Supervised Learning

Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction

Regression as Classification: Influence of Task Formulation on Neural Network Features

Contrastive Classification and Representation Learning with Probabilistic Interpretation

Multi-Granularity Optimization for Non-Autoregressive Translation

Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Condition

Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization

Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems

Federated Representation Learning via Maximal Coding Rate Reduction

CAMRI Loss: Improving Recall of a Specific Class without Sacrificing Accuracy

Meta Pattern Concern Score: A Novel Evaluation Measure with Human Values for Multi-classifiers

On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness