cs.LGNov 13, 2026

On numerical issues of divergences between nearly equal distributions

Authors: Brandon Amos

Organizations: Reflection AI

Abstract

Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.

Figures & tables

Explore similar work

CardsList
  1. A Fix for Softmax and Cross-Entropy

    Oct 22, 2026Jonathan T. Barron, Dan Gnanapragasam, Michael TschannenNeural Network Activation Functions

  2. Divergence controls entropy in distillation

    Oct 2, 2026Nicolas Zucchet, Scott W. LindermanKullback-Leibler DivergenceKnowledge Distillation