Kullback-Leibler Divergence

Also known as KL

Follow this topic Ask about this topic

Latest papers

27 papers9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Information Estimation with Discrete Diffusion
Oct 16, 2026Alberto Foresti, Giulio Franzese, Pietro MichiardiMutual Information Estimation Diffusion Models
Department of Data Science, EURECOM, 450 Route des Chappes, 06410 Biot, France
Divergence-Routed On-Policy Distillation for Language Models
Oct 14, 2026Tiantian Peng, Guannan Lv, Hongyang Li +5On-Policy Distillation Knowledge Distillation
Peking University · Kuaishou Technology · Tsinghua University
DisCo Algebra: Why Disintegration and Combination Are Not Inverses
Oct 10, 2026Xuexing LuKullback-Leibler Divergence
School of Mathematics and Statistics, Zaozhuang University, China
KL Before and After Mixing: A Round-Complexity Separation in Parallel Sampling
Oct 9, 2026Tianyi WuKullback-Leibler Divergence
Amazon
Universal Bounds for Self-Normalized Kullback-Leibler Estimation
Oct 9, 2026Buu Phan, Lester Mackey, David Alvarez-MelisKullback-Leibler Divergence Importance Sampling
University of Toronto · Microsoft Research · Harvard University
Divergence controls entropy in distillation
Oct 2, 2026Nicolas Zucchet, Scott W. LindermanKullback-Leibler Divergence Knowledge Distillation
Stanford University
How Divergence Becomes Decision Flips in Compressed Language Models
Sep 30, 2026Beatriz Almeida FelicioLLM Evaluation LLM Compression
Instituto de Informática (INF) Universidade Federal de Goiás (UFG)
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Sep 29, 2026Qiwei Di, Xuheng Li, Kaixuan Ji +3Multi-Teacher Knowledge Distillation On-Policy Distillation
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
The Risk-Sensitive Schrödinger Bridge: Is Not a KL Projection
Sep 23, 2026Hamidreza BehjooKullback-Leibler Divergence Schrödinger Bridges
TF-IDF and BM25 Are Exact KL Divergences
Sep 12, 2026Ivan SilajevKullback-Leibler Divergence Information Retrieval
Independent Researcher
Mode Coverage in Normalizing Flow Boltzmann Generators via Log-Ratio Variation
Sep 8, 2026Qi Feng, Rongjie Lai, Di Qi +1Normalizing Flows Molecular Generation
Department of Mathematics, Florida State University, Tallahassee, FL 32306 · Department of Mathematics, Purdue University, West Lafayette, IN 47907
A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation
Aug 26, 2026Bing Shao, Jiazheng Zhang, Long Ma +13Language Model Distillation On-Policy Distillation
College of Computer Science and Artificial Intelligence, Fudan University
Tightening the Score Matching Gap for Diffusion Models
Jul 5, 2026Benjamin Dupuis, Tyler Farghly, Maxime Haddouche +2Score Matching Score-Based Generative Modeling
INRIA - CNRS - D´epartement d’Informatique de l’Ecole Normale Sup´erieure - PSL Research University, France · Ecole Polytechnique - CMAP, IP Paris, Palaiseau, France
Beyond Global Divergences: A Local-Mass Perspective on Bayesian Inference
Jun 25, 2026Hanli Xu, Fengxiang He, Sarat MokaBayesian Inference Kullback-Leibler Divergence
UNSW Sydney · University of Edinburgh
Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
Jun 17, 2026Miloš Nikolić, Ali Hadi Zadeh, Enrique Torres Sanchez +1LLM Quantization LLM Evaluation
ByteShape · University of Toronto · Vector Institute for Artificial Intelligence
Diffusion Flow Matching: Dimension-Improved KL Bounds and Wasserstein Guarantees
Jun 15, 2026Marta Gentiloni Silveri, Giovanni Conforti, Alain DurmusDiffusion Sampling Flow-Based Generative Modeling
Ecole Polytechnique, Massy Palaiseau, France · Università degli Studi di Padova, Padua, Italy
Rethinking the Role of Temperature in Large Language Model Distillation
May 29, 2026Hoang-Chau Luong, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY, United States
A Spectral Framework for Closed-Form Relative Density Estimation
May 11, 2026Francis BachDensity Ratio Estimation Spectral Methods
SIERRA · Inria - Ecole Normale Supérieure · PSL Research University
Exponential families from a single KL identity
Apr 30, 2026Marc DymetmanInformation Geometry Exponential Family
Scientific Consultant
KL Divergence Between Gaussians: A Step-by-Step Derivation for the Variational Autoencoder Objective
Apr 13, 2026Andrés Muñoz, Rodrigo RameleKullback-Leibler Divergence Variational Autoencoders
Instituto Tecnol´ogico de Buenos Aires
Worst-Case Optimal $\sqrtε$ -Rate Stability of KL Divergence under Perturbations of Gaussian Distributions
Apr 13, 2026Jialu Pan, Yufeng Zhang, Nan Hu +3Kullback-Leibler Divergence Information Geometry
College of Computer Science and Electronic Engineering, Hunan University, Changsha, China. · National University of Defense Technology, China. · Department of Computer Science, State University of New York at New Paltz, 1 Hawk Drive, New Paltz, NY 12561, USA.
Robust Multi-Agent Target Tracking in Intermittent Communication Environments via Analytical Belief Merging
Apr 8, 2026Mohamed Abdelnaby, Samuel Honor, Kevin LeahyKullback-Leibler Divergence Multi-Agent Systems
Department of Robotics Engineering, Worcester Polytechnic Institute (WPI), Worcester, MA 01609, USA.
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
Mar 31, 2026Hoang-Chau Luong, Dat Ba Tran, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Rochester Institute of Technology, Rochester, NY, USA · Rowan University, Glassboro, NJ, USA
Surprisal-Rényi Free Energy
Mar 3, 2026Shion Matsumoto, Raul Castillo, Benjamin Prada +1Large Deviation Principles Kullback-Leibler Divergence
Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, Florida, USA
Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence
Mar 2, 2026Shiyuan Zhang, Qiwei Di, Xuheng Li +1Kullback-Leibler Divergence Markov Chain Monte Carlo
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models
Jan 20, 2026Qiang Sun, H. Vincent Poor, Wenyi ZhangDiffusion Sampling Kullback-Leibler Divergence
Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China · Department of Electrical Engineering, Princeton University, Princeton, NJ, USA
JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence
Jan 8, 2026Shengyin Sun, Yiming Li, Renxi Liu +5Speculative Decoding LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies

Oct 16, 2026J/K move · Enter open · S save

Information Estimation with Discrete Diffusion

Alberto Foresti, Giulio Franzese, Pietro Michiardi

Department of Data Science, EURECOM, 450 Route des Chappes, 06410 Biot, France

Mutual Information Estimation Diffusion Models Kullback-Leibler Divergence

Open paper AskPDF arXiv

Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion-based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models. Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the "embedding trick". Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.