Kullback Leibler Divergence

Kullback-Leibler (KL) divergence is a measure of the difference between two probability distributions, frequently used to quantify the information lost when one distribution is approximated by another. Current research focuses on improving KL divergence-based methods for various machine learning tasks, including generative modeling (e.g., using diffusion models and variational autoencoders), density estimation, and knowledge distillation, often incorporating techniques like Wasserstein gradient flows and $f$-divergences for enhanced performance and robustness. The accurate and efficient computation of KL divergence, and its generalizations, is crucial for advancing these fields, impacting applications ranging from image generation and natural language processing to Bayesian inference and statistical modeling.

Papers

May 30, 2024

On the Connection Between Non-negative Matrix Factorization and Latent Dirichlet Allocation
Benedikt Geiger, Peter J. Park
Matrix Factorization Kullback Leibler Divergence New Connection Latent Dirichlet Allocation Non Negative Matrix Factorization Latent Semantic Analysis

April 19, 2024

Risk Bounds for Mixture Density Estimation on Compact Domains via the $h$-Lifted Kullback--Leibler Divergence
Mark Chiu Chong, Hien Duy Nguyen, TrungTin Nguyen
Mixture Model Kullback Leibler Divergence Probability Density Function Risk Bound Density Evolution Likelihood Estimation Continuous Domain

April 3, 2024

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, Ngai Wong
Knowledge Distillation Kullback Leibler Divergence Divergence Estimation

March 20, 2024

Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs
Charles C. Margossian, Loucas Pillaud-Vivien, Lawrence K. Saul
General Analysis Uncertainty Quantification Variational Inference Kullback Leibler Divergence Variational Approximation R\'enyi Divergence

March 18, 2024

Variational Approach for Efficient KL Divergence Estimation in Dirichlet Mixture Models
Samyajoy Pal, Christian Heumann
Variational Method Kullback Leibler Divergence Inverse Divergence Compositional Data Dirichlet Process Mixture Model

March 15, 2024

Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference
Declan McNamara, Jackson Loper, Jeffrey Regier
Kullback Leibler Divergence Sequential Monte Carlo Variational Distribution Amortized Variational Inference Kullback Leibler

March 11, 2024

Efficient first-order algorithms for large-scale, non-smooth maximum entropy models with application to wildfire science
Gabriel P. Langlois, Jatan Buch, Jérôme Darbon
Application Proficiency High Efficiency Science Journalism Kullback Leibler Divergence Maximum Entropy First Order Algorithm MaxEnt RL

March 4, 2024

Soft-constrained Schrodinger Bridge: a Stochastic Control Approach
Jhanvi Garg, Xianyang Zhang, Quan Zhou
Kullback Leibler Divergence Schr\"odinger Bridge Stochastic Control

March 3, 2024

Limits to classification performance by relating Kullback-Leibler divergence to Cohen's Kappa
L. Crow, S. J. Watts
Machine Learning Continuum Limit Kullback Leibler Divergence Classification Algorithm Classification Performance Confusion Matrix

February 7, 2024

Wasserstein Gradient Flows for Moreau Envelopes of f-Divergences in Reproducing Kernel Hilbert Spaces
Sebastian Neumayer, Viktor Stein, Gabriele Steidl, Nicolaj Rux
Reproducing Kernel Hilbert Space Kullback Leibler Divergence Wasserstein Gradient Wasserstein Gradient Flow F$ Divergence Moreau Envelope

January 31, 2024

Convergence analysis of t-SNE as a gradient flow for point cloud on a manifold
Seonghyeon Jeong, Hau-Tieng Wu
Point Cloud Gradient Flow Convergence Analysis Kullback Leibler Divergence SNE Algorithm Weak Convergence

December 13, 2023

Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift
Mona Schirmer, Dan Zhang, Eric Nalisnick
System Performance Distribution Shift Distribution Data Kullback Leibler Divergence Inverse Divergence Human Agreement Predictive Distribution

November 29, 2023

Entropy and the Kullback-Leibler Divergence for Bayesian Networks: Computational Complexity and Efficient Implementation
Marco Scutari
Causal Inference Bayesian Network Kullback Leibler Divergence Computational Complexity Distributional Assumption Stable Entropy Graphical Structure Gaussian Bayesian Network

November 23, 2023

Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning
Seonghak Kim, Gyeongdo Ham, Yucheol Cho, Daeshik Kim
Knowledge Distillation Data Augmentation Strong Generalization Kullback Leibler Divergence Distance Correlation

November 7, 2023

Innovation and Word Usage Patterns in Machine Learning
Vítor Bandeira Borges, Daniel Oliveira Cajueiro
Machine Learning Kullback Leibler Divergence Latent Dirichlet Allocation Digital Innovation Inverse Divergence Research Trend Language Pattern

November 2, 2023

Time-Independent Information-Theoretic Generalization Bounds for SGLD
Futoshi Futami, Masahiro Fujisawa
Langevin Dynamic Kullback Leibler Divergence Generalization Error Bound Information Theoretic Generalization Bound Information Theoretic Generalization Time Invariant

October 13, 2023

October 11, 2023

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains
Qingyue Zhao, Banghua Zhu
Knowledge Transfer Maximum Likelihood Kullback Leibler Divergence Fundamental Limit Probabilistic Classifier Continuous Domain Naive Adaptation

October 3, 2023

Simulation-based Inference with the Generalized Kullback-Leibler Divergence
Benjamin Kurt Miller, Marco Federici, Christoph Weniger, Patrick Forré
Kullback Leibler Divergence Simulation Based Inference Neural Posterior Estimation Neural Ratio Estimation

Kullback Leibler Divergence

Papers

On the Connection Between Non-negative Matrix Factorization and Latent Dirichlet Allocation

Risk Bounds for Mixture Density Estimation on Compact Domains via the $h$-Lifted Kullback--Leibler Divergence

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs

Variational Approach for Efficient KL Divergence Estimation in Dirichlet Mixture Models

Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference

Efficient first-order algorithms for large-scale, non-smooth maximum entropy models with application to wildfire science

Soft-constrained Schrodinger Bridge: a Stochastic Control Approach

Limits to classification performance by relating Kullback-Leibler divergence to Cohen's Kappa

Wasserstein Gradient Flows for Moreau Envelopes of f-Divergences in Reproducing Kernel Hilbert Spaces

Convergence analysis of t-SNE as a gradient flow for point cloud on a manifold

Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift

Entropy and the Kullback-Leibler Divergence for Bayesian Networks: Computational Complexity and Efficient Implementation

Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning

Innovation and Word Usage Patterns in Machine Learning

Time-Independent Information-Theoretic Generalization Bounds for SGLD

Computing Marginal and Conditional Divergences between Decomposable Models with Applications

MINDE: Mutual Information Neural Diffusion Estimation

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains

Simulation-based Inference with the Generalized Kullback-Leibler Divergence