Kullback-Leibler Divergence

Follow this topic Ask about this topic

Recent momentum

-31%

9 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

7 new papers

A weekly snapshot of new work published in Kullback-Leibler Divergence.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Kullback-Leibler Divergence.

113 papers

Latest in Kullback-Leibler Divergence

Open your feed →

Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
Sep 15, 2026Haichen Hu, Yuheng Zhang, David Simchi-LeviKullback-Leibler Divergence
A Unified Per-Token Gating Family for On-Policy Distillation: FKL/RKL Mixing with Multi-Channel and Bias Coefficients
Sep 12, 2026Suwan Wu, Yumeng Lin, Pengcheng Yuan +1On-Policy Distillation Kullback-Leibler Divergence
Local Robustness Quantification for Naive Bayes Classifiers and Generative Forests: a General Approach
Sep 11, 2026Adrián Detavernier, Jasper De BockRandom Forest Empirical Risk Minimization
How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL
Sep 11, 2026Qifu Wen, Shuaijun Liu, Zihan Zhou +2Posterior Kullback-Leibler Divergence
Likelihood-free inference with nuisance parameters through normalizing flows
Sep 9, 2026Phil AsshetonNormalizing Flows Kullback-Leibler Divergence
Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation
Sep 9, 2026Hiroshi Tamano, Daichi MochihashiItem Response Theory Kullback-Leibler Divergence
FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Sep 9, 2026Yansen Han, Shengyi Liao, Peng Sun +4Flow-Matching Loss Preference Alignment
Mode Coverage in Normalizing Flow Boltzmann Generators via Log-Ratio Variation
Sep 8, 2026Qi Feng, Rongjie Lai, Di Qi +1Normalizing Flows Boltzmann Distributions
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Heterogeneous Knowledge Distillation Teacher
Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions
Aug 31, 2026Joonyong Park, Shinnosuke Takamichi, David M. Chan +3Neural Audio Codecs Calibration-Free Low-Bit Vector Quantization
SoftWater: Class-Aware Rate Allocation for Softmax Quantization
Aug 12, 2026Joao V. Cavalcanti, Ashia C. WilsonLow-Bit Quantization Gumbel-Softmax Relaxation
A Local Sinkhorn Framework for Conditional Distribution Reconstruction of Multidimensional Random Fields
Aug 12, 2026Mingtao Xia, Qijing ShenSinkhorn Conditional Distribution
PAC-Bayes Beyond Parameter Space: Behavioral Equivalence, Z-Information, and Exact Complexity Decomposition
Aug 11, 2026Vasant G. Honavar, Satish Kumar Keshri, Neil Ashtekar +1Bayesian Inference Approaches Predictive Coding
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Aug 10, 2026Zhuo Sun, Entong Li, Yanlong Zhao +7Unsupervised On-Policy Self-Distillation Self-Teacher
Exact Rank-Space KL Projection for Shared-Marginal Low-Rank Factors: Application to Doubly Stochastic Clustering
Aug 9, 2026Enliang HuLow-Rank Factorization Kullback-Leibler Regularization
Causal State-Space Model for Causal Inference: Estimating Longitudinal Individual Treatment Effects
Aug 8, 2026Abisoye Abidakun, Mingjun Zhong, Georgios LeontidisCausal Inferences Heterogeneous Treatment Effects
Finite Constant Frontiers and Auditable Regret Certificates for Average-Reward Reinforcement Learning
Aug 7, 2026Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Md Najmus SwaqeebMarkov Decision Processes Optimal Learner
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Heterogeneous Knowledge Distillation Dataset Distillation
Flux-OPD: On-Policy Distillation with Evolving Contexts
Jul 30, 2026Yuran Wang, Zekun Wang, Bohan Zeng +10On-Policy Distillation Self-Teacher
Trajectory-Regularized Stochastic Optimal Control via KL Divergence
Jul 24, 2026Mintae Kim, Koushil SreenathStochastic Optimal Control Kullback-Leibler Regularization
From Score Approximation to Distribution Approximation in Score-Based Diffusion Models
Jul 24, 2026Lan V. TruongScore-Based Diffusion Model Neural-Network Approximation
When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Heterogeneous Knowledge Distillation Summarization
Total Variation Distance Estimation in Autoregressive Models
Jul 21, 2026Eric Price, Kevin Tian, Zhiyang Xun +1Autoregressive Language Models Total Variation
Distilled Reinforcement Learning for LLM Post-training
Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4Reinforcement Learning Post-Training Large Language Model Post-Training
Approximate Relative Entropy Constraints for Nonlinear Covariance Steering Under Distribution Ambiguity
Jul 18, 2026Trevor N. Wolf, Jay W. McMahonCovariance Estimation Conditional-Value-At-Risk
Feedback Attribution and Representation Geometry: Metrics for Comparing Individual and Shared Rewards in MARL
Jul 17, 2026Tasha Pais, Richard HigginsMulti-Agent Reinforcement Learning Reward Signal
An Efficient Newton Algorithm for Nonnegative Matrix Factorization with the Kullback-Leibler Divergence
Jul 15, 2026Damien Lesens, Jérémy E. Cohen, Bora UçarNonnegative Matrix Factorization Kullback-Leibler Divergence
Can a Language Model Learn Facts Continually in Its Weights?
Jul 13, 2026Charles O'NeillLarge Language Model Memory Facts
A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems
Jul 7, 2026Fabian Schneider, Tapio Helin, Leila TaghizadehIll-Posed Inverse Problem Neural-Network Approximation
Lean-Quantum: Toward AI-Assisted Formalization of Quantum Information
Jul 6, 2026Kazumi Kasaura, Kei Tsukamoto, Kento Mori +6Information-Theoretic Limits Formalization
What Does a Discrete Diffusion Model Learn?
Jul 6, 2026Rodrigo Casado Noguerales, Bernhard Schölkopf, Thomas Hofmann +1Diffusion Models Kullback-Leibler Divergence
Tightening the Score Matching Gap for Diffusion Models
Jul 5, 2026Benjamin Dupuis, Tyler Farghly, Maxime Haddouche +2Score-Based Diffusion Model Kullback-Leibler Divergence
CouCE: A Unified Causal Framework for Debiased Deep Metric Learning
Jun 29, 2026Xin Yuan, Zhenyang Niu, Meiqi Wan +3State-Of-The-Art Zero-Shot Performance Meta-Learning
ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
Jun 29, 2026Zilong Liu, Xuewen Zhang, Jinrui Xing +3Dataset Distillation Kullback-Leibler Divergence
KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search
Jun 29, 2026Tao Feng, Xinke Jiang, Chao WuAgentic Reinforcement Learning Heterogeneous Knowledge Distillation
A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
Jun 29, 2026Liu ZewenModel Self-Assessment Preference Pairs
Beyond Global Divergences: A Local-Mass Perspective on Bayesian Inference
Jun 25, 2026Hanli Xu, Fengxiang He, Sarat MokaBayesian Inference Approaches Kullback-Leibler Divergence
AsyncOPD: How Stale Can On-Policy Distillation Be?
Jun 23, 2026Wonjun Kang, Kevin Galim, Seunghyuk Oh +9On-Policy Distillation Asynchronous Execution
KLip-PPO: A per-sample KL perspective on PPO-Clip
Jun 22, 2026Riccardo Colletti, Robin HolzingerProximal Policy Optimization Kullback-Leibler Divergence
StreamKL: Fast and Memory-Efficient KL Divergence for Boosting Attention Distillation
Jun 18, 2026Guangda Liu, Yiquan Wang, Chengwei Li +6Dynamic Sparse Attention Kullback-Leibler Divergence
QMaxCal: Path-Space Regularization for Open Quantum Control via Girsanov's Theorem
Jun 18, 2026Merijn Moody, Zier Mensch, Miranda C. N. Cheng +2Quantum Kullback-Leibler Regularization
Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
Jun 17, 2026Miloš Nikolić, Ali Hadi Zadeh, Enrique Torres Sanchez +1Large Language Model Quantization Kullback-Leibler Divergence
Expanding SPHERE-JEPA: A Family of Statistical Regularizers for the Hypersphere
Jun 16, 2026Léo Nicollier, Enric Meinhardt-Llopis, Max Dunitz +3Hypersphere Maximum Mean Discrepancy
Diffusion Flow Matching: Dimension-Improved KL Bounds and Wasserstein Guarantees
Jun 15, 2026Marta Gentiloni Silveri, Giovanni Conforti, Alain DurmusGenerative Models Kullback-Leibler Divergence
Not all Jensen-Shannon Divergence Estimators are Equal
Jun 15, 2026Alba Garrido, Alejandro Almodóvar, Mar Elizo +3Bregman Divergences Kullback-Leibler Divergence
Recovery thresholds for hidden weighted sparse graphs
Jun 12, 2026Zhe Hou, Jingcheng LiuExponential Random Graph Models Information-Theoretic Limits
Escaping the KL Agreement Trap in On-Policy Distillation
Jun 8, 2026Haoran Xin, Anhao Zhao, Ying Sun +3On-Policy Distillation Token-Level Supervision
Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
Jun 4, 2026Naïl B. Khelifa, Richard E. Turner, Ramji VenkataramananScore-Based Diffusion Model Diffusion Models
Dead Directions: Geometric Singular Learning
Jun 4, 2026Tejas Pradeep ShirodkarDirectionality Kullback-Leibler Divergence
BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation
Jun 2, 2026Xi Zhou, Famin Wu, Mingming Li +4Representation Learning Heterogeneity
Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
Jun 1, 2026Yuying Li, Leqi Zheng, Yongzi Yu +6On-Policy Distillation Kullback-Leibler Divergence
OPD+: Rethinking the Advantage Design for On-Policy Distillation
May 31, 2026Hanyang Zhao, Haoxian Chen, Han Lin +3On-Policy Distillation Rp-Opsd
Rethinking the Role of Temperature in Large Language Model Distillation
May 29, 2026Hoang-Chau Luong, Lingwei ChenKullback-Leibler Divergence Temperature
Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
May 29, 2026Sanae Lotfi, Polina Kirichenko, Steven Li +1Large Reasoning Models Quantization
Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints
May 29, 2026Shervin Khalafi, Alejandro Ribeiro, Dongsheng DingDiffusion Models Kullback-Leibler Divergence
Matching Rates and Optimal Allocation for Federated Probe-Logit Distillation under Heterogeneous Bandwidth Budgets
May 28, 2026Prasanjit Dubey, Xiaoming HuoBandwidth Extension Kullback-Leibler Divergence
Soft Specialists: $α$ -Rényi Ensembles for Uncertainty-Aware LLM Post-TrainingSoft Specialists: α-Rényi Ensembles for Uncertainty-Aware LLM Post-Training
May 26, 2026Paula Cordero-Encinar, Georgy Tyukin, Andrew B. DuncanLarge Language Model Uncertainty Large Language Model Post-Training
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6On-Policy Distillation Token-Level Supervision
Extreme Region Policy Distillation
May 25, 2026Changyu Chen, Xiting Wang, Rui YanOn-Policy Distillation Off-Policy Reinforcement Learning
TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models
May 22, 2026Arseny Ivanov, Sergei Kholkin, Vladislav Gromadskii +3Diffusion Language Models Diffusion Models

Sep 15, 2026 · cs.LGJ/K move · Enter open · S save

Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback

Haichen Hu, Yuheng Zhang, David Simchi-Levi

Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student. Direct imitation, however, can also transfer the teacher's systematic bias and errors. This challenge is particularly pronounced under covariate shift, when the teacher's reliability on target questions is uncertain and target-domain reward feedback is unavailable. We propose Coupled Calibration and Learning (CCL), an LLM distillation algorithm that couples teacher calibration with student updates through token-level branching, using reward feedback only on source questions. Each iteration calibrates the teacher using source feedback and then uses the calibrated teacher to train the student on target questions. The updated student, in turn, informs subsequent calibration. In an autoregressive policy framework, we prove that the output student's expected average Kullback-Leibler divergence to the oracle student converges to zero at a polynomial rate in the number of iterations. The oracle maximizes the true reference-regularized target reward within the student class, which need not represent the unrestricted optimal policy. Our analysis quantifies the progress of projected student gradient updates while controlling the error in teacher calibration. We further establish a separation from regularized direct matching: its error relative to the oracle student can remain bounded away from zero even when the teacher achieves higher regularized target reward than every student policy. These results demonstrate that LLM distillation can overcome persistent teacher bias and recover the optimal student through coupled calibration and learning, without target-domain reward feedback.