Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

Hoang-Chau Luong; Dat Ba Tran; Lingwei Chen

cs.LGMar 31, 2026

Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

Authors: Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen

Organizations: Rochester Institute of Technology, Rochester, NY, USA · Rowan University, Glassboro, NJ, USA

Abstract

Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL), particularly in regimes with large vocabularies and significant teacher-student capacity mismatch, where RKL focuses learning on dominant modes rather than enforcing dense alignment. However, RKL introduces a structural limitation that drives the student toward overconfident predictions. We first provide an analysis of RKL by decomposing its gradients into target and non-target components, and show that non-target gradients consistently push the target logit upward even when the student already matches the teacher, thereby reducing output diversity. In addition, RKL provides weak supervision over non-target classes, leading to poor tail alignment. To address these issues, we propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL. Extensive experiments across datasets and model families demonstrate that DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity-diversity trade-off.

Figures & tables

Panel of Figure 1 — Figure 1: FKL and RKL when fitting a teacher distribution under different output sizes. The x-axis shows the class index, and the y-axis shows the corresponding output probability. As the number of classes increases, the optimization difficulty grows substantially.

Panel of Figure 2 — Figure 2: (a, b) Fidelity vs. diversity: RKL reduces diversity (Negative Self-BLEU and Distinct-2), while DRKL achieves a better balance across methods. (c) ROUGE-L vs. prediction confidence: RKL produces overconfident predictions without improving ROUGE-L, while DRKL is better calibrated.

Figure 3: Losses comparison.

Figure 4: Performance of DRKL when combined with SRKL.

Method	Dolly	Self-Instruct	Vicuna	Super-NI	UnNI	Avg. ( $\uparrow$ )
GPT-2 XL (Teacher)	27.00 $\pm$ 0.19	14.07 $\pm$ 0.37	16.31 $\pm$ 0.32	26.46 $\pm$ 0.41	31.10 $\pm$ 0.06	22.99
GPT-2 XL (1.5B) $\to$ GPT-2 Base (0.1B)
FKL	23.80 $\pm$ 0.55	9.68 $\pm$ 0.49	14.53 $\pm$ 0.31	16.27 $\pm$ 0.24	19.03 $\pm$ 0.09	16.66
RKL ( Gu et al., 2024 )	24.67 $\pm$ 0.13	11.25 $\pm$ 0.35	15.82 $\pm$ 0.43	21.03 $\pm$ 0.21	23.94 $\pm$ 0.12	19.34
Sym-KL	24.38 $\pm$ 0.14	11.07 $\pm$ 0.43	15.72 $\pm$ 0.75	19.92 $\pm$ 0.25	22.21 $\pm$ 0.03	18.66
DKD ( Zhao et al., 2022 )	22.88 $\pm$ 0.23	10.28 $\pm$ 0.29	16.12 $\pm$ 0.31	16.57 $\pm$ 0.25	19.64 $\pm$ 0.13	17.10

Table 1: Comparison of distillation losses for GPT-2 and OPT teacher–student pairs. We report ROUGE-L, with mean and standard deviation computed over five random seeds. The best results are boldfaced and the second-best are underlined .

Panel of Figure 5 — Figure 5: (a) Validation performance of different distillation losses. (b) DRKL performance across different values of $\gamma$ . (c) Efficiency analysis.

Case study # 1
Instruction	Given a paragraph with different names for the same thing, write a question that asks about this.
Input	Passage: The United States of America is composed of fifty states, each with its own state capital. California’s capital is Sacramento, while Florida’s state capital is Tallahassee and Texas’ capital city is Austin.
Ground-truth	What is the capital of the United States of America?
FKL	Yes, the United States of America is made of fifty states, each having its own state capital. The states that comprise the United States of America are California, Florida, Pennsylvania, and Texas.
RKL	A passage describing the United States of America includes the phrase ”each with its own state capital”. This questions implies that the United States is composed of anywhere between 50 and 100 states, with each state capital being its own distinct entity.
Sym-KL	The United States of America is comprised of fifty states, each of which has its own state capital. <n> California’s state capital is Sacramento, while Georgia’s state capital is Atlanta and Texas’ capital city, Austin.

Table 2: Qualitative examples from the Unnatural ( Honovich et al., 2023 ) dataset for instruction following. The student OPT-1.3B model is distilled from the OPT-6.7B teacher. DRKL generates responses that better satisfy the given instructions compared to baseline methods.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Ratio of RKL to FKL gradient norms, illustrating gradient-scale mismatch.

Figure 7: Teacher–student alignment under different distillation losses.

Case study # 2
Instruction	In this task, you will be given a piece of text and asked to report what someone said. Your answer should contain all the necessary information about who said what, but it can be delivered in any format.
Input	”I’m so excited for the party tonight!” cried Sarah.
Ground-truth	Sarah said ”I’m so excited for the party tonight!”
FKL	”I’m so excited for tonight!” cried Sarah
RKL	Sarah is so excited for the party tomorrow! She lives in the city and is going to have a blast. The weather is finally good, so she can go out and play. She typically brings along her two little crushes, who are her little guards. She plans to buy them each a party band, in memory of her late boyfriend.
Sym-KL	Text: ”Brian! Brian, I’m so excited for the Party tonight!” cried Sarah.

Appendix

Table 3: Qualitative examples from the Unnatural ( Honovich et al., 2023 ) dataset for instruction following. The student OPT-1.3B model is distilled from the OPT-6.7B teacher. DRKL generates responses that better satisfy the given instructions compared to baseline methods.

Case study # 4
Instruction	Given a paragraph, write down all the proper nouns in the order they occur. A proper noun is a specific name for a person, place, or thing and is always capitalized.
Input	Passage: Toby thought that Mr. Krebs was being unfair when he gave out detentions for talking during class, but Mrs. Dvorak said that it was part of his job as a teacher.
Ground-truth	Toby, Mr. Krebs, Mrs. Dvorak
FKL	Toby, Ms. Dvorak, and Mr. Krebs are all proper nouns.
RKL	Toby, Mrs. Dvorak, Mr. Krebs, Class
Sym-KL	Toby, Mrs. Dvorak

Appendix

Table 4: Qualitative examples from the Unnatural ( Honovich et al., 2023 ) dataset for instruction following. The student OPT-1.3B model is distilled from the OPT-6.7B teacher. DRKL generates responses that better satisfy the given instructions compared to baseline methods.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 514 papers

Rethinking the Role of Temperature in Large Language Model Distillation
May 29, 2026Hoang-Chau Luong, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY, United States
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Data-Free Knowledge Distillation Language Model Distillation
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +3
Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation
Feb 24, 2026Sayantan Dasgupta, Trevor Cohn, Timothy BaldwinLanguage Model Distillation Knowledge Distillation
School of Computing and Information Systems, University of Melbourne, Melbourne, Australia.

May 29, 2026J/K move · Enter open · S save

Rethinking the Role of Temperature in Large Language Model Distillation

Hoang-Chau Luong, Lingwei Chen

Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY, United States

Language Model Distillation Knowledge Distillation Kullback-Leibler Divergence

Open paper AskPDF arXiv

Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature

τ

, overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and RKL. Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients, causing FKL to benefit much more from

τ

scaling than RKL. This asymmetry overturns the standard empirical conclusion: although RKL outperforms FKL at

τ=1

, FKL consistently surpasses RKL at higher temperatures across instruction-following benchmarks. Moreover, the impact of temperature is not limited to FKL; it improves a broader family of distillation objectives, enabling simple KL-based methods to achieve competitive performance against recent state-of-the-art LLM distillation approaches.

Abstract

Language Model Distillation

Explore similar work

Rethinking the Role of Temperature in Large Language Model Distillation

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation