Language Model Distillation

Latest papers 350

All topics
CardsList
  1. Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers

    Sep 4, 2026Yumiao Li, Peixin Liu, Donglin Di +2LLM EvaluationPolicy Evaluation

  2. Rethinking On-Policy Distillation of Large Language Models II: One Training Example

    Sep 3, 2026Zixuan Fu, Bingxiang He, Yuxin Zuo +10Language Model DistillationOn-Policy Distillation

  3. ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models

    Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model PretrainingAutoregressive Language Modeling

  4. Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation

    Sep 2, 2026Wenhui Chen, Zhifeng Li, Jie Zhou +5LLM CompressionLanguage Model Distillation

  5. Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

    Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Language Model DistillationFactual Knowledge in Language Models

  6. Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

    Sep 1, 2026Zhixuan Liu, Zhichen Dong, Yuyu Fan +2Supervised Fine-TuningSubliminal Learning

  7. SimCRAFT: Distilling Remote Sensing Agents via Synthetic Trajectories and Contextual Retrieval-Augmented Fine-Tuning

    Aug 31, 2026Haoran Wang, Jing Yao, Xu Yang +4Fine-TuningRemote Sensing

  8. Stratified Consistency Distillation for Natural Language Formalization

    Aug 31, 2026Zhichao Hou, Ferhat Erata, Joe Lilien +1LLM Fine-TuningConsistency Distillation

  9. Improving Argument Saliency Coverage in Small LLMs for Long Legal Opinion Summarization via Sequence-Level Distillation

    Aug 30, 2026Mohamed Elaraby, Ahmed Elhady, Diane LitmanText SummarizationLanguage Model Distillation

  10. Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation

    Aug 30, 2026Run Yang, Runpeng Dai, Jie Sun +5Language Model DistillationOn-Policy Distillation

  11. ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

    Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge DistillationCross-Architecture Knowledge Distillation

  12. MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation

    Aug 30, 2026Yangsong Lan, Renkai Hu, HongKai Zheng +4CoT DistillationLanguage Model Distillation

  13. Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed

    Aug 12, 2026Haokun Lin, Kaijie Zhu, Haobo Xu +4LLM QuantizationLLM Pruning

  14. ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation

    Aug 11, 2026Ximo Zhu, Ruiqi Liu, Rong Wang +8Language Model DistillationOn-Policy Distillation

  15. Mismatch Matters: On-Policy Distillation Beyond Token Agreement

    Aug 10, 2026Zichao Yu, Chengzhi Yu, Shengze Xu +4Distribution MatchingLanguage Model Distillation

  16. SR-OPSD: Self-Referenced On-Policy Self-Distillation

    Aug 10, 2026Zhuo Sun, Entong Li, Yanlong Zhao +9On-Policy Self-DistillationLanguage Model Distillation

  17. WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training

    Aug 10, 2026Zehao Chen, Gongxun Li, Tianxiang Ai +9Language Model DistillationOn-Policy Distillation

  18. Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

    Aug 10, 2026Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1On-Policy Self-DistillationLanguage Model Distillation

  19. Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast

    Aug 9, 2026Jiaxin Guo, Yanwei Yue, Xuanbo Fan +2Self-Training for Language ModelsLanguage Model Distillation

  20. Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

    Aug 8, 2026Yongkang Yang, Zhezheng Hao, Hong Zhang +8On-Policy Self-DistillationLanguage Model Distillation

  21. Adaptive Supervised Anchoring for On-Policy Self-Distillation

    Aug 8, 2026Meilin Yang, Zixuan Ding, Jianhao Nie +5Supervised Fine-TuningOn-Policy Self-Distillation

  22. RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

    Aug 6, 2026Xinye Wang, Junxiao Liu, Shujian HuangNumerical Reasoning in Language ModelsMultilingual Language Models

  23. Subliminal Learning is Non-Semantic Distillation

    Aug 6, 2026Ethan Hadley, Eren GultepeLLM AuditingSubliminal Learning