Knowledge Distillation

Ask about this topic

Momentum

17 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 196

All topics
CardsList

  1. Distillation of Tabular Foundation Models into Efficient Predictors

    Oct 1, 2026Minho Jeong, Dooho Lee, Jinmo Lee +1Tabular Foundation ModelsKnowledge Distillation

    Nums AI

  2. Distilling Directional Verification

    Oct 1, 2026Jungseob Lee, Sugyeong Eo, Seongtae Hong +4Knowledge DistillationFew-Step Distillation

    Korea University · Yonsei University Mirae Campus · Soongsil University +1

  3. MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models

    Sep 30, 2026Yanshu Li, Jiaqian Li, Canran Xiao +3Multimodal Large Language ModelsKnowledge Distillation

    Brown University · University of Alabama at Birmingham · Hanyang University

  4. Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

    Sep 30, 2026Qianfeng Yuan, Wenbing TaoKnowledge DistillationKullback-Leibler Divergence

    Huazhong University of Science and Technology

  5. K2P: Label-Free Knowledge to Prompt Distillation

    Sep 30, 2026Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1Knowledge DistillationTeacher

    Department of Statistics University of Georgia Athens, GA 30602

  6. Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses

    Sep 29, 2026Ziluowen Luo, Senzhang Wang, Chaozhuo Li +9Knowledge DistillationProduction Agentic Systems

    Central South University, ChangSha, Hunan, China · Beijing Academy of Artificial Intelligence, Beijing, China · Hong Kong Polytechnic University, Hongkong, China +2

  7. From UNI2-h to ConvNeXt-T: Lightweight Nuclei Instance Segmentation via Knowledge Distillation

    Sep 28, 2026Wenyan LiConvnextVision Transformer

    School of Computer Science, Wuhan University, Wuhan, China

  8. When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation

    Sep 28, 2026Minjae Park, Taesun Yeom, Jaeho LeeKnowledge DistillationTeacher

    Pohang University of Science and Technology (POSTECH)

  9. Distilling Visual Reasoning into Text Space

    Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1Recent Vision-Language ModelsMultimodal Reasoning Benchmarks

    University of California, Los Angeles · Cisco Research

  10. A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions

    Sep 27, 2026Luyang Fang, Haoran Lu, Jiazhang Cai +4Knowledge DistillationResidual Distillation

    Department of Statistics, University of Georgia, Athens, Georgia 30602, USA · Department of Biostatistics, Boston University, Boston, Massachusetts 02118, USA

  11. Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces

    Sep 22, 2026Minghui Liu, Thomas Magelinski, Dehao Yuan +2LLM Reasoning StrategiesThoughts

    University of Maryland, College Park · Capital One

  12. Robust Failure, Conservative Repair: Textual Knowledge Distillation from Cross-Model Failures

    Sep 21, 2026Andrew Ren, Haokun Liu, Chenhao TanKnowledge DistillationProbe-Logit Distillation

    University of Chicago

  13. Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability

    Sep 20, 2026Aleks Czufarow, Ihor BabinKnowledge DistillationInterpretability

    XIV High School of Stanislaw Staszic,Warsaw,Poland · Ukrainian Catholic University, Lviv, Ukraine

  14. Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement

    Sep 17, 2026Yitong Xing, Yuhao Cheng, Yanping Li +1Rt-DetrDetection Transformer

    School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China

  15. Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

    Sep 15, 2026Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro CavalcanteFederated LearningKnowledge Distillation

    Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden

  16. Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation

    Sep 15, 2026Vijay John, Amar DabajaAutomotive DetectionMultimodal Classification

    Department of Mathematics and Computer Science Lawrence Technological University, Southfield, MI, USA

  17. Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition

    Sep 14, 2026Yanjiang Shi, Peng Zhao, Nan Qi +1Knowledge DistillationStudent

    School of Computer Science and Technology, and with the National Engineering Laboratory for Big Data Analytics (NEL-BDA), Xi’an Jiaotong University, Xi’an 710049, China

  18. ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models

    Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Dataset DistillationKnowledge Distillation

    VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1

  19. Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

    Sep 1, 2026Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4Knowledge DistillationGossip

    University of California, Santa Cruz, USA · China University of Petroleum, China · The Chinese University of Hong Kong, Shenzhen, China +2

  20. Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

    Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Knowledge DistillationTeacher

    Meta AI · University of Washington · Princeton University

  21. ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

    Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Teacher-Student DistillationKnowledge Distillation

    1Harbin Institute of Technology · 2Peng Cheng Laboratory

  22. SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning

    Aug 27, 2026Zhuochun Li, Yuelyu Ji, Yiming Zeng +1Process Reward ModelKnowledge Distillation

    University of Pittsburgh, Pittsburgh, USA · University of Connecticut, Storrs, USA

  23. RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation

    Aug 12, 2026Rong Chao, Sung-Feng Huang, Moreno La Quatra +4Speech EnhancementMamba-Based Models

    Academia Sinica, Taiwan · National Taiwan University, Taiwan · NVIDIA +2

  24. Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed

    Aug 12, 2026Haokun Lin, Kaijie Zhu, Haobo Xu +4Small Large Language ModelsLarge Language Model Reliability

    Institute of Automation, CAS Beijing, China · Tsinghua University Beijing, China · Harvard Medical School Boston, U.S. +1

  25. Distillation of Foundation Models for Time-dependent PDEs

    Aug 12, 2026Daniel Musekamp, Boshra Ariguib, Andrei Manolache +1Autoregressive RolloutKnowledge Distillation

    University of Stuttgart · University of Cologne · Bitdefender

  26. Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer

    Aug 12, 2026Syed Muhammad Raza, Omer Tariq, Jeongbae SonKnowledge DistillationSymbolic Distillation

    Perception AI, Neubility Inc., Seoul, South Korea

  27. Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

    Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoVision TransformerKnowledge Distillation

    Hyundai Motor Company · School of Electronics and Avionics Engineering, Korea Aerospace University

  28. SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features

    Aug 11, 2026HyeonJun Lee, Hyeonsik Jo, Jinwoo Chung +1Quantization-Aware TrainingKnowledge Distillation

    Kookmin University

  29. UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

    Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Knowledge Distillation

    University of Electronic Science and Technology of China, Chengdu, China

  30. A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs

    Aug 8, 2026Abhinand Balachandran, Praveen PrashantKnowledge DistillationResnet-18

    Georgia Institute of Technology Atlanta, Georgia, USA

  31. STEMMA: An Adversarial Multi-Agent Framework for Evaluating Self-Identity Consistency in LLMs

    Aug 8, 2026Nuthakki Siva Gopala Krishna, Kanishka JainKnowledge DistillationAdversarial Prompts

    BML Munjal, Haryana, India · Indian Institute of Technology, Delhi, India

  32. NeuroGuard: Neural Gradient Update Aware of Representation Damage

    Aug 8, 2026Taigo Sakai, Kazuhito HottaClass-Incremental LearningBalanced Learning

    Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

  33. FedTransKD-IDS: Robust Federated Transfer Learning with Knowledge Distillation for Intrusion Detection in IoT

    Aug 6, 2026Mohammad Hosssein Gholamrezazadeh, Ahmadreza MontazerolghaemAhmadreza MontazerolghaemFederated LearningInternet Of Thing

    Faculty of Computer Engineering University of Isfahan Isfahan, Iran

  34. Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss

    Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Knowledge DistillationKullback-Leibler Divergence

    Multiverse Computing

  35. Lightweight 3D Object Detection via Mamba-Based Knowledge Distillation

    Aug 4, 2026Quoc Cuong Ninh, Huy Xuan Pham, Anh Tung Nguyen +13D Object DetectionVoxel

    Viettel AI, Viettel Group, Vietnam · Artificial Intelligence in Robotics Laboratory (AiR Lab), Department of Electrical and Computer Engineering, Aarhus University, 8000 Aarhus C, Denmark · VIST, Viettel Group, Vietnam

  36. Channel-wise Dynamic Knowledge Distillation via Adaptive Sample Generation for Action Recognition

    Aug 4, 2026Ping Li, Chenhao Ping, Jie Song +1Knowledge DistillationMotion

    School of Computer Science, Hangzhou Dianzi University, Hangzhou, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China

  37. Auditing Data Provenance in LLM Fine-tuning via Intrinsic Distributional Fingerprints

    Aug 3, 2026Zirui Huang, Yunlong Mao, Wei Tong +3Large Language Model Fine-TuningProvenance

    State Key Laboratory of Novel Software Technology, Nanjing University Nanjing, Jiangsu, China

  38. Towards Compact Unified Multimodal Tracking: Synergizing Knowledge Distillation with Structural Pruning

    Aug 2, 2026Yuqi Li, Yuedong Tan, Huiran Duan +7Cross-Modal DistillationMulti-Object Tracking

    The City College of New York, New York, NY, USA · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · Julius-Maximilians-Universität Würzburg, Würzburg, Germany +3

  39. Progressive2^2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression

    Jul 31, 2026Tiancong Cheng, Ying Zhang, Zhiwen Yu +2Teacher-Student DistillationKnowledge Distillation

    School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China

  40. CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation

    Jul 29, 2026Fengming Yu, Haiwei Pan, Kejia Zhang +3Knowledge DistillationTeacher

    College of Computer Science and Technology, Harbin Engineering University, Harbin, China

  41. From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search

    Jul 27, 2026Junlin Liu, Jiangwang Chen, Zixin Song +7Knowledge DistillationAgentic Search

    University of Chinese Academy of Sciences · 2Tsinghua University · 3Peking University +3

  42. SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

    Jul 25, 2026Aditya Dewan, Arjun Yogeswaran, Benjamin FedorukKnowledge DistillationHessian

    Carnegie Mellon University · Ontario Tech University

  43. When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

    Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Knowledge DistillationSummarization

    BRAC University, Dhaka, Bangladesh

  44. RF-Agent: A Practical Framework for Building Language Agents for RFIC Design

    Jul 21, 2026Yueqi Xing, Houbo He, Jolie Wang +5Table Reasoning DatasetsElectronic Design Automation

    ECE Department, Rice University, Houston, TX, USA · ECE Department, The George Washington University, Washington, D.C., USA

  45. Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection

    Jul 21, 2026Qiuli Zhou, Jingyuan Yao, Shengeng Tang +3Cross-Topic Arabic Stance DetectionMotion-Language Alignment

    School of Foreign Studies, Hefei University of Technology, China · School of Computer Science and Information Engineering, Hefei University of Technology, China · School of Computer Science and Information Engineering, Hefei University of Technology +1

  46. Efficient Audio-Visual Event Recognition via Knowledge Distillation and Dynamic INT8 Quantization of a Hybrid Cross-Attention Network

    Jul 18, 2026Parinaz Binandeh Dehaghani, Danilo Pena, A. Pedro AguiarDetection TransformerKnowledge Distillation

    University of Porto, Porto, Portugal · ResoSight, Montreal, Canada

  47. CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation

    Jul 18, 2026Satyam Kumar, Saurabh JhaEfficient On-Policy DistillationKnowledge Distillation

  48. On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures

    Jul 17, 2026Mohamed Amine KinaKnowledge DistillationGenerative Architectures

    Universität Bremen

  49. Symbiosis-Inspired Knowledge Distillation for Incremental Object Detection

    Jul 15, 2026Mingyue Zeng, De Cheng, Zhipeng Xu +3Class-Incremental LearningIntra-Class

    State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China

  50. MobileSAM2: Lightweight Segment Anything for Spatial Intelligence

    Jul 14, 2026Kai Jiang, Jiaxing Huang, Jingyi Zhang +5Knowledge DistillationMobilenetv2

    Hong Kong Polytechnic University, HK, China · Nanyang Technological University, Singapore · Xidian University, China +1

  51. Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

    Jul 13, 2026Xuyi Yu, Yaohua Liu, Ziming Song +3Knowledge DistillationMulti-Stage Training

    State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Taipa, Macau SAR, China +1

  52. LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models

    Jul 13, 2026Gangsu Kim, Won-Ki JeongPathology Foundation ModelsMedical Vision-Language Models

    Department of Computer Science and Engineering, College of Informatics, Korea University, Republic of Korea

  53. Knowledge Distillation for Automated AI Tutor Evaluation

    Jul 12, 2026Tahmid Al Hannan, Diego Garcia, Alex Njoroge +2TutorsPedagogical Frameworks

    Folsom Lake College Folsom, CA, USA

  54. BucketKD: A Safety-Aware Bucket-Based Knowledge Distillation Framework for End-to-End Motion Planning

    Jul 12, 2026Md Nahidul Islam, Mohd Hasan Ali, Dipankar Dasgupta +1Causality-Aware End-To-End Autonomous DrivingAutonomous Driving

    Department of Computer Science, University of Memphis, Memphis, TN, United States · Department of Electrical and Computer Engineering, University of Memphis, Memphis, TN, United States

  55. Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking

    Jul 11, 2026Shreeya Dasa Lakshminath, Shubhan SCross-EncoderAgentic Retrieval-Augmented Generation Systems

    Department of Computer Science and Engineering, B.M.S. College of Engineering, Bangalore, India

  56. CVKD-UDA: Cross-View Knowledge Distillation for 3D Unsupervised Domain Adaptive Segmentation

    Jul 11, 2026Zhimin Yuan, Ming Cheng, Shangshu Yu +4Interactive 3D SegmentationDomain Adaptation

    School of Artificial Intelligence, Nanyang Normal University, Nanyang 473061, China · Fujian Key Laboratory of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University, Xiamen 361005, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China

  57. ZipDepth: Bringing Lightweight Zero-Shot Monocular Depth Anywhere, on Any Device

    Jul 9, 2026Fabio Tosi, Luca Bartolomei, Matteo Poggi +1Monocular Depth EstimationMobilenetv2

    University of Bologna, Bologna, Italy

  58. SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation

    Jul 9, 2026Wangyu Wu, Xiaojian Lin, Rong Fu +4Text-To-SqlSmall Large Language Models

    The University of Liverpool · Microsoft · Tsinghua University +4

  59. Enhancing deep learning models for time series classification via knowledge distillation

    Jul 7, 2026Javidan Abdullayev, Maxime Devanne, Jonathan Weber +1Knowledge DistillationTime-Series Classification

    IRIMAS, University of Haute Alsace, Frères Lumière, Mulhouse 68200, France · Faculty of IT, Monash University, Wellington Road, Melbourne 3800, Australia

  60. Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

    Jul 7, 2026Cemil-Andrei Dilmac, Florinel-Alin Croitoru, Radu Tudor IonescuGeometry-Aware Uncertainty CoresetsKnowledge Distillation

    Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania