Knowledge Distillation

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

9 new papers

A weekly snapshot of new work published in Knowledge Distillation.

Period ending 2026-09-14

8 new papers

A weekly snapshot of new work published in Knowledge Distillation.

Period ending 2026-09-07

13 new papers

A weekly snapshot of new work published in Knowledge Distillation.

Inside this field

Focused directions

Focused topic

Knowledge Distillation

Within Knowledge Distillation

181 papersExplore

Focused topic

Chain-Of-Thought Distillation

Within Knowledge Distillation

113 papersExplore

Focused topic

Asymmetric Distillation

Within Knowledge Distillation

33 papersExplore

Focused topic

Multi-Teacher Distillation

Within Knowledge Distillation

31 papersExplore

Focused topic

Batch Distillation

Within Knowledge Distillation

30 papersExplore

Focused topic

Progressive Distillation

Within Knowledge Distillation

26 papersExplore

Focused topic

Prompt Distillation

Within Knowledge Distillation

24 papersExplore

Focused topic

Ontology-Amplified Distillation

Within Knowledge Distillation

22 papersExplore

418 papers

Latest in Knowledge Distillation

Browse all topics →

GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering
May 28, 2026Xin Sun, Jianan Xie, Zhongqi Chen +6Agentic Reinforcement-Learning Framework Deep Reinforcement Learning
TRACER: Persistent Regularization for Robust Multimodal Finetuning
May 28, 2026Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1Chain-Of-Thought Distillation Multimodal Pretraining
Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
May 28, 2026Junlin He, Yihong Tang, Tong Nie +5Chain-Of-Thought Distillation Large Language Model Reasoning
LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
May 28, 2026Shali Jiang, Hua Zheng, Boyang Liu +40Knowledge Distillation Wireless Foundation Models
Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility
May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +1Table Reasoning Datasets Asymmetric Distillation
Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation
May 27, 2026Yang Gao, Wuyang Li, Po-Chien Luan +13D Semantic Occupancy Prediction Ego-Motion
Entropy-aware Masking for Masked Language Modeling
May 27, 2026Gokul Srinivasagan, Kai Hartung, Munir GeorgesMasked Diffusion Language Models Knowledge Distillation
From Fact Overwriting to Knowledge Evolution: Causal Editing via On-Policy Self-Distillation
May 27, 2026Shuaike Li, Kai Zhang, Xianquan Wang +2Multimodal Knowledge Editing Knowledge Distillation
Pruning and Distilling Mixture-of-Experts into Dense Language Models
May 27, 2026Junhyuck Kim, Jihun Yun, Haechan Kim +3Mixture-Of-Expert Language Models Mixture-of-Experts Models
Context Distillation as Latent Memory Management
May 27, 2026Ziyang Zheng, Zeju Li, Xiangyu Wen +5Chain-Of-Thought Distillation LLM Inference Efficiency
Dual-branch Distilled Transformer for Efficient Asymmetric UAV Tracking
May 27, 2026Hongtao Yang, Bineng Zhong, Qihua Liang +4Unmanned Aerial Vehicle Tracking Multi-Object Tracking
Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
May 27, 2026Luyang Fang, Yongkai Chen, Jiazhang Cai +2Multi-Teacher Distillation Knowledge Distillation
Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
May 26, 2026Zehao Liu, Yuanpu Cao, Jinghui Chen +1Unsupervised On-Policy Self-Distillation Batch Distillation
Gradient Transformer: Learning to Generate Updates for LLMs
May 26, 2026Binh-Nguyen Nguyen, Khang Tran, NhatHai Phan +1Large Language Model Fine-Tuning Knowledge Distillation
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
May 26, 2026Ning Wu, Rui Liu, Xinkun Lin +5Medical Report Generation Token-Level Supervision
Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
May 26, 2026Junkai Chen, Yuhao He, Junxiang You +3Multi-Modal Large Language Models Chain-Of-Thought Distillation
OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
May 26, 2026Yunze Liu, Chi-Hao Wu, Enmin Zhou +1Multimodal Retrieval Cross-Modal Learning
O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding
May 26, 2026Peiran Wu, Yunze Liu, Chi-Hao Wu +2Audio-Visual Reasoning Video Understanding
The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
May 25, 2026Guanghui Wang, Kaiwen Lv Kacuila, Zhiyong Yang +5Knowledge Distillation Soft-Label
Llamion Technical Report
May 25, 2026Kisu Yang, Yoonna Jang, Hyeonseok Moon +6Lizad Multi-Teacher Distillation
Strong Teacher Not Needed? On Distillation in LLM Pretraining
May 22, 2026Taiming Lu, Zhuang LiuChain-Of-Thought Distillation Teacher-Student Log-Probability Gap
TaxDistill: Improving Metagenomic Taxonomic Annotation via Distilled Genomic Foundation Models
May 22, 2026Rongye Ye, Lun Li, Zheng Luo +2Bacterial Colony Counting Ontology-Amplified Distillation
DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA
May 21, 2026Jianing Yin, Tan TangMemdefrag Chain-Of-Thought Distillation
TextTeacher: What Can Language Teach About Images?
May 21, 2026Tobias Christian Nauen, Stanislav Frolov, Brian Bernhard Moser +3Imagenet Knowledge Distillation
Safeguarding Text-to-Image Generative Models Against Unauthorized Knowledge Distillation
May 21, 2026Yilan Gao, Sida Huang, Hongyuan Zhang +1Adversarial Examples Text-To-Image
Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
May 21, 2026Hongsin Lee, Hye Won ChungMulti-Teacher Distillation Chain-Of-Thought Distillation
Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning
May 20, 2026Emanuel Sommer, David RügamerBayesian Neural Networks Posterior
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Multi-Teacher Distillation Teacher
GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation
May 20, 2026Sixiang Chen, Zhaohu Xing, Tian Ye +7Image Generation Self-Evolution
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
May 20, 2026Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon +3Federated Learning Knowledge Distillation
GSA-YOLO: A High-Efficiency Framework via Structured Sparsity and Adaptive Knowledge Distillation for Real-Time X-ray Security Inspection
May 20, 2026Jiahao KongYolo26 Sparsity
Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?
May 20, 2026Muquan Li, Yingyi Ma, Yihong Huang +5Adversarial Transferability Chain-Of-Thought Distillation
Consistently Informative Soft-Label Temperature for Knowledge Distillation
May 19, 2026Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1Knowledge Distillation
Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimization
May 19, 2026Thien Le, Melanie WeberProgressive Distillation Combinatorial Optimization
LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models
May 19, 2026Hyunsoo Han, Sangyeop Yeo, Jaejun YooKnowledge Distillation Discrete Diffusion Model
Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
May 19, 2026Mahdi Naser MoghadasiRandom Forest Knowledge Distillation
Distilling Tabular Foundation Models for Structured Health Data
May 18, 2026Aditya Tanna, Nassim Bouarour, Mohamed Bouadi +2Tabular Foundation Models Electronic Health Record Foundation Models
UTOPYA: A Multimodal Deep Learning Framework for Physics-Informed Anomaly Detection and Time-Series Prediction
May 18, 2026Robson W. S. Pessoa, Julien Amblard, Alessandra Russo +1Anomaly Detection Multivariate Forecasting
SAS: Semantic-aware Sampling for Generative Dataset Distillation
May 18, 2026Mingzhuo Li, Guang Li, Linfeng Ye +4Chain-Of-Thought Distillation Separability
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
May 18, 2026Woongyeng Yeo, Yumin Choi, Taekyung Ki +1Long-Horizon Agents Hindsight
Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization
May 18, 2026Anh B. H. Nguyen, Ba Tho Phan, Viet Cuong TaKnowledge Distillation Balanced Learning
Agentic Cost-Aware Query Planning with Knowledge Distillation for Big Data Analytics
May 18, 2026Mahdi Naser-MoghadasiData Lakes Cost-Aware
Sometin Beta Pass Notin (SBPN): Improving Multilingual ASR for Nigerian Languages via Knowledge Distillation
May 18, 2026Sewade OgunMultilingual Automatic Speech Recognition Word Error Rate
SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation
May 17, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedTraining-Free Chain-Of-Thought Distillation
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
May 16, 2026Yihao Liang, Niraj K. JhaChain-Of-Thought Distillation Large Vision Language Models
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
May 16, 2026Jiarui Liu, Lechen Zhang, Yongjin Yang +5Knowledge Distillation Supervised Fine-Tuning
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation
May 16, 2026Anhao Zhao, Haoran Xin, Yingqi Fan +3On-Policy Distillation Knowledge Distillation
Asking Back: Interaction-Layer Antidistillation Watermarks
May 15, 2026Guang Yang, Amir Ghasemian, Fengchen Liu +3Language Model Evasion Attacks Watermarks
How to Choose Your Teacher for Fine Grained Image Recognition
May 15, 2026Oswin Gosal, Edwin Arkel Rios, Augusto Christian Surya +3Knowledge Distillation Fine-Grained Perception
DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation
May 15, 2026Jaehun Jung, Hyunwoo Kim, Brandon Cui +4Chain-Of-Thought Distillation On-Policy Distillation
Learning from Language Feedback via Variational Policy Distillation
May 14, 2026Yang Li, Erik Nijkamp, Semih Yavuz +1Unsupervised On-Policy Self-Distillation On-Policy Distillation
BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring
May 14, 2026Zixuan Shu, Tiancheng Cao, Hen-Wei HuangFederated Learning Knowledge Distillation
Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions
May 14, 2026Qirui Liu, Hao Chen, Weijie Shi +2Student Misconceptions Knowledge Distillation
TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability
May 14, 2026Krish Sharma, Omar Naim, Soumadeep Saha +3Out-Of-Distribution Chain-Of-Thought Distillation
Learning with Semantic Priors: Stabilizing Point-Supervised Infrared Small Target Detection via Hierarchical Knowledge Distillation
May 14, 2026Yuanhang Yao, Ping Qian, Zhu Liu +2Infrared Small Target Detection Detection Framework
Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
May 14, 2026Kai Sun, Peibo Duan, Yongsheng Huang +4Spiking Neural Networks Knowledge Distillation
Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines
May 13, 2026Katherine Lambert, Sasha LuccioniAsymmetric Distillation Progressive Distillation
STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes
May 13, 2026Chenjun Xu, Zhennan Zhou, Zhan Su +3Deepseek On-Policy
On the Generalization of Knowledge Distillation: An Information-Theoretic View
May 13, 2026Bingying Li, Haiyun HeKnowledge Distillation Ontology-Amplified Distillation
GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
May 13, 2026Kasidit Sermsri, Teerapong PanboonyuenProgressive Distillation

May 28, 2026 · cs.CLJ/K move · Enter open · S save

GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering

Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu+5

University of Science and Technology of China · 2NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · 3ShanghaiTech University · 4Ant Group

Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose token distribution is distilled back to the ordinary student policy over generated action-token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.

Agentic Reinforcement-Learning Framework Deep Reinforcement Learning On-Policy Distillation Asymmetric Distillation

Open paper Ask about it PDF