Knowledge Distillation

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

9 new papers

A weekly snapshot of new work published in Knowledge Distillation.

Period ending 2026-09-14

8 new papers

A weekly snapshot of new work published in Knowledge Distillation.

Period ending 2026-09-07

13 new papers

A weekly snapshot of new work published in Knowledge Distillation.

Inside this field

Focused directions

Focused topic

Knowledge Distillation

Within Knowledge Distillation

181 papersExplore

Focused topic

Chain-Of-Thought Distillation

Within Knowledge Distillation

113 papersExplore

Focused topic

Asymmetric Distillation

Within Knowledge Distillation

33 papersExplore

Focused topic

Multi-Teacher Distillation

Within Knowledge Distillation

31 papersExplore

Focused topic

Batch Distillation

Within Knowledge Distillation

30 papersExplore

Focused topic

Progressive Distillation

Within Knowledge Distillation

26 papersExplore

Focused topic

Prompt Distillation

Within Knowledge Distillation

24 papersExplore

Focused topic

Ontology-Amplified Distillation

Within Knowledge Distillation

22 papersExplore

418 papers

Latest in Knowledge Distillation

Browse all topics →

Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model
May 13, 2026Tianhao Qiu, Xiaojun ChenText-To-Sql Table Reasoning Datasets
Flow Augmentation and Knowledge Distillation for Lightweight Face Presentation Attack Detection
May 13, 2026Muhammad Shahid Jabbar, Muhammad Sohail Ibrahim, Taha Hasan Masood Siddique +2Optical Flow Knowledge Distillation
Lossless Anti-Distillation Sampling
May 12, 2026Zibo Diao, Jingchu Gai, Xinyue Ai +3Batch Distillation Generative Models
Fast Image Super-Resolution via Consistency Rectified Flow
May 12, 2026Jiaqi Xu, Wenbo Li, Haoze Sun +8Generative Super-Resolution Rectified Flow
STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation
May 12, 2026Shuhan Ye, Yi Yu, Qixin Zhang +5Spiking Neural Networks Knowledge Distillation
GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
May 12, 2026Sijia Li, Yuchen Huang, Zifan Liu +7Credit Assignment Group Relative Policy Optimization
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
May 12, 2026Guobin Shen, Xiang Cheng, Chenxiao Zhao +4Unsupervised On-Policy Self-Distillation Batch Distillation
A Study on Hidden Layer Distillation for Large Language Model Pre-Training
May 12, 2026Maxime Guigon, Lucas Dixon, Michaël E. SanderLarge Language Model Pretraining Knowledge Distillation
Uncovering Latent Pathological Signatures in Pulmonary CT via Cross-Window Knowledge Distillation
May 12, 2026Bo Peng, Wujian Xu, Kun Wang +8Computed Tomography Dataset Multimodal Clinical Data
Generative Diffusion Prior Distillation for Long-Context Knowledge Transfer
May 12, 2026Nilushika Udayangani, Kishor Nandakishor, Marimuthu PalaniswamiKnowledge Distillation Cross-Task Knowledge Transfer
Curriculum Learning-Guided Progressive Distillation in Large Language Models
May 11, 2026Jincheng Cao, Fanzhi Zeng, Leqi Liu +1Knowledge Distillation Curriculum
AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling
May 11, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Neural Audio Codecs Emotion Recognition
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
May 11, 2026Keitaro Sakamoto, Pierre Ablin, Federico Danieli +1Model-Agnostic Defense Model Weights
Step Rejection Fine-Tuning: A Practical Distillation Recipe
May 11, 2026Igor Slinko, Ilia Zavidnyi, Egor Bogomolov +1Swe-Bench Verified Progressive Distillation
LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
May 11, 2026Nikolaos Gkalelis, Vasileios MezarisKnowledge Distillation Large Vision Language Models
PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillation
May 11, 2026Ahmet Onur Akman, Rafał KucharskiMulti-Agent Reinforcement Learning Multi-Robot Systems
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
May 11, 2026Xuewen Zhang, Haixiao Zhang, Xinlong HuangKnowledge Distillation Neural Machine Translation Models
Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT
May 10, 2026Alaa Asfour, Christopher Indris, Leihan Chen +23D Spatial Reasoning Spatial Reasoning
Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasSpeculative Decoding Self-Speculative
Probability-Flow Distillation: Exact Wasserstein Gradient Flow for High-Fidelity 3D Generation
May 9, 2026Rohith Ramanan, A. N. Rajagopalan3D Generative Models Diffusion Priors
CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
May 9, 2026Soo Min Kwon, Ziteng Sun, Ananda Theertha Suresh +2Group Relative Policy Optimization Flow-Grpo
Reasoning Compression with Mixed-Policy Distillation
May 9, 2026Han Yang, Mingyan Wu, Bailan He +4Large Language Model Reasoning On-Policy Distillation
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Large Language Model Compression Large Language Model Pretraining
Multi-Level Graph Attention Network Contrastive Learning for Knowledge-Aware Recommendation
May 8, 2026Zhifei Hu, Feng XiaKnowledge Graph Embeddings Graph Neural Networks
Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +3Conditional Flow Matching Few-Step Generation
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
May 8, 2026Boyu Shi, YiCheng Jiang, Chang Liu +3Small Language Models Knowledge Distillation
Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models
May 8, 2026Bincheng Peng, Guang Li, Ping Liu +2Chain-Of-Thought Distillation Imagenet
Structural Rationale Distillation via Reasoning Space Compression
May 8, 2026Jialin Yang, Jiankun Wang, Jiajun Wu +3Reasoning Paths Chain-Of-Thought Distillation
Physical Fidelity Reconstruction via Improved Consistency-Distilled Flow Matching for Dynamical Systems
May 7, 2026Sicheng Ma, Tianyue Yang, Xiuzhe Wu +1High-Fidelity Reconstruction Quality
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
May 7, 2026Miao Rang, Zhenni Bi, Hang Zhou +6On-Policy Distillation Chain-Of-Thought Distillation
TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
May 7, 2026Bibin WilsonSelf-Supervised Chain-Of-Thought Distillation
Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
May 7, 2026Huizi Cui, Huan Ma, Qilin Wang +2Large Language Model Uncertainty Reliable Uncertainty Quantification
Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learning
May 6, 2026Michael Lanier, Luise Ge, Sastry Kompella +1Asymmetric Distillation Approximate Nearest-Neighbor Search
Deep Reprogramming Distillation for Medical Foundation Models
May 6, 2026Siyuan Du, Yuhang Zhou, Haolin Li +5Knowledge Distillation Wireless Foundation Models
Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUs
May 5, 2026Faraz Kayani, Sarmad Kayani, Asad Ahmed +2Neural Processing Units Mobilenetv2
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
May 5, 2026Wenjin Hou, Shangpin Peng, Weinong Wang +13On-Policy Distillation Multi-Teacher Distillation
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
May 5, 2026Qingzhuo Wang, Ruiyang Qin, Zhenxin Qin +2Knowledge Distillation Sparsity
Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning
May 5, 2026Taigo Sakai, Kazuhiro HottaDomain Incremental Learning Continual Learning Methods
Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability
May 4, 2026Yash Aggarwal, Atmika Gorti, Vinija Jain +3Large Language Model Bias Moral Reasoning
Memory-Efficient EDA Denoising via Knowledge Distillation for Wearable IoT Under Severe Motion Artifacts and Underwater Conditions
May 4, 2026Yongbin Lee, Andrew Peitzsch, Youngsun Kong +4Wearable Devices Surface Electromyography
Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection
May 4, 2026Mohamad Khajezade, Fatemeh H. Fard, Mohamed Sami ShehataKnowledge Distillation Deepseek
Exploring Data-Free LoRA Transferability for Video Diffusion Models
May 3, 2026Yuchen Wang, Wenliang Zhong, Lichen Bai +6Pre-Trained Video Diffusion Models Low-Rank Adaptation
Multilingual Safety Alignment via Self-Distillation
May 3, 2026Ruiyang Qin, Qingzhuo Wang, Dongrui Liu +3Multilingual Safety Large Language Model Adaptation
Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models
May 3, 2026Nikolaos Giarelis, Charalampos Mastrokostas, Nikos KaracapilidisMultilingual Large Language Models Large Language
EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer
May 3, 2026Hao Zhang, Zhibin Zhang, Guangxin Wu +3Knowledge Distillation Token-Level Uncertainty
Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detection
May 2, 2026Ceausescu Ciprian-Mihai, Anghelina Ion-Marian, Alexe Dumitru-BogdanMedical Imaging Datasets Medical Image Segmentation
Dynamics Distillation for Efficient and Transferable Control Learning
May 2, 2026Xunjiang Gu, Kashyap Chitta, Mahsa Golchoubian +2Autonomous Driving Simulation Nvidia Isaac Sim
LIE: LiDAR-only HD Map Construction with Intensity Enhancement via Online Knowledge Distillation
May 2, 2026Kanak Mazumder, Fabian B. FlohrLane-Level Map Generation Lidar
MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
May 2, 2026Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen +3Knowledge Distillation Latent Representation Alignment
SRA: Span Representation Alignment for Large Language Model Distillation
May 2, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4Knowledge Distillation Center
LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
May 1, 2026Shashank Kapadia, Deep Naryan Mishra, Sujal Reddy Alugubelli +4Chain-Of-Thought Distillation Model Pretraining
AsymTalker: Identity-Consistent Long-Term Talking Head Generation via Asymmetric Distillation
May 1, 2026Yuxin Lu, Jiayang Sun, Guibo Zhu +1Joint Audio-Video Generation Talk
Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration
May 1, 2026Luru Jing, Cong Cong, Yanyuan Chen +1Federated Learning Whole-Slide Images
Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation
May 1, 2026Kuan-Po Huang, Bo-Ru Lu, Byeonggeun Kim +8Audio Latent Zero-Shot Text-To-Speech
Investigating LLM's Problem Solving Capability -- a Study on Statics Questions
Apr 30, 2026Tanner Culleton, Hung-Fu ChangLarge Language Model Performance Engineering
Fair Dataset Distillation via Cross-Group Barycenter Alignment
Apr 30, 2026Mohammad Hossein Moslemi, Nima Hosseini Dashtbayaz, Zhimin Mei +2Fairness-Aware Learning Chain-Of-Thought Distillation
Energy-Efficient Plant Monitoring via Knowledge Distillation
Apr 29, 2026Ilyass Moummad, Reda Bensaid, Kawtar Zaher +5Tomato Diseases Knowledge Distillation
Edge AI for Automotive Vulnerable Road User Safety: Deployable Detection via Knowledge Distillation
Apr 29, 2026Akshay Karjol, Darrin M. HannaYolov8-Nano Model Heterogeneous Edge Devices
GaitKD: A Universal Decoupled Distillation Framework for Efficient Gait Recognition
Apr 29, 2026Yuqi Li, Qian Zhou, Huiran Duan +5Gait Recognition Knowledge Distillation
Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models
Apr 28, 2026Pengcheng Fang, Hongli Chen, Xiaohao CaiForesight World Models

May 13, 2026 · cs.CLJ/K move · Enter open · S save

Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model

Tianhao Qiu, Xiaojun Chen

Shenzhen University Shenzhen, China

Text-to-SQL converts natural language questions into executable SQL queries, enabling non-technical users to access relational databases for analytics and intelligent data services. In real-world scenarios, performance is often constrained by low-resource settings, where high-quality annotated \texttt{<question, SQL>} pairs are scarce, particularly for domain-specific databases. Additional challenges include opaque schema definitions, abbreviations, and implicit business logic that are not explicitly encoded in the schema. Existing data synthesis and prompting techniques improve coverage but often fail to produce task-specific, semantically grounded examples aligned with database constraints. To address these challenges, we propose a knowledge-aware Text-to-SQL framework that constructs task-specific knowledge base including schema semantics, abbreviations, business logic, and query patterns, and injects them into both training and inference. This framework generates diverse, contextually grounded synthetic training data and enhances inference through targeted knowledge retrieval. Experiments on seven benchmarks, covering both general and domain-specific datasets, demonstrate that our approach substantially improves the performance of open-source and closed-source large language models in Text-to-SQL tasks, especially in low-resource domain-specific settings, enhancing generalization, robustness, and adaptability.

Text-To-Sql Table Reasoning Datasets Natural Language Queries Large Language Model Baselines Knowledge Distillation

Open paper Ask about it PDF