Residual Distillation

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

Smaller Models, Better Rejects: Preference Distillation Scaling
Sep 30, 2026Rui Cai, Wenhui Zhu, Xiwen Chen +15Small Reasoning Model Residual Distillation
AI Agentic Modeling and Foundation Team, LinkedIn · University of California, Davis · Arizona State University +3
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Sep 28, 2026Xinke Jiang, Tao Feng, Zhibang Yang +4Residual Distillation
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +1
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Kullback-Leibler Divergence Residual Distillation
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +3
A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
Sep 27, 2026Luyang Fang, Haoran Lu, Jiazhang Cai +4Knowledge Distillation Residual Distillation
Department of Statistics, University of Georgia, Athens, Georgia 30602, USA · Department of Biostatistics, Boston University, Boston, Massachusetts 02118, USA
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Sep 7, 2026Xiao Ma, Hong Shen, Hui Tian +2Byzantine Attacks Residual Distillation
Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR, China · School of Engineering and Technology, Central Queensland University, Australia · School of Information and Communication Technology, Griffith University, Australia
Matryoshka Language Model Suites
Aug 10, 2026Nathan Godey, Yoav ArtziLanguage Modeling Matryoshka Representation Learning
Cornell University
When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills
Aug 4, 2026Yongli Xiang, Zhifang Zhang, Bojun Yang +4Persona Consistency Privacy
1Sydney AI Centre, The University of Sydney · University of Queensland · 3Southeast University +1
AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning
Aug 4, 2026Shengyang Li, Yiting Dong, Liuyang Song +5Spiking Neural Networks Federated Learning
School of Software and Microelectronics, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China
Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving
Jul 31, 2026Zeyu Dong, Yimin Zhu, Yu Wu +1Autonomous Driving Waypoints
Stony Brook University, USA · Rutgers University, USA · Sunrise Technology Inc., USA
LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation
Jul 29, 2026Rongxiang Zhang, Songhua LiuPrecise Lip Synchronization Talk
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology
Geometric Distillation from Rectified Stereo: Leveraging Epipolar Cues for Monocular Depth
Jul 17, 2026Jung-Hee Kim, Xiaoming LiuMonocular Depth Estimation Scale-Ambiguous Epipolar Geometry
Michigan State University, East Lansing, MI 48824, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC 27514, USA
Domain-Incremental Remote Sensing Change Detection via Difference-Guided Adaptation and Frequency-Decoupled Distillation
Jul 14, 2026Daifeng Peng, Yaning Li, Haiyan GuanChange Detection Class-Incremental Learning
School of Remote Sensing and Geomatics Engineering, Nanjing University of Information Science and Technology, Nanjing 210044, China
Understanding Layer Patching in Model Size Interpolation
Jul 9, 2026Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak +3Model Size Layer-Wise
1Harvard University · 2Kempner Institute · 3IST Austria
Building Multi-Task Agentic LLMs via Two-Phase Distillation
Jun 29, 2026Huaijie Wang, Shusheng Xu, Yi Wu +1Residual Distillation Artificial Intelligence Systems
Tsinghua University · Ant Group
From Sparse and Imperfect 2D Anchors to Consistent 3D Gaussian Street Scenes: Support-Aware Appearance
Jun 24, 2026Long Cao, Zhongquan Wang, Jie Li +43D Gaussian Appearance
College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing 400044, China
Channel Location Constrains the Auditability of Subliminal Learning
Jun 20, 2026Tamas MadlChannel Residual Distillation
Austrian Research Institute for Artificial Intelligence · University of Vienna
Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
Jun 5, 2026Debjyoti Saha Roy, Byron C. Wallace, Javed A. AslamMulti-Label Classification Zero-Shot
Khoury College of Computer Sciences, Northeastern University
PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization
Jun 2, 2026Ying Tang, Dong Li, Youjia Zhang +3Vision Foundation Models Prism
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
Logit Distillation on Manifolds: Mapping by Learning
May 30, 2026Yiru Yang, Junling Wang, Nishant Kumar Singh +2Residual Distillation Teacher
University of Zurich · ETH Zurich · Deutsche Bank Securities
GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering
May 28, 2026Xin Sun, Jianan Xie, Zhongqi Chen +6Agentic Reinforcement Learning Efficient On-Policy Distillation
University of Science and Technology of China · 2NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · 3ShanghaiTech University +1
Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
May 28, 2026Junlin He, Yihong Tang, Tong Nie +5Efficient Large Language Model Symbolic Distillation
The Hong Kong Polytechnic University, Hong Kong SAR, China · McGill University, Montreal, QC, Canada
One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration
May 20, 2026Chaoyang Wang, Yunhai TongDiffusion Models Probe-Logit Distillation
Peking University
Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimization
May 19, 2026Thien Le, Melanie WeberResidual Distillation Combinatorial Optimization
SEAS, Harvard University · Cambridge, MA
Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation
May 14, 2026Min Zhao, Hongzhou Zhu, Kaiwen Zheng +6Video Generation Autoregressive Model
Tsinghua University · ShengShu · Renmin University of China
Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines
May 13, 2026Katherine Lambert, Sasha LuccioniEnergy Consumption Residual Distillation
University of Toronto · 2Sustainable AI Group
Curriculum Learning-Guided Progressive Distillation in Large Language Models
May 11, 2026Jincheng Cao, Fanzhi Zeng, Leqi Liu +1Knowledge Distillation Large Language Model Training
The University of Texas at Austin · Google Research
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
May 11, 2026Keitaro Sakamoto, Pierre Ablin, Federico Danieli +1Model-Agnostic Defense Model Weights
Apple
Step Rejection Fine-Tuning: A Practical Distillation Recipe
May 11, 2026Igor Slinko, Ilia Zavidnyi, Egor Bogomolov +1Swe-Bench Verified Supervised Finetuning
JetBrains Research
CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
May 9, 2026Soo Min Kwon, Ziteng Sun, Ananda Theertha Suresh +2Flow-Grpo Knowledge Distillation
University of Michigan, Ann Arbor · *Work done during an internship at Google · Google Research +1
Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUs
May 5, 2026Faraz Kayani, Sarmad Kayani, Asad Ahmed +2Neural Processing Units Mobilenetv2
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Dynamics Distillation for Efficient and Transferable Control Learning
May 2, 2026Xunjiang Gu, Kashyap Chitta, Mahsa Golchoubian +2Autonomous Driving Simulation Autonomous Driving
University of Toronto · Vector Institute · NVIDIA +1
LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
May 1, 2026Shashank Kapadia, Deep Naryan Mishra, Sujal Reddy Alugubelli +4Transformer Architectures Residual Distillation
Walmart Inc. · Sam’s Club
AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generation
Apr 29, 2026Xu Wang, Zexian Li, Litong Gong +2Few-Step Distillation Few-Step Generation
Shanghai Jiao Tong University · Alimama Tech
Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
Apr 20, 2026Xiao Lingao, Yang HeDiffusion-Based Dataset Distillation Imagenet
CFAR, Agency for Science, Technology and Research, Singapore 138632 · IHPC, Agency for Science, Technology and Research, Singapore 138632 · National University of Singapore, Singapore 119077
TurboTalk: Progressive Distillation for One-Step Audio-Driven Talking Avatar Generation
Apr 16, 2026Xiangyu Liu, Feng Gao, Xiaomei Zhang +4Audio-Video Generation Dataset Distillation
1MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Meituan +2

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Smaller Models, Better Rejects: Preference Distillation Scaling

Rui Cai, Wenhui Zhu, Xiwen Chen, Jincheng Cao+14

AI Agentic Modeling and Foundation Team, LinkedIn · University of California, Davis · Arizona State University+3

Small Reasoning Model Residual Distillation

Open paper AskPDF arXiv

Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come from a model at least as large as the student, making generation costly at scale. We find neither assumption holds: across students from 7B to 72B, smaller frozen models generate rejects with less inference compute yet train stronger students than self-generated rejects, before and after sequence-level knowledge distillation, on code generation and mathematical reasoning. To explain this result, we derive a finite-horizon utility bound for Direct Preference Optimization in a linearized feature model. The bound characterizes favorable reject distributions and motivates three interventions. First, mixing rejects from smaller and student-scale models improves performance as the smaller model's share increases. Second, reassigning rejects to other prompts and shuffling their code tokens still outperform length-matched gibberish, showing that task structure contributes to reject utility. Third, selecting candidates with lower likelihood under the reference policy improves net transfer when higher-likelihood candidates provide less useful contrast. Lower-likelihood selections outperform higher-likelihood ones for every source. These results suggest that effective rejects preserve task structure while limiting coupling to the reference policy, and that smaller frozen models can provide them at low cost.

Momentum

Related topics

Latest papers 35