Self-Distillation Framework

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 433% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 70

All topics

The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
Oct 1, 2026Shuo Xing, Zilin Dai, Chengyuan Qian +7Mathematical Reasoning LLM Reasoning Strategies
Texas A&M University · Harvard University · Vanderbilt University +3
Adaptive Self-Consistency: From Black-Box Sampling to Distribution-Valued Feedback
Sep 30, 2026Jingkai Huang, Yunfan Zhang, Will Ma +2Self-Consistency Self-Distillation Framework
Stern School of Business New York University · School of Management Zhejiang University
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Sep 30, 2026Ilgee Hong, Changlong Yu, Zhenghao Xu +5Self-Distillation Framework Large Language Model Judges
Georgia Institute of Technology · Amazon · UC San Diego
Activation-Conditioned Self-Distillation
Sep 29, 2026Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati +2Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Rensselaer Polytechnic Institute · IBM Research
AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
Sep 29, 2026Rishabh Agrawal, Hejie Cui, Shasha Li +2Self-Distillation Framework
Google · University of Southern California
Beam Search as Test-Time Self-Distillation via Counterfactual Contexts
Sep 29, 2026Su Ee Tan, Xiaotong Ji, Rasul Tutunov +2Self-Distillation Framework Inference-Time
Huawei Noah’s Ark Lab · UCL Centre for AI
EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding
Sep 29, 2026Yuwei Miao, Xuesheng Zhang, Wenhao Zou +5Streaming Video Understanding Streaming
Baidu · University of Chinese Academy of Sciences · China University of Petroleum, Beijing
K-OPSD: Verifiable On-Policy Self-Distillation for Post-Training Vision-Language Models on AEC Drawings
Sep 28, 2026Yunfei Bai, Enrico Chionna, Akash Amol +2Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Amazon Web Services
Grounding Memory Summarization in Utility Intent
Sep 27, 2026Zhenyu Lei, Mingjia Shi, Xingbo Fu +3Conversational Memory Summarization
University of Virginia · Northeastern University
Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models
Sep 27, 2026Yadong Xi, Rongsheng Zhang, Tangjie Lv +2Confidence Calibration Large Reasoning Models
NetEase · Amazon · Singapore University of Technology and Design
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Sep 21, 2026Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4Self-Distillation Framework Self-Teacher
Imperial College London · Politecnico di Torino · UCL Centre for AI
PSD: Pseudo Self-Distillation of Memory Representation Capabilities for LLM Agents
Sep 20, 2026Pirzada Suhail, Menglin Xia, Xuchao Zhang +3Self-Distillation Framework Long-Term Agent Memory
Microsoft Research
Similarity Pairing with Energy Mover's Distance for Self-Supervised Pre-Training at the LHC
Sep 15, 2026Ho Fung Tsoi, Dylan RankinSelf-Supervised Learning Self-Distillation Framework
Department of Physics and Astronomy University of Pennsylvania, USA
Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models
Sep 12, 2026Rui Zhu, Minglong Cao, Chenyu Zhou +2Operations Research Self-Distillation Framework
Shanghai Jiao Tong University, Shanghai, China · Fudan University, Shanghai, China
Prototype Matters: Modality-unified Prototype Self-distillation for Unsupervised Visible-infrared Person Re-identification
Sep 11, 2026Menglin Wang, Xiaojin GongRe-Identification Cross-Modality
Nanjing Normal University Nanjing, China · Zhejiang University Hangzhou, China
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Sep 10, 2026Rongcan Pei, Zhepei Wei, Shuyao Xu +3Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Department of Computer Science, University of Virginia · Stanford University
Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics
Sep 1, 2026Samuel Young, César Jesús-Valls, Kazuhiro TeraoWireless Foundation Models Foundation Model
Stanford University · Stanford, CA 94305 · European Organization for Nuclear Research (CERN) +3
LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation
Aug 12, 2026Zhixin Zhang, Xinke Jiang, Zhibang Yang +5Long-Horizon Agents Agentic Reinforcement Learning
School of Computer Science, Peking University, Beijing, China · National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +2
Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models
Aug 10, 2026Xuanchen Li, Haitao Li, Yujia Zhou +5Self-Distillation Framework Selective
DCST, Tsinghua University · Tencent
On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3Unsupervised On-Policy Self-Distillation Self-Distillation Framework
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Aug 6, 2026Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10Agentic Reinforcement Learning Unsupervised On-Policy Self-Distillation
Tsinghua University · Meituan · Zhejiang University
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
Aug 5, 2026Yi Yang, Cong Qin, Xiaodan Liu +8Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Meituan LongCat Interaction · Nanjing University · Peking University +3
When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents
Aug 5, 2026Junzhuo Liu, Weiwei Li, Jun Ling +1Self-Teacher Self-Distillation Framework
University of Electronic Science and Technology of China
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Aug 3, 2026Haowei Liu, Jiamian Wang, Hsin-Tai Wu +2Hindsight Search Agents
Santa Clara University Santa Clara, CA · Rochester Institute of Technology Rochester, NY · Independent Researcher Sunnyvale, CA
EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents
Aug 2, 2026Jianan Xie, Xin Sun, Zhongqi Chen +4Self-Distillation Framework Search Agents
ShanghaiTech University · NLPR, MAIS, CASIA · Ant Group
TransMem: Transforming Hidden States into Memory for Large Language Models
Jul 31, 2026Haodong Lei, Junming Liu, Yirong Chen +4Large Language Model Memory Large Language Model Agents
Southeast University, Nanjing, Jiangsu, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
Group-Reflective Self-Distillation for Agentic Reinforcement Learning
Jul 30, 2026Binbin Zheng, Zijun Xie, Guanqun Zhao +4Agentic Reinforcement Learning Unsupervised On-Policy Self-Distillation
University of Science and Technology of China · 2Baidu Inc. · 3Peking University +3
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Jul 30, 2026Xingjian Wu, Junlin Liu, Xingchen Liu +6Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Meituan
From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
Jul 30, 2026Xu Xia, Jinghua Piao, Min Yang +3Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Southeast University · Zhongguancun Academy · Tsinghua University +2
Learning Semantic-Robust Change Detection via Semantic-Invariant Self-Distillation
Jul 21, 2026Jiuhe Qu, Yingping Liang, Ying FuChange Detection Self-Distillation Framework
Beijing Institute of Technology, Beijing, China
Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?
Jul 20, 2026Fan Yang, Rui Meng, Yuxin WenSelf-Distillation Framework Self-Teacher
Chapman University, Orange, CA, USA · Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Optimal Self-Distillation for Rectified Flow via Linear Probing
Jul 16, 2026Saptarshi Roy, Debepsita Mukherjee, Pratik PatilSelf-Distillation Framework Rectified Flow
University of Texas, Austin
Consensus as Privileged Context for Label-Free Self-Distillation
Jul 15, 2026John Gkountouras, Josip Jukić, Ivan TitovToken-Level Supervision Self-Distillation Framework
1ILLC, University of Amsterdam · 2ILCC, University of Edinburgh
Label-Free Target-Domain Adaptation for Unconstrained Event-Image Feature Matching via Dual-Stage Distillation
Jul 11, 2026Zhonghua Yi, Hao Shi, Qi Jiang +3Harder Better Faster Denser Feature Matching Cross-Modal Distillation
Zhejiang University · Ant Group · National University of Defense Technology +1
DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
Jul 8, 2026Xinyu Geng, Xuanhua He, Sixiang Chen +7Search Agents Self-Improving Agents
HKUST · Tencent · HKUST(GZ)
Geometric Self-Distillation for Reasoning Generalization
Jul 7, 2026Josip Jukić, Ivan TitovSelf-Distillation Framework Geogs-Slam
ILLC, University of Amsterdam · ILCC, University of Edinburgh
Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation
Jul 2, 2026Zhuowei Chen, Xiang Lorraine LiUnsupervised On-Policy Self-Distillation Self-Distillation Framework
University of Pittsburgh, United States
Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes
Jun 25, 2026Jeremias Ferrao, Niclas Müller-Hof, Iustin Sîrbu +2Large Language Model Safety Self-Distillation Framework
University of Groningen · University Politehnica of Bucharest · NVIDIA
ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models
Jun 24, 2026Kejing Wang, Toan Nguyen, Minh Hoang Nguyen +2Diffusion-Based Vision-Language-Actions Vision-Language Model Adaptation
Applied Artificial Intelligence Initiative Deakin University Geelong, Australia · Air Force Research Laboratory USA
Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models
Jun 18, 2026Durgesh Kalwar, Vardhan Palod, Jaya Adithya Pavuluri +1Self-Distillation Framework Large Reasoning Models
SCAI, Arizona State University
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
Jun 17, 2026Siyi Gu, Jialin Chen, Sophia Zhou +2Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Yale University
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
Jun 16, 2026Jingyuan Huang, Zuming Huang, Yucheng Shi +4Graphical User Interface Self-Distillation Framework
University of Georgia · 2INFLY Tech · *This work was done during an internship at INFLY Tech. +2
Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
Jun 10, 2026Tianyu Ding, Jianhong Xin, Juan Pablo De la Cruz WeinsteinBetter Call Group Relative Policy Optimization Policy Gradient
Amazon Web Services
Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers
Jun 10, 2026MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang +6Rubrics Reinforcement Learning With Verifiable Reward
Scale AI
The Role of Feedback Alignment in Self-Distillation
Jun 9, 2026Semih Kara, Oğuzhan ErsoySelf-Distillation Framework Self-Teacher
Gensyn
PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment
Jun 8, 2026Yang Tian, Rui Wang, Xumeng Wen +5Credit Assignment Reward-Only Policy Optimization Variants
School of AI, Shanghai Jiao Tong University · 2XYZ AI Lab
CoralBay: A Self-Supervised CT Foundation Model
Jun 2, 2026Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora +1Medical Imaging Volume
Echo-POSED: Geometric Self-Distillation for Echocardiography Guidance
May 30, 2026Elias Stenhede, Edvart Grüner Bjerke, Joanna Sulkowska +4Echocardiography Catheters
Medical Technology & E-Health, Akershus University Hospital, Norway · Department of Technology Systems, University of Oslo, Norway · K.G. Jebsen Center for Cardiac Biomarkers, University of Oslo, Norway +3
OISD: On-Policy Internal Self-Distillation of Language Models
May 27, 2026Xinyu Liu, Darryl Cherian Jacob, Yang Zhou +2Unsupervised On-Policy Self-Distillation Self-Distillation Framework
Auburn University · William & Mary
Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
May 26, 2026Zehao Liu, Yuanpu Cao, Jinghui Chen +1Unsupervised On-Policy Self-Distillation Self-Distillation Framework
College of Information Sciences and Technology Pennsylvania State University
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecMulti-Turn Large Language Model Conversations Conversational Context
Stanford
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
May 20, 2026Tony Lee, Percy LiangSelf-Distillation Framework Large Reasoning Models
Stanford University
AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
May 20, 2026Duy Nguyen, Hanqi Xiao, Archiki Prasad +7Self-Distillation Framework Teacher
UNC Chapel Hill · Capital One · The University of Texas at Austin
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
May 18, 2026Zhenlin Wei, Pu Jian, Yingzhuo Deng +6Self-Distillation Framework Credit Assignment
Meituan Beijing, China · University of Chinese Academy of Sciences
It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs
May 18, 2026Sangwoo Park, Woongyeong Yeo, Seanie Lee +6Privacy-Utility Trade-Offs Self-Distillation Framework
KAIST
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
May 18, 2026Woongyeng Yeo, Yumin Choi, Taekyung Ki +1Long-Horizon Agents Hindsight
KAIST
Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models
May 18, 2026Radu Lecoiu, Debarghya Mukherjee, Pragya SurCovariance Self-Distillation Framework
Department of Statistics, Harvard University · Department of Mathematics & Statistics, Boston University
Learning from Language Feedback via Variational Policy Distillation
May 14, 2026Yang Li, Erik Nijkamp, Semih Yavuz +1Unsupervised On-Policy Self-Distillation Online-Policy Distillation
Salesforce AI Research
Information theoretic underpinning of self-supervised learning by clustering
May 12, 2026Josef Kittler, Sara Atito, Muhammad AwaisSelf-Supervised Learning Kullback-Leibler Divergence
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, Surrey GU2 7XH, UK · Surrey Institute for People-Centred AI (PAI), University of Surrey, Guildford, Surrey GU2 7XH, UK
GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
May 12, 2026Sijia Li, Yuchen Huang, Zifan Liu +7Credit Assignment Self-Distillation Framework
Hong Kong University of Science and Technology · Microsoft Research Asia

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models

Shuo Xing, Zilin Dai, Chengyuan Qian, Fangzhou Lin+6

Texas A&M University · Harvard University · Vanderbilt University+3

Mathematical Reasoning LLM Reasoning Strategies Self-Distillation Framework

Open paper AskPDF arXiv

While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training. First, we introduce the notion of Mathematical Primitive to probe structural mathematical understanding and propose \hlei{}, a novel benchmark that evaluates mathematical reasoning along four distinct dimensions: Discovery, Generation, Digestion, and Execution. Second, our systematic diagnosis shows that solution accuracy masks distinct capability profiles, primitives unlock substantial latent execution capacity, and Discovery is the dominant bottleneck in mathematical reasoning. Our post-training analysis further shows that discovery-limited failures are particularly amenable to repair. Finally, building on these findings, we introduce \abs{}, a primitive-privileged self-distillation framework that selectively transfers primitive-guided reasoning into the student model. Extensive experiments demonstrate that \abs{} consistently improves mathematical reasoning over baselines across model scales and challenging benchmarks.

Momentum

Related topics

Latest papers 70