Context Distillation

Follow this topic Ask about this topic

Context distillation studies how models can absorb useful information from privileged prompts, interaction histories, or teacher feedback instead of repeatedly receiving it at inference time. Common approaches include on-policy self-distillation with KL losses, trajectory-based experience learning, and LoRA adapters or learned tokens that store and retrieve compressed memories. Researchers evaluate these methods in reasoning, coding and tool-use agents, long-context language modeling, and multimodal tasks, using benchmarks such as AIME, SWE-bench, and ALFWorld. The central challenge is balancing effective transfer and lower context costs against forgetting, distorted reasoning, or loss of details needed for future actions.

Latest papers

33 papers12 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Self-Distillation for Supervised Fine-Tuning
Oct 18, 2026Wenxuan Zhu, Bang An, Shuming Liu +6Supervised Fine-Tuning Language Model Distillation
King Abdullah University of Science and Technology · Shanghai Jiao Tong University
CD-OPD: Contextual Digest for On-Policy Distillation
Oct 18, 2026Mingyuan Li, Yucheng Li, Xiaxia Wang +1On-Policy Distillation Knowledge Distillation
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Department of Computer Science, University of Oxford
OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Oct 15, 2026Yuechen Luo, Zhiyi Lai, Miao Ziqi +9AI Coding Agents LLM Agent Training
Tsinghua University · Independent Researcher · Shanghai Artificial Intelligence Laboratory +1
Learning from Interaction Experience via Localized Context Distillation
Oct 15, 2026Zhezheng Hao, Yongkang Yang, Xilin Xia +9Context Distillation Continual Learning for LLMs
Zhejiang University · Workbuddy Team, Tencent · The Chinese University of Hong Kong +1
PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining
Oct 14, 2026Zhenhe Wu, Jian Yang, Yaping Jin +3Language Model Pretraining Language Model Distillation
Huawei Technologies · Beihang University
Privileged Context as Drift in On-Policy Self-Distillation
Oct 6, 2026Ravenor Davion, Nick RuiOn-Policy Self-Distillation Context Distillation
Stanford University
Internalizing Agent Experience into Diffusion Model Weights via On-Policy Context Distillation
Oct 5, 2026Wenxuan Wang, Zekai Liu, Weinan Zhang +2T2I Diffusion Models Context Distillation
Harbin Institute of Technology · Shanghai AI Laboratory · Shandong University +2
Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention
Sep 30, 2026Luis Zuin, Alexis Huet, Dario Rossi +1On-Policy Self-Distillation Self-Distillation
Huawei Technologies Co., Ltd., Paris, France
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Sep 30, 2026Ilgee Hong, Changlong Yu, Zhenghao Xu +5LLM-as-a-Judge Preference Modeling
Georgia Institute of Technology · Amazon · UC San Diego
Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation
Sep 29, 2026Xingtong Ge, Yutong Wang, Lunjie Zhu +7Streaming Video Generation Video Diffusion Models
The Hong Kong University of Science and Technology · Vivix Group Limited · The University of Sydney +1
Compress to Remember: Learning Compact Memory via On-Policy Distillation for Long Video Generation
Sep 28, 2026Xiaoyu Wu, Weihang Guo, Yifei Wang +3Video Diffusion Models Long Video Generation
Carnegie Mellon University · Rice University
Continuous Context Management
Sep 28, 2026William Hoy, Jingxuan Fan, Nurcin Celik +1Context Distillation Long-Horizon LLM Agents
University of Miami · Harvard University
Instruct, Not Answer: Using Instruction Privileges in On-Policy Context Distillation
Sep 26, 2026Hantao Yu, Xiaoxue Han, Udaya Ghai +4OOD Generalization On-Policy Distillation
Columbia University · Amazon Web Services
Compress What You See, Not What You Say: Anchored Context Distillation for Latent-Observation Software Engineering Agents
Sep 25, 2026Zhensheng Zou, Guoqing Wang, Dan HaoLLM Context Management Context Compression
Peking University
What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation
Sep 22, 2026Kanghui Tian, Siyuan Liu, Tianxiang Jiang +8On-Policy Self-Distillation Context Distillation
Fudan University · Shanghai Artificial Intelligence Laboratory · Nanjing University +4
Verbalizing Subliminal Learning Effects Using Text Optimization
Sep 15, 2026Nathan Hu, Sanmi Koyejo, Christopher PottsSubliminal Learning Context Distillation
Stanford University
Learning What to Remember: Test-Time Training via Context Distillation
Aug 3, 2026Zixuan Wang, Xingyu Dang, Rui-Jie Zhu +4Continual Learning for LLMs Long-Context Language Modeling
Princeton University · UC Santa Cruz · Carnegie Mellon University +1
Flux-OPD: On-Policy Distillation with Evolving Contexts
Jul 30, 2026Yuran Wang, Zekun Wang, Bohan Zeng +10Language Model Distillation On-Policy Distillation
Peking University · Kling Team · Tsinghua University +3
Sample-Efficient Learning from Agent Experience
Jul 23, 2026Chenhui Gou, Haoqin Tu, Yunhao Fang +2In-Context RL Language Model Distillation
Monash University · ByteDance Seed
Rethinking On-Policy Self-Distillation for Thinking Models
Jul 6, 2026Simran Kaur, Narutatsu Ri, Yinghui He +2LLM Self-Correction On-Policy Self-Distillation
Princeton Language and Intelligence, Princeton University
DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation
Jun 29, 2026Peyman Hosseini, Ondrej Bohdal, Ahmed Alajrami +6LLM Agents LLM Agent Training
Samsung R&D Institute UK · Queen Mary University of London
Doc-to-Atom: Learning to Compile and Compose Memory Atoms
Jun 10, 2026Xingjian Diao, Wenbo Li, Yashas Malur Saidutta +3Memory-Augmented Language Models Long-Context Language Model Inference
AI Center-Mountain View, Samsung Electronics · Dartmouth College
When Context Returns: Toward Robust Internalization in On-Policy Distillation
Jun 10, 2026Xun Wang, Ruishuo Chen, Zhuoran Li +2On-Policy Distillation Context Distillation
1IIIS, Tsinghua University
Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
Jun 3, 2026Jingwen Chen, Wenkai Yang, Shengda Fan +7Continual Learning for LLM Agents Continual Learning for LLMs
1Gaoling School of Artificial Intelligence, Renmin University of China · School of Software, Beihang University · 3Meituan
ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks
May 31, 2026Dhruv Saini, Rohan PandeyEfficient Language Model Reasoning Language Model Distillation
Bellevue High School · DigitalOcean
Context Distillation as Latent Memory Management
May 27, 2026Ziyang Zheng, Zeju Li, Xiangyu Wen +5Memory-Augmented Language Models Language Model Distillation
The Chinese University of Hong Kong · Huawei Noah’s Ark Lab
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
Apr 12, 2026Hao Wang, Guozhi Wang, Han Xiao +8RL for LLM Agents LLM Agent Skill Learning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong +2
TSUBASA: Improving Long-Horizon Personalization via Evolving Memory and Self-Learning with Context Distillation
Apr 9, 2026Xinliang Frederick Zhang, Lu WangPersonalized Language Models Memory-Augmented Language Models
Computer Science and Engineering, University of Michigan, Ann Arbor, MI
TED: Training-Free Experience Distillation for Multimodal Reasoning
Mar 25, 2026Shuozhi Yuan, Jinqing Wang, Zihao Liu +5Context Distillation Knowledge Distillation
China Telecom Digital Intelligence Technology Co.,Ltd., Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Structured Distillation for Personalized Agent Memory: 11x Token Reduction with Retrieval Preservation
Mar 13, 2026Sydney LewisLLM Agent Memory Context Distillation

Oct 18, 2026J/K move · Enter open · S save

Self-Distillation for Supervised Fine-Tuning

Wenxuan Zhu, Bang An, Shuming Liu, Wenqing Cui+5

King Abdullah University of Science and Technology · Shanghai Jiao Tong University

Supervised Fine-Tuning Language Model Distillation Self-Distillation Context Distillation

Open paper AskPDF arXiv

Supervised fine-tuning treats the expert demonstration or gold response as a one-hot target, suppressing valid alternative continuations that the model would otherwise assign probability to. We propose Gold-Conditioned Distillation (GCD), a method that turns expert demonstrations into model-native supervision. GCD uses one model as both teacher and student: the teacher conditions on a prompt containing the gold response in context, the student conditions on the original prompt as at inference, and the student is trained to minimize the forward KL divergence to the teacher's next-token distribution at each gold response token. This lets the model use its prior knowledge to teach itself the new task: correctness is anchored by the gold response, while the soft target preserves plausible alternatives. Across various training tasks on Qwen and Llama models, GCD improves new-task accuracy over SFT and generally reduces degradation on general benchmarks.