Context Distillation

Context distillation studies how models can absorb useful information from privileged prompts, interaction histories, or teacher feedback instead of repeatedly receiving it at inference time. Common approaches include on-policy self-distillation with KL losses, trajectory-based experience learning, and LoRA adapters or learned tokens that store and retrieve compressed memories. Researchers evaluate these methods in reasoning, coding and tool-use agents, long-context language modeling, and multimodal tasks, using benchmarks such as AIME, SWE-bench, and ALFWorld. The central challenge is balancing effective transfer and lower context costs against forgetting, distorted reasoning, or loss of details needed for future actions.