cs.AINov 14, 2026

Program Distillation: Transferring LLM Capability from System 2 to System 0

Authors: Pujun Zheng, Wanying Ren, Guoxiu He

Organizations: School of Economics and Management, East China Normal University, Shanghai, China · School of Statistics, East China Normal University, Shanghai, China

Abstract

Recent studies leverage large language models (LLMs) to construct executable programs through environment interaction, feedback analysis, and code revision. However, prior work has mainly treated LLMs as optimizers, overlooking their ability to transfer knowledge across system levels. We propose Program Distillation (PD), a framework that formalizes the transfer of abilities from System 2 to System 0. During learning, the LLM provides task knowledge and draft strategies; at deployment, a student program executes the resulting policy. We show that this approach can improve performance while reducing inference time. We introduce PDBench, a benchmark comprising 27 discrete- and continuous-control tasks with fixed initial policies, and PDMetaHarness, a model-harness-independent meta-system for monitoring and guiding the learning process. The strongest model achieves a normalized mean score of 0.728, surpassing PPO (0.656) and Jev-like decision models (0.239), while running exclusively on CPUs and reducing inference costs. Our analysis shows that performance is largely driven by the first breakthrough during learning, and structural modifications have a slightly lower success rate than parameter modifications but typically yield larger gains. We further introduce style distillation, which enables interpretable stylistic control over the learning process. Overall, we present an analytical framework for PD and demonstrate its practical viability.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion

    Feb 26, 2026Zhen Tan, Chengshuai Zhao, Song Wang +3Language Model DistillationReasoning Distillation

  2. Distilled Reinforcement Learning for LLM Post-training

    Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4RL for Language Model ReasoningCross-Architecture Knowledge Distillation

  3. ReAD: Reinforcement-Guided Capability Distillation for Large Language Models

    May 11, 2026Xueqi Cheng, Xugui Zhou, Tyler Derr +1LLM CompressionLanguage Model Distillation