Program Distillation: Transferring LLM Capability from System 2 to System 0
Authors: Pujun Zheng, Wanying Ren, Guoxiu He
Organizations: School of Economics and Management, East China Normal University, Shanghai, China · School of Statistics, East China Normal University, Shanghai, China
Recent studies leverage large language models (LLMs) to construct executable programs through environment interaction, feedback analysis, and code revision. However, prior work has mainly treated LLMs as optimizers, overlooking their ability to transfer knowledge across system levels. We propose Program Distillation (PD), a framework that formalizes the transfer of abilities from System 2 to System 0. During learning, the LLM provides task knowledge and draft strategies; at deployment, a student program executes the resulting policy. We show that this approach can improve performance while reducing inference time. We introduce PDBench, a benchmark comprising 27 discrete- and continuous-control tasks with fixed initial policies, and PDMetaHarness, a model-harness-independent meta-system for monitoring and guiding the learning process. The strongest model achieves a normalized mean score of 0.728, surpassing PPO (0.656) and Jev-like decision models (0.239), while running exclusively on CPUs and reducing inference costs. Our analysis shows that performance is largely driven by the first breakthrough during learning, and structural modifications have a slightly lower success rate than parameter modifications but typically yield larger gains. We further introduce style distillation, which enables interpretable stylistic control over the learning process. Overall, we present an analytical framework for PD and demonstrate its practical viability.
Figures & tables
Figure 1. Program distillation overview. A frozen System 2 LLM serves as both the distillation teacher and the learning optimizer. As a teacher, it supplies task-specific knowledge and uses its parametric knowledge to propose control strategies; as an optimizer, it organizes experiments, invokes tools, and interprets feedback. The environment evaluates candidate programs, and the final program is frozen and deployed as a System 0 policy.
Category
Count
Representative tasks
Primary decision requirement
Game scoring and rule-based decisions
2
Pong, Blackjack
Adversarial control and rule selection under hidden information
Map navigation and safe paths
5
Taxi, FrozenLake, CliffWalking
Path planning and risk management on known maps
Postural stability and swinging
6
CartPole, Acrobot, HumanoidStandup
Balancing, swinging up, and postural recovery
Goal reaching and manipulation
6
MountainCar, LunarLander, Pusher
Reaching goals, landing, and object manipulation
Sustained locomotion and propulsion
8
BipedalWalker, Hopper, Ant, Humanoid
Joint coordination and sustained locomotion
Total
27
—
Discrete and continuous control
Table 1. Task coverage of PDBench. Each of the 27 tasks is assigned to exactly one primary semantic category; counts and representative tasks summarize the five groups.
Method
Cont. control (16)
Discrete control (11)
Overall (27)
Build time (h)
Inference time (s)
Jev-like
SemIf-4B
0.240
0.237
0.239
–
264960
Laya
0.175
0.217
0.192
–
33480
Classic RL
PPO
0.603
0.733
0.656
5.1
147
SPO
0.558
0.643
0.593
5.1
146
Table 2. Performance comparison across the three baseline categories. The best and second-best values across the entire table are shown in bold and underlined, respectively. Build time is cumulative learning-session time. Inference time is the total testing duration for the entire batch; – denotes not applicable.
Figure 2. Normalized score versus inference time for the program distillation, reinforcement-learning, and online-decision routes. The best PD configuration attains a normalized score of 0.728 with shorter inference time than PPO, A3C, SemIf-4B, and Laya. The horizontal axis is logarithmic and runs from slower to faster.
Method
Contin. control (16)
Discrete control (11)
Overall (27)
Build time (h)
Inference time (s)
Alternatives
Meta
0.501
0.852
0.644
236.3
95
History
0.529
0.836
0.654
207.2
112
Population
0.569
0.866
0.690
176.5
124
Force
0.596
0.872
0.709
121.9
1567
PDMetaHarness
Table 3. Performance and cost of the four alternative learning-control paths (Meta, History, Population, and Force) and the Standard, Reset, and Swarm PDMetaHarness strategies. Scores are test-evaluation results after development-set selection for each task; costs include the initial execution of each alternative and all additional reruns following failures.
Model
Contin. control (16)
Discrete control (11)
Overall (27)
Build time (h)
Inference time (s)
GLM-5.3 (single)
0.618
0.864
0.718
38.0
110
GLM-5.3 + DeepSeek-V4.1-Flash
0.594
0.865
0.705
49.9
101
GLM-5.3 + Kimi-K2.6
0.501
0.848
0.642
43.8
116
GLM-5.3 + Qwen3.6-35B-A3B
0.550
0.785
0.646
25.3
134
Table 4. Test-evaluation performance when the GLM-5.3 tool agent is fixed as the optimizer and only the distillation teacher is replaced. The continuous and discrete columns are grouped by action type, so that corresponding settings can be compared directly across teachers. Failed tasks still enter test evaluation through their frozen submissions rather than being discarded, and the overall scores are computed over all 27 tasks.
Figure 3. Contribution of successive breakthroughs. (a) Mean share of total development gain by breakthrough order, with 95% bootstrap intervals and the cumulative share; (b) Distribution of the first-breakthrough share across 244 positive-gain runs.
Figure 4. Structural modifications as low-hit, high-reward events. (a) Attempt-level exceedance, the probability that an attempt’s payoff reaches a given share of run gain; structural and parameter modifications have similar success rates (33.1% versus 34.6%). (b) Distribution of event share conditional on setting a new record; the median contribution is 9.28% for structural modifications versus 1.44% for parameter modifications, and the 75th-percentile shares are 72.5% and 6.0%. Structural modifications succeed slightly less often but have a substantially heavier payoff tail.
Figure 5. Four representative learning trajectories: an early structural jump, a later structural jump, a parameter jump near the end of the budget, and a case with substantial gain but weak test evaluation. Gray marks candidate development scores, blue points mark accepted candidates that set records, the orange line marks the historical best, and the dashed line marks the initial policy. Gray shading indicates the pre-jump interval without a new record.
Figure 6. Measured style, internal Elo rank, and style strength, reported by assigned training target. (a) Measured style positions, where darker color indicates a higher internal Elo rank. (b) Style strength under each assigned target; across all training methods, the conservative-stable style is maintained more consistently than the other styles, indicating that this target is easier to preserve during training.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Environment ID
Action space
Maximum decision steps
Lower / upper anchor
atari_pong
ALE/Pong-v5
6 discrete actions
100,000; ends at 11 points
-11.000 / 11.000
cartpole
CartPole-v1
2 discrete actions
500
0.000 / 500.000
mountain_car
MountainCar-v0
3 discrete actions
200
-200.000 / -80.000
mountain_car_continuous
MountainCarContinuous-v0
1 dimension, range [-1, 1]
999
-100.000 / 100.000
pendulum
Pendulum-v1
1 dimension, range [-2, 2]
200
-1600.000 / 0.000
acrobot
Acrobot-v1
3 discrete actions
500
-500.000 / -80.000
Appendix
Table 5. Complete task protocol. Each task is listed with its environment ID, action space, maximum decision steps, and lower/upper scoring anchors. The gym_ prefix is omitted from task names.
Figure 7. Cumulative gain over observed attempts. (a) Mean marginal increment and cumulative share of total development gain by observed candidate attempt. (b) Distribution of cumulative gain at attempts 1, 3, and 10.
Figure 8. Complete leaderboard for a snapshot of the 500-player Echo Court league. Players are ordered by Elo, with wins, losses, and mean score difference per game retained.
Figure 9. A new player entering the local game; screenshot produced by off-screen rendering of the actual pygame window.
Figure 10. An actual match in the 50-match fixed-policy experience between Rally100, which uses the unoptimized initial program from the reference commit to control the right paddle, and a composite player; screenshot produced by off-screen rendering of the actual pygame window.
Metric
Result
CartPole return
500.000
Input tokens
212633
Output tokens
3342490
Cumulative model-request latency (h)
9.377
Appendix
Table 6. Return and execution investment for the single successful CartPole-v1 episode of the direct JSON-decision baseline. Request latencies come from usage records; extrema and the median follow the experiment report.
Figure 11. Test normalized scores for ten models across 27 tasks. Rows are grouped into five semantic task categories, the bar on the left distinguishes discrete and continuous actions, and model means appear at the bottom. The matrix supports comparison of model-task differences and action-type groups. Red indicates higher scores and blue lower scores; each cell is labeled to three decimal places.
Figure 12. The ablation experiments are executed on the normalized scores on each task in the test set, and the bottom shows the average of each method across 27 tasks.
Model
Candidate Rounds
Input tokens (M)
Output tokens (M)
Claude-Opus-5
645
86.421
1.897
MiMo-V2.6-Pro
404
449.347
7.091
GLM-5.3
453
385.133
4.052
DeepSeek-V4.1-Flash
457
63.432
1.533
GLM-5.3-Flash
324
235.451
3.789
GPT-6-Sol
512
61.230
0.433
Appendix
Table 7. Training token and round investment of ten models in selected sessions. Token units are in millions.
Dimension
System 0
System 1
System 2
System 3
Temporal scale
Super-fast
Fast
Slow
Super-slow
Underlying model
Physical dynamics
World model
Language model
Emergent symbolic system
Representation
Pre-representational
Implicit
Explicit and internal
External and shared
Cognitive process
Reflexive
Intuitive
Deliberate
Collective
Role of language
Non-linguistic
Sub-linguistic
Inner language
External language
Appendix
Table 8. Key characteristics of the System 0/1/2/3 framework.
Distilling robust reasoning capabilities from large language models (LLMs) into smaller, computationally efficient student models remains an unresolved challenge. Despite recent advances, distilled models frequently suffer from superficial pattern memorization and subpar generalization. To overcome these limitations, we introduce a novel distillation framework that moves beyond simple mimicry to instill a deeper conceptual understanding. Our framework features two key innovations. First, to address pattern memorization, Explanatory Inversion (EI) generates targeted ``explanatory probes'' that compel the student to articulate the underlying logic behind an answer, rather than just memorizing it. Second, to improve generalization, Explanatory GRPO (EXGRPO) uses a reinforcement learning algorithm with a novel Dialogue Structure Utility Bonus, which explicitly rewards the student for maintaining a coherent reasoning process across these probes. Extensive evaluations on 12 datasets demonstrate significant improvements. Using Gemma-7b as the student model, our method yields an average 20.39% increase over zero-shot performance and a 6.02% improvement over the state-of-the-art distillation baselines. Moreover, models distilled with our method show remarkable training efficiency (e.g., surpassing vanilla fine-tuning with 10-25% training data) and strong generalization to out-of-distribution tasks. Implementation is released at https://github.com/Zhen-Tan-dmml/ExGRPO.git.
Zhen Tan, Chengshuai Zhao, Song Wang +3
Arizona State University · University of Virginia · University of North Carolina at Chapel Hill
Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.
Chen Wang, Zhaochun Li, Jionghao Bai +4
College of Elite Engineers, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +4
Capability distillation applies knowledge distillation to selected model capabilities, aiming to compress a large language model (LLM) into a smaller one while preserving the abilities needed for a downstream task. However, most existing methods treat capabilities as independent training targets and overlook how improving one capability can reshape the student's broader capability profile, especially when multiple abilities jointly determine task success. We study capability distillation under a fixed token budget and identify two consistent patterns: distillation induces systematic, budget-dependent cross-capability transfer, and additional budget often brings limited task-relevant gains while sometimes degrading other useful abilities. Building on these insights, we propose ReAD, a Reinforcement-guided cApability Distillation framework that explicitly accounts for capability interdependence. ReAD first infers task-essential capabilities, then generates capability-targeted supervision on the fly, and finally uses an uncertainty-aware contextual bandit to adaptively allocate the distillation budget based on expected utility gains. Extensive experiments show that ReAD improves downstream utility under the same token budget while reducing harmful spillover and wasted distillation effort compared to strong baselines. Our code is publicly available at https://github.com/LabRAI/ReAD.
Xueqi Cheng, Xugui Zhou, Tyler Derr +1
Florida State University · Louisiana State University · Vanderbilt University