AtomSD: Procedural Atom-Guided Self-Distillation for LLM Agents

Bohan Yu; Pengfei Cao; Kai Tian; Chen Han; Shi-Yang Li; Chenxi Zhou; Junlin Liu; Cao Liu

cs.CLOct 17, 2026

AtomSD: Procedural Atom-Guided Self-Distillation for LLM Agents

Authors: Bohan Yu, Pengfei Cao, Kai Tian, Chen Han, Shi-Yang Li, Chenxi Zhou, Junlin Liu, Cao Liu,

Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Meituan

Abstract

Large language models (LLMs) possess broad factual knowledge and strong reasoning capabilities, but solving complex tasks also requires procedural knowledge about how to act. Existing approaches often represent procedural knowledge through trajectories or bundled skills, limiting flexible reuse across interaction histories. This paper introduces Procedural Atoms, fine-grained and independently applicable units that represent Intent, When, Do, and Avoid guidance. Building on this representation, we propose AtomSD, an Atom-guided Self-Distillation framework in which the learner continually extracts Procedural Atoms from its own successful trajectories and distills them into the evolving policy through Atom-conditioned supervision and task-reward optimization. Comprehensive experiments show that AtomSD consistently outperforms strong reward-based and distillation methods, exceeding the state-of-the-art baseline by up to 9.31 points in average performance. Further analyses demonstrate that the proposed Procedural Atoms provide effective guidance, highlighting the value of atom-level procedural knowledge for policy learning.

Figures & tables

Figure 1: Procedural Atoms enable fine-grained reuse and composition of procedural knowledge.

Figure 2: Overview of AtomSD. Procedural Atoms are continually abstracted from the student’s own successful trajectories and distilled back into the evolving policy through Atom-guided self-distillation.

Panel of Table 1 — Table 1: Main results across four benchmarks. denotes methods requiring no additional training; denotes methods that update model parameters. Higher is better in every column. Avg. is the average of the six reported scores; green arrows ( $\boldsymbol{\uparrow}$ ) indicate our absolute gains over the highest baseline Avg. within each backbone. Best , Second-best , and Third-best scores are highlighted within each backbone.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Task characteristics	Task types	Training	Evaluation
ALFWorld	Household interaction with object search, navigation, and prerequisite-aware manipulation	6 families	3,553	140 seen / 134 unseen
ScienceWorld	Multi-step scientific experimentation involving transformations, measurement, circuits, and biological reasoning	30 types	3,592	194 seen / 211 unseen
WebShop	Interactive product search, attribute matching, option selection, and purchase	Open-ended	10,587	200 test
StableToolBench	Multi-turn tool use involving API selection, parameterization, and execution	3 groups	10,129	169 solvable
Total			27,861	1,048

Appendix

Table 2: Dataset characteristics and task counts used for training and evaluation.

Parameter	Setting
Shared training settings
Backbones	Qwen3.5-4B / Qwen3.5-9B
Tasks per iteration	48 (12 per benchmark)
Rollouts per task	4
Optimization mini-batch	12 tasks / 48 trajectories
Optimization epochs	1

Appendix

Table 3: Shared training hyperparameters and AtomSD-specific settings.

Method	GRPO reward loss	Distillation weight	Teacher update	Distillation configuration
GRPO	Yes	—	—	Task-reward optimization only.
OPSD	No	1.0	Fixed	Sampled-token reverse-KL surrogate with successful same-task demonstrations.
GRPO+OPSD	Yes	0.001	Fixed	Same demonstration-conditioned distillation objective as OPSD.
SDPO	No	1.0	EMA ( $\alpha=0.95$ )	Distribution-level self-distillation over the top- $K$ tokens and the aggregated tail probability, with $K=100$ .
Skill-SD	Yes	0.001	Full synchronization	Differentiable importance-weighted $k_{3}$ divergence.
SDAR	Yes	0.01	Full synchronization	Detached sigmoid gating with $\beta=5.0$ .

Appendix

Table 4: Method-specific configurations for the training baselines.

Model	Stage	Hist.	$N$	$S_{\mathrm{max}}$ (%)	$D_{\alpha}$ (%)	$D_{\beta}$ (%)	$D_{\alpha\beta}$ (%)
4B	Early	Same	175	36.9	68.4	64.9	67.4
	Early	Cross	144	16.9	70.8	72.9	75.0
	Late	Same	175	21.2	52.7	50.0	56.3
	Late	Cross	144	5.8	47.1	46.1	44.2
9B	Early	Same	173	45.2	59.0	56.0	54.4
	Early	Cross	149	25.9	69.9	68.7	61.9

Appendix

Table 5: Procedural Atom internalization across model scales, training stages, and history settings.

N

denotes the number of analysis units;

S_{\mathrm{max}}

measures maximum-token support change;

D_{\alpha}

D_{\beta}

, and

D_{\alpha\beta}

measure guidance-direction agreement for Atom

\omega_{\alpha}

, Atom

\omega_{\beta}

, and their composition, respectively.

Procedural Pattern	Core Dependency	Representative Case
Observation-Conditioned Branching	A downstream action depends on newly observed information.	ScienceWorld: AtomSD measures a substance at $-31\,^{\circ}\mathrm{C}$ and selects the destination specified for temperatures below $50\,^{\circ}\mathrm{C}$ , completing the task.
Constraint-Driven Revision	An earlier choice must be revised after detecting a violated constraint.	WebShop: AtomSD rejects an over-budget product, returns to search, and selects a cheaper item satisfying the requested attributes.
Subgoal-State Dependency	A later subgoal requires the state produced by an earlier operation.	ALFWorld: AtomSD acquires a potato, heats it, and only then places the heated object in the target receptacle.
Feedback-Guided API Recovery	Uninformative API responses require adapting the query strategy based on returned feedback.	StableToolBench: After an empty response, AtomSD revises its pagination strategy, verifies the returned results, and reports nine products grounded in the API output.

Appendix

Table 6: Representative cases of recurring procedural patterns observed in evaluation trajectories.

Intent	When	Do	Avoid
Collect ingredients	Required ingredients have been identified from the recipe.	Move each required ingredient into a suitable common container.	Leaving ingredients outside the container or mixing prematurely.
Mix the contents	All required ingredients are in one accessible, stable container, and the recipe calls for mixing.	Mix the container and observe whether the target substance is produced.	Mixing incomplete or separated ingredients, or ignoring mixing errors.
Acknowledge the product	The desired substance has been created and its location is known.	Focus on the created substance and confirm task completion.	Focusing on the wrong object or continuing without acknowledging the goal.

Appendix

Table 7: Paraphrases of Procedural Atoms extracted from successful training trajectories.

Method	Score	Outcome
GRPO	33	Incomplete
OPSD	25	Incomplete
GRPO+OPSD	33	Incomplete
SDPO	8	Incomplete
Skill-SD	33	Incomplete
SDAR	33	Incomplete

Appendix

Table 8: Episode-level comparison on the dough task.

Figure 8: Prompt template for procedural Atom extraction.

Figure 9: Atom-conditioned teacher input template.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 396 papers

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Jul 1, 2026Ye Liu, Srijan Bansal, Bo Pang +6Self-Reflection in Language Model Agents Language Model Distillation
Salesforce AI Research
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
Apr 12, 2026Hao Wang, Guozhi Wang, Han Xiao +8RL for LLM Agents LLM Agent Skill Learning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong +2
SKILL-KD: Contrastive Skill Distillation for LLM Agents
Jul 30, 2026Qiming Shi, Yibo Dou, Jiawen Zhu +5Skill Distillation LLM Agent Skill Learning
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University +1

Jul 1, 2026J/K move · Enter open · S save

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

Ye Liu, Srijan Bansal, Bo Pang, Yang Li+5

Salesforce AI Research

Self-Reflection in Language Model Agents Language Model Distillation Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.

Abstract

LLM Agent Skill Learning

Explore similar work

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

SKILL-KD: Contrastive Skill Distillation for LLM Agents