SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning

Yuxuan Zhu; Mohammad Mohammadi Amiri; Pin-Yu Chen; Keerthiram Murugesan

cs.AIOct 27, 2026

SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning

Authors: Yuxuan Zhu, Mohammad Mohammadi Amiri, Pin-Yu Chen, Keerthiram Murugesan

Organizations: Rensselaer Polytechnic Institute · IBM Research

Abstract

Latent multi-agent systems reduce inference cost by keeping intermediate reasoning and inter-agent communication in continuous representations.\ Reusable skills, however, are commonly supplied as text or stored as layer-wise Key-Value states or skill-specific model parameters, introducing repeated processing or substantial per-skill storage.\ We introduce SkillMem, which represents each reusable skill as a small external latent memory and makes it directly available during latent multi-agent reasoning.\ A role-conditioned cross-attention reader allows the same memory to guide different agents according to their roles and current hidden states.\ We train the skill memories and shared reader through behavioral distillation, matching the change in token predictions induced by a textual skill while keeping the backbone frozen.\ For our experiments, we obtain textual skills from paired successful and failed multi-agent trajectories, although existing skills can also be compiled directly.\ Across six benchmarks and Qwen3 models ranging from 4B to 14B parameters, SkillMem provides strong gains for smaller backbones while preserving efficient latent multi-agent inference across model scales.\ We use the same $8\times256$ memory configuration across all evaluated model scales, requiring 9.6,KB of skill-specific storage per skill, including its retrieval embedding.\ For a library of 100 skills, SkillMem reduces total storage by more than $38\times$ compared with alternative skill representations and achieves up to a $4.23\times$ end-to-end inference speedup.\ These results show that compact procedural memories can be reused without sacrificing the efficiency of latent multi-agent reasoning.

Figures & tables

Figure 1: Accuracy and efficiency comparison. (a) Accuracy changes of SkillMem relative to TextMAS and LatentMAS using Qwen3-4B. (b) Total storage required by different skill representations as the skill library grows. (c) End-to-end inference speedup relative to TextMAS.

Figure 2: Overview of SkillMem . Reusable textual skills are discovered from paired multi-agent trajectories. Each skill is compiled into a compact latent memory by matching the behavioral effect of skill injection. At inference time, the retrieved memory is shared across agents and consumed through a role-conditioned reader while inter-agent communication remains entirely latent.

Figure 3: Behavioral skill distillation. For each skill, agent role, and calibration task, the frozen backbone generates a teacher response with the textual skill included in the role prompt. We score the same response with and without the textual skill to measure its token-level effect. We then replace the skill text with its memory and train the memory and role-conditioned reader to reproduce the skill-supported behavior while preserving the base behavior at positions with limited skill support.

Tasks	Metrics	Qwen3-4B						Qwen3-8B						Qwen3-14B
Tasks	Metrics	Text	Latent	KV	SP	S2L	SkillMem	Text	Latent	KV	SP	S2L	SkillMem	Text	Latent	KV	SP	S2L	SkillMem
GSM8K	Acc. $\uparrow$	89.40	72.40	84.90	89.20	84.13	89.90	90.80	92.33	92.60	92.20	92.27	93.07	94.00	94.30	94.30	93.70	94.00	92.60
	Token $\downarrow$	229	215	205	208	185	169	240	196	206	195	223	202	198	188	195	195	187	175
	Speed $\downarrow$	19.1	6.8	6.2	6.1	7.6	5.5	35.1	6.0	12.2	6.0	7.9	6.6	26.8	9.4	11.8	9.7	9.7	8.9
GPQA-D	Acc. $\uparrow$	29.94	17.97	22.46	30.24	27.54	35.63	33.53	36.53	40.72	35.33	34.33	37.72	37.72	31.74	28.14	35.33	34.13	35.33
	Token $\downarrow$	488	222	268	380	366	420	495	398	408	427	407	374	501	445	456	365	428	476

Table 1: Main results across model scales. We report task accuracy (%, “Acc.”), average output tokens (“Token”), and inference time in seconds (“Speed”). “Text” and “Latent” denote TextMAS and LatentMAS, respectively, and “SP” denotes Soft Prompt. SkillMem accuracy is bolded when it exceeds either TextMAS or LatentMAS.

Method	Per-Skill	$\mathbf{N=10}$	$\mathbf{N=100}$	$\mathbf{N=1000}$
Text Skill	173 B	1.7 KB	17.3 KB	173 KB
LoRA Skill	30.7 MB	306.9 MB	3.07 GB	30.69 GB
KV Skill	13.0 MB	135.9 MB	1.36 GB	13.59 GB
SkillMem: Reader	shared	34.5 MB	34.5 MB	34.5 MB
SkillMem: Memory	9.6 KB	96 KB	0.96 MB	9.6 MB
SkillMem: Total	–	34.6 MB	35.5 MB	44.1 MB

Table 2: (a) Skill storage scaling with bank size. (b) Inference and training efficiency.

Method	GSM8K	GPQA-D	ARC-E	ARC-C	MedQA	Avg.
Qwen3-4B
SkillMem	89.90	35.63	97.53	91.20	61.50	75.15
Role Agnostic	75.70	32.94	97.87	89.73	40.61	67.37
Constant Emb.	87.40	34.14	97.53	90.07	59.39	73.71
Shuffled Emb.	85.00	34.13	97.33	87.80	57.75	72.40
Per-role Memory	83.90	32.94	96.60	90.27	58.92	72.52

Table 3: Ablation studies of SkillMem . Results report task accuracy (%).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Qwen3-4B	Qwen3-8B	Qwen3-14B
GSM8K	10	10	8
GPQA-D	9	8	8
ARC-E	10	10	10
ARC-C	9	9	9
MedQA	8	10	8
MBPP+	10	10	10

Appendix

Table 4: Number of discovered skills per benchmark and model scale after filtering instance-specific candidates and deduplicating overlapping skills.

Dataset	Skill	Description
GSM8K	Rate vs Total Confusion	When dealing with per-unit rates, ensure that the total amount is not confused with the rate itself.
	Quantity Tracking	Track named quantities, updates, and dependencies in a word problem.
	Equation Construction	Translate verbal relations into arithmetic equations before solving.
	Rate and Proportion Reasoning	Handle per-unit rates, totals, fractions, percentages, and repeated events.
	Intermediate Result Verification	Check intermediate arithmetic and whether each number answers the intended subquestion.
	Target Quantity Restatement	Restate the question’s target quantity before the final step to avoid answering a different quantity.

Appendix

Table 5: Discovered skills per benchmark (Qwen3-4B backbone).

Setting	Value
Sampling temperature	0.6
Top- $p$	0.95
Latent reasoning steps per agent	8
Max. new tokens (judger decode), default	512
Evaluation seeds (averaged)	41, 42, 43

Appendix

Table 6: Decoding configuration used for evaluation.

Hyperparameter	Value
Memory slots per skill ( $M$ )	8
Memory dimension ( $d_{\text{mem}}$ )	256
Reader dimension ( $d_{r}$ )	256 (defaults to $d_{\text{mem}}$ )
Reader insertion layers	$\{L/4,\ L/2,\ 3L/4,\ L-1\}$
Qwen3-4B / 8B ( $L{=}36$ )	layers $\{9,18,27,35\}$
Qwen3-14B ( $L{=}40$ )	layers $\{10,20,30,39\}$

Appendix

Table 7: SkillMem training hyperparameters (default configuration,

M=8

Hyperparameter	Value
LoRA rank $r$	16
LoRA $\alpha$	32
LoRA dropout	0.0
Target modules	$W_{q}$ , $W_{v}$
Learning rate	$1\mathrm{e}{-4}$
Batch size	4

Appendix

Table 8: Skill-to-LoRA (S2L) baseline hyperparameters.

Benchmark	Contrastive extraction $\Delta$	Latent generation $\Delta$
MedQA	$+0.00$ pp	$-1.57$ pp
GSM8K	$+0.27$ pp	$-0.13$ pp
GPQA-D	$-32.53$ pp	$-4.96$ pp

Appendix

Table 9: 3-seed accuracy delta vs. no-skill LatentMAS for the two zero-training, latent-space skill discovery variants (Qwen3-8B).

Procedural Memory for LLM Agents

See all 13 papers

SPEM: Structure-Aware Procedural Memory for Efficient Skill Use in LLM Agents
Oct 19, 2026Ran Zhou, Zehui Kong, Fanyu Lin +3Procedural Memory for LLM Agents LLM Agent Memory
School of Intelligence Science and Technology, Nanjing University
LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
Jun 4, 2026Aofan Yu, Chenyu Zhou, Tianyi Xu +8LLM Agent Skill Learning In-Context Learning
Shanghai Jiao Tong University · Sun Yat-Sen University · Shanghai Innovation Institute +1
HyperSkill: Self-Evolving LLM Agents via Hypergraph-Structured Skill Memory
Aug 17, 2026Ruiyao Xu, Tiankai Yang, Wei-Chieh HuangAgent Memory LLM Agent Memory
Northwestern University · University of Southern California · University of Illinois at Chicago

Oct 19, 2026J/K move · Enter open · S save

SPEM: Structure-Aware Procedural Memory for Efficient Skill Use in LLM Agents

Ran Zhou, Zehui Kong, Fanyu Lin, Jiaheng Liu+2

School of Intelligence Science and Technology, Nanjing University

Procedural Memory for LLM Agents LLM Agent Memory LLM Agent Skill Retrieval LLM Agent Efficiency LLM Agents Graph-Based Agent Memory

Open paper AskPDF arXiv

LLM-based agents increasingly rely on reusable skills. However, scaling skill libraries makes efficient and accurate skill use difficult. Full skill loading incurs high token and runtime overhead. In contrast, flat semantic retrieval treats skills as independent text entries, ignoring dependencies, execution order, state transitions, and data flow constraints. We propose SPEM, a lightweight structure-aware procedural memory framework for efficient skill use in LLM-based agents. SPEM maintains Atomic Skill Memory for individual skills and Skill-DAG Memory for reusable compositions derived from successful executions. For a new task, SPEM reuses high-confidence Skill-DAG templates when available. Otherwise, it retrieves atomic skills and invokes Skill-GNN for state- and structure-aware skill selection. Successful executions are written back as new Skill-DAGs, allowing procedural memory to accumulate over time. Experiments on ALFWorld and SkillsBench show that SPEM achieves the best overall effectiveness--efficiency trade-off, reaching 87.1% ALFWorld success and 38.38% SkillsBench reward while reducing token and runtime costs against strong baselines.

Abstract

Procedural Memory for LLM Agents

Explore similar work

SPEM: Structure-Aware Procedural Memory for Efficient Skill Use in LLM Agents

LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents

HyperSkill: Self-Evolving LLM Agents via Hypergraph-Structured Skill Memory