Latent multi-agent systems reduce inference cost by keeping intermediate reasoning and inter-agent communication in continuous representations.\ Reusable skills, however, are commonly supplied as text or stored as layer-wise Key-Value states or skill-specific model parameters, introducing repeated processing or substantial per-skill storage.\ We introduce SkillMem, which represents each reusable skill as a small external latent memory and makes it directly available during latent multi-agent reasoning.\ A role-conditioned cross-attention reader allows the same memory to guide different agents according to their roles and current hidden states.\ We train the skill memories and shared reader through behavioral distillation, matching the change in token predictions induced by a textual skill while keeping the backbone frozen.\ For our experiments, we obtain textual skills from paired successful and failed multi-agent trajectories, although existing skills can also be compiled directly.\ Across six benchmarks and Qwen3 models ranging from 4B to 14B parameters, SkillMem provides strong gains for smaller backbones while preserving efficient latent multi-agent inference across model scales.\ We use the same 8×256 memory configuration across all evaluated model scales, requiring 9.6,KB of skill-specific storage per skill, including its retrieval embedding.\ For a library of 100 skills, SkillMem reduces total storage by more than 38× compared with alternative skill representations and achieves up to a 4.23× end-to-end inference speedup.\ These results show that compact procedural memories can be reused without sacrificing the efficiency of latent multi-agent reasoning.
Figures & tables
Figure 1: Accuracy and efficiency comparison. (a) Accuracy changes of SkillMem relative to TextMAS and LatentMAS using Qwen3-4B. (b) Total storage required by different skill representations as the skill library grows. (c) End-to-end inference speedup relative to TextMAS.
Figure 2: Overview of SkillMem . Reusable textual skills are discovered from paired multi-agent trajectories. Each skill is compiled into a compact latent memory by matching the behavioral effect of skill injection. At inference time, the retrieved memory is shared across agents and consumed through a role-conditioned reader while inter-agent communication remains entirely latent.
Figure 3: Behavioral skill distillation. For each skill, agent role, and calibration task, the frozen backbone generates a teacher response with the textual skill included in the role prompt. We score the same response with and without the textual skill to measure its token-level effect. We then replace the skill text with its memory and train the memory and role-conditioned reader to reproduce the skill-supported behavior while preserving the base behavior at positions with limited skill support.
Tasks
Metrics
Qwen3-4B
Qwen3-8B
Qwen3-14B
Text
Latent
KV
SP
S2L
SkillMem
Text
Latent
KV
SP
S2L
SkillMem
Text
Latent
KV
SP
S2L
SkillMem
GSM8K
Acc. ↑
89.40
72.40
84.90
89.20
84.13
89.90
90.80
92.33
92.60
92.20
92.27
93.07
94.00
94.30
94.30
93.70
94.00
92.60
Token ↓
229
215
205
208
185
169
240
196
206
195
223
202
198
188
195
195
187
175
Speed ↓
19.1
6.8
6.2
6.1
7.6
5.5
35.1
6.0
12.2
6.0
7.9
6.6
26.8
9.4
11.8
9.7
9.7
8.9
GPQA-D
Acc. ↑
29.94
17.97
22.46
30.24
27.54
35.63
33.53
36.53
40.72
35.33
34.33
37.72
37.72
31.74
28.14
35.33
34.13
35.33
Token ↓
488
222
268
380
366
420
495
398
408
427
407
374
501
445
456
365
428
476
Table 1: Main results across model scales. We report task accuracy (%, “Acc.”), average output tokens (“Token”), and inference time in seconds (“Speed”). “Text” and “Latent” denote TextMAS and LatentMAS, respectively, and “SP” denotes Soft Prompt. SkillMem accuracy is bolded when it exceeds either TextMAS or LatentMAS.
Method
Per-Skill
N=10
N=100
N=1000
Text Skill
173 B
1.7 KB
17.3 KB
173 KB
LoRA Skill
30.7 MB
306.9 MB
3.07 GB
30.69 GB
KV Skill
13.0 MB
135.9 MB
1.36 GB
13.59 GB
SkillMem: Reader
shared
34.5 MB
34.5 MB
34.5 MB
SkillMem: Memory
9.6 KB
96 KB
0.96 MB
9.6 MB
SkillMem: Total
–
34.6 MB
35.5 MB
44.1 MB
Table 2: (a) Skill storage scaling with bank size. (b) Inference and training efficiency.
LLM-based agents increasingly rely on reusable skills. However, scaling skill libraries makes efficient and accurate skill use difficult. Full skill loading incurs high token and runtime overhead. In contrast, flat semantic retrieval treats skills as independent text entries, ignoring dependencies, execution order, state transitions, and data flow constraints. We propose SPEM, a lightweight structure-aware procedural memory framework for efficient skill use in LLM-based agents. SPEM maintains Atomic Skill Memory for individual skills and Skill-DAG Memory for reusable compositions derived from successful executions. For a new task, SPEM reuses high-confidence Skill-DAG templates when available. Otherwise, it retrieves atomic skills and invokes Skill-GNN for state- and structure-aware skill selection. Successful executions are written back as new Skill-DAGs, allowing procedural memory to accumulate over time. Experiments on ALFWorld and SkillsBench show that SPEM achieves the best overall effectiveness--efficiency trade-off, reaching 87.1% ALFWorld success and 38.38% SkillsBench reward while reducing token and runtime costs against strong baselines.
Ran Zhou, Zehui Kong, Fanyu Lin +3
School of Intelligence Science and Technology, Nanjing University
Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 64.1% fewer prefill tokens, and improves Search-QA exact match by 3.0 points with 72.2% lower skill-token overhead. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be precisely controlled via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.
Aofan Yu, Chenyu Zhou, Tianyi Xu +8
Shanghai Jiao Tong University · Sun Yat-Sen University · Shanghai Innovation Institute +1
As agentic tasks grow in complexity, LLM agents increasingly rely on experiential memory to reuse procedural knowledge across tasks. Effective memory design must jointly address what to store, how memory is structured and retrieved, and how memory evolves. Existing systems tackle each only partially: they store trajectories, insights, or workflows as isolated entries, discarding compositional relationships among subtasks and reusable skills; retrieve by flat embedding similarity that ignores relational signals; and maintain memory without leveraging its relational structure. We propose HyperSkill, a hypergraph-based memory framework that jointly improves all three. HyperSkill represents memory as a hypergraph with two node types, subtask steps and reusable skills, where each hyperedge links the subtasks and skills from a single trajectory. Dual-path retrieval queries both subtask and trajectory levels, ranking skills by co-occurrence across retrieved trajectories. Periodic structure-informed maintenance prunes low-utility nodes and merges redundant skills via quality-weighted propagation. Across xBench, GAIA, and WebWalkerQA with GPT-4o and Qwen3-30B-A3B, HyperSkill outperforms ten memory baselines, yielding gains of up to +11.51 on GAIA and +11.18 on WebWalkerQA.
Ruiyao Xu, Tiankai Yang, Wei-Chieh Huang
Northwestern University · University of Southern California · University of Illinois at Chicago