WASP-Mem: Let Agent Organize Procedural Memory. Human Spend Effort on Diagnosis
Organizations: University of Illinois Urbana-Champaign · Amazon
Abstract
LLM agents struggle to automate complex enterprise procedures because static, pre-built harness cannot capture proprietary policies, undocumented behaviors, or state-dependent recovery paths that surface only during execution. Procedural memory accrued from agent experience offers a training-free path to closing this gap -- yet unlike episodic records or semantic facts, it must consolidate repeated successes and diagnosed failures into reusable actions and recovery strategies. We introduce WASP-Mem (Wiki-based Agent-Synthesized Procedural Memory), a nimble but effective method that distills execution traces and feedback into persistent, interlinked wiki pages, enabling continual agent learning. On TheAgentCompany benchmark, WASP-Mem raises the checkpoint pass rate by 4.6-6.1% over the strongest baseline we compared across three models while requiring 57-61% less storage. We further present a systematic controlled study spanning consolidation method, feedback type, trace ingestion scope, and refinement rounds. Our results reveal that prescriptive knowledge organization such as human defined schemas and explicit deduplication rules hurts performance rather than closing the procedural-knowledge gap. In contrast, human diagnosis converts failed traces into corrective knowledge and drives sustained improvement: a single round of human diagnosis outperforms four rounds of self-judgment by 7.0-10.9%. These findings point to a simple division of labor: let agents organize procedural memory, and invest human effort where it matters most -- diagnosis.
Figures & tables
| TheAgentCompany | -Bench | |||||
| Method | Checkpoint pass | Fully passed | Store (KB) | Checkpoint pass | Fully passed | Store (KB) |
| Claude Opus 4.6 | ||||||
| No Memory | 58.5% | 36.2% | n/a | 66.7% | 40.0% | n/a |
| No Consolidation | 61.3% (+2.8) | 40.2% (+4.0) | 7,979 | 69.3% (+2.6) | 42.5% (+2.5) | 1,428 |
| SkillX | 65.3% (+6.8) | 45.7% (+9.5) | 1,125 | 81.2% (+14.5) | 50.0% (+10.0) | 336 |
| Insights | 65.7% (+7.2) | 45.7% (+9.5) | 396 | 78.1% (+11.4) | 55.0% (+15.0) | 96 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Episodic | Semantic | Procedural | |
|---|---|---|---|
| Stored content | Specific event and context | General fact or concept | Condition-action procedure |
| Repeated trials | Remain distinct events | Duplicate facts merge | Refine one reusable procedure |
| Failure value | Another event | Wrong facts are discarded | Reveals missing conditions or error paths |
| Generalization target | Event recall | Concept-level recall | Execution across task instantiations |
| Primary evaluation | Recall accuracy | Recall accuracy | Execution success |
| Method | Durable representation | Input grouping | Existing store access | Organization / deduplication |
|---|---|---|---|---|
| No Memory | none | none | no | none |
| No Consolidation | raw rollout | rollout | no | raw-rollout retrieval |
| SkillX | planning/functional/atomic skills | rollout independently | no | three levels, no deduplication |
| Insights | strategy insight | rollout independently | no | flat, no deduplication |
| Schema Form-ED | typed wiki page | independent extraction then merge | merge step only | fixed schema, deterministic matching |
| Schema Form | typed wiki page | sequential rollouts | yes | fixed schema, implicit deduplication |
| Representation | Task-level retrieval | Subgoal retrieval | Context expansion |
|---|---|---|---|
| No Memory | none | none | none |
| No Consolidation | raw-rollout retrieval | none | retrieved rollout trace |
| Insights | search/index, up to 5 files | search, up to 3 files | full insight files |
| SkillX | search/routing, up to 5 files | search/routing, up to 3 files | full files across skill tiers |
| Schema variants | top procedure from 3 candidates | top 5 actions per query | linked procedures, actions, and lessons |
| Free Form | search top 5 per query/index | search/index, up to 3 pages | full wiki pages |
| Feedback type | Information available during consolidation |
|---|---|
| Self-Judge | Agent self-assessment without an external outcome signal. |
| Task-binary | External pass/fail for the whole task. |
| Checkpoint-binary | External pass/fail for each checkpoint. |
| Checkpoint-binary + Human | Checkpoint outcomes plus human failure diagnosis. |
| Experiment | Varied dimension | Method | Feedback | Scope | Rounds |
|---|---|---|---|---|---|
| 1 | Consolidation method | seven conditions | Checkpoint-binary | Both | R1 |
| 2 | Feedback type | Free Form | four conditions | Both | R1 |
| 3 | Ingestion scope | Free Form | Self-Judge and Checkpoint-binary + Human | Success-only, Both | R1 |
| 4 | Rounds | Free Form | Self-Judge and Checkpoint-binary + Human | Both | R0 to R4 |
| Model | Method | Checkpoint pass | (pp) | Fully passed | Store (KB) | Consol. cost |
|---|---|---|---|---|---|---|
| Claude Opus 4.6 | No Memory | 317/542 (58.5%) | n/a | 46/127 (36.2%) | n/a | n/a |
| No Consolidation | 332/542 (61.3%) | +2.8 | 51/127 (40.2%) | 7,979 | n/a | |
| SkillX | 354/542 (65.3%) | +6.8 | 58/127 (45.7%) | 1,125 | $173 | |
| Insights | 356/542 (65.7%) | +7.2 | 58/127 (45.7%) | 396 | $148 | |
| Schema Form-ED | 356/542 (65.7%) | +7.2 | 57/127 (44.9%) | 789 | $380 | |
| Schema Form | 359/542 (66.2%) | +7.7 | 62/127 (48.8%) | 565 | $230 |
| Model | Feedback | Checkpoint pass | (pp) | Fully passed | Store (KB) | Consol. cost |
|---|---|---|---|---|---|---|
| Claude Opus 4.6 | Self-Judge | 352/542 (64.9%) | 0.0 | 59/127 (46.5%) | 212 | $174 |
| Task-binary | 391/542 (72.1%) | +7.2 | 64/127 (50.4%) | 237 | $205 | |
| Checkpoint-binary | 392/542 (72.3%) | +7.4 | 61/127 (48.0%) | 234 | $198 | |
| Checkpoint-binary + Human | 411/542 (75.8%) | +10.9 | 71/127 (55.9%) | 328 | $196 | |
| GPT-5.4 | Self-Judge | 352/542 (64.9%) | 0.0 | 56/127 (44.1%) | 200 | $156 |
| Task-binary | 384/542 (70.8%) | +5.9 | 66/127 (52.0%) | 224 | $186 |
| Model | Feedback | Scope | Checkpoint pass | (pp) | Fully passed | Store (KB) | Consol. cost |
|---|---|---|---|---|---|---|---|
| Claude Opus 4.6 | CP-binary + Human | Success-only | 359/542 (66.2%) | 0.0 | 53/127 (41.7%) | 119 | $180 |
| CP-binary + Human | Both | 411/542 (75.8%) | +9.6 | 71/127 (55.9%) | 328 | $196 | |
| Self-Judge | Success-only | 351/542 (64.8%) | 0.0 | 54/127 (42.5%) | 193 | $180 | |
| Self-Judge | Both | 352/542 (64.9%) | +0.1 | 59/127 (46.5%) | 212 | $174 | |
| GPT-5.4 | CP-binary + Human | Success-only | 361/542 (66.6%) | 0.0 | 60/127 (47.2%) | 135 | $170 |
| CP-binary + Human | Both | 418/542 (77.1%) | +10.5 | 78/127 (61.4%) | 335 | $201 |
| Model | Round | Checkpoint pass | (pp) | Marginal (pp) | Fully passed | Store (KB) | Consol. cost |
|---|---|---|---|---|---|---|---|
| Claude Opus 4.6 | R0 | 317/542 (58.5%) | 0.0 | n/a | 46/127 (36.2%) | n/a | n/a |
| R1 | 411/542 (75.8%) | +17.3 | +17.3 | 71/127 (55.9%) | 328 | $196 | |
| R2 | 430/542 (79.3%) | +20.8 | +3.5 | 79/127 (62.2%) | 460 | $171 | |
| R3 | 446/542 (82.3%) | +23.8 | +3.0 | 86/127 (67.7%) | 492 | $180 | |
| R4 | 464/542 (85.6%) | +27.1 | +3.3 | 92/127 (72.4%) | 583 | $179 | |
| GPT-5.4 | R0 | 325/542 (60.0%) | 0.0 | n/a | 51/127 (40.2%) | n/a | n/a |
| Model | Round | Checkpoint pass | (pp) | Fully passed | Store (KB) | Consol. cost |
|---|---|---|---|---|---|---|
| Claude Opus 4.6 | R0 | 317/542 (58.5%) | 0.0 | 46/127 (36.2%) | n/a | n/a |
| R1 | 352/542 (64.9%) | +6.4 | 59/127 (46.5%) | 212 | $174 | |
| R2 | 349/542 (64.4%) | +5.9 | 52/127 (40.9%) | 297 | $141 | |
| R3 | 362/542 (66.8%) | +8.3 | 56/127 (44.1%) | 273 | $136 | |
| R4 | 373/542 (68.8%) | +10.3 | 60/127 (47.2%) | 291 | $135 | |
| GPT-5.4 | R0 | 325/542 (60.0%) | 0.0 | 51/127 (40.2%) | n/a | n/a |
| Model | Condition | Checkpoint pass | (pp) | Fully passed |
|---|---|---|---|---|
| Claude Opus 4.6 | Pooled R0 to R4, No Consolidation | 408/542 (75.3%) | +16.8 | 70/127 (55.1%) |
| Pooled R0 to R4, Insights | 431/542 (79.5%) | +21.0 | 78/127 (61.4%) | |
| Free Form, R2 | 430/542 (79.3%) | +20.8 | 79/127 (62.2%) | |
| Free Form, R4 | 464/542 (85.6%) | +27.1 | 92/127 (72.4%) | |
| GPT-5.4 | Pooled R0 to R4, No Consolidation | 412/542 (76.0%) | +16.0 | 77/127 (60.6%) |
| Pooled R0 to R4, Insights | 433/542 (79.9%) | +19.9 | 84/127 (66.1%) |
| Behavior | Observed pattern | Procedural consequence |
|---|---|---|
| Implicit merge | Free Form revises one GitLab authentication wiki page with new pagination and error handling evidence. | Repeated trials become one richer procedure. |
| Independent duplication | SkillX creates 15 separate pages describing substantially the same GitLab API authentication procedure. | Retrieval must sift through redundant fragments. |
| Schema omission | On ds-stock-analysis-slides , Schema Form records step names but drops the code and formatting details retained by Free Form. | The action is named but not reliably executable. |
| Explicit false merge | Schema Form-ED merges “grade ML exams with Python” and “extract video frames with Python” under action-bash-run-python . | Unrelated instructions can be retrieved together. |
| Model | No Memory execution | Free Form execution | Retrieval | Free Form total |
|---|---|---|---|---|
| Claude Opus 4.6 | $106.6 | $88.9 | $60.4 | $149.3 |
| GPT-5.4 | $108.5 | $87.1 | $60.6 | $147.8 |
| GLM-5 | $56.3 | $47.2 | $18.6 | $65.9 |