LLM agents struggle to automate complex enterprise procedures because static, pre-built harness cannot capture proprietary policies, undocumented behaviors, or state-dependent recovery paths that surface only during execution. Procedural memory accrued from agent experience offers a training-free path to closing this gap -- yet unlike episodic records or semantic facts, it must consolidate repeated successes and diagnosed failures into reusable actions and recovery strategies. We introduce WASP-Mem (Wiki-based Agent-Synthesized Procedural Memory), a nimble but effective method that distills execution traces and feedback into persistent, interlinked wiki pages, enabling continual agent learning. On TheAgentCompany benchmark, WASP-Mem raises the checkpoint pass rate by 4.6-6.1% over the strongest baseline we compared across three models while requiring 57-61% less storage. We further present a systematic controlled study spanning consolidation method, feedback type, trace ingestion scope, and refinement rounds. Our results reveal that prescriptive knowledge organization such as human defined schemas and explicit deduplication rules hurts performance rather than closing the procedural-knowledge gap. In contrast, human diagnosis converts failed traces into corrective knowledge and drives sustained improvement: a single round of human diagnosis outperforms four rounds of self-judgment by 7.0-10.9%. These findings point to a simple division of labor: let agents organize procedural memory, and invest human effort where it matters most -- diagnosis.
Figures & tables
Figure 1: Workflow and key results. (a) The multi-round loop enables continual learning by consolidate agent execution traces and feedback into reusable procedural memory. (b) Memory-store size and checkpoint performance across consolidation methods and model variants. WASP-Mem achieves higher checkpoint pass rate with smaller memory store size.
TheAgentCompany
τ2 -Bench
Method
Checkpoint pass
Fully passed
Store (KB)
Checkpoint pass
Fully passed
Store (KB)
Claude Opus 4.6
No Memory
58.5%
36.2%
n/a
66.7%
40.0%
n/a
No Consolidation
61.3% (+2.8)
40.2% (+4.0)
7,979
69.3% (+2.6)
42.5% (+2.5)
1,428
SkillX
65.3% (+6.8)
45.7% (+9.5)
1,125
81.2% (+14.5)
50.0% (+10.0)
336
Insights
65.7% (+7.2)
45.7% (+9.5)
396
78.1% (+11.4)
55.0% (+15.0)
96
Table 1: Consolidation method across model variants. Bold marks the best value for each metric within a block, and the shaded row is the Free Form WASP-Mem .
Figure 2: Feedback type and ingestion scope. Checkpoint pass rate (a) and fully-passed rate (b) when consolidating successful traces only versus both successful and failed traces under Self-Judge, Checkpoint-binary, and Checkpoint-binary + Human feedback. Colored points denote model variants; black diamonds and whiskers show the cross-model mean and min–max range.
Figure 3: Multi-round refinement and task completion. (a) Checkpoint pass rate across R0–R4 for all three models under Checkpoint-binary + Human, Checkpoint-binary, and Self-Judge feedback. (b) Claude Opus 4.6 task scores under Checkpoint-binary + Human feedback. Segment labels are task counts.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Episodic
Semantic
Procedural
Stored content
Specific event and context
General fact or concept
Condition-action procedure
Repeated trials
Remain distinct events
Duplicate facts merge
Refine one reusable procedure
Failure value
Another event
Wrong facts are discarded
Reveals missing conditions or error paths
Generalization target
Event recall
Concept-level recall
Execution across task instantiations
Primary evaluation
Recall accuracy
Recall accuracy
Execution success
Appendix
Table 2: Memory types and evaluation requirements. Procedural memory differs because repeated trials should refine a reusable action policy, and its quality is established by execution rather than recall.
Method
Durable representation
Input grouping
Existing store access
Organization / deduplication
No Memory
none
none
no
none
No Consolidation
raw rollout
rollout
no
raw-rollout retrieval
SkillX
planning/functional/atomic skills
rollout independently
no
three levels, no deduplication
Insights
strategy insight
rollout independently
no
flat, no deduplication
Schema Form-ED
typed wiki page
independent extraction then merge
merge step only
fixed schema, deterministic matching
Schema Form
typed wiki page
sequential rollouts
yes
fixed schema, implicit deduplication
Appendix
Table 3: Consolidation-method definitions. “Existing store access” refers to whether the consolidator reads the current memory store before proposing an update. SkillX and Insights are alternative consolidation representations, whereas Schema Form-ED , Schema Form , and Free Form are LLM wiki variants.
Figure 4: Controlled procedural-memory design space. The consolidation methods varied in Experiment 1 appear above. Feedback type (Experiment 2) and ingestion scope (Experiment 3) appear below at left and right, respectively. The repeated workflow in Figure 1(a) defines the round dimension in Experiment 4.
Representation
Task-level retrieval
Subgoal retrieval
Context expansion
No Memory
none
none
none
No Consolidation
raw-rollout retrieval
none
retrieved rollout trace
Insights
search/index, up to 5 files
search, up to 3 files
full insight files
SkillX
search/routing, up to 5 files
search/routing, up to 3 files
full files across skill tiers
Schema variants
top procedure from 3 candidates
top 5 actions per query
linked procedures, actions, and lessons
Free Form
search top 5 per query/index
search/index, up to 3 pages
full wiki pages
Appendix
Table 4: Representation-specific retrieval adapters. The executor loop is shared, while each adapter exposes the native organization of its memory representation. “Search” is semantic search over file contents, and “index” and “routing” are the representation’s own entry-point file.
Feedback type
Information available during consolidation
Self-Judge
Agent self-assessment without an external outcome signal.
Task-binary
External pass/fail for the whole task.
Checkpoint-binary
External pass/fail for each checkpoint.
Checkpoint-binary + Human
Checkpoint outcomes plus human failure diagnosis.
Appendix
Table 5: Feedback types. Rich feedback denotes Checkpoint-binary + Human , and weak feedback denotes Self-Judge .
Experiment
Varied dimension
Method
Feedback
Scope
Rounds
1
Consolidation method
seven conditions
Checkpoint-binary
Both
R1
2
Feedback type
Free Form
four conditions
Both
R1
3
Ingestion scope
Free Form
Self-Judge and Checkpoint-binary + Human
Success-only, Both
R1
4
Rounds
Free Form
Self-Judge and Checkpoint-binary + Human
Both
R0 to R4
Appendix
Table 6: Controlled experimental matrix. Each experiment varies one focal dimension while fixing the remaining dimensions.
Model
Method
Checkpoint pass
Δ (pp)
Fully passed
Store (KB)
Consol. cost
Claude Opus 4.6
No Memory
317/542 (58.5%)
n/a
46/127 (36.2%)
n/a
n/a
No Consolidation
332/542 (61.3%)
+2.8
51/127 (40.2%)
7,979
n/a
SkillX
354/542 (65.3%)
+6.8
58/127 (45.7%)
1,125
$173
Insights
356/542 (65.7%)
+7.2
58/127 (45.7%)
396
$148
Schema Form-ED
356/542 (65.7%)
+7.2
57/127 (44.9%)
789
$380
Schema Form
359/542 (66.2%)
+7.7
62/127 (48.8%)
565
$230
Appendix
Table 7: Complete Experiment 1 results. Δ is checkpoint-pass improvement over No Memory for the same model. Cost covers consolidation API calls only.
Model
Feedback
Checkpoint pass
Δ (pp)
Fully passed
Store (KB)
Consol. cost
Claude Opus 4.6
Self-Judge
352/542 (64.9%)
0.0
59/127 (46.5%)
212
$174
Task-binary
391/542 (72.1%)
+7.2
64/127 (50.4%)
237
$205
Checkpoint-binary
392/542 (72.3%)
+7.4
61/127 (48.0%)
234
$198
Checkpoint-binary + Human
411/542 (75.8%)
+10.9
71/127 (55.9%)
328
$196
GPT-5.4
Self-Judge
352/542 (64.9%)
0.0
56/127 (44.1%)
200
$156
Task-binary
384/542 (70.8%)
+5.9
66/127 (52.0%)
224
$186
Appendix
Table 8: Complete Experiment 2 results. Δ is relative to Self-Judge for the same model.
Model
Feedback
Scope
Checkpoint pass
Δ (pp)
Fully passed
Store (KB)
Consol. cost
Claude Opus 4.6
CP-binary + Human
Success-only
359/542 (66.2%)
0.0
53/127 (41.7%)
119
$180
CP-binary + Human
Both
411/542 (75.8%)
+9.6
71/127 (55.9%)
328
$196
Self-Judge
Success-only
351/542 (64.8%)
0.0
54/127 (42.5%)
193
$180
Self-Judge
Both
352/542 (64.9%)
+0.1
59/127 (46.5%)
212
$174
GPT-5.4
CP-binary + Human
Success-only
361/542 (66.6%)
0.0
60/127 (47.2%)
135
$170
CP-binary + Human
Both
418/542 (77.1%)
+10.5
78/127 (61.4%)
335
$201
Appendix
Table 9: Complete Experiment 3 results. Δ is relative to Success-only under the same model and feedback type. CP-binary + Human abbreviates Checkpoint-binary + Human .
Model
Round
Checkpoint pass
Δ (pp)
Marginal (pp)
Fully passed
Store (KB)
Consol. cost
Claude Opus 4.6
R0
317/542 (58.5%)
0.0
n/a
46/127 (36.2%)
n/a
n/a
R1
411/542 (75.8%)
+17.3
+17.3
71/127 (55.9%)
328
$196
R2
430/542 (79.3%)
+20.8
+3.5
79/127 (62.2%)
460
$171
R3
446/542 (82.3%)
+23.8
+3.0
86/127 (67.7%)
492
$180
R4
464/542 (85.6%)
+27.1
+3.3
92/127 (72.4%)
583
$179
GPT-5.4
R0
325/542 (60.0%)
0.0
n/a
51/127 (40.2%)
n/a
n/a
Appendix
Table 10: Experiment 4 with rich feedback. Δ is relative to R0. Marginal gain is relative to the preceding round.
Model
Round
Checkpoint pass
Δ (pp)
Fully passed
Store (KB)
Consol. cost
Claude Opus 4.6
R0
317/542 (58.5%)
0.0
46/127 (36.2%)
n/a
n/a
R1
352/542 (64.9%)
+6.4
59/127 (46.5%)
212
$174
R2
349/542 (64.4%)
+5.9
52/127 (40.9%)
297
$141
R3
362/542 (66.8%)
+8.3
56/127 (44.1%)
273
$136
R4
373/542 (68.8%)
+10.3
60/127 (47.2%)
291
$135
GPT-5.4
R0
325/542 (60.0%)
0.0
51/127 (40.2%)
n/a
n/a
Appendix
Table 11: Experiment 4 with weak feedback. Δ is relative to R0 for the same model.
Model
Condition
Checkpoint pass
Δ (pp)
Fully passed
Claude Opus 4.6
Pooled R0 to R4, No Consolidation
408/542 (75.3%)
+16.8
70/127 (55.1%)
Pooled R0 to R4, Insights
431/542 (79.5%)
+21.0
78/127 (61.4%)
Free Form, R2
430/542 (79.3%)
+20.8
79/127 (62.2%)
Free Form, R4
464/542 (85.6%)
+27.1
92/127 (72.4%)
GPT-5.4
Pooled R0 to R4, No Consolidation
412/542 (76.0%)
+16.0
77/127 (60.6%)
Pooled R0 to R4, Insights
433/542 (79.9%)
+19.9
84/127 (66.1%)
Appendix
Table 12: Rollout-volume control. The pooled methods consume 635 rollouts from rounds R0 to R4, more than the 508 the evaluated Free Form R4 store was built from, and process them in a single non-incremental pass while Free Form consolidates one batch at a time. Δ is relative to No Memory for the same model in Table 7 .
Figure 5: Rollout volume does not account for the R4 result. Colored markers show individual model variants. Black diamonds and whiskers show the cross-model mean and min to max range. The pooled controls consume 635 rollouts from R0 to R4, more than the 508 the Free Form R4 store was built from, and consolidate them in one pass while Free Form consolidates one batch at a time. Exact results appear in Table 12 .
Figure 6: Free Form versus Schema Form by task-score bucket for Claude Opus 4.6. The checkpoint-level advantage of Free Form is distributed across partially completed tasks, while Schema Form narrowly leads in the all-or-nothing fully-passed rate. This difference motivates reporting both metrics.
Behavior
Observed pattern
Procedural consequence
Implicit merge
Free Form revises one GitLab authentication wiki page with new pagination and error handling evidence.
Repeated trials become one richer procedure.
Independent duplication
SkillX creates 15 separate pages describing substantially the same GitLab API authentication procedure.
Retrieval must sift through redundant fragments.
Schema omission
On ds-stock-analysis-slides , Schema Form records step names but drops the code and formatting details retained by Free Form.
The action is named but not reliably executable.
Explicit false merge
Schema Form-ED merges “grade ML exams with Python” and “extract video frames with Python” under action-bash-run-python .
Unrelated instructions can be retrieved together.
Appendix
Table 13: Representative filtering behaviors observed in generated stores. These examples motivate, rather than independently prove, the Filter Hypothesis.
Model
No Memory execution
Free Form execution
Retrieval
Free Form total
Claude Opus 4.6
$106.6
$88.9
$60.4
$149.3
GPT-5.4
$108.5
$87.1
$60.6
$147.8
GLM-5
$56.3
$47.2
$18.6
$65.9
Appendix
Table 14: API cost of one complete 127-task evaluation batch. Execution is the task-execution phase, retrieval is the memory-query phase, and Free Form total is their sum. Consolidation costs are reported separately in the experiment tables. Evaluator, embedding, host-compute, and human-feedback costs are outside these values.
Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. We introduce AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones. The benchmark includes controlled evaluation settings for local improvement, cross-task transfer, cross-role transfer, and cross-model generalization. Experiments show that procedural memory delivers consistent gains in industrial workflows: a single refinement round improves aggregate performance by 3.7-6.7 points, while skills evolved from diverse multi-model execution traces achieve 73.1% cross-model test accuracy, outperforming all single-model trace sources. We further find that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer. These results provide practical guidance for building, evaluating, and deploying procedural memory systems in production agent platforms.
External memory enables LLM agents to improve from experience by distilling trajectories of past tasks into concise lessons for new tasks. However, memory can also lead to memory-induced degradation, where an agent fails on tasks it would solve without memory, often because it follows a lesson that is incorrect, or misapplies or misreads a valid one. This is because lessons are inferred from trajectories labeled only with task success or failure, yet are typically provided without the trajectories they came from, leaving the agent no way to check them. To address this, we introduce WikiTrace, an evidence-linked memory system designed to let agents check each lesson against its original trajectory. Specifically, motivated by the finding that a trajectory often still holds the evidence needed to catch an erroneous lesson, WikiTrace stores each lesson as a concise insight linked to a summary of why its trajectory succeeded or failed and to the raw trajectory itself. Then, on a new task, the agent starts from all insights and, whenever one lacks details needed to act, conflicts with another, or comes from a failed task, descends through its links with terminal tools to fill in missing details or correct the insight before acting. We validate WikiTrace on five agentic benchmarks, showing that it consistently outperforms existing memory approaches and a baseline given the same insights alone, while also reducing memory-induced degradation. Our code is available at https://github.com/yukyeongleee/wikitrace
LLM-based agents offer new opportunities for automating business process execution beyond the limits of rule-based systems. However, general-purpose LLMs lack the organization-specific knowledge required for reliable execution, which is typically fragmented across human-oriented artifacts such as policies, process models, and standard operating procedures. While such knowledge can technically be encoded in individual prompts or agent-specific retrieval setups, this approach does not scale in enterprises, as it gives rise to knowledge silos and rule duplicates, and makes consistent updates and learning across agents difficult. We argue that this calls for an organizational memory for agentic business process execution: a shared, governed, and agent-consumable reference layer of evolving organization-specific procedural knowledge about how work should be executed. We derive requirements for such a memory, propose an architecture for its curation and consumption, and demonstrate its effectiveness in a proof-of-concept based on a procurement scenario.
Lukas Kirchdorfer, Adrian Rebmann, Christian Warmuth +3