Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Hongyang Du; Lan Yan; Christian Flores; Asim Kadav

cs.AISep 18, 2026

Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Authors: Hongyang Du, Lan Yan, Christian Flores, Asim Kadav

Organizations: Adobe · Brown University

Abstract

Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.

Figures & tables

Figure 1: System overview. Left — rollout and reward. A brief (from a real User Prompt or in-context LLM Prompter) enters the graphic design agent; the Skill Bank retrieves the top skills with and filters the tool list. The agent executes a chain of tools through intermediate document states to an output image, which the grader turns into a scalar reward. Right — reflection and evolution. Per-skill statistics and recent call histories are the system’s assets; skills implicated in failures are edited or rewritten and bad histories are summarized into new skills, each verified by replay gate.

Figure 2: Widening pipeline. Uncovered subtasks are clustered into recurring coverage gaps, distilled into candidate skills, and admitted to the bank through the replay gate (§ 3.3 ).

Figure 3: Deepening pipeline. Failure-prone skills are revised by contrasting failed and successful histories; a candidate skill enters the bank only if it wins at the replay gate (§ 3.3 ) on the skill’s worst failing prompts. A rejected candidate triggers an optional one-shot exploration cycle.

Panel of Figure 4 — Figure 4: Skill-bank evolution over five rounds. (a) Skill embeddings ( BGE-small-en-v1.5 , UMAP, cosine); shape and colour indicate the target application, with hollow markers for the $76$ cold-start skills distilled from internal documentation and filled markers for the $63$ skills minted from user trajectories. (b) Per-round Added (Widening) and Rewritten (Deepening) skills, split into committed and gate-rejected. R1 is nearly all repair—the gap store has not yet accumulated enough recurring misses to mint from—while minting peaks in R2–R3 and tapers as coverage saturates.

Panel of Figure 5 — Figure 5: Iterative skill bank evolution drives consistent completeness gains (Claude-Sonnet-4, $n{=}200$ ). Retention $R(\tau)$ denotes the percentage of cases satisfying completeness $\geq\tau$ . Subfigure ( 5(a) ) shows the gap to the no-skill baseline $\Delta R(\tau)$ , while Table ( 5(b) ) details the underlying absolute values (row max in bold ). Later rounds (especially R5) dominate across nearly all thresholds, with the largest improvement at $\tau{=}0.9$ ( $+13$ pp over Base).

Agent	Method	GenEval2 $\uparrow$	DPG-Bench $\uparrow$	OneIG-EN $\uparrow$	OneIG-ZH $\uparrow$	Avg $\uparrow$
Claude-Opus-4.6	Base	61.24 (96.7)	77.58 (89.3)	60.66 (91.3)	68.67 (93.3)	67.04 (92.7)
	Evolve	57.92 (98.0)	86.54 (84.7)	68.79 (94.0)	70.21 (94.0)	70.87 (92.7)
	$\Delta$	-3.32 (+1.3)	+8.96 (-4.6)	+8.13 (+2.7)	+1.54 (+0.7)	+3.83 (0.0)
Claude-Sonnet-4	Base	34.26 (72.7)	68.28 (82.7)	52.00 (98.0)	52.00 (96.7)	51.63 (87.5)
	Evolve	46.25 (99.3)	83.79 (100)	53.47 (96.0)	53.79 (96.7)	59.33 (98.0)
	$\Delta$	+11.99 (+26.6)	+15.51 (+17.3)	+1.47 (-2.0)	+1.79 (0.0)	+7.70 (+10.5)

Table 1: Quantitative results on general T2I benchmarks. Each cell reports the generation quality followed by the success rate (%) . Agents equipped with the Evolve skill bank improve generation quality and success on most benchmarks across backbones compared to the no-skill baseline.

Figure 6: Generation latency comparison. Mean wall-clock time per successful generation. The Evolve framework introduces minimal computational overhead across most backbones.

Agent	OpenCOLE	GraphicBench	CreatiDesign	BannerRequest400	Overall
Claude-Opus-4.6	64.0% +14.0%	63.5% +13.5%	71.7% +21.7%	71.3% +21.3%	67.6% +17.6%
Claude-Sonnet-4	66.0% +16.0%	56.8% +6.8%	68.1% +18.1%	56.2% +6.2%	61.8% +11.8%
Qwen3.6-27B	62.2% +12.2%	49.0% -1.0%	70.9% +20.9%	69.2% +19.2%	62.8% +12.8%

Table 2: Pairwise win rates on specialized graphic design tasks. Evolve -vs- Base win rates judged by GPT-5.4 . Values in green show the margin over a 50% tie baseline.

Panel of Figure 7 — Figure 7: Qualitative comparison on six design briefs. Each column is a brief; rows are Base (no skill) and Ours ( Evolve ). Asset search runs once per brief and both arms are shown the same candidate pool, so retrieval is held fixed and the rows differ only in which assets the agent selects and how it edits them. All outputs are uncurated one-shot claude-opus-4.6 rollouts at low thinking effort, with no re-sampling and no human intervention, center-cropped to square for display.

		Quality $\uparrow$		Critique $\downarrow$		Tokens $\downarrow$
Bank	#Sk.	Compl.	Aesth.	Major	Minor	Prompt	Out	Tools	WR	SR
Base	0	69.08	65.92	3.93	8.02	346.8k	4979	33.5	—–	95.0
Cold start	76	68.62	65.66	3.88	7.87	445.2k	5298	35.4	46.4	96.5
+ rewrite	76	69.02	65.98	3.70	8.14	450.1k	5369	35.9	48.6	97.5
+ new skills	139	69.79	64.77	3.70	7.93	445.4k	5342	36.4	49.4	98.5
Evolve	139	74.04	66.53	3.57	8.13	436.6k	5167	36.7	58.5	98.0

Table 3: Ablation of skill-bank update mechanisms. All arms share identical prompts and evaluation settings. Neither rewriting nor adding new skills alone separates from cold start; applied together, they produce a superadditive interaction (

+3.85

completeness). Best per column in bold .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Agent	Method	OpenCOLE	GraphicBench	CreatiDesign	BannerRequest400	Avg
Claude-Opus-4.6	Base	100.0%	98.7%	94.7%	100.0%	98.3%
	Evolve	100.0%	100.0%	96.7%	100.0%	99.2%
	$\Delta$	0.0%	+1.3%	+2.0%	0.0%	+0.9%
Claude-Sonnet-4	Base	98.0%	97.3%	94.0%	74.7%	91.0%
	Evolve	100.0%	99.3%	100.0%	100.0%	99.8%
	$\Delta$	+2.0%	+2.0%	+6.0%	+25.3%	+8.8%

Appendix

Table 4: Success rates on specialized graphic design tasks. Percentage of prompts completed with a valid design within 900 s. On hosted Claude backbones, Evolve reaches near-perfect completion. Results for Qwen3.6-27B reflect local deployment stability constraints (see text).

Panel of Figure 8 — Figure 8: Three personalization modes. Each row is a user-flagged prompt run without the personalized skill ( Base , left) and with it (Ours, right), on the same frozen backbone. Top — method preference: the default renders snow as a sparse particle field; the skill, enforced via preference-conditioned retrieval and tool scoping, reproduces the user’s layered noise technique. Middle — perception/precision limit: both attempts at a recursive in-frame Droste effect leave the nested frames imperfectly seated; the skill helps but the residual misalignment reflects a verification gap that still needs a human in the loop. Bottom — vibe drawing: for the base unit tile of a De Nigris “3/4 Decorative” pattern, the Base agent drifts into an uncontrolled design, while the distilled skill reconstructs the user’s hand-tuned tile from a single prompt. The mosaic prompt is given together with the target pattern shown above it (a De Nigris “3/4 Decorative” tiling); the skill’s tile is a correct unit that tessellates into it, whereas the Base vibe drawing does not.

Panel of Figure 9 — Figure 9: Pushing past the base tile to the full pattern. (a) The full “3/4 Decorative” pattern a user builds by hand—the annotated target. (b) What the agent produces from that same hand-annotated skill: the tiling layout is roughly preserved, but the three-tone gray fills and fine structure are lost, so the result is a discounted version of the target. Fidelity leaks at each of the three stages—annotating the demonstration, distilling it into a compact skill, and getting the frozen model to follow a long skill—and the losses compound.

Figure 10: Opus 4.6 backbone (low thinking). All $193$ held-out prompts rendered with the round-5 skill bank, in random order. Prompt text is withheld (internal test set); aspect ratios are preserved and no panel is selected for quality.

Figure 11: Sonnet 4 backbone (low thinking). The same $193$ prompts as Figure 10 , rendered on the smaller backbone under the same protocol.

Procedural Memory for LLM Agents

See all 10 papers

Mar 19, 2026

Memento-Skills: Let Agents Design Agents

We introduce Memento-Skills, a generalist, continually-learnable LLM agent system that functions as an agent-designing agent: it autonomously constructs, adapts, and improves task-specific agents through experience. The system is built on a memory-based reinforcement learning framework with stateful prompts, where reusable skills (stored as structured markdown files) serve as persistent, evolving memory. These skills encode both behaviour and context, enabling the agent to carry forward knowledge across interactions. Starting from simple elementary skills (like Web search and terminal operations), the agent continually improves via the Read--Write Reflective Learning mechanism introduced in Memento~2~\cite{wang2025memento2}. In the read phase, a behaviour-trainable skill router selects the most relevant skill conditioned on the current stateful prompt; in the write phase, the agent updates and expands its skill library based on new experience. This closed-loop design enables continual learning without updating LLM parameters, as all adaptation is realised through the evolution of externalised skills and prompts. Unlike prior approaches that rely on human-designed agents, Memento-Skills enables a generalist agent to design agents end-to-end for new tasks. Through iterative skill generation and refinement, the system progressively improves its own capabilities. Experiments on the General AI Assistants benchmark and Humanity's Last Exam demonstrate sustained gains, achieving 26.2% and 116.2% relative improvements in overall accuracy, respectively. Code is available at https://github.com/Memento-Teams/Memento-Skills.

Huichi Zhou, Siyuan Guo, Anjie Liu +14

Agentic AI LLM Agent Skill Learning Continual Learning for LLM Agents Memory-Augmented Language Model Agents RL for LLM Agents LLM Agent Skill Retrieval

Ask PDF

Jun 22, 2026

Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. We introduce AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones. The benchmark includes controlled evaluation settings for local improvement, cross-task transfer, cross-role transfer, and cross-model generalization. Experiments show that procedural memory delivers consistent gains in industrial workflows: a single refinement round improves aggregate performance by 3.7-6.7 points, while skills evolved from diverse multi-model execution traces achieve 73.1% cross-model test accuracy, outperforming all single-model trace sources. We further find that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer. These results provide practical guidance for building, evaluating, and deploying procedural memory systems in production agent platforms.

Julia Belikova, Rauf Parchiev, Evgeny Egorov +4

LLM Agent Memory LLM Agent Evaluation LLM Agent Skill Learning AI Agent Benchmarks Procedural Memory for LLM Agents Memory-Augmented Language Model Agents

Ask PDF

Sep 8, 2026

Procedural Graphs: Self-Evolving Execution Structures for LLM Agents

Large language models are increasingly deployed as agents that plan over long horizons and act through external tools. Most agents select actions through unconstrained generation over an accumulating history, leaving implicit the procedural knowledge of what to do, in what order, and under which conditions. As trajectories lengthen, agents can lose track of their objectives, invoke tools out of order, and repeat unproductive actions. We introduce the Procedural Graph: just as a knowledge graph organizes factual knowledge into (entity, relation, entity) triplets for what-is questions, a Procedural Graph organizes procedural knowledge into (procedure, relation, procedure) triplets for what-to-do questions. At each decision step, the framework localizes the agent's active node, and a guidance model translates the surrounding subgraph into step-level situational guidance that biases the solver's next action without dictating it. The graph is self-evolving: an LLM refiner contrasts failed trajectories with successful ones and edits the graph's topology and attributes, committing edits that preserve or improve held-out validation performance while retaining rejected ones to discourage repetition. Starting from a minimal skeleton, the loop builds graphs that match or surpass hand-designed ones. It can also repair a flawed expert prior. Across multiple datasets, task types, and LLMs, the Procedural Graph delivers consistent gains over memory-based baselines, and self-evolution further improves performance without manual engineering.

Yuxing Lu, Yicheng Chen, Shanchan Wu +1

Google · Georgia Institute of Technology · Peking University

Graph-Based Agent Memory Long-Horizon LLM Agents LLM Agent Planning Self-Evolving Agents

Ask PDF

Abstract

Procedural Memory for LLM Agents

Explore similar work

Memento-Skills: Let Agents Design Agents

Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

Procedural Graphs: Self-Evolving Execution Structures for LLM Agents