cs.AISep 18, 2026
SaveDesigner-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design
Organizations: Adobe · Brown University
Abstract
Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.
Figures & tables
Figure 1: System overview. Left — rollout and reward. A brief (from a real User Prompt or in-context LLM Prompter) enters the graphic design agent; the Skill Bank retrieves the top skills with and filters the tool list. The agent executes a chain of tools through intermediate document states to an output image, which the grader turns into a scalar reward. Right — reflection and evolution. Per-skill statistics and recent call histories are the system’s assets; skills implicated in failures are edited or rewritten and bad histories are summarized into new skills, each verified by replay gate.
Figure 2: Widening pipeline. Uncovered subtasks are clustered into recurring coverage gaps, distilled into candidate skills, and admitted to the bank through the replay gate (§ 3.3 ).
Figure 3: Deepening pipeline. Failure-prone skills are revised by contrasting failed and successful histories; a candidate skill enters the bank only if it wins at the replay gate (§ 3.3 ) on the skill’s worst failing prompts. A rejected candidate triggers an optional one-shot exploration cycle.
Figure 4: Skill-bank evolution over five rounds. (a) Skill embeddings ( BGE-small-en-v1.5 , UMAP, cosine); shape and colour indicate the target application, with hollow markers for the cold-start skills distilled from internal documentation and filled markers for the skills minted from user trajectories. (b) Per-round Added (Widening) and Rewritten (Deepening) skills, split into committed and gate-rejected. R1 is nearly all repair—the gap store has not yet accumulated enough recurring misses to mint from—while minting peaks in R2–R3 and tapers as coverage saturates.
Figure 5: Iterative skill bank evolution drives consistent completeness gains (Claude-Sonnet-4, ). Retention denotes the percentage of cases satisfying completeness . Subfigure ( 5(a) ) shows the gap to the no-skill baseline , while Table ( 5(b) ) details the underlying absolute values (row max in bold ). Later rounds (especially R5) dominate across nearly all thresholds, with the largest improvement at ( pp over Base).
| Agent | Method | GenEval2 | DPG-Bench | OneIG-EN | OneIG-ZH | Avg |
|---|---|---|---|---|---|---|
| Claude-Opus-4.6 | Base | 61.24 (96.7) | 77.58 (89.3) | 60.66 (91.3) | 68.67 (93.3) | 67.04 (92.7) |
| Evolve | 57.92 (98.0) | 86.54 (84.7) | 68.79 (94.0) | 70.21 (94.0) | 70.87 (92.7) | |
| -3.32 (+1.3) | +8.96 (-4.6) | +8.13 (+2.7) | +1.54 (+0.7) | +3.83 (0.0) | ||
| Claude-Sonnet-4 | Base | 34.26 (72.7) | 68.28 (82.7) | 52.00 (98.0) | 52.00 (96.7) | 51.63 (87.5) |
| Evolve | 46.25 (99.3) | 83.79 (100) | 53.47 (96.0) | 53.79 (96.7) | 59.33 (98.0) | |
| +11.99 (+26.6) | +15.51 (+17.3) | +1.47 (-2.0) | +1.79 (0.0) | +7.70 (+10.5) |
Table 1: Quantitative results on general T2I benchmarks. Each cell reports the generation quality followed by the success rate (%) . Agents equipped with the Evolve skill bank improve generation quality and success on most benchmarks across backbones compared to the no-skill baseline.
Figure 6: Generation latency comparison. Mean wall-clock time per successful generation. The Evolve framework introduces minimal computational overhead across most backbones.
| Agent | OpenCOLE | GraphicBench | CreatiDesign | BannerRequest400 | Overall |
|---|---|---|---|---|---|
| Claude-Opus-4.6 | 64.0% +14.0% | 63.5% +13.5% | 71.7% +21.7% | 71.3% +21.3% | 67.6% +17.6% |
| Claude-Sonnet-4 | 66.0% +16.0% | 56.8% +6.8% | 68.1% +18.1% | 56.2% +6.2% | 61.8% +11.8% |
| Qwen3.6-27B | 62.2% +12.2% | 49.0% -1.0% | 70.9% +20.9% | 69.2% +19.2% | 62.8% +12.8% |
Table 2: Pairwise win rates on specialized graphic design tasks. Evolve -vs- Base win rates judged by GPT-5.4 . Values in green show the margin over a 50% tie baseline.
Figure 7: Qualitative comparison on six design briefs. Each column is a brief; rows are Base (no skill) and Ours ( Evolve ). Asset search runs once per brief and both arms are shown the same candidate pool, so retrieval is held fixed and the rows differ only in which assets the agent selects and how it edits them. All outputs are uncurated one-shot claude-opus-4.6 rollouts at low thinking effort, with no re-sampling and no human intervention, center-cropped to square for display.
| Quality | Critique | Tokens | ||||||||
| Bank | #Sk. | Compl. | Aesth. | Major | Minor | Prompt | Out | Tools | WR | SR |
| Base | 0 | 69.08 | 65.92 | 3.93 | 8.02 | 346.8k | 4979 | 33.5 | —– | 95.0 |
| Cold start | 76 | 68.62 | 65.66 | 3.88 | 7.87 | 445.2k | 5298 | 35.4 | 46.4 | 96.5 |
| + rewrite | 76 | 69.02 | 65.98 | 3.70 | 8.14 | 450.1k | 5369 | 35.9 | 48.6 | 97.5 |
| + new skills | 139 | 69.79 | 64.77 | 3.70 | 7.93 | 445.4k | 5342 | 36.4 | 49.4 | 98.5 |
| Evolve | 139 | 74.04 | 66.53 | 3.57 | 8.13 | 436.6k | 5167 | 36.7 | 58.5 | 98.0 |
Table 3: Ablation of skill-bank update mechanisms. All arms share identical prompts and evaluation settings. Neither rewriting nor adding new skills alone separates from cold start; applied together, they produce a superadditive interaction ( completeness). Best per column in bold .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent | Method | OpenCOLE | GraphicBench | CreatiDesign | BannerRequest400 | Avg |
|---|---|---|---|---|---|---|
| Claude-Opus-4.6 | Base | 100.0% | 98.7% | 94.7% | 100.0% | 98.3% |
| Evolve | 100.0% | 100.0% | 96.7% | 100.0% | 99.2% | |
| 0.0% | +1.3% | +2.0% | 0.0% | +0.9% | ||
| Claude-Sonnet-4 | Base | 98.0% | 97.3% | 94.0% | 74.7% | 91.0% |
| Evolve | 100.0% | 99.3% | 100.0% | 100.0% | 99.8% | |
| +2.0% | +2.0% | +6.0% | +25.3% | +8.8% |
Table 4: Success rates on specialized graphic design tasks. Percentage of prompts completed with a valid design within 900 s. On hosted Claude backbones, Evolve reaches near-perfect completion. Results for Qwen3.6-27B reflect local deployment stability constraints (see text).
Figure 8: Three personalization modes. Each row is a user-flagged prompt run without the personalized skill ( Base , left) and with it (Ours, right), on the same frozen backbone. Top — method preference: the default renders snow as a sparse particle field; the skill, enforced via preference-conditioned retrieval and tool scoping, reproduces the user’s layered noise technique. Middle — perception/precision limit: both attempts at a recursive in-frame Droste effect leave the nested frames imperfectly seated; the skill helps but the residual misalignment reflects a verification gap that still needs a human in the loop. Bottom — vibe drawing: for the base unit tile of a De Nigris “3/4 Decorative” pattern, the Base agent drifts into an uncontrolled design, while the distilled skill reconstructs the user’s hand-tuned tile from a single prompt. The mosaic prompt is given together with the target pattern shown above it (a De Nigris “3/4 Decorative” tiling); the skill’s tile is a correct unit that tessellates into it, whereas the Base vibe drawing does not.
Figure 9: Pushing past the base tile to the full pattern. (a) The full “3/4 Decorative” pattern a user builds by hand—the annotated target. (b) What the agent produces from that same hand-annotated skill: the tiling layout is roughly preserved, but the three-tone gray fills and fine structure are lost, so the result is a discounted version of the target. Fidelity leaks at each of the three stages—annotating the demonstration, distilling it into a compact skill, and getting the frozen model to follow a long skill—and the losses compound.
Figure 10: Opus 4.6 backbone (low thinking). All held-out prompts rendered with the round-5 skill bank, in random order. Prompt text is withheld (internal test set); aspect ratios are preserved and no panel is selected for quality.
Figure 11: Sonnet 4 backbone (low thinking). The same prompts as Figure 10 , rendered on the smaller backbone under the same protocol.