Designer-RSI: A Design Agent That Remembers Its Own Fixes

A new paper on arXiv, "Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design" (arXiv:2609.22086), authored by Hongyang Du, Lan Yan, Christian Flores, and Asim Kadav, presents a framework in which a language model operates design software using more than 230 tools while continuously building a library of reusable design procedures — a form of procedural memory that grows and refines itself as the agent handles real user requests.

Why procedural memory, not episodic memory

Most agent systems today lean on episodic or conversational memory: remembering what was said in this session, what was just done. That works for short-lived context, but it does little for an agent that must repeatedly operate the same design tools across hundreds of similar-but-not-identical requests.

Once a conversation ends, whatever sequence of tool calls the agent improvised to solve that task typically vanishes with it. Designer-RSI's core bet is that what deserves to persist is not "what happened" but "how it was solved" — an abstracted, nameable, reusable procedure. That is closer to how human experts actually work: an experienced designer does not rediscover software functionality from scratch on every job, but draws on an internalized set of workflows that gets adapted to the task at hand.

A regression-safety mechanism for a self-editing skill library

One notable design choice is a mechanism that prevents improvements to existing procedures from introducing regressions — ensuring a new fix does not silently break something that already worked. This matters enormously for a system that keeps rewriting its own skill library over time.

Without such a guard, every round of "refinement" carries the risk of quietly eroding capability that was already earned, meaning a growing skill bank could accumulate risk just as easily as it accumulates competence. The approach echoes, in spirit, how software engineering leans on regression testing to keep code changes from breaking what already works — except here it is applied to an agent's own evolving library of procedures rather than to source code.

Design as a uniquely hard proving ground for agent self-improvement

The authors argue that procedural memory offers a practical pathway for adapting agents when feedback is noisy and difficult to verify — and that framing is precisely what makes graphic design such a demanding test case. Math problems have a single correct answer; code either runs or it doesn't.

A design output, by contrast, is judged by taste, subjective preference, and context, with no ground-truth score to check against. That means a self-improving system in this domain has to tell genuine progress apart from noise without ever getting a clean, binary signal to lean on. Succeeding at self-improvement under those conditions is arguably a harder problem than succeeding at it in domains where correctness is cleanly verifiable.

What the numbers suggest

Across 1,406 real user design requests and 1,869 automated trajectories, the skill bank expanded from 76 to 139 procedures — nearly doubling — while one reported metric climbed from 72.7% to 99.3%, alongside win rates of 61.8% to 67.6% against baseline models. Analysis confirmed that both skill acquisition (learning to handle previously unaddressed tasks) and procedure refinement (improving what already existed) contributed meaningfully, rather than one mechanism carrying the whole gain.

Taken together, these numbers point to a broader gap: most tool-using agents deployed today have no built-in way to learn from repetition at all. Every call looks like the first call, and the headroom left on the table by that omission may be considerably larger than commonly assumed.