cs.ROOct 30, 2026
SaveRoboArchitect: Automating the Design of Robotic Agents with Persistent Memory
Organizations: Stanford University
Abstract
Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.
Figures & tables
Figure 1: Wrapping a frozen policy in an evolved harness raises its held-out success rate from 9.7% to 43.3% on unseen scenes and from 12.7% to 40.2% on unseen tasks, often past the same policy finetuned on the benchmark ( Section 4.2 ). All scores are evaluated on the held-out episodes never seen during the harness’s optimization. Unseen scenes: held-out episodes evaluate the same tasks under varied scene arrangements. Unseen tasks: held-out episodes evaluate distinct tasks to test cross-task skill transfer ( Section 4.1 ).
Figure 2: RoboArchitect turns a frozen policy into a performant robotic agent without updating its weights: a coding model iteratively refines the harness from execution traces. Starting from a bare policy (VLA), RoboArchitect evolves a robotic agent equipped with auxiliary tools and persistent memory. The right side illustrates experiment conditions ( Section 4.1 ) that test how well the evolved harness generalizes.
| Unseen Scenes (%) | Unseen Tasks (%) | |||||||||
| Bare | Harness (ours) | Bare | Harness (ours) | |||||||
| Benchmark | Policy | Trained On | Policy | One-Shot | Iterative | Policy | One-Shot | Iterative | ||
| LIBERO-Pro-10 | MolmoAct-2 | LIBERO | 10.0 | 12.0 | 32.0 | +22.0 | 2.5 | 2.5 | 15.0 | +12.5 |
| OpenVLA-7B | LIBERO-10 | 0.0 | 2.0 | 26.0 | +26.0 | 0.0 | 0.0 | 22.5 | +22.5 | |
| LIBERO | 8.0 | 4.0 | 24.0 | +16.0 | 0.0 | 0.0 | 2.5 | +2.5 | ||
| RLinf- | LIBERO-130 | 2.0 | 2.0 | 40.0 | +38.0 | 0.0 | 0.0 | 10.0 | +10.0 | |
Table 1: RoboArchitect ’s iteratively evolved harness raises mean held-out success by +345% on unseen scenes and +216% on unseen tasks. Bare Policy : the VLA policy alone. Harness (ours) : the same frozen policy inside the harness RoboArchitect designed, after a single design attempt ( One-Shot , iteration 1) or after iterative refinement over up to 5 iterations ( Iterative ). : gain of the iterative harness over the bare policy. Rows marked show the finetuned version of the same policy evaluated on the same held-out set. Training-source distance between the policy and the evaluation benchmark is color-coded: black = in-distribution; gold = spatially swapped; orange = cross-benchmark; dark red = out-of-domain. All scores are success rates in %.
| Training Pool (%) procedurally generated | Held-Out (%) standard benchmark episodes | |||||||||
| Bare | Harness (ours) | Bare | Harness (ours) | |||||||
| Policy | Trained On | Benchmark | Policy | One-Shot | Iterative | Policy | One-Shot | Iterative | ||
| General X-VLA | general robot data | LIBERO-Pro | 0.0 | 3.5 | 29.4 | +29.4 | 0.1 | 3.5 | 24.1 | +24.0 |
| VLABench | 7.1 | 14.8 | 17.5 | +10.4 | 7.5 | 17.3 | 21.6 | +14.2 | ||
| SimplerEnv WidowX | 0.0 | 65.8 | 68.3 | +68.3 | 0.0 | 45.8 | 24.0 | +24.0 | ||
| SimplerEnv Google | 0.0 | 22.0 | 51.2 | +51.2 | 0.0 | 20.0 | 45.6 | +45.6 | ||
Table 2: A unified evolved harness serves different benchmarks: around the general X-VLA it substantially raises held-out success, and around the finetuned X-VLA specialists it preserves their competence. Columns and labels read as in Table 1 .
Figure 3: Evolved control routines fix concrete failure patterns. Each row shows an episode under an earlier harness (starting with bare VLA policy) on the left, and under the revised RoboArchitect harness on the right. Row (a) shows detecting and clearing arm jams, (b) synthesizing missing motion primitives, (c) rescaling contact forces, and (d) redirecting to target objects instead of memorized locations. Fig. 5 shows more examples. See discussions on these revisions in Section 4.3 and Appendix A .
Figure 4: Persistent memories emerge from harness evolution and inform action decisions. Each row illustrates one form of memory mechanism; rows (a) and (b) show reusable execution experience, (c) task and environment context, and (d) embodiment-specific calibration. See discussions on these memories in Section 4.4 and Appendix A .
| Setting | Memory Entries | Reads per Episode | Actionable Reads |
|---|---|---|---|
| X-VLA on VLABench | finger depths that have been lifting objects | 3.4 | 98% |
| X-VLA on SimplerEnv WidowX | height at which to close the fingers, per measured object height | 3.5 | 96% |
| MolmoAct-2 on LIBERO-Pro-10 | average depth of successful grasps below an object’s top | 9.9 | 99% |
Table 3: Auto-designed persistent memories are frequently accessed and yield highly useful information to guide later actions. Reads per Episode : average frequency of memory reads in held-out episodes. Actionable Reads : the share of these reads that find stored info.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: More evolved control routines. Rows read as in Fig. 3 . Row (e) shows synthesizing missing motion primitives, and (f) tethering mobile bases to active workspaces.
Figure 6: Evolved harnesses also enhance the robotic agent’s perception precision and reliability. Panels read as in Fig. 3 , discussions in Section 4.3 . Rows (g) and (h) show adapting to perception limitations, and (i) replacing an unusable policy with a code-as-policy agent ( Section 4.2 ).
Figure 7: The unified harness learns to take over the arm command from a failing policy and to leave a progressing one alone, and calibrates itself to each robot’s configurations. Each row shows an episode under the harness before (left) and after (right) an iteration’s revision.
Figure 8: Held-out episodes differ from the optimization episodes either in scene arrangement ( unseen scenes ) or in the task itself ( unseen tasks ) ( Section 4.1 ). Left, unseen scenes: the same tasks with scene configurations dealt out alternately. Right, unseen tasks: different tasks that require similar underlying skills.
| Benchmark | Trials | Optimization Tasks | Held-Out Tasks |
|---|---|---|---|
| LIBERO-Pro-10 | 10 | turn on the stove and put the moka pot on it ; put the black bowl in the bottom drawer of the cabinet and close it ; put both the alphabet soup and the tomato sauce in the basket ; put both the cream cheese box and the butter in the basket ; put the white mug on the left plate and put the yellow and white mug on the right plate | put the yellow and white mug in the microwave and close it ; put both moka pots on the stove ; put both the alphabet soup and the cream cheese box in the basket ; put the white mug on the plate and put the chocolate pudding to the right of the plate |
| RoboCerebra | 10 | cream cheese, popcorn and butter, each into a white storage box; cream cheese into a white storage box, then the frypan onto the rack; plate, popcorn and cookies rearranged on the dining set, then wine poured onto the plate; chocolate pudding from the cabinet top onto the plate, then wine poured onto the plate; chocolate pudding and butter into a black bowl, the stove turned on, then the frypan onto the stove | cookies into a basket, tomato sauce and bbq sauce poured onto the frypan, wine poured into a mug, then another mug into a wooden tray; cream cheese and popcorn into a white bowl, then the frypan onto the cabinet top; the short cabinet opened, butter into the cabinet, then tomato sauce poured onto the plate; the short cabinet opened, cream cheese into the cabinet, chocolate pudding into a black bowl, then wine poured into a mug; white bowl from the cabinet top, cookies and chocolate pudding into the bowl, the microwave opened, then the bowl into the microwave |
| VLABench | 10 | select the named fruit; select the named mahjong tile; select the named poker card | select the named drink out of a fridge; select the -th largest poker card; select the mahjong tile of the unique type |
| SimplerEnv WidowX | 24 | put the spoon on the towel ; stack the green block on the yellow block | put carrot on plate ; put eggplant into yellow basket |
| RoboCasa-365 | 50 | close the toaster oven door; open the cabinet door; open the drawer; pick an object from the counter and place it in the cabinet; press the start button on the microwave | close the fridge door; pick an object from the plate and place it in the pan on the stove; fully slide the top dishwasher rack out; press down the lever to turn on the electric kettle |
Table 4: Optimization and held-out tasks under the unseen tasks protocol. On RoboCerebra a task is a chain of sub-task instructions stated here in brief. On VLABench and RoboCasa-365 the instruction of a task changes with the sampled objects. Trials is the number of trials ( i.e., episodes) per task.
Figure 10: Under unseen scenes , the harness improves over iterations on the training set, and the final harness outperforms the one-shot harness on nearly every held-out set. Left of the dotted line is the training episode set, iteration by iteration; iteration 0 is the bare VLA policy, filled markers denote accepted harness revisions that improved task success over prior iterations, and open markers denote revisions that were not accepted. Right of the dotted line are the scores on the held-out set: the bare policy, the harness after iteration 1 (a single-attempt agent design) and the final iteratively refined harness. The held-out set evaluates identical tasks under different scene arrangements ( Section 4.1 ).
Figure 11: Under unseen tasks , the harness improves over iterations on the training set, and the final harness outperforms the bare VLA policy and the one-shot harness on every held-out set. Labels and panels read as in Fig. 10 . A cross marks a revision that could not be evaluated; when this is the first revision, no single-attempt harness is available for held-out evaluation. The held-out set evaluates unseen tasks requiring shared underlying skills ( Section 4.1 ).
Figure 12: Around the general X-VLA, the unified harness improves on every benchmark of the training pool over iterations; around the finetuned specialists, it leaves their already competent scores intact. Left column, using the general X-VLA checkpoint as policy; right column, using benchmark-finetuned X-VLA specialists as policy. The first row is the mean over the benchmarks and each later row is one benchmark. Left of the dotted line shows the performance on the procedurally generated training pool, iteration by iteration (iteration 0 is the bare policy; filled markers denote accepted harness revisions). Right of it are the scores on the benchmark’s standard episodes (unseen in the training pool): the bare policy, the harness after iteration 1 (a single-attempt agent design) and the final iteratively refined harness.
More in this topic
Agent Harness Optimization
Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.
See all 197 papers