cs.ROOct 30, 2026

RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory

Authors: Shengguang Wu, Suning Huang, Cherie Ho, Ria Doshi, Hao Zhu, Serena Yeung-Levy

Organizations: Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.

Figures & tables

Figure 1: Wrapping a frozen policy in an evolved harness raises its held-out success rate from 9.7% to 43.3% on unseen scenes and from 12.7% to 40.2% on unseen tasks, often past the same policy finetuned on the benchmark ( Section 4.2 ). All scores are evaluated on the held-out episodes never seen during the harness’s optimization. Unseen scenes: held-out episodes evaluate the same tasks under varied scene arrangements. Unseen tasks: held-out episodes evaluate distinct tasks to test cross-task skill transfer ( Section 4.1 ).
Figure 2: RoboArchitect turns a frozen policy into a performant robotic agent without updating its weights: a coding model iteratively refines the harness from execution traces. Starting from a bare policy (VLA), RoboArchitect evolves a robotic agent equipped with auxiliary tools and persistent memory. The right side illustrates experiment conditions ( Section 4.1 ) that test how well the evolved harness generalizes.
   Unseen Scenes (%)Unseen Tasks (%)
   BareHarness (ours)Δ\DeltaBareHarness (ours)Δ\Delta
BenchmarkPolicyTrained OnPolicyOne-ShotIterativePolicyOne-ShotIterative
LIBERO-Pro-10MolmoAct-2LIBERO10.012.032.0+22.02.52.515.0+12.5
 OpenVLA-7BLIBERO-100.02.026.0+26.00.00.022.5+22.5
 π0.5\pi_{0.5}LIBERO8.04.024.0+16.00.00.02.5+2.5
 RLinf- π0.5\pi_{0.5}LIBERO-1302.02.040.0+38.00.00.010.0+10.0
Table 1: RoboArchitect ’s iteratively evolved harness raises mean held-out success by +345% on unseen scenes and +216% on unseen tasks. Bare Policy : the VLA policy alone. Harness (ours) : the same frozen policy inside the harness RoboArchitect designed, after a single design attempt ( One-Shot , iteration 1) or after iterative refinement over up to 5 iterations ( Iterative ). Δ\Delta : gain of the iterative harness over the bare policy. Rows marked ↪\hookrightarrow show the finetuned version of the same policy evaluated on the same held-out set. Training-source distance between the policy and the evaluation benchmark is color-coded: black = in-distribution; gold = spatially swapped; orange = cross-benchmark; dark red = out-of-domain. All scores are success rates in %.
   Training Pool (%) procedurally generatedHeld-Out (%) standard benchmark episodes
   BareHarness (ours)Δ\DeltaBareHarness (ours)Δ\Delta
PolicyTrained OnBenchmarkPolicyOne-ShotIterativePolicyOne-ShotIterative
General X-VLAgeneral robot dataLIBERO-Pro0.03.529.4+29.40.13.524.1+24.0
VLABench7.114.817.5+10.47.517.321.6+14.2
SimplerEnv WidowX0.065.868.3+68.30.045.824.0+24.0
SimplerEnv Google0.022.051.2+51.20.020.045.6+45.6
Table 2: A unified evolved harness serves different benchmarks: around the general X-VLA it substantially raises held-out success, and around the finetuned X-VLA specialists it preserves their competence. Columns and labels read as in Table 1 .
Figure 3: Evolved control routines fix concrete failure patterns. Each row shows an episode under an earlier harness (starting with bare VLA policy) on the left, and under the revised RoboArchitect harness on the right. Row (a) shows detecting and clearing arm jams, (b) synthesizing missing motion primitives, (c) rescaling contact forces, and (d) redirecting to target objects instead of memorized locations. Fig. 5 shows more examples. See discussions on these revisions in Section 4.3 and Appendix A .
Figure 4: Persistent memories emerge from harness evolution and inform action decisions. Each row illustrates one form of memory mechanism; rows (a) and (b) show reusable execution experience, (c) task and environment context, and (d) embodiment-specific calibration. See discussions on these memories in Section 4.4 and Appendix A .
SettingMemory EntriesReads per EpisodeActionable Reads
X-VLA on VLABenchfinger depths that have been lifting objects3.498%
 
X-VLA on SimplerEnv WidowXheight at which to close the fingers, per measured object height3.596%
 
MolmoAct-2 on LIBERO-Pro-10average depth of successful grasps below an object’s top9.999%
 
Table 3: Auto-designed persistent memories are frequently accessed and yield highly useful information to guide later actions. Reads per Episode : average frequency of memory reads in held-out episodes. Actionable Reads : the share of these reads that find stored info.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: More evolved control routines. Rows read as in Fig. 3 . Row (e) shows synthesizing missing motion primitives, and (f) tethering mobile bases to active workspaces.
Figure 6: Evolved harnesses also enhance the robotic agent’s perception precision and reliability. Panels read as in Fig. 3 , discussions in Section 4.3 . Rows (g) and (h) show adapting to perception limitations, and (i) replacing an unusable policy with a code-as-policy agent ( Section 4.2 ).
Figure 7: The unified harness learns to take over the arm command from a failing policy and to leave a progressing one alone, and calibrates itself to each robot’s configurations. Each row shows an episode under the harness before (left) and after (right) an iteration’s revision.
Figure 8: Held-out episodes differ from the optimization episodes either in scene arrangement ( unseen scenes ) or in the task itself ( unseen tasks ) ( Section 4.1 ). Left, unseen scenes: the same tasks with scene configurations dealt out alternately. Right, unseen tasks: different tasks that require similar underlying skills.
BenchmarkTrialsOptimization TasksHeld-Out Tasks
LIBERO-Pro-1010turn on the stove and put the moka pot on it ; put the black bowl in the bottom drawer of the cabinet and close it ; put both the alphabet soup and the tomato sauce in the basket ; put both the cream cheese box and the butter in the basket ; put the white mug on the left plate and put the yellow and white mug on the right plateput the yellow and white mug in the microwave and close it ; put both moka pots on the stove ; put both the alphabet soup and the cream cheese box in the basket ; put the white mug on the plate and put the chocolate pudding to the right of the plate
RoboCerebra10cream cheese, popcorn and butter, each into a white storage box; cream cheese into a white storage box, then the frypan onto the rack; plate, popcorn and cookies rearranged on the dining set, then wine poured onto the plate; chocolate pudding from the cabinet top onto the plate, then wine poured onto the plate; chocolate pudding and butter into a black bowl, the stove turned on, then the frypan onto the stovecookies into a basket, tomato sauce and bbq sauce poured onto the frypan, wine poured into a mug, then another mug into a wooden tray; cream cheese and popcorn into a white bowl, then the frypan onto the cabinet top; the short cabinet opened, butter into the cabinet, then tomato sauce poured onto the plate; the short cabinet opened, cream cheese into the cabinet, chocolate pudding into a black bowl, then wine poured into a mug; white bowl from the cabinet top, cookies and chocolate pudding into the bowl, the microwave opened, then the bowl into the microwave
VLABench10select the named fruit; select the named mahjong tile; select the named poker cardselect the named drink out of a fridge; select the nn -th largest poker card; select the mahjong tile of the unique type
SimplerEnv WidowX24put the spoon on the towel ; stack the green block on the yellow blockput carrot on plate ; put eggplant into yellow basket
RoboCasa-36550close the toaster oven door; open the cabinet door; open the drawer; pick an object from the counter and place it in the cabinet; press the start button on the microwaveclose the fridge door; pick an object from the plate and place it in the pan on the stove; fully slide the top dishwasher rack out; press down the lever to turn on the electric kettle
Appendix
Table 4: Optimization and held-out tasks under the unseen tasks protocol. On RoboCerebra a task is a chain of sub-task instructions stated here in brief. On VLABench and RoboCasa-365 the instruction of a task changes with the sampled objects. Trials is the number of trials ( i.e., episodes) per task.
Figure 10: Under unseen scenes , the harness improves over iterations on the training set, and the final harness outperforms the one-shot harness on nearly every held-out set. Left of the dotted line is the training episode set, iteration by iteration; iteration 0 is the bare VLA policy, filled markers denote accepted harness revisions that improved task success over prior iterations, and open markers denote revisions that were not accepted. Right of the dotted line are the scores on the held-out set: the bare policy, the harness after iteration 1 (a single-attempt agent design) and the final iteratively refined harness. The held-out set evaluates identical tasks under different scene arrangements ( Section 4.1 ).
Figure 11: Under unseen tasks , the harness improves over iterations on the training set, and the final harness outperforms the bare VLA policy and the one-shot harness on every held-out set. Labels and panels read as in Fig. 10 . A cross marks a revision that could not be evaluated; when this is the first revision, no single-attempt harness is available for held-out evaluation. The held-out set evaluates unseen tasks requiring shared underlying skills ( Section 4.1 ).
Figure 12: Around the general X-VLA, the unified harness improves on every benchmark of the training pool over iterations; around the finetuned specialists, it leaves their already competent scores intact. Left column, using the general X-VLA checkpoint as policy; right column, using benchmark-finetuned X-VLA specialists as policy. The first row is the mean over the benchmarks and each later row is one benchmark. Left of the dotted line shows the performance on the procedurally generated training pool, iteration by iteration (iteration 0 is the bare policy; filled markers denote accepted harness revisions). Right of it are the scores on the benchmark’s standard episodes (unseen in the training pool): the bare policy, the harness after iteration 1 (a single-attempt agent design) and the final iteratively refined harness.
More in this topic

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 197 papers

Explore similar work

CardsList
Oct 7, 2026

EmbodiedRSI: Active Continual Robot Learning Through Hypothesis-Guided Co-Evolution

Robot foundation models provide strong visuomotor control, yet their performance can degrade when object positions or task instructions change. Further improvements often require post-training on substantial robot data, which can be costly to collect through methods such as teleoperation. Agentic harnesses can adapt around the model, but current self-evolving harnesses use robot trials inefficiently when deciding which code and skill changes to pursue. We introduce EmbodiedRSI, a self-evolving agentic harness that autonomously decides where to explore next and turns the resulting physical interaction into improved code and skills. EmbodiedRSI realizes this through a Fast-Slow Dual-System Architecture, in which competing code and skill hypotheses are maintained in a Hypothesis Graph. Value-of-Information Experiment Selection chooses physical experiments that can distinguish these hypotheses. Their outcomes guide Code-Skill Co-Evolution. The Slow System builds Hierarchical Memory, and Reward-Grounded Memory Learning selects effective memory according to their value for later Fast-System improvement. On RoboCasa365, EmbodiedRSI reaches 77.0% overall success and 71.3% on Composite-Unseen, compared with 40.1% for the best baseline. EmbodiedRSI also reaches 86.8% overall success on LIBERO-Pro. Beyond benchmark performance, EmbodiedRSI transfers zero-shot to real-world robot, achieving 71.3% overall success across multiple challenging tasks.

Python Song, Zhixuan Liang, Kelsey Fu +3

Columbia University · Princeton University

Continual Robot LearningRobot Skill LearningSelf-Evolving AgentsActive LearningRobot Foundation Models
AskPDF
Sep 27, 2026

Recursive Harness Distillation across Agents for Robot Manipulation

A central goal in robotics is to enable manipulation across changing tasks and environments. Vision-language-action (VLA) models provide broad manipulation capabilities but can struggle when execution requires diagnosing failures and adapting behavior. Strong agents can discover effective interventions through interaction with these policies. We propose Recursive Harness Distillation to accumulate this experience as reusable guidance across agents. A strong agent distills its experience into a playbook for a light agent, then recursively refines the playbook using the light agent's execution feedback. The resulting playbook enables agents to reuse accumulated intervention knowledge in new task instances without updating model parameters. In real-world manipulation, the harness improves success from 37.3% to 64.0%. On SimplerEnv Bridge, the light agent with the playbook achieves 66.7% success, compared with 41.7% for the GR00T-only baseline, and outperforms the strong agent without a playbook. The same playbook also benefits the strong agent, which reaches 79.2% success. These results demonstrate the feasibility of harness distillation for robotics: intervention experience can be accumulated, refined through execution, and reused across agents to improve manipulation.

Seungyeon Kim, Junhoo Lee, Minkyu Kim +2

Seoul National University · Korea Advanced Institute of Science and Technology

Robotic ManipulationRobot Policy GeneralizationTest-Time Adaptation of VLA ModelsVLMs for Robotics
AskPDF
Jun 15, 2026

RHO: Your Coding Agent is Secretly a Roboticist

Code-as-Policies (CaP) has shown that large language models (LLMs) can write code to solve robotics tasks by composing perception, planning, and control primitives. Recent CaP systems, however, rely on multi-turn code-generation loops at test time, which is often infeasible for real-time robot control. We introduce Robotics Harness Optimization (RHO), a novel paradigm in which tool-enabled coding agents, at training time, propose and search for interpretable, neurosymbolic multi-file policy repositories (Repositories-as-Policies) that compose these primitives rather than a single prompt, function, or file. RHO searches with reflective feedback from environment reward and execution rather than teleoperation demonstrations. It generalizes to perturbed pick-and-place settings like LIBERO-PRO, where OpenVLA scores 0.0% and π0.5π_{0.5} averages 12.83%. Using the same low-level primitives, RHO reaches a 45.0% success rate, 2.5x higher than the strongest multi-turn agentic system, and 3.5x higher than π0.5π_{0.5}. On Robosuite, RHO sets a new state-of-the-art of 70.0%, exceeding the prior multi-turn record of 68.29% using single-turn execution with no corrective LLM code edits at deployment. When an LLM is used in the control loop, as on RAI's O3DE benchmark, RHO optimizes the deployed agent's multi-file harness of prompts, tools, and control code, improving held-out success from 23.5% to 44.3% with 20% less wall-clock time and 27% fewer tool calls.

Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3

University of California, Berkeley · Embodied Science · AMD

LLM Agent HarnessesAI ControlRobotic ManipulationRobot Policy LearningAI Coding AgentsLarge Language Model-Based Robot Planning+5
AskPDF