From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

Tingwen Du; Haoyu Guo; Hanxue Zhang; Xuanxu Wang; Junting Dong; Chao Gao; Mingfei Han; Tao Lu

cs.ROOct 21, 2026

From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

Authors: Tingwen Du, Haoyu Guo, Hanxue Zhang, Xuanxu Wang, Junting Dong, Chao Gao, Mingfei Han, Tao Lu,

Organizations: University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Mohamed bin Zayed University of Artificial Intelligence ·

Abstract

Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.

Figures & tables

Figure 1: From planner to agent for history-dependent manipulation. Top: Similar observations can require different decisions depending on task progress. (a) Memory-augmented policies equip VLA models with memory mechanisms to retain and use historical information. (b) Hierarchical planning separates high-level planning from low-level execution. (c) SERA uses an editable harness to organize the VLM as an agent that maintains task context from observation and interaction history. Execution-driven self-evolution refines the harness using rollout feedback. The dashed arrow denotes the selected harness, fixed before testing.

Figure 2: Agentic execution and harness self-evolution. (a) The harness maintains task context and uses a VLM to plan commands for a VLA executor, incorporating execution feedback in subsequent rounds. (b) Execution traces guide revisions to planning guidance, assessment criteria, and control rules. A verifier evaluates candidates and informs further revision and selection; the selected harness is fixed before testing. Tokens denote information categories and image stacks show observation sequences.

VLA baselines
Method	Counting	Permanence	Reference	Imitation	Average $\uparrow$
$\pi_{0.5}$	28.78	17.00	17.17	8.78	17.93
$\pi_{0.5}$ w/ action history	29.09	22.75	14.59	11.17	19.40
Memory-augmented policies
SAM2Act	35.33	26.00	16.83	7.33	21.37
FrameSamp+Modul	65.22	25.11	36.33	51.39	44.51

Table 1: RoboMME main results. Task-averaged success rates (%; higher is better). Memory-augmented policies equip action models with mechanisms to retain and use history; hierarchical policies separate subgoal planning from low-level execution, including systems that also use memory. SERA is grouped with hierarchical policies. Baselines use official leaderboard results; SERA uses autonomous selection. Bold and underlined values denote the best and second-best scores across all listed methods.

Figure 3: Self-evolution results and complementary execution cases. (a) RoboMME success rates before and after harness revision. (b) VideoPlaceOrder: the base agent analyzes a demonstration to identify the intended target despite a swap, then uses retained task context to predict its current location and successfully place the cube. (c) PatternLock: self-evolution diagnoses direction-mapping errors and missing action candidates, revises the direction guidance and candidate set, and the revised agent successfully completes the sequence. VideoPlaceOrder shows a test episode; PatternLock shows a self-evolution repair example. Direction guides and candidate colors are schematic; paths show motion and target order.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Suite	Task	MemER (reproduced)	SERA
Counting	BinFill	30/50	24/50
Counting	PickXtimes	41/50	35/50
Counting	SwingXtimes	36/50	36/50
Counting	StopCube	2/50	3/50
Permanence	VideoUnmask	43/50	48/50
Permanence	VideoUnmaskSwap	18/50	46/50

Appendix

Table 2: Per-task RoboMME results. Successful episodes out of 50 per task. MemER is our local reproduction using released checkpoints, distinct from the public leaderboard results in Table 1 . SERA uses autonomous harness selection. Bold denotes the higher count in each row, including ties.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models
May 13, 2026Zixing Lei, Changxing Liu, Yichen Xiong +5Long-Horizon Robotic Manipulation Tool-Using Vision-Language Agents
Shanghai Jiao Tong University · Zhongguancun Academy · Beihang University
AdaHVLA: Adaptive Harnesses for Long-Horizon Vision-Language-Action Execution
Sep 24, 2026Junyi Tang, Jie Peng, Zezhen Ding +2Long-Horizon Robotic Manipulation Agent Harness Optimization
UNC · USTC · HKUST +1
Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
Tsinghua University · Striding AI · Purdue University +4

May 13, 2026J/K move · Enter open · S save

Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models

Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong+4

Shanghai Jiao Tong University · Zhongguancun Academy · Beihang University

Long-Horizon Robotic Manipulation Tool-Using Vision-Language Agents Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models are effective robot action executors, but they remain limited on long-horizon tasks due to the dual burden of extended closed-loop planning and diverse physical operations. We therefore propose VLAs-as-Tools, a strategy that distributes this burden across a high-level vision language model (VLM) agent for temporal reasoning and a family of specialized VLA tools for diverse local physical operations. The VLM handles scene analysis, global planning, and recovery, while each VLA tool executes a bounded subtask. To tightly couple agent planning with VLA tool execution in long-horizon tasks, we introduce a VLA tool-family interface that exposes explicit tool selection and in-execution progress feedback, enabling efficient event-triggered agent replanning without continuous agent polling. To obtain diverse specialized VLA tools that faithfully follow agent invocations, we further propose Tool-Aligned Post-Training (TAPT), which constructs invocation-aligned training units for instruction following and adopts tool-family residual adapters for efficient tool specialization. Across five simulated benchmarks, VLAs-as-Tools improves the success rate of pi05 by 8.9 percentage points on average and enhances invocation fidelity by 15.0 points in Non-biased Rate. On three real-world Franka tasks, it improves success rate from 16.7% to 58.3% under matched data and compute. Code will be released.

Abstract

Robotic Manipulation

Explore similar work

Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models

AdaHVLA: Adaptive Harnesses for Long-Horizon Vision-Language-Action Execution

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents