Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection

Zhen Liu; Xinyu Ning; Zhe Hu; Xinxin Xie; Weize Li; Zhipeng Tang; Chongyu Wang; Zejun Yang

cs.ROApr 15, 2026

Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection

Authors: Zhen Liu, Xinyu Ning, Zhe Hu, Xinxin Xie, Weize Li, Zhipeng Tang, Chongyu Wang, Zejun Yang, +3 more

Organizations: Beijing University of Posts and Telecommunications · InspireOmni AI · Tsinghua University

Abstract

Recent vision-language-action (VLA) systems have demonstrated strong capabilities in embodied manipulation. However, most existing VLA policies rely on limited observation windows and end-to-end action prediction, which makes them brittle in long-horizon, memory-dependent tasks with partial observability, occlusions, and multi-stage dependencies. Such tasks require not only precise visuomotor control, but also persistent memory, adaptive task decomposition, and explicit recovery from execution failures. To address these limitations, we propose a dual-system framework for long-horizon embodied manipulation. Our framework explicitly separates high-level semantic reasoning from low-level motor execution. A high-level planner, implemented as a VLM-based agentic module, maintains structured task memory and performs goal decomposition, outcome verification, and error-driven correction. A low-level executor, instantiated as a VLA-based visuomotor controller, carries out each sub-task through diffusion-based action generation conditioned on geometry-preserving filtered observations. Together, the two systems form a closed loop between planning and execution, enabling memory-aware reasoning, adaptive replanning, and robust online recovery. Experiments on representative RMBench tasks show that the proposed framework substantially outperforms representative baselines, achieving a 32.4% average success rate compared with 9.8% for the strongest baseline. Ablation studies further confirm the importance of structured memory and closed-loop recovery for long-horizon manipulation.

Figures & tables

Figure 1: A long-horizon manipulation example. The robot moves a Coke can to a scanner, but the first scan fails because the barcode is not visible. Reflection diagnoses a wrong orientation, and the remaining plan is rewritten to rotate and align the can before rescanning.

Figure 2: Overview of Goal2Skill. The high-level system integrates adaptive planning, proactive memory, progress verification, and reflective recovery. The low-level executor performs task-conditioned visual grounding and generates action chunks, returning execution evidence for verification and memory updates. The verifier guides task progression by determining when to continue execution, advance to the next sub-task, or trigger corrective reasoning. This closed loop supports targeted plan correction during long-horizon manipulation.

$\bullet$ Memory-Augmented VLA Policies:
Method	Pick Xtimes	Button UnmaskSwap	Video Repick	Video PlaceOrder	Pick Highlight	Video PlaceButton	Average
$\bullet$ Base Policies:
$\pi_{0.5}$	42.0	6.0	0.0	26.0	12.0	32.0	19.7
$\pi_{0.5}$ w/ past actions	58.0	16.0	8.0	18.0	12.0	24.0	22.7
MME-VLA (FrameSamp)	82.0	12.0	32.0	28.0	24.0	54.0	38.7
MemER	78.0	24.0	32.0	36.0	82.0	16.0	44.7

Table 1: Success rates (%) on RoboMME tasks. Average denotes the average over the six selected tasks. Best and second-best results among non-oracle methods are shown in bold and underline , respectively. Complete results on all 16 tasks are reported in Appendix Table 7 . All RoboMME baselines are reproduced using the officially released checkpoints.

Figure 3: Success rates by difficulty, averaged over all 16 RoboMME tasks.

Method	M(1) Tasks		M(n) Tasks			Average
Method	Observe and Pick Up	Rearrange Blocks	Battery Try	Blocks Ranking Try	Press Button	M(1)	M(n)	Total
$\bullet$ Base Policies:
DP	1.0	0.0	10.0	10.0	0.0	0.5	6.7	4.2
ACT	1.0	29.0	19.0	0.0	0.0	15.0	6.3	9.8
$\pi_{0.5}$	9.0	13.0	16.0	6.0	0.0	11.0	7.3	8.8
X-VLA	9.0	13.0	26.0	1.0	0.0	11.0	9.0	9.8

Table 2: Detailed results on five RMBench tasks. We report success rates (%) together with the corresponding group-wise and overall averages. Baseline results are taken from Yang et al. (2026a) . To better isolate the contribution of high-level reasoning and memory, we only compare with methods that use standard low-level policies without specific architectural modifications.

Figure 4: A case indicating how the verification-driven skill evolution improves performance for swing counting.

Figure 5: Reflection-based replanning after a verification failure.

Hyperparameter	Value
Base policy	$\pi_{0.5}$
Policy configuration	LIBERO
Number of training tasks	16
Demonstrations per task	100
Masked demonstrations per task	20
Unmasked demonstrations per task	80

Table 3: Training configuration of the low-level policy on RoboMME. We follow the original RoboMME training recipe and incorporate task-conditioned masked demonstrations into the joint multi-task training data.

Hyperparameter	Value
Base policy	$\pi_{0.5}$
Initialization	$\pi_{0.5}$ base checkpoint
Demonstrations per task	50
Masked demonstrations per task	10 (20%)
Unmasked demonstrations per task	40 (80%)
Batch size	64

Table 4: Training configuration of the low-level policy on RMBench. We follow the official

\pi_{0.5}

training configuration and incorporate task-conditioned masked demonstrations into the training data.

Figure 6: Reflection-driven recovery on ButtonUnmask (hard) from RoboMME. The initial plan terminates while the container covering the blue cube remains unopened. Reflection identifies the inconsistent completion record and revises the remaining execution.

Figure 7: Reflection-driven correction on PatternLock (easy) from RoboMME. Reflection detects that the demonstrated trajectory remains incomplete after the initial plan terminates and replans the missing motion segment.

Figure 8: Reflection-driven recovery on VideoUnmask (medium) from RoboMME. A nominally completed container-lifting stage is invalidated by the unsuccessful terminal task state, triggering replanning until the target cube is uncovered.

Figure 9: Verification-driven progression on VideoPlaceOrder (medium) from RoboMME. The verifier compares stage-level execution evidence and confirms that the blue cube reaches the required target location before advancing the plan.

Figure 10: Verification- and reflection-guided plan expansion on PickHighlight (easy) from RoboMME. Verification detects the scene transition induced by the button press, after which reflection extends the plan to manipulate the newly highlighted cubes.

Figure 11: Verification-driven stage transition on VideoPlaceButton (easy) from RoboMME. The verifier confirms that the green cube has been securely grasped before execution proceeds to the placement stage.

Figure 12: Sample trajectories of our framework on RMBench tasks. From top to bottom, the task sequences correspond to Observe and Pick Up, Rearrange Blocks, Battery Try, Blocks Ranking Try, and Press Button. Each row shows key frames from one rollout and illustrates that the method can complete representative memory-dependent manipulation tasks with different sequential interaction requirements.

Setting	Observe and Pick Up	Rearrange Blocks	Blocks Ranking Try	Avg.
Goal2Skill	8%	38%	60%	35.3%
$-\mathcal{E}_{t}$	6%	36%	42%	28.0%
$-\mathcal{E}_{t}-\mathcal{W}_{t}$	8%	21%	54%	27.7%
$-\mathcal{E}_{t}-\mathcal{W}_{t}-\mathcal{H}_{t}$	4%	10%	6%	6.7%

Table 5: Memory ablation on RMBench. We progressively remove components of structured memory to evaluate their contributions to tasks requiring target retention, intermediate state tracking, or multi-stage bookkeeping.

Setting	Battery Try	Press Button	Avg.
Goal2Skill	46%	10%	28.0%
$-\mathrm{Reflection}$	30%	5%	17.5%

Table 6: Reflection ablation on RMBench.

Base Policies
Method	Counting				Permanence				Reference				Imitation				Overall
Method	Bin Fill	Pick Xtimes	Swing Xtimes	Stop Cube	Video Unmask	Video Unmask Swap	Button Unmask	Button Unmask Swap	Pick Highlight	Video Repick	Video Place Button	Video Place Order	Move Cube	Insert Peg	Pattern Lock	Route Stick	Overall
$\pi_{0.5}$	30.0	42.0	36.0	6.0	20.0	18.0	22.0	6.0	12.0	0.0	32.0	26.0	26.0	2.0	2.0	4.0	17.8
$\pi_{0.5}$ w/ past actions	26.0	58.0	26.0	4.0	30.0	20.0	24.0	16.0	12.0	8.0	24.0	18.0	34.0	2.0	4.0	6.0	19.5
Memory-Augmented VLA Policies
MME-VLA (FrameSamp)	38.0	82.0	94.0	50.0	38.0	28.0	26.0	12.0	24.0	32.0	54.0	28.0	48.0	2.0	54.0	66.0	42.3

Table 7: Complete per-task success rates (%) on all 16 RoboMME tasks ( Dai et al., 2026 ) . Best and second-best results among non-oracle methods are shown in bold and underline , respectively. GroundSG+Oracle is included as an oracle reference.

Base Policies
Method	Difficulty	Counting				Permanence
Method	Difficulty	Bin Fill	Pick Xtimes	Swing Xtimes	Stop Cube	Video Unmask	Video Unmask Swap	Button Unmask	Button Unmask Swap
$\pi_{0.5}$	Easy	38.5	46.2	34.6	7.7	23.1	19.2	23.1	7.7
	Medium	25.0	41.7	41.7	8.3	16.7	16.7	25.0	8.3
	Hard	16.7	33.3	33.3	0.0	16.7	16.7	16.7	0.0
$\pi_{0.5}$ w/ past actions	Easy	34.6	61.5	26.9	3.8	34.6	19.2	26.9	19.2

Table 8: Difficulty-stratified evaluation on the Counting and Permanence tasks of RoboMME ( Dai et al., 2026 ) . We report success rates (%) for Easy, Medium, and Hard episodes. Each task contains 50 test episodes, including 26 Easy, 12 Medium, and 12 Hard episodes. GroundSG+Oracle is included as an oracle reference.

Base Policies
Method	Difficulty	Reference				Imitation
Method	Difficulty	Pick Highlight	Video Repick	Video Place Button	Video Place Order	Move Cube	Insert Peg	Pattern Lock	Route Stick
$\pi_{0.5}$	Easy	15.4	0.0	34.6	26.9	26.9	3.8	3.8	7.7
	Medium	8.3	0.0	33.3	25.0	25.0	0.0	0.0	0.0
	Hard	8.3	0.0	25.0	25.0	25.0	0.0	0.0	0.0
$\pi_{0.5}$ w/ past actions	Easy	15.4	7.7	26.9	19.2	34.6	3.8	3.8	7.7

Table 9: Difficulty-stratified evaluation on the Reference and Imitation tasks of RoboMME ( Dai et al., 2026 ) . We report success rates (%) for Easy, Medium, and Hard episodes. Each task contains 50 test episodes, including 26 Easy, 12 Medium, and 12 Hard episodes. GroundSG+Oracle is included as an oracle reference.

Explore similar work

HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control
Jul 3, 2026Li Ji, Siyin Wang, Pengfang Qian +5Long-Horizon Robotic Manipulation Vision-Language-Action Models
1Fudan University · 2Shanghai Innovation Institute · 3East China Normal University
Long-Horizon Manipulation via Trace-Conditioned VLA Planning
Apr 23, 2026Isabella Liu, An-Chieh Cheng, Rui Yan +7Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
University of California, San Diego · NVIDIA
HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation
Apr 20, 2026Zijian Zeng, Fei Ding, Huiming Yang +1Long-Horizon Robotic Manipulation Robot Failure Recovery
Tsinghua University · Alibaba Group · Bengbu University

Jul 3, 2026J/K move · Enter open · S save

HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control

Li Ji, Siyin Wang, Pengfang Qian, Xiaopeng Yu+4

1Fudan University · 2Shanghai Innovation Institute · 3East China Normal University

Long-Horizon Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents

Open paper AskPDF arXiv

Current Vision-Language-Action (VLA) models excel at robotic manipulation but often struggle with non-Markovian tasks requiring long-term memory and reasoning due to their reliance on immediate observations. Existing solutions face a ''frequency-competence paradox,'' where stronger reasoning models are too slow for real-time control, while faster models lack sufficient reasoning capabilities. To resolve this architectural misalignment, we propose HiMe, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy. We also introduce a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms, allowing robots to maintain memory plasticity through ''Add, Update, and Delete'' operations. This hierarchical design effectively balances the conflict between real-time execution and slow thinking planning, significantly improving success rates in long-horizon tasks. Experiments demonstrate that this approach not only outperforms flat memory baselines but also exhibits the novel ability to self-correct its internal knowledge based on human preferences.

Abstract

Explore similar work

HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control

Long-Horizon Manipulation via Trace-Conditioned VLA Planning

HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation