Recent vision-language-action (VLA) systems have demonstrated strong capabilities in embodied manipulation. However, most existing VLA policies rely on limited observation windows and end-to-end action prediction, which makes them brittle in long-horizon, memory-dependent tasks with partial observability, occlusions, and multi-stage dependencies. Such tasks require not only precise visuomotor control, but also persistent memory, adaptive task decomposition, and explicit recovery from execution failures. To address these limitations, we propose a dual-system framework for long-horizon embodied manipulation. Our framework explicitly separates high-level semantic reasoning from low-level motor execution. A high-level planner, implemented as a VLM-based agentic module, maintains structured task memory and performs goal decomposition, outcome verification, and error-driven correction. A low-level executor, instantiated as a VLA-based visuomotor controller, carries out each sub-task through diffusion-based action generation conditioned on geometry-preserving filtered observations. Together, the two systems form a closed loop between planning and execution, enabling memory-aware reasoning, adaptive replanning, and robust online recovery. Experiments on representative RMBench tasks show that the proposed framework substantially outperforms representative baselines, achieving a 32.4% average success rate compared with 9.8% for the strongest baseline. Ablation studies further confirm the importance of structured memory and closed-loop recovery for long-horizon manipulation.
Figures & tables
Figure 1: A long-horizon manipulation example. The robot moves a Coke can to a scanner, but the first scan fails because the barcode is not visible. Reflection diagnoses a wrong orientation, and the remaining plan is rewritten to rotate and align the can before rescanning.
Figure 2: Overview of Goal2Skill. The high-level system integrates adaptive planning, proactive memory, progress verification, and reflective recovery. The low-level executor performs task-conditioned visual grounding and generates action chunks, returning execution evidence for verification and memory updates. The verifier guides task progression by determining when to continue execution, advance to the next sub-task, or trigger corrective reasoning. This closed loop supports targeted plan correction during long-horizon manipulation.
Method
Pick Xtimes
Button UnmaskSwap
Video Repick
Video PlaceOrder
Pick Highlight
Video PlaceButton
Average
∙ Base Policies:
π0.5
42.0
6.0
0.0
26.0
12.0
32.0
19.7
π0.5 w/ past actions
58.0
16.0
8.0
18.0
12.0
24.0
22.7
∙ Memory-Augmented VLA Policies:
MME-VLA (FrameSamp)
82.0
12.0
32.0
28.0
24.0
54.0
38.7
MemER
78.0
24.0
32.0
36.0
82.0
16.0
44.7
Table 1: Success rates (%) on RoboMME tasks. Average denotes the average over the six selected tasks. Best and second-best results among non-oracle methods are shown in bold and underline , respectively. Complete results on all 16 tasks are reported in Appendix Table 7 . All RoboMME baselines are reproduced using the officially released checkpoints.
Figure 3: Success rates by difficulty, averaged over all 16 RoboMME tasks.
Method
M(1) Tasks
M(n) Tasks
Average
Observe and Pick Up
Rearrange Blocks
Battery Try
Blocks Ranking Try
Press Button
M(1)
M(n)
Total
∙ Base Policies:
DP
1.0
0.0
10.0
10.0
0.0
0.5
6.7
4.2
ACT
1.0
29.0
19.0
0.0
0.0
15.0
6.3
9.8
π0.5
9.0
13.0
16.0
6.0
0.0
11.0
7.3
8.8
X-VLA
9.0
13.0
26.0
1.0
0.0
11.0
9.0
9.8
Table 2: Detailed results on five RMBench tasks. We report success rates (%) together with the corresponding group-wise and overall averages. Baseline results are taken from Yang et al. (2026a) . To better isolate the contribution of high-level reasoning and memory, we only compare with methods that use standard low-level policies without specific architectural modifications.
Figure 4: A case indicating how the verification-driven skill evolution improves performance for swing counting.
Figure 5: Reflection-based replanning after a verification failure.
Hyperparameter
Value
Base policy
π0.5
Policy configuration
LIBERO
Number of training tasks
16
Demonstrations per task
100
Masked demonstrations per task
20
Unmasked demonstrations per task
80
Table 3: Training configuration of the low-level policy on RoboMME. We follow the original RoboMME training recipe and incorporate task-conditioned masked demonstrations into the joint multi-task training data.
Hyperparameter
Value
Base policy
π0.5
Initialization
π0.5 base checkpoint
Demonstrations per task
50
Masked demonstrations per task
10 (20%)
Unmasked demonstrations per task
40 (80%)
Batch size
64
Table 4: Training configuration of the low-level policy on RMBench. We follow the official π0.5 training configuration and incorporate task-conditioned masked demonstrations into the training data.
Figure 6: Reflection-driven recovery on ButtonUnmask (hard) from RoboMME. The initial plan terminates while the container covering the blue cube remains unopened. Reflection identifies the inconsistent completion record and revises the remaining execution.
Figure 7: Reflection-driven correction on PatternLock (easy) from RoboMME. Reflection detects that the demonstrated trajectory remains incomplete after the initial plan terminates and replans the missing motion segment.
Figure 8: Reflection-driven recovery on VideoUnmask (medium) from RoboMME. A nominally completed container-lifting stage is invalidated by the unsuccessful terminal task state, triggering replanning until the target cube is uncovered.
Figure 9: Verification-driven progression on VideoPlaceOrder (medium) from RoboMME. The verifier compares stage-level execution evidence and confirms that the blue cube reaches the required target location before advancing the plan.
Figure 10: Verification- and reflection-guided plan expansion on PickHighlight (easy) from RoboMME. Verification detects the scene transition induced by the button press, after which reflection extends the plan to manipulate the newly highlighted cubes.
Figure 11: Verification-driven stage transition on VideoPlaceButton (easy) from RoboMME. The verifier confirms that the green cube has been securely grasped before execution proceeds to the placement stage.
Figure 12: Sample trajectories of our framework on RMBench tasks. From top to bottom, the task sequences correspond to Observe and Pick Up, Rearrange Blocks, Battery Try, Blocks Ranking Try, and Press Button. Each row shows key frames from one rollout and illustrates that the method can complete representative memory-dependent manipulation tasks with different sequential interaction requirements.
Setting
Observe and Pick Up
Rearrange Blocks
Blocks Ranking Try
Avg.
Goal2Skill
8%
38%
60%
35.3%
−Et
6%
36%
42%
28.0%
−Et−Wt
8%
21%
54%
27.7%
−Et−Wt−Ht
4%
10%
6%
6.7%
Table 5: Memory ablation on RMBench. We progressively remove components of structured memory to evaluate their contributions to tasks requiring target retention, intermediate state tracking, or multi-stage bookkeeping.
Setting
Battery Try
Press Button
Avg.
Goal2Skill
46%
10%
28.0%
−Reflection
30%
5%
17.5%
Table 6: Reflection ablation on RMBench.
Method
Counting
Permanence
Reference
Imitation
Overall
Bin Fill
Pick Xtimes
Swing Xtimes
Stop Cube
Video Unmask
Video Unmask Swap
Button Unmask
Button Unmask Swap
Pick Highlight
Video Repick
Video Place Button
Video Place Order
Move Cube
Insert Peg
Pattern Lock
Route Stick
Base Policies
π0.5
30.0
42.0
36.0
6.0
20.0
18.0
22.0
6.0
12.0
0.0
32.0
26.0
26.0
2.0
2.0
4.0
17.8
π0.5 w/ past actions
26.0
58.0
26.0
4.0
30.0
20.0
24.0
16.0
12.0
8.0
24.0
18.0
34.0
2.0
4.0
6.0
19.5
Memory-Augmented VLA Policies
MME-VLA (FrameSamp)
38.0
82.0
94.0
50.0
38.0
28.0
26.0
12.0
24.0
32.0
54.0
28.0
48.0
2.0
54.0
66.0
42.3
Table 7: Complete per-task success rates (%) on all 16 RoboMME tasks ( Dai et al., 2026 ) . Best and second-best results among non-oracle methods are shown in bold and underline , respectively. GroundSG+Oracle is included as an oracle reference.
Method
Difficulty
Counting
Permanence
Bin Fill
Pick Xtimes
Swing Xtimes
Stop Cube
Video Unmask
Video Unmask Swap
Button Unmask
Button Unmask Swap
Base Policies
π0.5
Easy
38.5
46.2
34.6
7.7
23.1
19.2
23.1
7.7
Medium
25.0
41.7
41.7
8.3
16.7
16.7
25.0
8.3
Hard
16.7
33.3
33.3
0.0
16.7
16.7
16.7
0.0
π0.5 w/ past actions
Easy
34.6
61.5
26.9
3.8
34.6
19.2
26.9
19.2
Table 8: Difficulty-stratified evaluation on the Counting and Permanence tasks of RoboMME ( Dai et al., 2026 ) . We report success rates (%) for Easy, Medium, and Hard episodes. Each task contains 50 test episodes, including 26 Easy, 12 Medium, and 12 Hard episodes. GroundSG+Oracle is included as an oracle reference.
Method
Difficulty
Reference
Imitation
Pick Highlight
Video Repick
Video Place Button
Video Place Order
Move Cube
Insert Peg
Pattern Lock
Route Stick
Base Policies
π0.5
Easy
15.4
0.0
34.6
26.9
26.9
3.8
3.8
7.7
Medium
8.3
0.0
33.3
25.0
25.0
0.0
0.0
0.0
Hard
8.3
0.0
25.0
25.0
25.0
0.0
0.0
0.0
π0.5 w/ past actions
Easy
15.4
7.7
26.9
19.2
34.6
3.8
3.8
7.7
Table 9: Difficulty-stratified evaluation on the Reference and Imitation tasks of RoboMME ( Dai et al., 2026 ) . We report success rates (%) for Easy, Medium, and Hard episodes. Each task contains 50 test episodes, including 26 Easy, 12 Medium, and 12 Hard episodes. GroundSG+Oracle is included as an oracle reference.
Current Vision-Language-Action (VLA) models excel at robotic manipulation but often struggle with non-Markovian tasks requiring long-term memory and reasoning due to their reliance on immediate observations. Existing solutions face a ''frequency-competence paradox,'' where stronger reasoning models are too slow for real-time control, while faster models lack sufficient reasoning capabilities. To resolve this architectural misalignment, we propose HiMe, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy. We also introduce a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms, allowing robots to maintain memory plasticity through ''Add, Update, and Delete'' operations. This hierarchical design effectively balances the conflict between real-time execution and slow thinking planning, significantly improving success rates in long-horizon tasks. Experiments demonstrate that this approach not only outperforms flat memory baselines but also exhibits the novel ability to self-correct its internal knowledge based on human preferences.
Li Ji, Siyin Wang, Pengfang Qian +5
1Fudan University · 2Shanghai Innovation Institute · 3East China Normal University
Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip
Vision-Language-Action (VLA) models fail systematically on long-horizon manipulation tasks despite strong short-horizon performance. We show that this failure is not resolved by extending context length alone in the current reactive execution setting; instead, it stems from three recurring execution-loop deficiencies: the memory gap, the verification gap, and the recovery gap. We present HELM, a model-agnostic framework that addresses these deficiencies with three components: an Episodic Memory Module (EMM) that retrieves key task history via CLIP-indexed keyframes, a learned State Verifier (SV) that predicts action failure before execution from observation, action, subgoal, and memory-conditioned context, and a Harness Controller (HC) that performs rollback and replanning. The SV is the core learning contribution: it consistently outperforms rule-based feasibility checks and ensemble uncertainty baselines, and its effectiveness depends critically on access to episodic memory. On LIBERO-LONG, HELM improves task success rate by 23.1 percentage points over OpenVLA (58.4% to 81.5%), while extending the context window to H=32 yields only a 5.4-point gain and same-budget LoRA adaptation remains 12.2 points below HELM. HELM also improves long-horizon performance on CALVIN and substantially boosts recovery success under controlled perturbations. Ablations and mechanism analyses isolate the contribution of each component, and we release LIBERO-Recovery as a perturbation-injection protocol for evaluating failure recovery in long-horizon manipulation.
Zijian Zeng, Fei Ding, Huiming Yang +1
Tsinghua University · Alibaba Group · Bengbu University