Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

Youpeng Wen; Gang Wang; Tianyu Huang; Weiwei Zhang; Zhijian Huang; Wei Chen; Xieyuanli Chen; Yunhui Liu

cs.ROOct 13, 2026

Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

Authors: Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu

Organizations: The Chinese University of Hong Kong · Xiaomi Corporation · National University of Defense Technology

Abstract

A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $π_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra.

Figures & tables

Figure 1: Overview of Orchestra. (a) Long-horizon instruction following requires task continuity and correct grounding. (b) Orchestra coordinates action generation and subtask transitions through a shared execution context. (c) Evaluation spans RMBench, RMBench-Compose, and real-world tasks. (d) Orchestra improves success rate and Average Success Length over the indicated baselines.

Figure 2: Orchestra framework. (a) Let the Mind Conceive. The agent decomposes multimodal instructions into subtasks, grounds relevant entities, and maintains task records and recent RGB observations. (b) Let the Hand Achieve. An action policy and an independent progress evaluator use the active subtask, observations, and spatial grounding to generate action chunks and predict progress. A terminal-category prediction triggers the next subtask.

Base policies
Method	Observe and Pick Up	Rearrange Blocks	Put Back Block	Swap Blocks	Swap T	Battery Try	Blocks Ranking Try	Cover Blocks	Press Button	Total average
DP	1%	0%	0%	11%	20%	10%	10%	0%	0%	5.78%
ACT	1%	29%	0%	2%	2%	19%	0%	0%	0%	5.89%
$\pi_{0.5}$	9%	13%	11%	24%	15%	16%	6%	0%	0%	10.44%
X-VLA	9%	13%	18%	16%	3%	26%	1%	2%	0%	9.78%
QwenOFT	0%	0%	0%	0%	0%	14%	37%	0%	0%	5.67%

Table 1: Success rates on RMBench. The best result in each column is shown in bold.

Figure 3: Atomic and compositional performance on RMBench-Compose. (a–c) Unweighted five-environment averages of Atomic SR, SSR, and ASL. (d) Per-environment ASL; dashed segments indicate task lengths. Orchestra with

\pi_{0.5}

changes the action policy; the Qwen variant changes the orchestration agent. Spatially conditioned configurations use environment-provided grounding.

Figure 4: Component ablations and simulation examples on RMBench-Compose. (a) Mean and per-environment ASL for component removals and state-based oracle transitions. Heatmap cells report raw ASL; colors encode ASL/ $n$ . (b) Selected subtask frames illustrating Cover and Uncover Block and Swap Block.

Figure 5: Real-world performance and grounded subtask examples. (a) SSR and ASL on four tasks over $20$ episodes per task and configuration. Dashed lines mark task lengths ( $n=4,3,3,5$ ). (b) Selected subtask examples for Fruit Bowl Placement and Number Ordering, with corresponding object masks overlaid on RGB frames.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Task-orchestration agent prompt and output schema. The template specifies the scene, instruction, and capability inputs; seven rules for subtask decomposition and entity identification; and the required JSON output format. Each output entry pairs an executable manipulation instruction with descriptions of its relevant entities. Spatial localization, action generation, and completion detection are delegated to downstream components.

Figure 7: Short-horizon visual memory inside SigLIP. (a) Each camera supplies a six-frame window with stride three, spanning observation indices $t-15$ through $t$ . Patch embedding and spatial positional encoding are applied independently to each frame. Four spatial layers (S) alternate with a causal temporal-attention operation (T), giving temporal mixing after layers 4, 8, 12, 16, and 20. After layer 20, historical tokens are discarded and only current-frame features pass through the remaining spatial layers. The VLM receives 256 visual tokens per camera, matching the single-frame baseline. (b) Spatial attention is bidirectional within a frame; temporal attention operates across frames at the same patch index, with causal and valid-history masks. The temporal pathway reuses the corresponding spatial-attention weights and adds fixed sinusoidal age encodings, with zero encoding for the current frame. This adaptation introduces no new learned memory parameters. Patch grids are schematic.

Figure 8: Architecture of the progress evaluator. A learned query aggregates the multimodal visual and language prefix, and a normalized linear head predicts one of 11 progress categories.

Task	Collection units	Eval. episodes	Eval. length
Breakfast Preparation	120	20	4
Fruit Bowl Placement	120	20	3
Number Ordering	144	20	3
Trash Bin Placement	120	20	5

Appendix

Table 2: Real-world data collection and evaluation. Collection amounts use complete-task accounting units assembled from subtask-level collection. Evaluation counts are per configuration.

Environment	$m$	Demos ( $50m$ )	$n$	Budget
Pick Up Object	5	250	10	250
Cover and Uncover Block	6	300	12	300
Press Button	3	150	6	150
Swap Block	7	350	14	300
Place Block on Mat	9	450	18	250

Appendix

Table 3: RMBench-Compose collection and evaluation settings.

m

is the minimum task length used for collection scaling;

n=2m

defines a deliberately long evaluation sequence designed to stress-test the policy’s limits in sustained execution. Budgets are maximum executed action steps per subtask, not inference-call counts. Each environment is evaluated for 100 episodes per configuration.

Figure 9: Custom dual-arm robot platform used in the real-world experiments.

Figure 10: Per-environment success rates on RMBench-Compose. (a) Atomic SR measures isolated atomic-task execution. (b) SSR measures success among attempted subtasks in composed episodes; unreached subtasks are excluded. Results are shown for all five evaluated configurations and all five environments. Task-level ASL is reported in Fig. 3 .

Figure 11: Qualitative execution examples on RMBench-Compose. From top to bottom: Pick Up Object, Cover and Uncover Block, Swap Block, Press Button, and Place Block on Mat. Each row shows selected observations from a composed manipulation sequence, with blue arrows indicating temporal progression. The examples visualize object manipulation and scene changes across successive subtasks; they do not depict every step of the full evaluation instructions.

Figure 12: Qualitative real-world execution with spatial grounding and progress visualization. (a) Breakfast Preparation (four subtasks). (b) Fruit Bowl Placement (three subtasks). (c) Number Ordering (three subtasks). (d) Trash Bin Placement (five subtasks). Each subtask contains selected RGB observations with pointing markers, corresponding mask overlays, and a progress trace on a normalized scale from 0 to 1. Subtasks are arranged from left to right, and the horizontal axes of the progress plots indicate execution steps.

Figure 13: Failure analysis of Orchestra in real-world manipulation. (a) Task decomposition repeats the yellow-edged snack bag and omits the white-and-green box. (b) Object localization selects a different object from the requested snack bag. (c) Segmentation captures only part of the central target region on the blue mat. (d) The evaluator fails to recognize completion promptly after bread placement, delaying the subtask transition. (e) Action execution fails through inaccurate placement, an incorrect grasp pose, unintended contact, or unstable placement. (f) Distribution of 65 recorded failures of the complete Orchestra system across the five categories.

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 431 papers

LADDER: A Long-Horizon Robot Manipulation Benchmark
Oct 13, 2026Petko Petkov, Giuliano Albanese, Nikolay Nikolov +3Long-Horizon Robotic Manipulation Robot Manipulation Benchmarks
INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA
Exploring the Limits of Combinatorial Generalization in Embodied Manipulation
Oct 11, 2026Xupeng Zhang, Yantai Yang, Chang Guo +2Generalization in Robotic Manipulation Long-Horizon Robotic Manipulation
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University
MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation
Feb 18, 2026Mehrshad Taji, Arad Mahdinezhad Kashani, Iman Ahmadi +3Large Language Model-Based Robot Planning Robotic Manipulation
Department of Electrical Engineering Sharif University of Technology

Oct 13, 2026J/K move · Enter open · S save

LADDER: A Long-Horizon Robot Manipulation Benchmark

Petko Petkov, Giuliano Albanese, Nikolay Nikolov, Luc Van Gool+2

INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA

Long-Horizon Robotic Manipulation Robot Manipulation Benchmarks Robotic Policy Evaluation Robot Task Planning

Open paper AskPDF arXiv

General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-

\infty

-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai

Abstract

Long-Horizon Robotic Manipulation

Explore similar work

LADDER: A Long-Horizon Robot Manipulation Benchmark

Exploring the Limits of Combinatorial Generalization in Embodied Manipulation

MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation