A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of 68.22% across nine RMBench tasks, outperforming Mem-0 by 26.22 percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of 7.35 subtasks, 2.46 times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by 25.0% over Orchestra using the base π0.5 policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra.
Figures & tables
Figure 1: Overview of Orchestra. (a) Long-horizon instruction following requires task continuity and correct grounding. (b) Orchestra coordinates action generation and subtask transitions through a shared execution context. (c) Evaluation spans RMBench, RMBench-Compose, and real-world tasks. (d) Orchestra improves success rate and Average Success Length over the indicated baselines.
Figure 2: Orchestra framework. (a) Let the Mind Conceive. The agent decomposes multimodal instructions into subtasks, grounds relevant entities, and maintains task records and recent RGB observations. (b) Let the Hand Achieve. An action policy and an independent progress evaluator use the active subtask, observations, and spatial grounding to generate action chunks and predict progress. A terminal-category prediction triggers the next subtask.
Method
Observe and Pick Up
Rearrange Blocks
Put Back Block
Swap Blocks
Swap T
Battery Try
Blocks Ranking Try
Cover Blocks
Press Button
Total average
Base policies
DP
1%
0%
0%
11%
20%
10%
10%
0%
0%
5.78%
ACT
1%
29%
0%
2%
2%
19%
0%
0%
0%
5.89%
π0.5
9%
13%
11%
24%
15%
16%
6%
0%
0%
10.44%
X-VLA
9%
13%
18%
16%
3%
26%
1%
2%
0%
9.78%
QwenOFT
0%
0%
0%
0%
0%
14%
37%
0%
0%
5.67%
Table 1: Success rates on RMBench. The best result in each column is shown in bold.
Figure 3: Atomic and compositional performance on RMBench-Compose. (a–c) Unweighted five-environment averages of Atomic SR, SSR, and ASL. (d) Per-environment ASL; dashed segments indicate task lengths. Orchestra with π0.5 changes the action policy; the Qwen variant changes the orchestration agent. Spatially conditioned configurations use environment-provided grounding.
Figure 4: Component ablations and simulation examples on RMBench-Compose. (a) Mean and per-environment ASL for component removals and state-based oracle transitions. Heatmap cells report raw ASL; colors encode ASL/ n . (b) Selected subtask frames illustrating Cover and Uncover Block and Swap Block.
Figure 5: Real-world performance and grounded subtask examples. (a) SSR and ASL on four tasks over 20 episodes per task and configuration. Dashed lines mark task lengths ( n=4,3,3,5 ). (b) Selected subtask examples for Fruit Bowl Placement and Number Ordering, with corresponding object masks overlaid on RGB frames.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Task-orchestration agent prompt and output schema. The template specifies the scene, instruction, and capability inputs; seven rules for subtask decomposition and entity identification; and the required JSON output format. Each output entry pairs an executable manipulation instruction with descriptions of its relevant entities. Spatial localization, action generation, and completion detection are delegated to downstream components.
Figure 7: Short-horizon visual memory inside SigLIP. (a) Each camera supplies a six-frame window with stride three, spanning observation indices t−15 through t . Patch embedding and spatial positional encoding are applied independently to each frame. Four spatial layers (S) alternate with a causal temporal-attention operation (T), giving temporal mixing after layers 4, 8, 12, 16, and 20. After layer 20, historical tokens are discarded and only current-frame features pass through the remaining spatial layers. The VLM receives 256 visual tokens per camera, matching the single-frame baseline. (b) Spatial attention is bidirectional within a frame; temporal attention operates across frames at the same patch index, with causal and valid-history masks. The temporal pathway reuses the corresponding spatial-attention weights and adds fixed sinusoidal age encodings, with zero encoding for the current frame. This adaptation introduces no new learned memory parameters. Patch grids are schematic.
Figure 8: Architecture of the progress evaluator. A learned query aggregates the multimodal visual and language prefix, and a normalized linear head predicts one of 11 progress categories.
Task
Collection units
Eval. episodes
Eval. length
Breakfast Preparation
120
20
4
Fruit Bowl Placement
120
20
3
Number Ordering
144
20
3
Trash Bin Placement
120
20
5
Appendix
Table 2: Real-world data collection and evaluation. Collection amounts use complete-task accounting units assembled from subtask-level collection. Evaluation counts are per configuration.
Environment
m
Demos ( 50m )
n
Budget
Pick Up Object
5
250
10
250
Cover and Uncover Block
6
300
12
300
Press Button
3
150
6
150
Swap Block
7
350
14
300
Place Block on Mat
9
450
18
250
Appendix
Table 3: RMBench-Compose collection and evaluation settings. m is the minimum task length used for collection scaling; n=2m defines a deliberately long evaluation sequence designed to stress-test the policy’s limits in sustained execution. Budgets are maximum executed action steps per subtask, not inference-call counts. Each environment is evaluated for 100 episodes per configuration.
Figure 9: Custom dual-arm robot platform used in the real-world experiments.
Figure 10: Per-environment success rates on RMBench-Compose. (a) Atomic SR measures isolated atomic-task execution. (b) SSR measures success among attempted subtasks in composed episodes; unreached subtasks are excluded. Results are shown for all five evaluated configurations and all five environments. Task-level ASL is reported in Fig. 3 .
Figure 11: Qualitative execution examples on RMBench-Compose. From top to bottom: Pick Up Object, Cover and Uncover Block, Swap Block, Press Button, and Place Block on Mat. Each row shows selected observations from a composed manipulation sequence, with blue arrows indicating temporal progression. The examples visualize object manipulation and scene changes across successive subtasks; they do not depict every step of the full evaluation instructions.
Figure 12: Qualitative real-world execution with spatial grounding and progress visualization. (a) Breakfast Preparation (four subtasks). (b) Fruit Bowl Placement (three subtasks). (c) Number Ordering (three subtasks). (d) Trash Bin Placement (five subtasks). Each subtask contains selected RGB observations with pointing markers, corresponding mask overlays, and a progress trace on a normalized scale from 0 to 1. Subtasks are arranged from left to right, and the horizontal axes of the progress plots indicate execution steps.
Figure 13: Failure analysis of Orchestra in real-world manipulation. (a) Task decomposition repeats the yellow-edged snack bag and omits the white-and-green box. (b) Object localization selects a different object from the requested snack bag. (c) Segmentation captures only part of the central target region on the blue mat. (d) The evaluator fails to recognize completion promptly after bread placement, delaying the subtask transition. (e) Action execution fails through inaccurate placement, an incorrect grasp pose, unintended contact, or unstable placement. (f) Distribution of 65 recorded failures of the complete Orchestra system across the five categories.
General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-∞-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai
Petko Petkov, Giuliano Albanese, Nikolay Nikolov +3
INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA
In embodied manipulation, Vision Language Action (VLA) models can execute atomic skills, but long-horizon manipulation requires recombining these skills without demonstrations for every skill combination. The full instruction route treats skill chaining as long-horizon imitation. In a controlled study, full instruction training on limited skill combination sequences reaches 40.0% ordered success on covered combinations but only 7.7% on recombination, suggesting memorization of sequence templates rather than robust skill recombination. We introduce Hierarchical Subtask Execution (HSE), a diagnostic framework that exposes skill recombination through a shared modular interface over the planner, trigger, and policy, decomposing long-horizon failures into separable modes. We introduce RoboCombine, a benchmark for skill recombination without combined skill training trajectories, with metrics for ordered success and transition conversion across subtask boundaries. RoboCombine reveals that atomic competence alone does not ensure ordered execution. Failures concentrate at transitions where downstream subtasks launch from handoff states from preceding subtasks rather than canonical atomic starts. We identify this mismatch as the start state gap. To address it, we introduce Pose Robust Task Initialization (PRTI), a lightweight atomic task post-training strategy that broadens start state coverage beyond canonical starts. Under oracle timed HSE on RoboCombine, PRTI reaches 18.16% ordered success, compared with 4.21% under random perturbation and 9.47% under a robot-reset oracle, without combined skill training trajectories. These results show that combinatorial generalization requires handoff robust atomic skills that convert first stage progress into reliable downstream execution, not merely decomposition into known subtasks. Code: https://github.com/AutoLab-SAI-SJTU/RoboCombine.
Xupeng Zhang, Yantai Yang, Chang Guo +2
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University
Task planning for robotic manipulation with large language models (LLMs) is an emerging area. Prior approaches rely on specialized models, fine tuning, or prompt tuning, and often operate in an open loop manner without robust environmental feedback, making them fragile in dynamic settings. MALLVI presents a Multi Agent Large Language and Vision framework that enables closed-loop feedback driven robotic manipulation. Given a natural language instruction and an image of the environment, MALLVI generates executable atomic actions for a robot manipulator. After action execution, a Vision Language Model (VLM) evaluates environmental feedback and decides whether to repeat the process or proceed to the next step. Rather than using a single model, MALLVI coordinates specialized agents, Decomposer, Localizer, Thinker, and Reflector, to manage perception, localization, reasoning, and high level planning. An optional Descriptor agent provides visual memory of the initial state. The Reflector supports targeted error detection and recovery by reactivating only relevant agents, avoiding full replanning. Experiments in simulation and real-world settings show that iterative closed loop multi agent coordination improves generalization and increases success rates in zero shot manipulation tasks. Code available at https://github.com/iman1234ahmadi/MALLVI .
Mehrshad Taji, Arad Mahdinezhad Kashani, Iman Ahmadi +3
Department of Electrical Engineering Sharif University of Technology