General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-∞-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai
Figures & tables
Figure 1: Overview of our benchmark and key findings. Across all suites, orchestrating policy execution with sequential subgoal decomposition and visual verification substantially outperforms issuing the full instruction directly to the policy (bottom right). See Tables 1 and 3 .
Benchmark
Sim-to-Real
Tasks
Avg Subgoals
Max Subgoals
Tasks with ≥6 subgoals
Incremental Subgoal Tasks
CALVIN ( Mees et al. )
×
34
1.0
1
0
0
LIBERO-Long ( Liu et al. )
×
10
1.6
2
0
0
SimplerEnv ( Li et al. )
✓
4
1.0
1
0
0
RobotArena- ∞ ( Jangir et al. )
✓
74
1.0
2
0
0
RoboLab ( Yang et al. )
✓
120
1.9
11
4
56
RoboVoLo ( Chen et al. )
✓
126
2.0
7
2
2
Table 1: Comparison with existing manipulation benchmarks. Sim-to-Real indicates whether policy performance in the simulator transfers to or correlates with physical robots. CALVIN chains five instructions but issues each one separately, so each of its tasks has one subgoal. Avg Subgoals is the average number of object placements per task. Incremental Subgoal Tasks counts tasks that extend a shorter task of the same scene with more subgoals, or are extended by a longer one, to isolate the effect of task horizon.
Figure 2: Manipulation task categories. Example scenes and natural language instructions across diverse task structures. Visual overlays indicate target placements and task constraints.
Figure 3: Overview of the orchestrator architecture. The top box shows the task input (initial camera view and full instruction). Orange numbered boxes mark the four orchestrator decisions evaluated in our study (Table 2 ). The blue box denotes the execution backend. Dashed lines show baselines without orchestration ( low-level policy ) or without verification ( unverified ).
System
Backend
Plan
Trigger
Verification
Recovery
Low-level policy
Learned policy
None
None
None
None
Periodic trigger
Learned policy
Text subgoals
Periodic
None
Advance
Proprioceptive trigger
Learned policy
Text subgoals
Proprioceptive
None
Advance
Proprioceptive verifier
Learned policy
Text subgoals
Proprioceptive
Visual (VLM)
Retry
VoLo ( Chen et al., 2026 )
Learned policy
Text subgoals
Periodic
Visual (VLM)
Replan
Privileged verifier
Learned policy
Ground-truth subgoals
Privileged
Simulator state
Retry
Table 2: Overview of the evaluated systems according to the proposed characterization. Each configuration maps to the orchestrator decisions in Figure 3 . Top: systems using a low-level policy backend. Bottom: systems using alternative execution backends.
Tasks
Low-level policy
Periodic trigger
Proprioceptive trigger
Proprioceptive verifier
VoLo
Privileged verifier
CaP-X
TiPToP
PSP-Agent
GPT-Policy
SimplerEnv-Long
8.7 ± 1.7
17.4 ± 2.2
21.5 ± 2.3
25.2 ± 2.4
25.5 ± 2.4
29.5 ± 2.5
28.7 ± 2.5
6.7 ± 1.5
37.1 ± 2.7
60.3 ± 2.7
RobotArena- ∞ -Long
10.9 ± 3.6
15.2 ± 4.0
16.2 ± 4.1
24.2 ± 4.6
21.3 ± 4.4
25.5 ± 4.6
16.2 ± 4.1
11.2 ± 3.6
39.9 ± 5.0
66.8 ± 4.9
RoboLab-Long
19.6 ± 4.5
25.6 ± 4.8
28.1 ± 4.9
25.6 ± 4.8
29.0 ± 4.9
34.4 ± 5.1
20.7 ± 4.5
17.3 ± 4.3
51.4 ± 5.2
62.8 ± 5.2
Single destination
14.9 ± 2.9
23.5 ± 3.4
25.0 ± 3.4
31.0 ± 3.6
31.0 ± 3.6
33.8 ± 3.7
23.5 ± 3.4
12.9 ± 2.8
44.0 ± 3.8
61.8 ± 3.7
Transfer
16.7 ± 6.9
14.6 ± 6.7
14.6 ± 6.7
15.3 ± 6.8
27.1 ± 7.8
26.4 ± 7.7
43.8 ± 8.2
10.4 ± 6.1
36.1 ± 8.1
68.8 ± 8.0
Split destination
7.1 ± 3.5
11.5 ± 4.1
17.6 ± 4.7
17.2 ± 4.7
20.6 ± 5.0
24.3 ± 5.2
16.2 ± 4.6
12.2 ± 4.2
50.7 ± 5.7
60.5 ± 5.7
Table 3: Task success rate (%) of every system. Top: per suite, over all tasks (1 to 10 subgoals). Middle: per task category. Bottom: all tasks. Middle and bottom combine the episodes of the three suites. ± covers the 95% Wilson interval. Green marks the best learned-policy method, blue the best tool selection method ( CaP-X , TiPToP, PSP-Agent or GPT-Policy). Privileged verifier is a privileged reference for learning methods using this specific low-level policy.
Figure 4: Performance scaling across task horizon. Task success rate (top) and subgoal progress (bottom) as the horizon increases from 1 to 10 subgoals across all three benchmark suites (all ten systems: Tables 11 and 13 ).
Figure 5: Robot time and compute. (a) Share of all subgoals completed as the simulated time budget elapses. (b) Task success against VLM calls per episode. Both combine all suites (per suite: Table 15 ; tokens and API cost: Appendix G ).
Figure 6: Why subgoals fail. Share of subgoals not completed at the end of the episode, by cause (Appendix H ). A loss that an orchestrator decision made final is drawn as an orchestrator cause (grey; CaP-X and GPT-Policy log no decisions), every other loss by what happened to the object. (a) One bar per system, over all its episodes. (b) One point per task length, over all ten systems. Every value is computed within each suite and then averaged over the three suites (per suite: Figure 8 ).
Planning
Subgoal completion
State tracking
Next subgoal
Tokens
Model
exact match (%)
TP rate (%)
TN rate (%)
all correct (%)
valid (%)
per query
Qwen3-VL-2B
75.2 ± 7.6
68.2 ± 7.0
34.1 ± 5.9
9.3 ± 4.0
58.9 ± 5.3
–
Qwen3-VL-8B
91.0 ± 5.8
69.0 ± 6.8
79.9 ± 5.7
47.9 ± 5.5
72.3 ± 4.4
–
Qwen3.5-9B
86.9 ± 6.5
44.4 ± 6.3
96.9 ± 2.5
11.7 ± 3.7
63.2 ± 4.8
5,173
Qwen3.8-27B
91.0 ± 5.8
76.2 ± 6.3
84.8 ± 4.3
58.2 ± 4.8
79.3 ± 4.1
1,904
Gemini 3.1 Pro
89.7 ± 6.0
54.8 ± 6.6
96.0 ± 2.9
68.7 ± 4.3
75.2 ± 4.5
583
Table 4: Offline evaluation of Vision-Language Models across orchestrator capabilities. Scores in %; ± covers the 95% Wilson interval, widened for the correlation of frames from the same scene. TP/TN rates measure detecting finished vs. unfinished subgoals. Tokens: generated per query over the four capabilities, reasoning included (Table 16 ).
Figure 7: Real-robot evaluation. Left: task success (%) on the real robots per task category, 12 episodes per task and system. Right: overall task success in simulation (Table 3 ) and on the real robots. Both rank the three systems the same.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Task category
Requirement
Example instruction
Tasks
Single destination
every object goes to the same destination
“put the carrot and the spoon on the plate”
44
Transfer
every object starts on one destination and must be moved to another
“move the carrot and the spoon from the plate to the blue towel”
8
Split destination
different objects go to different destinations
“put the carrot on the plate and put the spoon on the blue towel”
25
Relay
objects pass through an intermediate destination
“first put the carrot on the plate, then put the carrot on the blue towel”
20
Revisit
objects return to a destination they have already left
“first put the spoon on the cloth, then put the spoon in the pot, then put the spoon on the cloth”
13
Exclusion
some objects must be left where they are
“put everything except the eggplant on the plate”
13
Appendix
Table 6: The eight kinds of task , with an example instruction for each. The kind of task decides what one instruction has to express, and so what an orchestrator can add. Tasks counts the tasks of that kind across the three suites; Figure 2 shows one frame per kind, with what its instruction asks for drawn on the scene.
Task
Subgoals
Instruction
Single destination
all1_plate
1
“put the carrot on the plate.”
all1_towel
1
“put the spoon on the blue towel.”
all2_plate
2
“put the carrot and the spoon on the plate”
all2_towel
2
“put the carrot and the spoon on the blue towel”
all3_plate
3
“put the carrot, the spoon and the eggplant on the plate”
Appendix
Table 7: Every SimplerEnv-Long task , with the instruction the policy is given, grouped by kind of task and ordered by subgoal count.
Task
Subgoals
Instruction
Single destination
scene36_1goal
1
“put the fork in the pan.”
scene45_1goal
1
“put the blue bowl on the purple cloth.”
scene54_1goal
1
“put the spoon on the cloth.”
scene76_1goal
1
“put the metal pot on the yellow cloth.”
scene36_2goal
2
“put the fork and the corn cob in the pan”
Appendix
Table 8: Every RobotArena- ∞ -Long task , with the instruction the policy is given, grouped by kind of task and ordered by subgoal count. The scene45_all_N tasks give the instruction of scene45_Ngoal , word for word for N=2,3 and with two objects named in the opposite order for N=4 to 6 . For N=3 to 6 they also share its scene, so scene45_all_3 repeats scene45_3goal ; scene45_all_2 has a larger cloth and moved objects. In scene54_transfer_2 both objects start on the table rather than on the cloth.
Task
Subgoals
Instruction
Single destination
comp_bin_1
1
“put the banana in the bin.”
comp_plate_1
1
“put the banana on the plate.”
comp_bin_2
2
“put the banana and the orange in the bin”
comp_bin_6
6
“put the banana, the orange, the lemon, the lime, the avocado and the onion in the bin”
comp_bin_10
10
“put the banana, both oranges, the lemon, the lime, the avocado, the onion, the lychee, the pumpkin and the pomegranate in the bin”
Appendix
Table 9: Every RoboLab-Long task , with the instruction the policy is given, grouped by kind of task and ordered by subgoal count. The suffix of comp_relay_N and comp_revisit_N counts objects, so these tasks have 2N and 3N subgoals.
Low-level policy
Proprioceptive verifier
PSP-Agent
Task category
Subgoals
Episodes
Progress
Success
Progress
Success
Progress
Success
Embodiment A (WidowX, Bridge baseline)
1 subgoal, control
1
24
66.7
66.7
75.0
75.0
100.0
100.0
Single destination
2–3
24
18.3
0.0
56.7
33.3
95.0
87.5
Split destination
2–3
24
21.7
0.0
63.3
29.2
86.7
83.3
Exclusion
2
12
8.3
0.0
62.5
58.3
95.8
91.7
Appendix
Table 10: Real-robot evaluation per task category. Subgoal progress (%) and task success rate (%) on the WidowX (Bridge baseline) and Franka (DROID baseline) physical platforms, 12 episodes per task and system (3 layouts, 4 trials each). PSP-Agent ran a month after the other two systems on the same tasks, with layouts restored from the same reference images. On the Franka single-destination task the low-level policy and the Proprioceptive verifier ran with 150 steps per subgoal (950 steps, with 200 more for the first subgoal) and PSP-Agent with 300 (1,700 steps); every PSP-Agent success there placed its fourth object after step 1,200. PSP-Agent ran that task twice, with no reason for the repeat recorded, and its first run (0/12 successes, 38.3% progress) is not included. On the real robots the Proprioceptive verifier used Gemini Robotics-ER 2 for planning and verification, and PSP-Agent Gemini 3.7 Flash. † Run with PSP-Agent only, and not part of the All multi-subgoal row.
Tasks
Low-level policy
Periodic trigger
Proprioceptive trigger
Proprioceptive verifier
VoLo
CaP-X
TiPToP
PSP-Agent
GPT-Policy
Privileged verifier
Episodes
SimplerEnv-Long
1 subgoal
89.6
91.7
91.7
89.6
93.8
87.5
72.9
89.6
85.4
85.4
48
2 subgoals
28.1
56.2
62.5
68.8
64.1
67.2
15.1
74.0
87.0
74.5
192
3 subgoals
6.0
29.6
36.1
40.7
49.5
47.2
9.7
57.4
75.0
50.5
216
4 subgoals
1.8
4.8
11.9
19.0
18.5
26.8
1.2
36.3
63.7
22.6
168
5 subgoals
0.0
0.6
7.7
10.7
6.0
11.3
0.0
25.6
48.8
11.9
168
Appendix
Table 11: Task success rate (%) per suite, system and number of subgoals (Figure 4 , top).
Tasks
Low-level policy
Periodic trigger
Proprioceptive trigger
Proprioceptive verifier
VoLo
CaP-X
TiPToP
PSP-Agent
GPT-Policy
Privileged verifier
Episodes
SimplerEnv-Long
Single destination
9.6
18.8
22.7
26.9
27.1
23.8
11.7
41.2
54.4
30.0
480
Transfer
9.2
8.3
7.5
9.2
22.5
44.2
1.7
25.8
67.5
21.7
120
Split destination
12.5
15.3
21.5
23.6
25.7
21.5
8.3
41.7
61.1
29.2
144
Relay
3.2
13.0
19.0
25.5
18.1
39.8
0.0
23.1
74.5
28.2
216
Revisit
0.0
0.0
0.0
0.0
1.0
9.4
0.0
8.3
39.6
1.0
96
Appendix
Table 12: Task success rate (%) per suite, system and task category. Table 3 combines the three suites; here each suite is shown on its own.
Tasks
Low-level policy
Periodic trigger
Proprioceptive trigger
Proprioceptive verifier
VoLo
CaP-X
TiPToP
PSP-Agent
GPT-Policy
Privileged verifier
Episodes
SimplerEnv-Long
1 subgoal
89.6
91.7
91.7
89.6
93.8
87.5
72.9
89.6
85.4
85.4
48
2 subgoals
49.2
73.7
77.9
81.8
76.0
78.9
29.9
83.1
92.7
83.6
192
3 subgoals
37.2
62.0
65.9
66.0
69.0
69.8
29.2
76.4
89.8
70.2
216
4 subgoals
29.9
43.3
48.2
54.2
54.9
56.0
11.3
67.3
87.5
54.5
168
5 subgoals
30.7
38.2
46.9
51.2
46.3
43.2
1.8
61.5
84.8
49.0
168
Appendix
Table 13: Subgoal progress (%) per suite, system and number of subgoals (Figure 4 , bottom).
Planning
Subgoal completion
State tracking
Next subgoal
Model
exact match (%)
TP rate (%)
TN rate (%)
all correct (%)
valid (%)
SimplerEnv-Long scenes
Qwen3-VL-2B
61.1
52.1
40.4
10.6
39.9
Qwen3-VL-8B
92.6
58.5
86.5
58.6
65.2
Qwen3.5-9B
96.3
45.7
97.1
13.6
60.6
Qwen3.8-27B
92.6
63.8
92.3
71.2
79.3
Appendix
Table 14: Offline evaluation of VLMs per suite. Bold marks the best model within each suite.
Low-level policy
Periodic trigger
Proprioceptive trigger
Proprioceptive verifier
VoLo
CaP-X
TiPToP
PSP-Agent
GPT-Policy
Privileged verifier
VLM calls per episode
SimplerEnv-Long
0.0
0.0
0.0
12.7
24.8
3.8
1.0
18.0
50.1
0.0
RobotArena- ∞ -Long
0.0
0.0
0.0
9.3
26.4
5.6
1.0
20.2
41.7
0.0
RoboLab-Long
0.0
0.0
0.0
9.3
19.2
6.3
1.0
31.8
70.8
0.0
VLM calls per completed subgoal
SimplerEnv-Long
0.0
0.0
0.0
4.9
10.1
1.6
2.8
6.1
11.8
0.0
Appendix
Table 15: Efficiency of every system, per suite. Subgoals completed are shares of all subgoals, measured when half of the simulated time budget has elapsed and at its end (the latter is the subgoal progress of Figure 4 ).
Plan decomposition
Subgoal completion
State tracking
Next subgoal
Model
input
generated
input
generated
input
generated
input
generated
Qwen3-VL-2B
–
–
–
–
–
–
–
–
Qwen3-VL-8B
–
–
–
–
–
–
–
–
Qwen3.5-9B
655
2,951
875
2,404
3,445
10,325
653
3,339
Qwen3.8-27B
690
943
617
967
2,321
3,790
690
1,189
Gemini 3.1 Pro
1,474
663
1,727
231
6,965
970
1,471
529
Appendix
Table 16: Tokens per query of the offline benchmark. Mean input and generated tokens (reasoning included) per scored query, from each API’s own usage counts. A query is one scene for plan decomposition and one frame for the other capabilities; state tracking asks about each subgoal of the frame separately. Input counts follow each provider’s own tokenizer and image accounting, so they compare within a provider only. The two local Qwen3-VL models report no usage.
Figure 8: Why subgoals fail, per benchmark. Share of each suite’s subgoals not completed, split by cause. Top: one bar per system, over all tasks of the suite. Bottom: all ten systems together, by number of subgoals in the task; on RobotArena- ∞ -Long each length from 7 up rests on one or two tasks. Systems keep the same order in every panel.
Grasping
Carrying
Placing
Undone later
Never started
System
Lost
Orch.
Missed
Lost grip
Dropped
Timed out
Wrong place
Off target
Picked up
Knocked off
Slid off
Not reached
Untouched
SimplerEnv-Long
Low-level policy
74.7
–
9.1
3.3
1.7
0.7
6.1
8.3
0.6
0.9
3.3
0.0
40.6
TiPToP
92.9
66.6
4.0
1.1
0.7
0.0
3.4
1.3
0.1
0.2
0.3
0.0
15.2
CaP-X
54.1
–
5.0
2.7
1.9
1.1
3.2
6.4
1.5
2.0
4.5
0.0
26.0
Periodic trigger
60.1
47.8
1.4
0.5
0.3
0.2
0.2
2.9
0.5
0.3
0.7
0.0
5.2
Appendix
Table 17: Where each system loses its subgoals , as a share of all its subgoals (per suite, over all its tasks), split exactly as Figure 8 draws it. Lost : not completed at the end of the episode (one minus the subgoal progress). Orch. : lost to an orchestrator mistake (Table 18 ); every other column counts only the remaining losses. The other columns abbreviate the causes defined above.
System
Lost
Orch.
Not planned
No plan
Stopped
Moved on
Deferred
Stuck
Re-issued
SimplerEnv-Long
Low-level policy
74.7
–
–
–
–
–
–
–
–
TiPToP
92.9
66.6
0.0
66.6
0.0
0.0
0.0
0.0
0.0
CaP-X
54.1
–
–
–
–
–
–
–
–
Periodic trigger
60.1
47.8
0.0
0.0
0.0
47.8
0.0
0.0
0.0
Proprioceptive trigger
54.2
34.2
0.0
0.0
0.0
33.2
0.0
1.0
0.0
Appendix
Table 18: Subgoals lost to an orchestrator mistake , as a share of all subgoals (per suite, over all its tasks), over the episodes of Table 17 . Each lost subgoal is attributed to the orchestrator decision that made its loss final; the rest of Lost was lost while the robot carried out a correct command. A dash marks a system that logs no orchestrator decisions (low-level policy, CaP-X and GPT-Policy). The columns are the decisions defined above; Stopped ( PSP-Agent only) means the agent made no tool call in the last 30% of the episode.
Robotic manipulation policies often degrade over extended horizons, yet existing benchmarks provide limited insight into why such failures occur. Most prior benchmarks are either simulation-based or report aggregate success, making it difficult to disentangle the distinct sources of temporal difficulty in real-world execution. We introduce LongBench, a real-world benchmark for evaluating long-horizon manipulation. LongBench consists of over 1,000 real-world episodes, covering two complementary regimes: Context-Independent (fully observable) and Context-Dependent (ambiguity-driven). By organizing tasks into capability- and ambiguity-specific subsets, LongBench enables mechanism-aware evaluation of execution robustness, temporal consistency, and context-dependent reasoning. Evaluating six state-of-the-art policies reveals that long-horizon performance is not governed by a single factor. We observe that performance in fully observable settings is more strongly associated with execution robustness, while contextual difficulty varies across tasks and is not consistently improved by memory-based methods. We hope that LongBench serves as a useful benchmark for studying long-horizon manipulation and for developing policies with stronger robustness across both execution and contextual challenges.
As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/.
Kyochul Jang, Seohyeon Park, Ohchul Kwon +9
Seoul National University · University of Massachusetts Amherst · Google Research
Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natural language instruction. Learning and evaluating these skills present three challenges: similar observations under a fixed task instruction may make skill selection ambiguous; even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution; and task-level metrics hinder skill-specific diagnosis, while early failures leave later skills untested. We therefore introduce ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities. Its dataset contains 137,899 segments across 21 skill types and 417 subtasks, and its benchmark contains 1,260 test instances. Correspondingly, ManiUnit pairs each segment with an explicit subtask instruction; measures sensitivity to perturbations of the robot's starting base position or joint configuration; and restores intermediate simulator states and defines local success conditions so that each skill can be evaluated without executing preceding stages. Evaluations of representative vision-language-action (VLA) policies show that similar aggregate scores can hide substantial per-skill differences. The tested starting-state perturbations also degrade execution: on the full benchmark, joint perturbations reduce success rates by approximately 56% relative to those from demonstrated starting states. On two long-horizon activities, a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations. The trained skills further support complete-task execution on these activities, as coordinating the task and skill policies through a planner raises full-task success from 4.0% to 18.0%.
Guoting Wei, Dawei Yan, Xia Yuan +10
Nanjing University of Science and Technology · Intellifusion · Northwestern Polytechnical University