LADDER: A Long-Horizon Robot Manipulation Benchmark

Petko Petkov; Giuliano Albanese; Nikolay Nikolov; Luc Van Gool; Jonathan Tremblay; Danda Pani Paudel

cs.ROOct 13, 2026

LADDER: A Long-Horizon Robot Manipulation Benchmark

Authors: Petko Petkov, Giuliano Albanese, Nikolay Nikolov, Luc Van Gool, Jonathan Tremblay, Danda Pani Paudel

Organizations: INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA

Abstract

General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena- $\infty$ -Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai

Figures & tables

Figure 1: Overview of our benchmark and key findings. Across all suites, orchestrating policy execution with sequential subgoal decomposition and visual verification substantially outperforms issuing the full instruction directly to the policy (bottom right). See Tables 1 and 3 .

Benchmark	Sim-to-Real	Tasks	Avg Subgoals	Max Subgoals	Tasks with $\geq 6$ subgoals	Incremental Subgoal Tasks
CALVIN ( Mees et al. )	$\times$	34	1.0	1	0	0
LIBERO-Long ( Liu et al. )	$\times$	10	1.6	2	0	0
SimplerEnv ( Li et al. )	✓	4	1.0	1	0	0
RobotArena- $\infty$ ( Jangir et al. )	✓	74	1.0	2	0	0
RoboLab ( Yang et al. )	✓	120	1.9	11	4	56
RoboVoLo ( Chen et al. )	✓	126	2.0	7	2	2

Table 1: Comparison with existing manipulation benchmarks. Sim-to-Real indicates whether policy performance in the simulator transfers to or correlates with physical robots. CALVIN chains five instructions but issues each one separately, so each of its tasks has one subgoal. Avg Subgoals is the average number of object placements per task. Incremental Subgoal Tasks counts tasks that extend a shorter task of the same scene with more subgoals, or are extended by a longer one, to isolate the effect of task horizon.

Figure 2: Manipulation task categories. Example scenes and natural language instructions across diverse task structures. Visual overlays indicate target placements and task constraints.

Figure 3: Overview of the orchestrator architecture. The top box shows the task input (initial camera view and full instruction). Orange numbered boxes mark the four orchestrator decisions evaluated in our study (Table 2 ). The blue box denotes the execution backend. Dashed lines show baselines without orchestration ( low-level policy ) or without verification ( unverified ).

System	Backend	Plan	Trigger	Verification	Recovery
Low-level policy	Learned policy	None	None	None	None
Periodic trigger	Learned policy	Text subgoals	Periodic	None	Advance
Proprioceptive trigger	Learned policy	Text subgoals	Proprioceptive	None	Advance
Proprioceptive verifier	Learned policy	Text subgoals	Proprioceptive	Visual (VLM)	Retry
VoLo ( Chen et al., 2026 )	Learned policy	Text subgoals	Periodic	Visual (VLM)	Replan
Privileged verifier	Learned policy	Ground-truth subgoals	Privileged	Simulator state	Retry

Table 2: Overview of the evaluated systems according to the proposed characterization. Each configuration maps to the orchestrator decisions in Figure 3 . Top: systems using a low-level policy backend. Bottom: systems using alternative execution backends.

Tasks	Low-level policy	Periodic trigger	Proprioceptive trigger	Proprioceptive verifier	VoLo	Privileged verifier	CaP-X	TiPToP	PSP-Agent	GPT-Policy
SimplerEnv-Long	8.7 $\pm$ 1.7	17.4 $\pm$ 2.2	21.5 $\pm$ 2.3	25.2 $\pm$ 2.4	25.5 $\pm$ 2.4	29.5 $\pm$ 2.5	28.7 $\pm$ 2.5	6.7 $\pm$ 1.5	37.1 $\pm$ 2.7	60.3 $\pm$ 2.7
RobotArena- $\infty$ -Long	10.9 $\pm$ 3.6	15.2 $\pm$ 4.0	16.2 $\pm$ 4.1	24.2 $\pm$ 4.6	21.3 $\pm$ 4.4	25.5 $\pm$ 4.6	16.2 $\pm$ 4.1	11.2 $\pm$ 3.6	39.9 $\pm$ 5.0	66.8 $\pm$ 4.9
RoboLab-Long	19.6 $\pm$ 4.5	25.6 $\pm$ 4.8	28.1 $\pm$ 4.9	25.6 $\pm$ 4.8	29.0 $\pm$ 4.9	34.4 $\pm$ 5.1	20.7 $\pm$ 4.5	17.3 $\pm$ 4.3	51.4 $\pm$ 5.2	62.8 $\pm$ 5.2
Single destination	14.9 $\pm$ 2.9	23.5 $\pm$ 3.4	25.0 $\pm$ 3.4	31.0 $\pm$ 3.6	31.0 $\pm$ 3.6	33.8 $\pm$ 3.7	23.5 $\pm$ 3.4	12.9 $\pm$ 2.8	44.0 $\pm$ 3.8	61.8 $\pm$ 3.7
Transfer	16.7 $\pm$ 6.9	14.6 $\pm$ 6.7	14.6 $\pm$ 6.7	15.3 $\pm$ 6.8	27.1 $\pm$ 7.8	26.4 $\pm$ 7.7	43.8 $\pm$ 8.2	10.4 $\pm$ 6.1	36.1 $\pm$ 8.1	68.8 $\pm$ 8.0
Split destination	7.1 $\pm$ 3.5	11.5 $\pm$ 4.1	17.6 $\pm$ 4.7	17.2 $\pm$ 4.7	20.6 $\pm$ 5.0	24.3 $\pm$ 5.2	16.2 $\pm$ 4.6	12.2 $\pm$ 4.2	50.7 $\pm$ 5.7	60.5 $\pm$ 5.7

Table 3: Task success rate (%) of every system. Top: per suite, over all tasks (1 to 10 subgoals). Middle: per task category. Bottom: all tasks. Middle and bottom combine the episodes of the three suites.

\pm

covers the 95% Wilson interval. Green marks the best learned-policy method, blue the best tool selection method ( CaP-X , TiPToP, PSP-Agent or GPT-Policy). Privileged verifier is a privileged reference for learning methods using this specific low-level policy.

Figure 4: Performance scaling across task horizon. Task success rate (top) and subgoal progress (bottom) as the horizon increases from 1 to 10 subgoals across all three benchmark suites (all ten systems: Tables 11 and 13 ).

Figure 5: Robot time and compute. (a) Share of all subgoals completed as the simulated time budget elapses. (b) Task success against VLM calls per episode. Both combine all suites (per suite: Table 15 ; tokens and API cost: Appendix G ).

Figure 6: Why subgoals fail. Share of subgoals not completed at the end of the episode, by cause (Appendix H ). A loss that an orchestrator decision made final is drawn as an orchestrator cause (grey; CaP-X and GPT-Policy log no decisions), every other loss by what happened to the object. (a) One bar per system, over all its episodes. (b) One point per task length, over all ten systems. Every value is computed within each suite and then averaged over the three suites (per suite: Figure 8 ).

	Planning	Subgoal completion		State tracking	Next subgoal	Tokens
Model	exact match (%)	TP rate (%)	TN rate (%)	all correct (%)	valid (%)	per query
Qwen3-VL-2B	75.2 $\pm$ 7.6	68.2 $\pm$ 7.0	34.1 $\pm$ 5.9	9.3 $\pm$ 4.0	58.9 $\pm$ 5.3	–
Qwen3-VL-8B	91.0 $\pm$ 5.8	69.0 $\pm$ 6.8	79.9 $\pm$ 5.7	47.9 $\pm$ 5.5	72.3 $\pm$ 4.4	–
Qwen3.5-9B	86.9 $\pm$ 6.5	44.4 $\pm$ 6.3	96.9 $\pm$ 2.5	11.7 $\pm$ 3.7	63.2 $\pm$ 4.8	5,173
Qwen3.8-27B	91.0 $\pm$ 5.8	76.2 $\pm$ 6.3	84.8 $\pm$ 4.3	58.2 $\pm$ 4.8	79.3 $\pm$ 4.1	1,904
Gemini 3.1 Pro	89.7 $\pm$ 6.0	54.8 $\pm$ 6.6	96.0 $\pm$ 2.9	68.7 $\pm$ 4.3	75.2 $\pm$ 4.5	583

Table 4: Offline evaluation of Vision-Language Models across orchestrator capabilities. Scores in %;

\pm

covers the 95% Wilson interval, widened for the correlation of frames from the same scene. TP/TN rates measure detecting finished vs. unfinished subgoals. Tokens: generated per query over the four capabilities, reasoning included (Table 16 ).

Figure 7: Real-robot evaluation. Left: task success (%) on the real robots per task category, 12 episodes per task and system. Right: overall task success in simulation (Table 3 ) and on the real robots. Both rank the three systems the same.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Task category	Requirement	Example instruction	Tasks
Single destination	every object goes to the same destination	“put the carrot and the spoon on the plate”	44
Transfer	every object starts on one destination and must be moved to another	“move the carrot and the spoon from the plate to the blue towel”	8
Split destination	different objects go to different destinations	“put the carrot on the plate and put the spoon on the blue towel”	25
Relay	objects pass through an intermediate destination	“first put the carrot on the plate, then put the carrot on the blue towel”	20
Revisit	objects return to a destination they have already left	“first put the spoon on the cloth, then put the spoon in the pot, then put the spoon on the cloth”	13
Exclusion	some objects must be left where they are	“put everything except the eggplant on the plate”	13

Appendix

Table 6: The eight kinds of task , with an example instruction for each. The kind of task decides what one instruction has to express, and so what an orchestrator can add. Tasks counts the tasks of that kind across the three suites; Figure 2 shows one frame per kind, with what its instruction asks for drawn on the scene.

Task	Subgoals	Instruction
Single destination
all1_plate	1	“put the carrot on the plate.”
all1_towel	1	“put the spoon on the blue towel.”
all2_plate	2	“put the carrot and the spoon on the plate”
all2_towel	2	“put the carrot and the spoon on the blue towel”
all3_plate	3	“put the carrot, the spoon and the eggplant on the plate”

Appendix

Table 7: Every SimplerEnv-Long task , with the instruction the policy is given, grouped by kind of task and ordered by subgoal count.

Task	Subgoals	Instruction
Single destination
scene36_1goal	1	“put the fork in the pan.”
scene45_1goal	1	“put the blue bowl on the purple cloth.”
scene54_1goal	1	“put the spoon on the cloth.”
scene76_1goal	1	“put the metal pot on the yellow cloth.”
scene36_2goal	2	“put the fork and the corn cob in the pan”

Appendix

Table 8: Every RobotArena-

\infty

-Long task , with the instruction the policy is given, grouped by kind of task and ordered by subgoal count. The scene45_all_N tasks give the instruction of scene45_Ngoal , word for word for

N=2,3

and with two objects named in the opposite order for

N=4

6

. For

N=3

6

they also share its scene, so scene45_all_3 repeats scene45_3goal ; scene45_all_2 has a larger cloth and moved objects. In scene54_transfer_2 both objects start on the table rather than on the cloth.

Task	Subgoals	Instruction
Single destination
comp_bin_1	1	“put the banana in the bin.”
comp_plate_1	1	“put the banana on the plate.”
comp_bin_2	2	“put the banana and the orange in the bin”
comp_bin_6	6	“put the banana, the orange, the lemon, the lime, the avocado and the onion in the bin”
comp_bin_10	10	“put the banana, both oranges, the lemon, the lime, the avocado, the onion, the lychee, the pumpkin and the pomegranate in the bin”

Appendix

Table 9: Every RoboLab-Long task , with the instruction the policy is given, grouped by kind of task and ordered by subgoal count. The suffix of comp_relay_N and comp_revisit_N counts objects, so these tasks have

2N

and

3N

subgoals.

			Low-level policy		Proprioceptive verifier		PSP-Agent
Task category	Subgoals	Episodes	Progress	Success	Progress	Success	Progress	Success
Embodiment A (WidowX, Bridge baseline)
1 subgoal, control	1	24	66.7	66.7	75.0	75.0	100.0	100.0
Single destination	2–3	24	18.3	0.0	56.7	33.3	95.0	87.5
Split destination	2–3	24	21.7	0.0	63.3	29.2	86.7	83.3
Exclusion	2	12	8.3	0.0	62.5	58.3	95.8	91.7

Appendix

Table 10: Real-robot evaluation per task category. Subgoal progress (%) and task success rate (%) on the WidowX (Bridge baseline) and Franka (DROID baseline) physical platforms, 12 episodes per task and system (3 layouts, 4 trials each). PSP-Agent ran a month after the other two systems on the same tasks, with layouts restored from the same reference images. On the Franka single-destination task the low-level policy and the Proprioceptive verifier ran with 150 steps per subgoal (950 steps, with 200 more for the first subgoal) and PSP-Agent with 300 (1,700 steps); every PSP-Agent success there placed its fourth object after step 1,200. PSP-Agent ran that task twice, with no reason for the repeat recorded, and its first run (0/12 successes, 38.3% progress) is not included. On the real robots the Proprioceptive verifier used Gemini Robotics-ER 2 for planning and verification, and PSP-Agent Gemini 3.7 Flash. † Run with PSP-Agent only, and not part of the All multi-subgoal row.

Tasks	Low-level policy	Periodic trigger	Proprioceptive trigger	Proprioceptive verifier	VoLo	CaP-X	TiPToP	PSP-Agent	GPT-Policy	Privileged verifier	Episodes
SimplerEnv-Long
1 subgoal	89.6	91.7	91.7	89.6	93.8	87.5	72.9	89.6	85.4	85.4	48
2 subgoals	28.1	56.2	62.5	68.8	64.1	67.2	15.1	74.0	87.0	74.5	192
3 subgoals	6.0	29.6	36.1	40.7	49.5	47.2	9.7	57.4	75.0	50.5	216
4 subgoals	1.8	4.8	11.9	19.0	18.5	26.8	1.2	36.3	63.7	22.6	168
5 subgoals	0.0	0.6	7.7	10.7	6.0	11.3	0.0	25.6	48.8	11.9	168

Appendix

Table 11: Task success rate (%) per suite, system and number of subgoals (Figure 4 , top).

Tasks	Low-level policy	Periodic trigger	Proprioceptive trigger	Proprioceptive verifier	VoLo	CaP-X	TiPToP	PSP-Agent	GPT-Policy	Privileged verifier	Episodes
SimplerEnv-Long
Single destination	9.6	18.8	22.7	26.9	27.1	23.8	11.7	41.2	54.4	30.0	480
Transfer	9.2	8.3	7.5	9.2	22.5	44.2	1.7	25.8	67.5	21.7	120
Split destination	12.5	15.3	21.5	23.6	25.7	21.5	8.3	41.7	61.1	29.2	144
Relay	3.2	13.0	19.0	25.5	18.1	39.8	0.0	23.1	74.5	28.2	216
Revisit	0.0	0.0	0.0	0.0	1.0	9.4	0.0	8.3	39.6	1.0	96

Appendix

Table 12: Task success rate (%) per suite, system and task category. Table 3 combines the three suites; here each suite is shown on its own.

Tasks	Low-level policy	Periodic trigger	Proprioceptive trigger	Proprioceptive verifier	VoLo	CaP-X	TiPToP	PSP-Agent	GPT-Policy	Privileged verifier	Episodes
SimplerEnv-Long
1 subgoal	89.6	91.7	91.7	89.6	93.8	87.5	72.9	89.6	85.4	85.4	48
2 subgoals	49.2	73.7	77.9	81.8	76.0	78.9	29.9	83.1	92.7	83.6	192
3 subgoals	37.2	62.0	65.9	66.0	69.0	69.8	29.2	76.4	89.8	70.2	216
4 subgoals	29.9	43.3	48.2	54.2	54.9	56.0	11.3	67.3	87.5	54.5	168
5 subgoals	30.7	38.2	46.9	51.2	46.3	43.2	1.8	61.5	84.8	49.0	168

Appendix

Table 13: Subgoal progress (%) per suite, system and number of subgoals (Figure 4 , bottom).

SimplerEnv-Long scenes
	Planning	Subgoal completion		State tracking	Next subgoal
Model	exact match (%)	TP rate (%)	TN rate (%)	all correct (%)	valid (%)
Qwen3-VL-2B	61.1	52.1	40.4	10.6	39.9
Qwen3-VL-8B	92.6	58.5	86.5	58.6	65.2
Qwen3.5-9B	96.3	45.7	97.1	13.6	60.6
Qwen3.8-27B	92.6	63.8	92.3	71.2	79.3

Appendix

Table 14: Offline evaluation of VLMs per suite. Bold marks the best model within each suite.

	Low-level policy	Periodic trigger	Proprioceptive trigger	Proprioceptive verifier	VoLo	CaP-X	TiPToP	PSP-Agent	GPT-Policy	Privileged verifier
VLM calls per episode
SimplerEnv-Long	0.0	0.0	0.0	12.7	24.8	3.8	1.0	18.0	50.1	0.0
RobotArena- $\infty$ -Long	0.0	0.0	0.0	9.3	26.4	5.6	1.0	20.2	41.7	0.0
RoboLab-Long	0.0	0.0	0.0	9.3	19.2	6.3	1.0	31.8	70.8	0.0
VLM calls per completed subgoal
SimplerEnv-Long	0.0	0.0	0.0	4.9	10.1	1.6	2.8	6.1	11.8	0.0

Appendix

Table 15: Efficiency of every system, per suite. Subgoals completed are shares of all subgoals, measured when half of the simulated time budget has elapsed and at its end (the latter is the subgoal progress of Figure 4 ).

	Plan decomposition		Subgoal completion		State tracking		Next subgoal
Model	input	generated	input	generated	input	generated	input	generated
Qwen3-VL-2B	–	–	–	–	–	–	–	–
Qwen3-VL-8B	–	–	–	–	–	–	–	–
Qwen3.5-9B	655	2,951	875	2,404	3,445	10,325	653	3,339
Qwen3.8-27B	690	943	617	967	2,321	3,790	690	1,189
Gemini 3.1 Pro	1,474	663	1,727	231	6,965	970	1,471	529

Appendix

Table 16: Tokens per query of the offline benchmark. Mean input and generated tokens (reasoning included) per scored query, from each API’s own usage counts. A query is one scene for plan decomposition and one frame for the other capabilities; state tracking asks about each subgoal of the frame separately. Input counts follow each provider’s own tokenizer and image accounting, so they compare within a provider only. The two local Qwen3-VL models report no usage.

Figure 8: Why subgoals fail, per benchmark. Share of each suite’s subgoals not completed, split by cause. Top: one bar per system, over all tasks of the suite. Bottom: all ten systems together, by number of subgoals in the task; on RobotArena-

\infty

-Long each length from 7 up rests on one or two tasks. Systems keep the same order in every panel.

SimplerEnv-Long
			Grasping		Carrying		Placing		Undone later			Never started
System	Lost	Orch.	Missed	Lost grip	Dropped	Timed out	Wrong place	Off target	Picked up	Knocked off	Slid off	Not reached	Untouched
Low-level policy	74.7	–	9.1	3.3	1.7	0.7	6.1	8.3	0.6	0.9	3.3	0.0	40.6
TiPToP	92.9	66.6	4.0	1.1	0.7	0.0	3.4	1.3	0.1	0.2	0.3	0.0	15.2
CaP-X	54.1	–	5.0	2.7	1.9	1.1	3.2	6.4	1.5	2.0	4.5	0.0	26.0
Periodic trigger	60.1	47.8	1.4	0.5	0.3	0.2	0.2	2.9	0.5	0.3	0.7	0.0	5.2

Appendix

Table 17: Where each system loses its subgoals , as a share of all its subgoals (per suite, over all its tasks), split exactly as Figure 8 draws it. Lost : not completed at the end of the episode (one minus the subgoal progress). Orch. : lost to an orchestrator mistake (Table 18 ); every other column counts only the remaining losses. The other columns abbreviate the causes defined above.

SimplerEnv-Long
System	Lost	Orch.	Not planned	No plan	Stopped	Moved on	Deferred	Stuck	Re-issued
Low-level policy	74.7	–	–	–	–	–	–	–	–
TiPToP	92.9	66.6	0.0	66.6	0.0	0.0	0.0	0.0	0.0
CaP-X	54.1	–	–	–	–	–	–	–	–
Periodic trigger	60.1	47.8	0.0	0.0	0.0	47.8	0.0	0.0	0.0
Proprioceptive trigger	54.2	34.2	0.0	0.0	0.0	33.2	0.0	1.0	0.0

Appendix

Table 18: Subgoals lost to an orchestrator mistake , as a share of all subgoals (per suite, over all its tasks), over the episodes of Table 17 . Each lost subgoal is attributed to the orchestrator decision that made its loss final; the rest of Lost was lost while the robot carried out a correct command. A dash marks a system that logs no orchestrator decisions (low-level policy, CaP-X and GPT-Policy). The columns are the decisions defined above; Stopped ( PSP-Agent only) means the agent made no tool call in the last 30% of the episode.

Explore similar work

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks
Apr 18, 2026Xueyao Chen, Jingkai Jia, Tong Yang +3Long-Horizon Robotic Manipulation Robotic Policy Evaluation
Fudan University
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Oct 1, 2026Kyochul Jang, Seohyeon Park, Ohchul Kwon +9Mobile Manipulation Robotic Tool Use
Seoul National University · University of Massachusetts Amherst · Google Research
ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks
Oct 8, 2026Guoting Wei, Dawei Yan, Xia Yuan +10Long-Horizon Robotic Manipulation Robot Manipulation Benchmarks
Nanjing University of Science and Technology · Intellifusion · Northwestern Polytechnical University

Apr 18, 2026J/K move · Enter open · S save

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

Xueyao Chen, Jingkai Jia, Tong Yang, Yibo Fu+2

Fudan University

Long-Horizon Robotic Manipulation Robotic Policy Evaluation Robotic Manipulation Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Robotic manipulation policies often degrade over extended horizons, yet existing benchmarks provide limited insight into why such failures occur. Most prior benchmarks are either simulation-based or report aggregate success, making it difficult to disentangle the distinct sources of temporal difficulty in real-world execution. We introduce LongBench, a real-world benchmark for evaluating long-horizon manipulation. LongBench consists of over 1,000 real-world episodes, covering two complementary regimes: Context-Independent (fully observable) and Context-Dependent (ambiguity-driven). By organizing tasks into capability- and ambiguity-specific subsets, LongBench enables mechanism-aware evaluation of execution robustness, temporal consistency, and context-dependent reasoning. Evaluating six state-of-the-art policies reveals that long-horizon performance is not governed by a single factor. We observe that performance in fully observable settings is more strongly associated with execution robustness, while contextual difficulty varies across tasks and is not consistently improved by memory-based methods. We hope that LongBench serves as a useful benchmark for studying long-horizon manipulation and for developing policies with stronger robustness across both execution and contextual challenges.

Abstract

Explore similar work

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution

ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks