ManiLadder: Benchmarking Robot Manipulation Through a Categorized and Multi-Difficulty Level Task Ladder
Authors: Chongkai Gao, Jieao Shi, Zhaoyu Mu, Yudi Lin, Xuanwei Liu, Wei Ken Ng, Baiye Cheng, Anqi Chen, Kerun Liu, Zhuoran Li, Xuchuan Huang, Haoyang Li, Shengjia Zhu, Nga Teng Chan, Yuxuan Zhang, Xingyu Chen, Lin Shao
Organizations: National University of Singapore · Nanyang Technological University · University of Toronto · Huazhong University of Science and Technology · Nanjing University · KTH · Shanghai Jiao Tong University · Peking University · HKUST
Recent advances in robotic manipulation have enabled increasingly complex and dexterous behaviors, yet existing benchmarks still focus on simple pick-and-place tasks and lack a systematic framework for measuring progress in robot learning along the axis of task difficulty. We present ManiLadder, a large-scale and organized simulation benchmark for robotic manipulation. ManiLadder contains 112 carefully designed tasks stratified into 4 difficulty levels, with 8 categories in each level, spanning rigid-body and deformable-object manipulation, single-arm and dual-arm settings, and gripper and dexterous-hand end-effectors. To calibrate difficulty, we fix the data amount the same for every task and use a unified flow- matching imitation learning policy under the same training and evaluation protocols, then use its evaluation success rate as an operational empirical difficulty signal. All tasks are iteratively refined until they fall into target difficulty ranges, which are then verified across different random seeds, policies, and agentic learning systems. We further analyze the task properties, compare different policy performances and training settings, analyze task transfer and data scaling behaviors, and conduct real-world experiments showing that the designed tasks preserve their difficulty ordering beyond simulation. ManiLad- der provides a calibrated ladder for measuring, comparing, and advancing robotic manipulation capabilities. More results are available at https://maniladder-benchmark.github.io/.
Figures & tables
Benchmark
Simulator
No. of tasks
Auto Demo
Difficulty Taxonomy
Deformable objects
Robots
MetaWorld [ 1 ]
Mujoco [ 2 ]
50
✓
✗
✗
S,G
RoboDojo [ 3 ]
Isaac Sim [ 4 ]
42
✓
✗
✓
D,G
COLOSSEUM [ 5 ]
V-Rep [ 6 ]
20
✓
✗
✗
S,G
RoboCasa [ 7 ]
Mujoco [ 2 ]
100
✓
✗
✗
S,X,G,D,M
Behavior 1K [ 8 ]
Isaac Sim [ 4 ]
1000 (100 released)
✗
✗
✓
S,G,M
LIBERO [ 9 ]
Mujoco [ 2 ]
120
✗
✓
✗
S,G
TABLE I : Comparison between ManiLadder and other simulation robot manipulation benchmarks focusing on task taxonomy and difficulty levels. Auto Demo means supporting automatic data generation. We use G for robots with grippers, X for robots with dexterous hands, S for single-arm robots, D for dual-arm robots, and M for mobile robots.
TABLE II : Manipulation skill coverage of ManiLadder. We group skills according to prior literature on manipulation-primitives and contact-rich manipulation taxonomy [ 43 , 44 , 45 , 46 ] .
Fig. 2 : The objects and robots in ManiLadder. Four kinds of robots and diverse kinds of rigid-body and deformable objects are included in ManiLadder.
Fig. 3 : Left: the basic scene setup of task FoldShirt , serving as an example illustrating a typical task scenario in ManiLadder. Middle left: the domain randomization on the lighting, background, and distracting objects of FoldShirt . Middle: in each rigid-body object task suite, we design different tasks focusing on different manipulation skills, such as HangMug for pick-and-place, OpenBoxRetrival for articulated object manipulation, and PivotBook for non-prehensile manipulation. In each deformable object task suite, we design tasks including 1D, 2D, and 3D deformable objects, such as StraightenRope for rope manipulation, UnfoldShirt for cloth manipulation, and ShapeTtoI for clay manipulation. Middle right: two mobile manipulation tasks in ManiLadder. Right: the task horizon and stage number histograms.
Fig. 4 : All task success rates in ManiLadder of the small flow-matching policy, π0.5 , and GPT6-Astra agentic script policy.
Fig. 5 : Examples of task calibration and refinement. Top: the HangMug task: we progressively modify the initialization regions of the mug and rack to make the task more challenging. Bottom: the DualArmCloseBox task: we first reduce the box size, then introduce an initialization pose that requires coordinated dual-arm rotation before the box can be closed.
Setting
ρ↑
Δ (pp) ↓
A (%) ↑
3 random seeds
0.958±0.006
5.67±0.361
96.4±3.100
π0.5
0.828
12.95
73.3
GPT6-Astra
0.605
18.26
60.9
TABLE III : Reproducibility of difficulty calibration under demonstration recollection and retraining, and two new calibration policies. Each metric is computed over all 112 tasks.
Fig. 6 : Task success rates with different task properties for all tasks in ManiLadder.
Rigid-Body Manipulation
Deformable-Object Manipulation
SG
SX
DG
DX
SG
SX
DG
DX
Level 1
48.0%( ↓ 23.5% )
95.5%( ↑ 27.5% )
83.0%( ↑ 9.5% )
59.5%( ↓ 13.5% )
70.7%( ↑ 0.7% )
92.7%( ↑ 16.7% )
68.0%( ↓ 2.0% )
90.7%( ↑ 20.7% )
Level 2
38.0%( ↑ 2.5% )
52.5%( ↑ 17.0% )
41.5%( ↑ 2.5% )
13.5%( ↓ 27.5% )
50.7%( ↑ 13.3% )
75.3%( ↑ 36.0% )
30.7%( ↓ 2.0% )
28.7%( ↓ 10.0% )
Level 3
11.5%( ↑ 3.0% )
9.0%(0.0%)
3.5%( ↓ 5.0% )
6.0%( ↓ 3.5% )
48.7%( ↑ 34.0% )
28.0%( ↑ 11.3% )
17.3%( ↑ 4.0% )
12.7%( ↓ 6.0% )
Level 4
5.0%( ↑ 5.0% )
2.0%( ↑ 2.0% )
0.5%( ↑ 0.5% )
2.0%( ↑ 2.0% )
2.7%( ↑ 2.7% )
2.0%( ↑ 0.7% )
0.7%( ↑ 0.7% )
1.3%( ↑ 1.3% )
TABLE IV : Success rates of π0.5 [ 16 ] and changes relative to the small flow-matching policy on each task suite in ManiLadder. Each suite score is the unweighted mean over its tasks; changes in parentheses are percentage-point differences. We use G for grippers, X for dexterous hands, S for single- arm robots, and D for dual-arm robots.
Rigid-Body Manipulation
Deformable-Object Manipulation
SG
SX
DG
DX
SG
SX
DG
DX
Level 1
55.5%( ↑ 7.5% )
78.0%( ↓ 17.5% )
83.5%( ↑ 0.5% )
33.0%( ↓ 26.5% )
54.7%( ↓ 16.0% )
95.3%( ↑ 2.7% )
69.3%( ↑ 1.3% )
76.7%( ↓ 14.0% )
Level 2
44.5%( ↑ 6.5% )
65.5%( ↑ 13.0% )
25.0%( ↓ 16.5% )
51.5%( ↑ 38.0% )
30.7%( ↓ 20.0% )
49.3%( ↓ 26.0% )
94.0%( ↑ 63.3% )
8.0%( ↓ 20.7% )
Level 3
41.5%( ↑ 30.0% )
27.0%( ↑ 18.0% )
10.5%( ↑ 7.0% )
37.3%( ↑ 31.3% )
37.3%( ↓ 11.3% )
34.7%( ↑ 6.7% )
14.6%( ↓ 2.7% )
0.0%( ↓ 12.7% )
Level 4
0.0%( ↓ 5.0% )
7.0%( ↑ 5.0% )
38.0%( ↑ 37.5% )
0.0%( ↓ 2.0% )
3.3%( ↑ 0.7% )
13.3%( ↑ 11.3% )
0.0%( ↓ 0.7% )
15.3%( ↑ 14.0% )
TABLE V : Success rates of GPT6-Astra and changes relative to π0.5 on each task suite in ManiLadder. Each suite score is the unweighted mean over its tasks; changes in parentheses are percentage-point differences. We use G for grippers, X for dexterous hands, S for single-arm robots, and D for dual-arm robots.
Setting
All Tasks
Rigid-Body
Deformable
Isolated L1
57.9%
57.5%
58.5%
L2 → L1
62.0%( ↑ 4.1% )
59.3%( ↑ 1.8% )
65.7%( ↑ 7.2% )
Isolated L2
24.1%
19.8%
30.0%
L1 → L2
21.8%( ↓ 2.3% )
20.4%( ↑ 0.6% )
23.7%( ↓ 6.3% )
L3 → L2
19.7%( ↓ 4.4% )
19.0%( ↓ 0.8% )
20.7%( ↓ 9.3% )
Isolated L3
0.5%
0.3%
1.0%
TABLE VI : Transfer across difficulty levels with π0.5 . A→B denotes we first finetune π0.5 on all tasks of level A , then finetune and evaluate it on all tasks of level B ; Isolated trains on level B only. Scores are means over tasks.
Setting
Isolated
Transfer
Deformable → Rigid-Body
17.3%
19.9%( ↑ 2.6% )
Rigid-Body → Deformable
16.7%
25.8%( ↑ 9.2% )
Dexterous → Gripper
16.6%
22.7%( ↑ 6.1% )
Gripper → Dexterous
21.0%
28.0%( ↑ 7.0% )
Dual-Arm → Single-Arm
24.1%
24.0%( ↓ 0.1% )
Single-Arm → Dual-Arm
12.0%
13.9%( ↑ 1.9% )
TABLE VII : Transfer across task suites within Level 2 with π0.5 . A→B denotes that we first finetune π0.5 on all tasks of suite A , then finetune and evaluate it on suite B ; Isolated trains on suite B only.
Rigid-Body Manipulation
Deformable-Object Manipulation
SG
SX
DG
DX
SG
SX
DG
DX
Level 1
15.5%( ↓ 32.5% )
44.0%( ↓ 51.5% )
16.5%( ↓ 66.5% )
29.0%( ↓ 30.5% )
12.0%( ↓ 58.7% )
36.7%( ↓ 56.0% )
15.3%( ↓ 52.7% )
37.3%( ↓ 53.4% )
Level 2
10.0%( ↓ 28.0% )
18.0%( ↓ 34.5% )
8.0%( ↓ 33.5% )
1.0%( ↓ 12.5% )
4.7%( ↓ 46.0% )
7.3%( ↓ 68.0% )
3.3%( ↓ 27.4% )
12.0%( ↓ 16.7% )
Level 3
0.0%( ↓ 11.5% )
0.0%( ↓ 9.0% )
0.0%( ↓ 3.5% )
0.0%( ↓ 6.0% )
0.7%( ↓ 48.0% )
0.0%( ↓ 28.0% )
0.0%( ↓ 17.3% )
0.0%( ↓ 12.7% )
Level 4
0.0%( ↓ 5.0% )
1.0%( ↓ 1.0% )
0.0%( ↓ 0.5% )
0.0%( ↓ 2.0% )
0.0%( ↓ 2.7% )
0.0%( ↓ 2.0% )
0.0%( ↓ 0.7% )
0.0%( ↓ 1.3% )
TABLE VIII : Success rates of π0.5 trained jointly on all 112 tasks of ManiLadder ( MT-A ) and changes relative to single-task training ( ST , Table IV ) on each task suite. Each suite score is the mean over its tasks. We use G for grippers, X for dexterous hands, S for single-arm robots, and D for dual-arm robots.
Fig. 7 : We choose 8 tasks from four different task difficulty levels for real-world experiments: PickPlacePen , HangMug , Stack6Blocks , PingPongTransport , UnfoldShirt , FoldShirt , BagHanging , and BuildTent , with corresponding simulation tasks at the bottom. Results are in Table IX .
Task
PickPlacePen
HangMug
Stack6Blocks
PingPongTransport
UnfoldShirt
FoldShirt
BagHanging
BuildTent
Real-world
100%
40%
10%
0%
80%
40%
10%
0%
Simulation
80%
44%
10%
0%
70%
44%
6%
0%
TABLE IX : Comparison between real-world and the corresponding simulation task success rates.
Fig. 8 : Data scaling of π0.5 on the 16 Level-2 rigid-body tasks. One policy is trained jointly on all tasks with 20–200 demonstrations per task. The black curve is the mean success rate over all 16 tasks; colored curves are suite means (4 tasks, 50 episodes each).
General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-∞-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai
Petko Petkov, Giuliano Albanese, Nikolay Nikolov +3
INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA
Humanoid robots hold great promise for operating in human-centered environments, yet ladder climbing remains one of the most challenging tasks due to sparse footholds and handholds, complex whole-body coordination, and sensitivity to perception and control errors. We present LadderMan, a unified system that enables humanoid robots to robustly climb diverse ladders and perform manipulation under such constrained conditions. Our climbing policy is built on a scalable two-stage learning pipeline, where we use hybrid motion tracking to learn multiple climbing experts from a single reference motion, and distill these experts into a unified depth-based visuomotor climbing policy via hybrid imitation and reinforcement learning. To enable real-world deployment, we leverage vision foundation models to bridge the sim-to-real gap in depth perception. Building on the learned climbing policy, we further train a separate manipulation policy using a dual-agent formulation, allowing stable on-ladder manipulation via teleoperation. Experiments demonstrate that LadderMan achieves robust ladder climbing across a wide range of geometries, successfully transfers to real-world hardware in a zero-shot manner, and supports various manipulation tasks under challenging ladder constraints. Video results are available at https://ladderman-robot.github.io .
Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.