ManiLadder: Benchmarking Robot Manipulation Through a Categorized and Multi-Difficulty Level Task Ladder
Authors: Chongkai Gao, Jieao Shi, Zhaoyu Mu, Yudi Lin, Xuanwei Liu, Wei Ken Ng, Baiye Cheng, Anqi Chen, Kerun Liu, Zhuoran Li, Xuchuan Huang, Haoyang Li, Shengjia Zhu, Nga Teng Chan, Yuxuan Zhang, Xingyu Chen, Lin Shao
Organizations: National University of Singapore · Nanyang Technological University · University of Toronto · Huazhong University of Science and Technology · Nanjing University · KTH · Shanghai Jiao Tong University · Peking University · HKUST
Recent advances in robotic manipulation have enabled increasingly complex and dexterous behaviors, yet existing benchmarks still focus on simple pick-and-place tasks and lack a systematic framework for measuring progress in robot learning along the axis of task difficulty. We present ManiLadder, a large-scale and organized simulation benchmark for robotic manipulation. ManiLadder contains 112 carefully designed tasks stratified into 4 difficulty levels, with 8 categories in each level, spanning rigid-body and deformable-object manipulation, single-arm and dual-arm settings, and gripper and dexterous-hand end-effectors. To calibrate difficulty, we fix the data amount the same for every task and use a unified flow- matching imitation learning policy under the same training and evaluation protocols, then use its evaluation success rate as an operational empirical difficulty signal. All tasks are iteratively refined until they fall into target difficulty ranges, which are then verified across different random seeds, policies, and agentic learning systems. We further analyze the task properties, compare different policy performances and training settings, analyze task transfer and data scaling behaviors, and conduct real-world experiments showing that the designed tasks preserve their difficulty ordering beyond simulation. ManiLad- der provides a calibrated ladder for measuring, comparing, and advancing robotic manipulation capabilities. More results are available at https://maniladder-benchmark.github.io/.
Figures & tables
Benchmark
Simulator
No. of tasks
Auto Demo
Difficulty Taxonomy
Deformable objects
Robots
MetaWorld [ 1 ]
Mujoco [ 2 ]
50
✓
✗
✗
S,G
RoboDojo [ 3 ]
Isaac Sim [ 4 ]
42
✓
✗
✓
D,G
COLOSSEUM [ 5 ]
V-Rep [ 6 ]
20
✓
✗
✗
S,G
RoboCasa [ 7 ]
Mujoco [ 2 ]
100
✓
✗
✗
S,X,G,D,M
Behavior 1K [ 8 ]
Isaac Sim [ 4 ]
1000 (100 released)
✗
✗
✓
S,G,M
LIBERO [ 9 ]
Mujoco [ 2 ]
120
✗
✓
✗
S,G
TABLE I : Comparison between ManiLadder and other simulation robot manipulation benchmarks focusing on task taxonomy and difficulty levels. Auto Demo means supporting automatic data generation. We use G for robots with grippers, X for robots with dexterous hands, S for single-arm robots, D for dual-arm robots, and M for mobile robots.
TABLE II : Manipulation skill coverage of ManiLadder. We group skills according to prior literature on manipulation-primitives and contact-rich manipulation taxonomy [ 43 , 44 , 45 , 46 ] .
Fig. 2 : The objects and robots in ManiLadder. Four kinds of robots and diverse kinds of rigid-body and deformable objects are included in ManiLadder.
Fig. 3 : Left: the basic scene setup of task FoldShirt , serving as an example illustrating a typical task scenario in ManiLadder. Middle left: the domain randomization on the lighting, background, and distracting objects of FoldShirt . Middle: in each rigid-body object task suite, we design different tasks focusing on different manipulation skills, such as HangMug for pick-and-place, OpenBoxRetrival for articulated object manipulation, and PivotBook for non-prehensile manipulation. In each deformable object task suite, we design tasks including 1D, 2D, and 3D deformable objects, such as StraightenRope for rope manipulation, UnfoldShirt for cloth manipulation, and ShapeTtoI for clay manipulation. Middle right: two mobile manipulation tasks in ManiLadder. Right: the task horizon and stage number histograms.
Fig. 4 : All task success rates in ManiLadder of the small flow-matching policy, π0.5 , and GPT6-Astra agentic script policy.
Fig. 5 : Examples of task calibration and refinement. Top: the HangMug task: we progressively modify the initialization regions of the mug and rack to make the task more challenging. Bottom: the DualArmCloseBox task: we first reduce the box size, then introduce an initialization pose that requires coordinated dual-arm rotation before the box can be closed.
Setting
ρ↑
Δ (pp) ↓
A (%) ↑
3 random seeds
0.958±0.006
5.67±0.361
96.4±3.100
π0.5
0.828
12.95
73.3
GPT6-Astra
0.605
18.26
60.9
TABLE III : Reproducibility of difficulty calibration under demonstration recollection and retraining, and two new calibration policies. Each metric is computed over all 112 tasks.
Fig. 6 : Task success rates with different task properties for all tasks in ManiLadder.
Rigid-Body Manipulation
Deformable-Object Manipulation
SG
SX
DG
DX
SG
SX
DG
DX
Level 1
48.0%( ↓ 23.5% )
95.5%( ↑ 27.5% )
83.0%( ↑ 9.5% )
59.5%( ↓ 13.5% )
70.7%( ↑ 0.7% )
92.7%( ↑ 16.7% )
68.0%( ↓ 2.0% )
90.7%( ↑ 20.7% )
Level 2
38.0%( ↑ 2.5% )
52.5%( ↑ 17.0% )
41.5%( ↑ 2.5% )
13.5%( ↓ 27.5% )
50.7%( ↑ 13.3% )
75.3%( ↑ 36.0% )
30.7%( ↓ 2.0% )
28.7%( ↓ 10.0% )
Level 3
11.5%( ↑ 3.0% )
9.0%(0.0%)
3.5%( ↓ 5.0% )
6.0%( ↓ 3.5% )
48.7%( ↑ 34.0% )
28.0%( ↑ 11.3% )
17.3%( ↑ 4.0% )
12.7%( ↓ 6.0% )
Level 4
5.0%( ↑ 5.0% )
2.0%( ↑ 2.0% )
0.5%( ↑ 0.5% )
2.0%( ↑ 2.0% )
2.7%( ↑ 2.7% )
2.0%( ↑ 0.7% )
0.7%( ↑ 0.7% )
1.3%( ↑ 1.3% )
TABLE IV : Success rates of π0.5 [ 16 ] and changes relative to the small flow-matching policy on each task suite in ManiLadder. Each suite score is the unweighted mean over its tasks; changes in parentheses are percentage-point differences. We use G for grippers, X for dexterous hands, S for single- arm robots, and D for dual-arm robots.
Rigid-Body Manipulation
Deformable-Object Manipulation
SG
SX
DG
DX
SG
SX
DG
DX
Level 1
55.5%( ↑ 7.5% )
78.0%( ↓ 17.5% )
83.5%( ↑ 0.5% )
33.0%( ↓ 26.5% )
54.7%( ↓ 16.0% )
95.3%( ↑ 2.7% )
69.3%( ↑ 1.3% )
76.7%( ↓ 14.0% )
Level 2
44.5%( ↑ 6.5% )
65.5%( ↑ 13.0% )
25.0%( ↓ 16.5% )
51.5%( ↑ 38.0% )
30.7%( ↓ 20.0% )
49.3%( ↓ 26.0% )
94.0%( ↑ 63.3% )
8.0%( ↓ 20.7% )
Level 3
41.5%( ↑ 30.0% )
27.0%( ↑ 18.0% )
10.5%( ↑ 7.0% )
37.3%( ↑ 31.3% )
37.3%( ↓ 11.3% )
34.7%( ↑ 6.7% )
14.6%( ↓ 2.7% )
0.0%( ↓ 12.7% )
Level 4
0.0%( ↓ 5.0% )
7.0%( ↑ 5.0% )
38.0%( ↑ 37.5% )
0.0%( ↓ 2.0% )
3.3%( ↑ 0.7% )
13.3%( ↑ 11.3% )
0.0%( ↓ 0.7% )
15.3%( ↑ 14.0% )
TABLE V : Success rates of GPT6-Astra and changes relative to π0.5 on each task suite in ManiLadder. Each suite score is the unweighted mean over its tasks; changes in parentheses are percentage-point differences. We use G for grippers, X for dexterous hands, S for single-arm robots, and D for dual-arm robots.
Setting
All Tasks
Rigid-Body
Deformable
Isolated L1
57.9%
57.5%
58.5%
L2 → L1
62.0%( ↑ 4.1% )
59.3%( ↑ 1.8% )
65.7%( ↑ 7.2% )
Isolated L2
24.1%
19.8%
30.0%
L1 → L2
21.8%( ↓ 2.3% )
20.4%( ↑ 0.6% )
23.7%( ↓ 6.3% )
L3 → L2
19.7%( ↓ 4.4% )
19.0%( ↓ 0.8% )
20.7%( ↓ 9.3% )
Isolated L3
0.5%
0.3%
1.0%
TABLE VI : Transfer across difficulty levels with π0.5 . A→B denotes we first finetune π0.5 on all tasks of level A , then finetune and evaluate it on all tasks of level B ; Isolated trains on level B only. Scores are means over tasks.
Setting
Isolated
Transfer
Deformable → Rigid-Body
17.3%
19.9%( ↑ 2.6% )
Rigid-Body → Deformable
16.7%
25.8%( ↑ 9.2% )
Dexterous → Gripper
16.6%
22.7%( ↑ 6.1% )
Gripper → Dexterous
21.0%
28.0%( ↑ 7.0% )
Dual-Arm → Single-Arm
24.1%
24.0%( ↓ 0.1% )
Single-Arm → Dual-Arm
12.0%
13.9%( ↑ 1.9% )
TABLE VII : Transfer across task suites within Level 2 with π0.5 . A→B denotes that we first finetune π0.5 on all tasks of suite A , then finetune and evaluate it on suite B ; Isolated trains on suite B only.
Rigid-Body Manipulation
Deformable-Object Manipulation
SG
SX
DG
DX
SG
SX
DG
DX
Level 1
15.5%( ↓ 32.5% )
44.0%( ↓ 51.5% )
16.5%( ↓ 66.5% )
29.0%( ↓ 30.5% )
12.0%( ↓ 58.7% )
36.7%( ↓ 56.0% )
15.3%( ↓ 52.7% )
37.3%( ↓ 53.4% )
Level 2
10.0%( ↓ 28.0% )
18.0%( ↓ 34.5% )
8.0%( ↓ 33.5% )
1.0%( ↓ 12.5% )
4.7%( ↓ 46.0% )
7.3%( ↓ 68.0% )
3.3%( ↓ 27.4% )
12.0%( ↓ 16.7% )
Level 3
0.0%( ↓ 11.5% )
0.0%( ↓ 9.0% )
0.0%( ↓ 3.5% )
0.0%( ↓ 6.0% )
0.7%( ↓ 48.0% )
0.0%( ↓ 28.0% )
0.0%( ↓ 17.3% )
0.0%( ↓ 12.7% )
Level 4
0.0%( ↓ 5.0% )
1.0%( ↓ 1.0% )
0.0%( ↓ 0.5% )
0.0%( ↓ 2.0% )
0.0%( ↓ 2.7% )
0.0%( ↓ 2.0% )
0.0%( ↓ 0.7% )
0.0%( ↓ 1.3% )
TABLE VIII : Success rates of π0.5 trained jointly on all 112 tasks of ManiLadder ( MT-A ) and changes relative to single-task training ( ST , Table IV ) on each task suite. Each suite score is the mean over its tasks. We use G for grippers, X for dexterous hands, S for single-arm robots, and D for dual-arm robots.
Fig. 7 : We choose 8 tasks from four different task difficulty levels for real-world experiments: PickPlacePen , HangMug , Stack6Blocks , PingPongTransport , UnfoldShirt , FoldShirt , BagHanging , and BuildTent , with corresponding simulation tasks at the bottom. Results are in Table IX .
Task
PickPlacePen
HangMug
Stack6Blocks
PingPongTransport
UnfoldShirt
FoldShirt
BagHanging
BuildTent
Real-world
100%
40%
10%
0%
80%
40%
10%
0%
Simulation
80%
44%
10%
0%
70%
44%
6%
0%
TABLE IX : Comparison between real-world and the corresponding simulation task success rates.
Fig. 8 : Data scaling of π0.5 on the 16 Level-2 rigid-body tasks. One policy is trained jointly on all tasks with 20–200 demonstrations per task. The black curve is the mean success rate over all 16 tasks; colored curves are suite means (4 tasks, 50 episodes each).