ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

Guoting Wei; Dawei Yan; Xia Yuan; Gengming Zhang; Yelin He; Guodong Du; Jiaquan Ye; Heng Zhang

cs.ROOct 8, 2026

ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

Authors: Guoting Wei, Dawei Yan, Xia Yuan, Gengming Zhang, Yelin He, Guodong Du, Jiaquan Ye, Heng Zhang, +5 more

Organizations: Nanjing University of Science and Technology · Intellifusion · Northwestern Polytechnical University

Abstract

Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natural language instruction. Learning and evaluating these skills present three challenges: similar observations under a fixed task instruction may make skill selection ambiguous; even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution; and task-level metrics hinder skill-specific diagnosis, while early failures leave later skills untested. We therefore introduce ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities. Its dataset contains 137,899 segments across 21 skill types and 417 subtasks, and its benchmark contains 1,260 test instances. Correspondingly, ManiUnit pairs each segment with an explicit subtask instruction; measures sensitivity to perturbations of the robot's starting base position or joint configuration; and restores intermediate simulator states and defines local success conditions so that each skill can be evaluated without executing preceding stages. Evaluations of representative vision-language-action (VLA) policies show that similar aggregate scores can hide substantial per-skill differences. The tested starting-state perturbations also degrade execution: on the full benchmark, joint perturbations reduce success rates by approximately 56% relative to those from demonstrated starting states. On two long-horizon activities, a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations. The trained skills further support complete-task execution on these activities, as coordinating the task and skill policies through a planner raises full-task success from 4.0% to 18.0%.

Figures & tables

Figure 1: ManiUnit overview. Left : Long-horizon execution poses three challenges for skill learning and evaluation. Center : Correspondingly, ManiUnit provides explicit subtask instructions to guide manipulation. Controlled tests measure starting-state sensitivity, while restored states and local BDDL goals enable independent skill evaluation. Right : ManiUnit organizes manipulation segments by skill type and subtask. Bottom : In a focused study on two long-horizon activities, skill-level training improves local manipulation success, and combining the task and skill policies through a planner improves full-task success.

Figure 2: ManiUnit dataset and benchmark construction. Top: Source demonstrations are converted into manipulation segments with curated subtask instructions. Bottom: Each evaluation anchor marks a selected operation start, whose restored state, subtask instruction, and local BDDL goal enable independent skill evaluation across original and perturbed starting states.

Panel of Figure 3 — Figure 3: Baseline performance on ManiUnit. (a) and (c) show success rates averaged equally across skill types under each robot starting-state condition; Overall averages the three conditions. (b) and (d) show the performance of PI and GR00T on selected skills, averaged across conditions.

Model	Original	Pos.	Color	Dist.
StarVLA-PI	68.6%	40.0%	60.0%	62.9%
StarVLA-GR00T	71.4%	20.0%	65.7%	62.9%
StarVLA-Cosmos2	60.0%	22.9%	54.3%	60.0%
$\pi_{0.5}$	91.4%	74.3%	80.0%	80.0%

Table 1: Pick Up success rates for Original and three scene variations: target position (Pos.), target color, and distractor presence (Dist.).

Figure 4: Skill-level vs. task-level learning. (a) shows overall success rates and success rates under each robot starting-state condition for the task and skill policies. (b) shows their success rates on the five subtasks, averaged across starting-state conditions.

	Task policy		Combined system
Activity	Full-task success	Mean Q-score	Full-task success	Mean Q-score
Radio	4.0%	4.0%	24.0%	24.0%
Trash	4.0%	18.0%	12.0%	26.0%
Overall	4.0%	11.0%	18.0%	25.0%

Table 2: Long-horizon task execution. Full-task success rates and mean Q-scores for the task policy alone and the combined system, which uses a planner to coordinate the task and skill policies. Overall averages the results across Turning On Radio and Picking Up Trash.

Dataset	Mobile	Training		Evaluation
Dataset		Skill labels	Language instr.	Independent skill eval.	Robot-state perturb.
LIBERO ( Liu et al., 2023 )	$\times$	$\times$	$\checkmark$	$\times$	$\times$
CALVIN ( Mees et al., 2022 )	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
RoboTwin 2.0 ( Chen et al., 2026 )	$\times$	$\times$	$\checkmark$	$\times$	$\times$
RoboCasa365 ( Nasiriany et al., 2026 )	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$
BEHAVIOR-1K ( Li et al., 2023 )	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$

Table 3: Comparison of manipulation datasets and benchmarks in terms of mobility, training annotations, and skill evaluation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Skill type	Source activities	Subtasks	Segments
Pick Up	50	153	52,251
Place	45	184	49,574
Chop	6	18	7,624
Open Door	24	9	7,476
Close Door	20	7	6,048
Pour	7	14	2,993

Appendix

Table 4: Skill-type inventory of ManiUnit-Full.

Parameter	Value
Monitored arm joints	First 3 per arm
Reference window	500 frames
Smoothing window	5 frames
Minimum joint departure	$15^{\circ}$
Minimum qualifying duration	5 frames
Gripper position-change threshold	$0.002\,\mathrm{m}$

Appendix

Table 5: Default parameters for temporal boundary refinement.

	StarVLA-PI				StarVLA-GR00T
Skill	Original	Base	Joint	Mean	Original	Base	Joint	Mean
Attach	21.7%	11.7%	3.3%	12.2%	26.7%	1.7%	0.0%	9.4%
Chop	41.7%	31.7%	46.7%	40.0%	43.3%	31.7%	41.7%	38.9%
Close Door	83.3%	83.3%	50.0%	72.2%	90.0%	83.3%	46.7%	73.3%
Close Drawer	86.7%	95.0%	35.0%	72.2%	98.3%	98.3%	46.7%	81.1%
Close Lid	73.3%	66.7%	31.7%	57.2%	75.0%	61.7%	35.0%	57.2%

Appendix

Table 6: Per-skill success rates on ManiUnit-Full (%). Base and Joint denote Base Perturbation and Joint Perturbation.

	StarVLA-PI			StarVLA-GR00T
Skill	Original	Base	Joint	Original	Base	Joint
Close Door	95.0%	97.5%	45.0%	95.0%	90.0%	42.5%
Open Door	80.0%	96.7%	13.3%	93.3%	83.3%	10.0%
Pick Up	68.6%	77.1%	8.6%	71.4%	80.0%	14.3%
Place	91.4%	74.3%	80.0%	85.7%	57.1%	68.6%
Pour	22.9%	5.7%	8.6%	57.1%	25.7%	37.1%

Appendix

Table 7: Per-skill success rates on ManiUnit-Lite (%). Base and Joint denote Base Perturbation and Joint Perturbation.

Figure 5: Successful rollouts of (a) Turning On Radio and (b) Picking Up Trash. A planner coordinates a task policy for navigation and task-level control with a ManiUnit-trained skill policy for manipulation. Colors indicate the active policy.

Figure 6: Object handling: recorded examples with their subtask instructions.

Figure 7: Object handling: recorded examples with their subtask instructions.

Figure 8: Opening and closing: recorded examples with their subtask instructions.

Figure 9: Opening and closing: recorded examples with their subtask instructions.

Figure 10: Activation and tool use: recorded examples with their subtask instructions.

Figure 11: Activation and tool use: recorded examples with their subtask instructions.

Explore similar work

Aug 31, 2026cs.RO

Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks

Reliable execution of long-horizon mobile manipulation tasks remains challenging because overall task success depends on the successful completion of multiple constituent skills. Existing benchmarks, however, still rely primarily on full-task rollouts and aggregate task-level metrics, making intermediate failures difficult to observe and analyze. We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks around executable constituent skills. It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories. Each instance pairs a skill instruction with an aligned observation-action segment, and is further associated with a restorable intermediate state and a skill success condition to enable independent evaluation under valid preconditions. We further introduce trajectory-level and skill-level metrics to characterize policy capability beyond aggregate task success. Extensive experiments across representative VLA policies including pi0.5 and GR00T on the complete 50-task benchmark show that failures are highly non-uniform across skills, with contact-rich manipulation skills forming persistent bottlenecks. These results demonstrate that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies. Behavior-Skill is publicly available at https://github.com/nubot-nudt/Behavior-Skill.

Chunyun Ma, Lun Luo, Xingjian Luo +7

College of Intelligence Science and Technology and National Key Laboratory of Equipment State Sensing and Smart Support, National University of Defense Technology, Changsha, China. · XPeng Inc., Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong, China. +1

Long-Horizon Robotic Manipulation Mobile Manipulation Robotic Policy Evaluation Robot Manipulation Benchmarks

Ask PDF

Apr 18, 2026cs.RO

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

Robotic manipulation policies often degrade over extended horizons, yet existing benchmarks provide limited insight into why such failures occur. Most prior benchmarks are either simulation-based or report aggregate success, making it difficult to disentangle the distinct sources of temporal difficulty in real-world execution. We introduce LongBench, a real-world benchmark for evaluating long-horizon manipulation. LongBench consists of over 1,000 real-world episodes, covering two complementary regimes: Context-Independent (fully observable) and Context-Dependent (ambiguity-driven). By organizing tasks into capability- and ambiguity-specific subsets, LongBench enables mechanism-aware evaluation of execution robustness, temporal consistency, and context-dependent reasoning. Evaluating six state-of-the-art policies reveals that long-horizon performance is not governed by a single factor. We observe that performance in fully observable settings is more strongly associated with execution robustness, while contextual difficulty varies across tasks and is not consistently improved by memory-based methods. We hope that LongBench serves as a useful benchmark for studying long-horizon manipulation and for developing policies with stronger robustness across both execution and contextual challenges.

Xueyao Chen, Jingkai Jia, Tong Yang +3

Fudan University

Long-Horizon Robotic Manipulation Robotic Policy Evaluation Robotic Manipulation Robot Manipulation Benchmarks

Ask PDF

Feb 5, 2026cs.RO

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first framework to verify VLA architectures before real-world deployment and introduce MobileManiBench, a large-scale benchmark for mobile-based robotic manipulation. Built on NVIDIA Isaac Sim and powered by reinforcement learning, our pipeline autonomously generates diverse manipulation trajectories with rich annotations (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions). MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories. This design enables controlled, scalable studies of robot embodiments, sensing modalities, and policy architectures, accelerating research on data efficiency and generalization. We benchmark representative VLA models and report insights into perception, reasoning, and control in complex simulated environments, with all code, datasets, and models publicly released.

Wenbo Wang, Fangyun Wei, QiXiu Li +5

University of Sydney · 1Microsoft Research Asia · 3Tsinghua University

Mobile Manipulation Robotics Simulation Synthetic Data Generation for Robotics Vision-Language-Action Models Robot Manipulation Benchmarks

Ask PDF