Recent robot manipulation benchmarks have enabled systematic evaluation of general-purpose robot policies across diverse objects, scenes, instructions, and embodiments. However, most existing benchmarks still assume that the robot is the only active agent and that task semantics are determined primarily by the instruction and object state. This assumption overlooks a common source of real-world variation: humans actively sharing the workspace. In such settings, humans may reveal an implicit goal, complete part of the task, temporarily change object availability, or constrain safe robot motion while working nearby. We introduce HARP-Bench, a benchmark for evaluating robot manipulation policies in workspaces shared with active humans. HARP-Bench treats human activity as part of the manipulation problem, rather than as background scene variation. It organizes human-present manipulation into four conditions: Cueing, Contribution, Interruption, and Co-presence. These conditions are instantiated through everyday manipulation tasks, including paired human-present variants for controlled comparison with standard manipulation. HARP-Bench also provides a demonstration-generation pipeline for dynamic human-present settings. We evaluate VLA models, imitation-learning methods, and heuristic policies on HARP-Bench. Human-present variations challenge current policies, and higher task success does not consistently improve safety. Quantitative failure analysis identifies manipulation and recovery as the dominant remaining challenges. Training with HARP-Bench demonstrations improves human-present performance in simulation. A real-world study on one task family shows improved performance under human contribution, but not interruption, providing initial evidence of the data's utility and identifying limits of transfer. Code is available at https://github.com/UMass-Embodied-AGI/harp.
Figures & tables
Figure 1: Task design in HARP-Bench. Human activity is organized into four evaluation conditions. Cueing changes what the robot should do, Contribution changes which subtasks remain, Interruption changes when or how the robot can act, and Co-presence adds motion constraints while leaving the original goal unchanged.
Benchmark
Manip.
Human
Active
Visual
Data
RoboTwin2 [ 9 ]
✓
✗
✗
✓
✓
VLABench [ 7 ]
✓
✗
✗
✓
✓
RoboCasa [ 6 ]
✓
✗
✗
✓
✓
RoboWits [ 8 ]
✓
✗
✗
✓
✓
PARTNR [ 11 ]
✗
✓
✓
✗
✗
HumanTHOR [ 12 ]
✗
✓
✓
✓
✗
Table 1: Comparison with related benchmarks. Manip. denotes low-level manipulation; Human denotes active humans in the workspace; Active denotes human activity during policy rollout; Visual denotes VLA-style visual observations; Data denotes demonstration or dataset generation support.
Figure 2: Overview of the HARP-Bench pipeline. Each task initializes a scene with objects, a robot embodiment, and avatar-driven human activity. The same task setup supports two usage modes: closed-loop evaluation, where avatar events occur during execution and metrics are computed, and data generation, where scripted rollouts are recorded for robot learning.
Task
Full-state expert
No-intent expert
ACT
DP
π0
SmolVLA
Cueing
Deliver to Human
1.00
0.30
0.00
0.00
0.00
0.00
Pour Water
0.99
0.91
0.59
0.70
0.63
0.27
Oil Bottle Recovery
0.95
0.01
0.00
0.09
0.07
0.09
Open Microwave
1.00
1.00
0.20
0.37
0.39
0.45
Feed Documents
1.00
0.14
0.00
0.00
0.00
0.00
Table 2: Task-level policy evaluation on HARP-Bench. We report collision-free success for each of the 37 tasks and averages over each condition and all tasks. Best nonzero values among the four learned policies in each row are highlighted, including ties.
Policy
Succ. ↑
Violat. ↓
Min. Dist. (m) ↑
Collision ↓
Full-state expert
0.99
0.057
0.158
0.000
No-intent expert
0.48
0.094
0.126
0.299
ACT
0.10
0.080
0.133
0.102
DP
0.09
0.095
0.120
0.174
π0
0.19
0.088
0.095
0.207
SmolVLA
0.06
0.093
0.104
0.214
Table 3: Task success and safety of policies. Results cover the 37 benchmark tasks. Success requires task completion without robot-avatar collision. Violat. is the mean fraction of timesteps with robot-avatar distance below 0.10 m; Min. Dist. is the mean episode minimum distance in meters; Collision is the fraction of episodes with robot-avatar contact. Bold marks the best value among the four learned policies in each column.
Training data
Base
Contribution
Interruption
Co-presence
Human Avg.
Base-only
0.42
0.10
0.05
0.14
0.10
Full-task
0.41 ( −0.01 )
0.18 ( +0.08 )
0.08 ( +0.03 )
0.19 ( +0.05 )
0.15 ( +0.05 )
Table 4: Effect of human-present data with per-task ACT training. Unlike Table 2 , ACT is trained separately for each task, using 300 demonstrations per task (7,500 total per setting). Results are averaged within each human-present condition; Human Avg. summarizes these conditions. Parentheses show absolute gains over Base-only training.
Figure 3: Case studies in simulation and real-world experiments. Representative rollouts illustrate missed human contributions, recovery failures after human intervention, and successful execution with human-present training data.
Training data
Base
Contribution
Interruption
Base only
0.40
0.13
0.00
Matched human-present
–
0.33
0.00
Table 5: Real-world evaluation on the xArm dump-bin task family. ACT success over 30 trials per evaluated setting, with matched training data sizes and steps.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Description
Cueing
Deliver to Human
A requested object is placed on the table. The robot must pick up the object and deliver it to the human’s hand.
Pour Water
A human drinks from a cup, cueing the robot to grasp a mug and pour water into the cup.
Oil Bottle Recovery
An oil bottle is knocked over near the cooking area. The robot must recover the bottle, place it upright, and move it to a safe region.
Open Microwave
The human carries a plate near a microwave. The robot must grasp the microwave handle, open the door, and leave the loading path clear.
Feed Documents
The human stamps two documents in sequence while the robot clears the first stamped document and pushes the second into the stamping zone.
Appendix
Table A2: Descriptions of all 37 HARP-Bench tasks across Cueing, Contribution, Interruption, and Co-presence settings.
Figure A1: Object asset pool used in HARP-Bench. The pool contains 147 object asset directories, including rigid tabletop objects, articulated fixtures, containers, food items, tools, appliances, blocks, and table variants. Some assets are adapted from existing simulation asset sources, including RoboTwin-OD [ 9 ] and SAPIEN [ 50 , 51 , 52 ] , while others are curated or organized for HARP-Bench. Object identity is used to define task semantics and provide visual and physical variation, but not every asset appears in every rollout.
Human-present condition
Number of tasks
Cueing
7
Contribution
10
Interruption
10
Co-presence
10
Total
37
Appendix
Table A3: Task counts in the 37-task HARP-Bench benchmark.
Figure A2: Avatar appearance pool used in HARP-Bench. We use 46 skinned avatar models from Virtual Community [ 43 ] . Avatar identity is used as visual appearance variation and is not tied to task semantics.
Figure A3: Avatar motion and control primitives used in HARP-Bench.
Learning robot manipulation policies from human video demonstrations constitutes a promising avenue for scalable robot learning. However, comparing different human-to-robot (H2R) transfer methods remains challenging, as existing approaches are evaluated under different settings, including differing task suites, scene layouts, object instances, and amounts of robot supervision. To address this challenge, we present H2RBench, a Real2Sim benchmark for evaluating H2R transfer methods. H2RBench provides a standardized protocol built on real human video demonstrations and simulated robot demonstrations, and includes four manipulation tasks spanning diverse interaction requirements. We evaluate multiple representative H2R transfer methods, each adopting a different strategy for bridging the embodiment gap. Using H2RBench, we systematically characterize how each method scales with the amount of human demonstrations, revealing that methods differ substantially in their ability to leverage additional human data. We further show that simulation performance is broadly predictive of real-world robot performance, with an overall Pearson correlation of r = 0.89, Spearman correlation of \r{ho} = 0.85 and Mean Maximum Rank Violation (MMRV) of 0.06 across method-task configurations. These results establish H2RBench as a practical and scalable benchmark for comparative H2R evaluation prior to real-world deployment.
Chuyang Xiao, Haotian Zhan, Sriram Krishna +4
Robotics Institute, Carnegie Mellon University · University of Michigan · Toyota Research Institute
As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/.
Kyochul Jang, Seohyeon Park, Ohchul Kwon +9
Seoul National University · University of Massachusetts Amherst · Google Research
Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.