Recent robot manipulation benchmarks have enabled systematic evaluation of general-purpose robot policies across diverse objects, scenes, instructions, and embodiments. However, most existing benchmarks still assume that the robot is the only active agent and that task semantics are determined primarily by the instruction and object state. This assumption overlooks a common source of real-world variation: humans actively sharing the workspace. In such settings, humans may reveal an implicit goal, complete part of the task, temporarily change object availability, or constrain safe robot motion while working nearby. We introduce HARP-Bench, a benchmark for evaluating robot manipulation policies in workspaces shared with active humans. HARP-Bench treats human activity as part of the manipulation problem, rather than as background scene variation. It organizes human-present manipulation into four conditions: Cueing, Contribution, Interruption, and Co-presence. These conditions are instantiated through everyday manipulation tasks, including paired human-present variants for controlled comparison with standard manipulation. HARP-Bench also provides a demonstration-generation pipeline for dynamic human-present settings. We evaluate VLA models, imitation-learning methods, and heuristic policies on HARP-Bench. Human-present variations challenge current policies, and higher task success does not consistently improve safety. Quantitative failure analysis identifies manipulation and recovery as the dominant remaining challenges. Training with HARP-Bench demonstrations improves human-present performance in simulation. A real-world study on one task family shows improved performance under human contribution, but not interruption, providing initial evidence of the data's utility and identifying limits of transfer. Code is available at https://github.com/UMass-Embodied-AGI/harp.
Figures & tables
Figure 1: Task design in HARP-Bench. Human activity is organized into four evaluation conditions. Cueing changes what the robot should do, Contribution changes which subtasks remain, Interruption changes when or how the robot can act, and Co-presence adds motion constraints while leaving the original goal unchanged.
Benchmark
Manip.
Human
Active
Visual
Data
RoboTwin2 [ 9 ]
✓
✗
✗
✓
✓
VLABench [ 7 ]
✓
✗
✗
✓
✓
RoboCasa [ 6 ]
✓
✗
✗
✓
✓
RoboWits [ 8 ]
✓
✗
✗
✓
✓
PARTNR [ 11 ]
✗
✓
✓
✗
✗
HumanTHOR [ 12 ]
✗
✓
✓
✓
✗
Table 1: Comparison with related benchmarks. Manip. denotes low-level manipulation; Human denotes active humans in the workspace; Active denotes human activity during policy rollout; Visual denotes VLA-style visual observations; Data denotes demonstration or dataset generation support.
Figure 2: Overview of the HARP-Bench pipeline. Each task initializes a scene with objects, a robot embodiment, and avatar-driven human activity. The same task setup supports two usage modes: closed-loop evaluation, where avatar events occur during execution and metrics are computed, and data generation, where scripted rollouts are recorded for robot learning.
Task
Full-state expert
No-intent expert
ACT
DP
π0
SmolVLA
Cueing
Deliver to Human
1.00
0.30
0.00
0.00
0.00
0.00
Pour Water
0.99
0.91
0.59
0.70
0.63
0.27
Oil Bottle Recovery
0.95
0.01
0.00
0.09
0.07
0.09
Open Microwave
1.00
1.00
0.20
0.37
0.39
0.45
Feed Documents
1.00
0.14
0.00
0.00
0.00
0.00
Table 2: Task-level policy evaluation on HARP-Bench. We report collision-free success for each of the 37 tasks and averages over each condition and all tasks. Best nonzero values among the four learned policies in each row are highlighted, including ties.
Policy
Succ. ↑
Violat. ↓
Min. Dist. (m) ↑
Collision ↓
Full-state expert
0.99
0.057
0.158
0.000
No-intent expert
0.48
0.094
0.126
0.299
ACT
0.10
0.080
0.133
0.102
DP
0.09
0.095
0.120
0.174
π0
0.19
0.088
0.095
0.207
SmolVLA
0.06
0.093
0.104
0.214
Table 3: Task success and safety of policies. Results cover the 37 benchmark tasks. Success requires task completion without robot-avatar collision. Violat. is the mean fraction of timesteps with robot-avatar distance below 0.10 m; Min. Dist. is the mean episode minimum distance in meters; Collision is the fraction of episodes with robot-avatar contact. Bold marks the best value among the four learned policies in each column.
Training data
Base
Contribution
Interruption
Co-presence
Human Avg.
Base-only
0.42
0.10
0.05
0.14
0.10
Full-task
0.41 ( −0.01 )
0.18 ( +0.08 )
0.08 ( +0.03 )
0.19 ( +0.05 )
0.15 ( +0.05 )
Table 4: Effect of human-present data with per-task ACT training. Unlike Table 2 , ACT is trained separately for each task, using 300 demonstrations per task (7,500 total per setting). Results are averaged within each human-present condition; Human Avg. summarizes these conditions. Parentheses show absolute gains over Base-only training.
Figure 3: Case studies in simulation and real-world experiments. Representative rollouts illustrate missed human contributions, recovery failures after human intervention, and successful execution with human-present training data.
Training data
Base
Contribution
Interruption
Base only
0.40
0.13
0.00
Matched human-present
–
0.33
0.00
Table 5: Real-world evaluation on the xArm dump-bin task family. ACT success over 30 trials per evaluated setting, with matched training data sizes and steps.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Description
Cueing
Deliver to Human
A requested object is placed on the table. The robot must pick up the object and deliver it to the human’s hand.
Pour Water
A human drinks from a cup, cueing the robot to grasp a mug and pour water into the cup.
Oil Bottle Recovery
An oil bottle is knocked over near the cooking area. The robot must recover the bottle, place it upright, and move it to a safe region.
Open Microwave
The human carries a plate near a microwave. The robot must grasp the microwave handle, open the door, and leave the loading path clear.
Feed Documents
The human stamps two documents in sequence while the robot clears the first stamped document and pushes the second into the stamping zone.
Appendix
Table A2: Descriptions of all 37 HARP-Bench tasks across Cueing, Contribution, Interruption, and Co-presence settings.
Figure A1: Object asset pool used in HARP-Bench. The pool contains 147 object asset directories, including rigid tabletop objects, articulated fixtures, containers, food items, tools, appliances, blocks, and table variants. Some assets are adapted from existing simulation asset sources, including RoboTwin-OD [ 9 ] and SAPIEN [ 50 , 51 , 52 ] , while others are curated or organized for HARP-Bench. Object identity is used to define task semantics and provide visual and physical variation, but not every asset appears in every rollout.
Human-present condition
Number of tasks
Cueing
7
Contribution
10
Interruption
10
Co-presence
10
Total
37
Appendix
Table A3: Task counts in the 37-task HARP-Bench benchmark.
Figure A2: Avatar appearance pool used in HARP-Bench. We use 46 skinned avatar models from Virtual Community [ 43 ] . Avatar identity is used as visual appearance variation and is not tied to task semantics.
Figure A3: Avatar motion and control primitives used in HARP-Bench.