cs.ROOct 13, 2026
SaveHelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution
Organizations: University of Auckland, New Zealand · MBZUAI, United Arab Emirates
Abstract
We introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Existing embodied household benchmarks typically assume that humans execute tasks correctly, but this assumption does not hold for cognitively impaired persons or older adults whose everyday activities produce structured execution errors. To address this gap, HelpMeOut operationalizes research on everyday action errors into 20000 paired activity episodes, covering nine neuropsychologically motivated error categories. In each episode, agents observe a streaming human activity trajectory and must monitor whether the execution is unfolding correctly, diagnosing any structured errors as early as possible. We evaluate learned-monitor, symbolic, change-point, and frontier LLM prompting baselines. Results show that these methods perform poorly as error monitors, often missing when execution goes wrong. These results suggest a gap left by standard goal-inference and task-completion benchmarks, making HelpMeOut a testbed for error-aware assistive agents.
Figures & tables
Figure 1: HelpMeOut task overview, shown with an instant-coffee task. a. We define the episode with goal, available objects, and initial scene. b. An episode consists of a pair of trajectories: the correct reference on top and the errorful observed execution on bottom, which contains a tool_substitution (fork used instead of spoon) and an omission (stirring skipped), with the latter confirmed only when the task ends. c. The system input is the task name, the initial scene, and the action prefix in symbolic PDDL notation; the output is an error probability, onset step, and error type. d. Evaluation labels and the four metrics (M1–M4) derived from the error injection metadata; we explain them in detail in Sec. 3.3 and Sec. 3.4 .
| Benchmark | Long- horiz. | Object interact. | Subgoal ctx. | Errorful human exec. | Neuropsych err. taxonomy | Per-step err. labels | Latency eval. | Streaming eval. |
| TextWorld [ 18 ] | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| ALFRED [ 19 ] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| VirtualHome [ 20 ] | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Watch-and-Help [ 21 ] | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| BEHAVIOR / 1K [ 22 , 23 ] | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| HandMeThat [ 24 ] | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
Table 1: Comparison with related benchmarks. ✓ present; ✗ absent; partial. Existing benchmarks evaluate goal inference or task completion under coherent human execution. HelpMeOut introduces errorful execution, a neuropsychological error taxonomy, step-level error labels, detection-latency evaluation, and streaming evaluation.
Figure 2: Neuropsychology-based error-generation pipeline. A neuropsychological profile drives the goal-control model to sample an error type and the subgoal block to edit; the edited trajectory is validated by symbolic replay, and onsets are derived by category-specific rules. The temporal view shows an omission deleting one subgoal block and shifting later steps left; the omission is confirmed at the task-end event.
| M2 | M3 | ||||||||
| Method | M1 | all | end | mid | Missed | Early | On target | FA | M4 |
| Random | 0.500 | 0.036 | 0.274 | 0.023 | 0.00 | 0.89 | 0.05 | 1.00 | 0.116 |
| Human † | 0.967 | 0.748 | – | – | 0.12 | 0.15 | 0.73 | 0.17 | 0.778 |
| Learned and change-point monitors | |||||||||
| Seq2Seq | 0.496 | 0.376 | 0.656 | 0.132 | 0.99 | 0.00 | 0.00 | 0.10 | 0.502 |
| Seq2Seq+SG | 0.554 | 0.394 | 0.665 | 0.151 | 0.98 | 0.00 | 0.00 | 0.05 | 0.548 |
Table 2: Main results on the 500-episode test set (metrics in Sec. 3.4 ). M3 at over the 175 episodes with an error before task end; FA over the 188 correct episodes (re-planner: timed-out episodes excluded). Random: positive-step rate for M1 and M2, a uniformly random monitor for M3 and M4 (mean of 20 seeds). † Human reference on the 60-episode subset (M2 random 0.027; M3 over 26 errorful and 30 correct episodes). Symbolic references use extra knowledge (Sec. 4 ).
Figure 3: M1 AUROC and M2 AUPRC of the thirteen LLM configurations on the 60-episode subset; dashed lines mark the Human, SGT, and Random references, and color marks whether reasoning is enabled. Confidence intervals and full M1–M4 results are in the supplementary material.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The taxonomy is organized by what an error disrupts: progression-level errors break the subgoal structure and ordering of the task, whereas current-action errors corrupt the selection of the current object, tool, location, or action. This organizing axis is distinct from the Goal-Control generator’s factorization (goal activation vs. executive control; Section A.6 ), which we use to sample errors.
| (a) Omission vs. commission (share of all errors) | |
| Omission | 0.44 |
| Commission | 0.56 |
| (b) Commission category weight (sums to 1.00) | |
| Sequence | 0.22 |
| Object substitution | 0.17 |
| Perseveration | 0.15 |
Table 4: Per-category error proportions used by the HelpMeOut generator in the nine-category taxonomy: (a) the omission–commission split and (b) the commission-category weights . We set these proportions from everyday-action error rates reported for clinical populations [ 10 , 9 , 2 ] , pooling the left-CVA ( ), right-CVA ( ), TBI ( ), and dementia ( ) groups. We then make small simulator-specific adjustments and add categories supported by the broader action-error literature [ 13 ] . Because an error type can be realized only when it is feasible (Section A.6 ), the released episodes were selected per category so that the released distribution approximates these targets (Figure 7 ).
| Class | #Types | Subclasses |
| location | 31 | ontop , inside (floor, countertop, table, desk, shelf, stove, bed, sofa, toilet, cabinet, refrigerator, microwave, oven, dishwasher, sink, washing_machine, dryer, toaster, closet, drawer, medicine_cabinet, bathtub, …) |
| receptacle | 31 | tableware, vessel, cooking_vessel, bag, box, basket, tray, bucket, ashcan |
| food | 49 | fruit, vegetable, drink, protein, bread_grain, spread, condiment, sauce, prepared_food, snack |
| tool | 31 | cutlery, cleaning_tool, cleansing, wrapping_tool, writing_tool, toiletry, laundry_tool |
| thing | 45 | clothing, linen, soft_furnishing, footwear, headwear, accessory, wrapping_material, paper_product, personal_item, medication |
Table 5: Object hierarchy. Five classes spanning 187 types; the 156 movable types ( receptacle / food / tool / thing ) are the “object types” of the main text, and the 31 location types are the non-movable ontop / inside locations.
| Operator (& key arguments) | Semantics |
|---|---|
| human-move (from,to) | Move between two non-movable locations. |
| human-pick-up-at-location (obj,loc) | Pick an object off a location (hand must be empty). |
| human-pick-up-from-receptacle-at-location (obj,rec,loc) | Pick an object out of/off a receptacle at a location. |
| human-put-inside-location (obj,loc) | Put the held object inside a location. |
| human-put-ontop-location (obj,loc) | Put the held object on top of a location. |
| human-put-inside-receptacle-at-location (obj,rec,loc) | Put the held object inside a receptacle. |
Table 6: The 38 action operators in HelpMeOut . The semantics column gives an informal description of each operator’s effect.
| # | Template | Anchor | Complexity |
|---|---|---|---|
| F1: Food preparation | |||
| 0 | making_toast_and_coffee | NAT | complex |
| 1 | slicing_fruit_and_refrigerating | NAT | medium |
| 2 | cooking_and_plating_protein | NAT | simple |
| 3 | storing_fruits_and_vegetables | Lawton IADL | simple |
| 4 | making_a_salad | NAT | medium |
Table 7: The 50 ADL/IADL task templates, grouped into eight families.
Figure 5: An example of one executable task template. Template #0 ( making_toast_and_coffee ) expands into first-order subgoals—the subgoal blocks , here grouped into a toast and a coffee subtask—which a planner solves into a correct reference trajectory (the shaded bands are the ). The block boundary shown on the trajectory is the structure the generator uses for progression-level errors; in generated episodes the block order is randomized, and onsets are labelled by the category rules of Section A.5 . Only representative actions are shown.
Figure 6: A correct-vs-errorful micro-example for each of the nine error categories, grouped into progression-level and current-action errors. In every error row, the step(s) that diverge from the correct execution are highlighted in red. Examples use the household domain of Figure 5 .
| Symbol / name | Value | Meaning |
| 0.48 | baseline goal salience | |
| 0.20 | per-subgoal activation decay (cognitive load) | |
| 0.12 | commission-error probability | |
| ( OMIT_THRESHOLD ) | 0.30 | omission activation threshold |
| ( ACTIVATION_NOISE_STD ) | 0.10 | activation noise std. dev. |
| min. trajectory length | 10 | shorter trajectories are filtered |
Table 8: Generator hyperparameters.
| Category | Feasible when … |
| omission | a subgoal block can be deleted while the trajectory stays executable |
| sequence | a stir follows several additions to one container, or a step-order norm applies |
| anticipation | the task requires processing an item before it is stored, served, or packed, or requires closing a gift box’s flaps before its lid |
| perseveration | an addition (or dose) can be repeated after intervening actions, with all goals still met |
| object_substitution | a same-subclass confusable object is present in the scene |
| tool_substitution | a confusable alternative tool is present in the scene |
Table 9: Per-category feasibility gate : the structural condition under which each error type can be injected at step . A sampled error is realized only when its condition holds in the current state; otherwise the generator resamples.
Figure 7: Calibration check: the generator targets the clinical per-category rates of Table 4 (blue); the released distribution (orange) is within 2.3 percentage points of the target for every category except action addition (8.4% vs. 2.8%), which frequently occurs as the second error of a multi-error episode. Shares are over all injected errors (12,817 in the release) and each series sums to 100%.
Figure 8: Summary statistics of the released HelpMeOut benchmark: (a) episode composition (paired errorful/correct), (b) task coverage, (c) mean trajectory length, (d) injected errors per trajectory, (e) annotated step-level onsets, and (f) the most frequent error categories.
| Split | Episodes | Use |
| Released corpus | 20,000 | Full benchmark release |
| Training / validation | 17,478 / 2,022 | Training and checkpoint selection |
| Held-out test set | 500 | Main baseline evaluation |
| LLM subset | 60 | Frontier-LLM and human-reference evaluation and matched baseline subset scores |
Table 10: Evaluation splits used in the main paper. The 60-episode LLM subset is drawn from the held-out test set; all other baselines are reported both on the full test set and on this common subset for fair comparison with LLMs.
| Error category | Seq2Seq | Seq2Seq+SG | SGT |
| Omission | 0.893 | 0.888 | 0.916 |
| Sequence | 0.778 | 0.741 | 0.921 |
| Anticipation | 0.386 | 0.422 | 0.607 |
| Perseveration | 0.632 | 0.664 | 0.803 |
| Object substitution | 0.145 | 0.166 | 0.196 |
| Tool substitution | 0.113 | 0.122 | 0.000 |
Table 11: Per-class M4 F1 ( ) of the learned monitors on the 500-episode test set, with correct episodes included (true category set empty). The last row gives macro-F1 when only errorful episodes are scored.
| Error category | (episodes / steps) | Random | Seq2Seq | Seq2Seq+SG | SGT |
| Omission | 137 / 137 | 0.026 | 0.696 | 0.716 | 0.730 |
| Sequence | 38 / 38 | 0.010 | 0.186 | 0.215 | 0.814 |
| Anticipation | 17 / 17 | 0.005 | 0.023 | 0.085 | 0.359 |
| Perseveration | 26 / 26 | 0.007 | 0.093 | 0.095 | 0.502 |
| Object substitution | 30 / 30 | 0.008 | 0.018 | 0.012 | 0.038 |
| Tool substitution | 9 / 9 | 0.003 | 0.003 | 0.003 | 0.020 |
Table 12: Per-class step-level AUPRC (M2) of the learned monitors on the 500-episode test set. Each row pools the test episodes whose primary category is the given one with the 188 correct episodes; is the number of episodes and of error steps (multi-error episodes contribute all their error steps). Random equals the positive-step rate of the row.
| Method | M1 | M2 | M2-end | M2-mid |
| Seq2Seq | 0.496 [0.44, 0.55] | 0.376 [0.33, 0.43] | 0.656 [0.58, 0.73] | 0.132 [0.11, 0.17] |
| Seq2Seq+SG | 0.554 [0.50, 0.60] | 0.394 [0.34, 0.45] | 0.665 [0.58, 0.75] | 0.151 [0.13, 0.18] |
| SGT | 0.778 [0.73, 0.82] | 0.656 [0.61, 0.70] | 0.791 [0.73, 0.85] | 0.553 [0.49, 0.62] |
| CUSUM | 0.645 [0.59, 0.69] | 0.039 [0.04, 0.04] | 0.235 [0.20, 0.28] | 0.029 [0.03, 0.03] |
| Precondition check | 0.513 [0.50, 0.52] | 0.059 [0.05, 0.08] | 0.274 [0.24, 0.31] | 0.062 [0.04, 0.09] |
| Re-planner | 0.887 [0.86, 0.91] | 0.379 [0.32, 0.45] | 0.690 [0.63, 0.76] | 0.118 [0.07, 0.17] |
Table 13: Main results with 95% confidence intervals (500-episode test set). M2-end: task-end step (omissions); M2-mid: steps before task end (all other errors). Random: M1 0.500, M2 0.036, M2-end 0.274, M2-mid 0.023. ∗ Privileged: receives the episode’s grounded goal.
| Simple | Medium | Complex | ||||
| Method | M1 | M2-mid | M1 | M2-mid | M1 | M2-mid |
| Seq2Seq | 0.614 | 0.189 | 0.498 | 0.140 | 0.349 | 0.047 |
| Seq2Seq+SG | 0.706 | 0.229 | 0.536 | 0.152 | 0.391 | 0.154 |
| SGT | 0.844 | 0.690 | 0.797 | 0.546 | 0.573 | 0.406 |
| CUSUM | 0.666 | 0.178 | 0.654 | 0.039 | 0.633 | 0.031 |
| Precondition check | 0.507 | 0.055 | 0.516 | 0.076 | 0.509 | 0.039 |
Table 14: M1 and M2-mid by template complexity on the test set (121 simple, 291 medium, 88 complex episodes).
| Method | Missed | Early | On target | Late | False alarm | Omission: end / before |
| Seq2Seq | 0.99 | 0.00 | 0.00 | 0.01 | 0.10 | 0.46 / 0.00 |
| Seq2Seq+SG | 0.98 | 0.00 | 0.00 | 0.02 | 0.05 | 0.38 / 0.00 |
| SGT | 0.52 | 0.04 | 0.38 | 0.06 | 0.12 | 0.58 / 0.01 |
| CUSUM | 0.06 | 0.77 | 0.07 | 0.10 | 0.81 | 0.01 / 0.80 |
| Precondition check | 0.95 | 0.00 | 0.05 | 0.00 | 0.00 | 0.00 / 0.00 |
| Re-planner | 0.10 | 0.75 | 0.05 | 0.10 | 0.62 | 0.33 / 0.66 |
Table 15: M3 first-alarm distribution at on the test set. Missed, early ( ), on target ( ), and late ( ) are shares of the 175 episodes whose first error occurs before task end; false alarm is the share of correct episodes with an alarm. Omissions: share of the 137 omission episodes alarmed at the task-end step / before it. Episodes on which the re-planner timed out are excluded from its rows (145 / 125 / 170 episodes for the template-goal re-planner and 155 / 132 / 180 for the privileged one).
| Model | OpenRouter ID | Provider | Reasoning | Max tokens |
| Qwen3.6-35B-A3B | qwen/qwen3.6-35b-a3b | DeepInfra (fp8) | disabled | 512 |
| Gemini 3.1 Flash Lite | google/gemini-3.1-flash-lite | Google AI Studio | disabled | 512 |
| Kimi K2.6 | moonshotai/kimi-k2.6 | Moonshot AI | disabled a | 512 |
| Grok 4.3 | x-ai/grok-4.3 | xAI | disabled | 512 |
| GLM-5.1 | z-ai/glm-5.1 | Z.AI (fp8) | disabled | 512 |
| DeepSeek V4 Flash | deepseek/deepseek-v4-flash | Novita (fp8) | disabled | 512 |
Table 16: LLM configurations. Each configuration is pinned to one OpenRouter provider (fallback disabled); reasoning is set explicitly. a Reports one reasoning token per request with reasoning disabled, and no reasoning text. b The provider rejects requests with reasoning disabled (“reasoning is mandatory”), so the lowest effort is used.
| Model | M1 | M2 | M2-end | M2-mid | M4 |
| Kimi K2.6 | 0.718 [0.60, 0.84] | 0.140 [0.06, 0.25] | 0.067 | 0.154 | 0.179 |
| Claude Sonnet 4.6 (2,000 tokens) | 0.697 [0.56, 0.82] | 0.325 [0.20, 0.48] | 0.111 | 0.451 | 0.432 |
| DeepSeek V4 Pro (medium) | 0.679 [0.55, 0.81] | 0.286 [0.17, 0.44] | 0.218 | 0.478 | 0.480 |
| GPT-5.5 (low) | 0.631 [0.48, 0.77] | 0.238 [0.14, 0.38] | 0.133 | 0.394 | 0.489 |
| GPT-5.5 (medium) | 0.617 [0.47, 0.75] | 0.231 [0.15, 0.37] | 0.116 | 0.430 | 0.477 |
| Grok 4.3 | 0.584 [0.48, 0.69] | 0.086 [0.03, 0.18] | 0.067 | 0.093 | 0.107 |
Table 17: LLM results on the 60-episode subset, sorted by M1, with the other baselines re-scored on the same episodes. M4 includes correct episodes (Section C.1 ). 95% intervals from a trajectory-level bootstrap stratified by errorful and correct episodes (1,000 resamples).
| Model | Missed | Early | On target | Late | False alarm |
| Kimi K2.6 | 0.46 | 0.38 | 0.12 | 0.04 | 0.27 |
| Claude Sonnet 4.6 (2,000 tokens) | 0.31 | 0.08 | 0.50 | 0.12 | 0.30 |
| DeepSeek V4 Pro (medium) | 0.27 | 0.19 | 0.42 | 0.12 | 0.47 |
| GPT-5.5 (low) | 0.15 | 0.08 | 0.46 | 0.31 | 0.57 |
| GPT-5.5 (medium) | 0.23 | 0.12 | 0.46 | 0.19 | 0.50 |
| Grok 4.3 | 0.69 | 0.27 | 0.04 | 0.00 | 0.13 |
Table 18: M3 first-alarm distribution at on the 60-episode subset: shares of the 26 errorful episodes whose first error occurs before task end (missed, early, on target, late) and false-alarm rate on the 30 correct episodes; for the re-planner, 22 and 28 episodes, excluding timeouts.
More in this topic
AI Agent Monitoring
AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.
See all 281 papers