cs.ROSep 26, 2026

WSM-Aware HRI: An IoT-Enhanced Framework for Early Detection and Norm-Guided Repair of Failures with LLM Guidance

Authors: Hanlin Zhang, Yuquan Wang, Tianwei Zhang, Zhenglong Sun

Organizations: The Chinese University of Hong Kong, Shenzhen, China · X SQUARE Robot, China · Shenzhen Institute of Artificial Intelligence and Robotics for Society, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human-robot interaction (HRI) failures remain a major barrier to deploying robots in real-world environments. Prior work often treats failures as isolated technical faults or focuses on post-hoc recovery behaviors. In practice, many breakdowns arise because humans and robots operate under inconsistent assumptions about the current world state. We propose WSM-Aware HRI, an IoT-enhanced modular framework that unifies diverse HRI breakdowns as World-State Mismatches (WSMs) between a human's instruction-implied assumptions and a robot's grounded world model built from multimodal perception and digital augmentation. A Large Language Model (LLM) is used to make implicit assumptions explicit, map them to a small set of mismatch types, and specify the evidence needed for verification against the robot's world state. WSM-Aware HRI shifts failure handling from execution-time recovery to proactive mismatch detection during intention formation, enabling interventions guided by safety, norm compliance, and multi-user coordination with transparent explanations. We evaluate mismatch identification in ten everyday cases spanning both visual and latent-state mismatches. The system can accurately produce the expected output results, and ablations show that reliable identification depends on appropriate grounding representations and verification-oriented refinement. These results indicate that treating interaction breakdowns as explicit world-state mismatches enables earlier detection of impending failures and offers a principled mechanism for integrating external evidence and social constraints into human-robot interaction.

Figures & tables

Figure 1. WSM-Aware HRI Framework.
WSM TypeDefinition and Representative Example
Instruction Logic MismatchDefinition: The instruction is incompatible with task-level logic or hazard/causal constraints under the current context. Example: Ask the robot to extinguish a fire with water while the robot detects an active electrical source.
Perceptual / Affordance MismatchDefinition: The user’s instruction assumes physical feasibility or safe affordances that are contradicted by grounded sensing or verification. Example: Request the robot to pick up a cup with cold water while the robot detects hot liquid inside.
Temporal / Sequence MismatchDefinition: The instruction conflicts with temporal ordering or state-machine prerequisites. Example: Ask the robot to place an object before it has been grasped.
Social / Ethical (Norm) MismatchDefinition: The instruction violates applicable social norms, policies, or role-based permissions, regardless of physical feasibility. Example: Request the robot to enter a restricted or private area.
Multi-user ConflictDefinition: The instruction is incompatible with constraints or objectives posed by multiple stakeholders, requiring coordination or negotiation. Example: The teacher requests the robot to maintain classroom order while students request entertainment.
Table 1. World-State Mismatch (WSM) Types, Definitions and Representative Examples
Prior failure mechanismInstruction-implied constraintEvidence needed for verificationRepair implicationResulting WSM category
Perception/action mismatch and physical execution failure ( Honig and Oron-Gilad, 2018 ; Cameron et al., 2024 )The instruction assumes that the target object or action is physically feasible, compatible, and safe under the current context.Grounded perception, object state, affordance checks, force/temperature/weight sensing, or other physical verification.Verify the relevant property, adapt the plan, avoid unsafe manipulation, or suggest a safer alternative.Perceptual/Affordance Mismatch
Mismatched expectations and invalid causal/task assumptions ( Honig and Oron-Gilad, 2018 ; Salem et al., 2015 )The instruction presupposes a causal, task-level, or goal-logic condition that may not hold in the current world state.Task model, hazard relation, commonsense constraint, causal dependency, or context-specific safety rule.Clarify the intended goal, reject invalid causal assumptions, or propose a goal-preserving alternative.Instruction Logic Mismatch
Expectation-related timing, prerequisite, or ordering failure ( Tolmeijer et al., 2020 ; Cameron et al., 2024 )The instruction assumes that a required prior state has already been achieved or that the requested action is temporally admissible.Execution history, task-state record, state-machine status, prerequisite checks, or temporal ordering constraints.Insert missing prerequisites, reorder the plan, delay execution, or request confirmation before proceeding.Temporal/Sequence Mismatch
Social error, privacy violation, permission failure, or role/policy violation ( Tian and Oviatt, 2021 ; Nogueira et al., 2026 )The instruction assumes that the requested action is socially, ethically, or institutionally permissible.Role information, permission status, privacy policy, institutional rule, safety norm, or social constraint.Request consent or authorization, refuse the action with an explanation, or provide a norm-compliant alternative.Social/Ethical Norm Mismatch
Intertwined social causes and incompatible stakeholder goals ( Civit et al., 2025 ; Sebo et al., 2020 )The instruction assumes that the goals or constraints of multiple users are mutually compatible.Active user requests, stakeholder roles, priority rules, conflict status, or multi-user interaction history.Initiate negotiation, arbitrate according to priority rules, escalate to a human authority, or propose a compromise.Multi-user Conflict
Table 2. Mapping prior failure mechanisms to WSM categories, verification evidence, and repair responses.
Figure 2. LLM-assisted mismatch type identification with a two-call verifier loop. The first LLM call ( LLM-Hypothesize ) extracts an explicit assumption set and proposes candidate mismatch types together with executable evidence requests. The system then acquires evidence (sensing/IoT/policy/execution-state checks) to update the augmented world state. A second LLM call ( LLM-Revise ) revises the hypothesis conditioned on grounded evidence, after which supported mismatch types are filtered and a primary type is selected via precedence-based adjudication.
Figure 3. Illustrations of representative scenes used to elicit World-State Mismatches (WSMs) in our experiments. Scene A shows a toy and a paper bowl on the table, with the instruction “put the toy into the bowl.” In this scene, the human’s assumed world state (i.e., the toy can fit into the bowl) is consistent with the robot’s grounded observation, thus no WSM is expected. Scene B shows a toy and a paper cup on the table, under the same instruction “put the toy into the cup.” Here, the human’s assumed feasibility conflicts with the robot’s perception of relative size constraints (the toy is too large), forming an affordance/compatibility WSM. Scene C shows a paper cup with water and a container with holes on the table, with the instruction “pour the water into the container.” In this case, the human assumes the container can hold liquid, while the robot recognizes the functional constraint (holes), forming a functional affordance WSM. Scene D shows a paper cup with water with an opaque body, under the instruction “I’m thirsty. Pass me the water.” Since the liquid level is not directly observable, a potential WSM arises when the human assumes the cup contains water but the robot’s verification reveals that it is nearly empty.
Figure 4. Demonstration of WSM-aware reasoning enabling the robot to respond to the cup with different liquid conditions. Case a: the cup is too light. The robot’s verification indicates that the cup does not contain enough water, which conflicts with the human’s assumed world state implied by the instruction (i.e., the water is sufficient). The system therefore identifies a World-State Mismatch (WSM) and informs the user (“Not enough water left”). The robot asks whether it should add some water or deliver other beverage instead. Case b: the cup contains liquid. The verified world state aligns with the user’s assumed condition, so no WSM is triggered and the robot proceeds to hand over the cup.
System variantWSM type matchEvidence coveragePre-execution interventionRepair match
Post-hoc recovery28%N/A10%32%
Affordance/action-grounding54%48%48%52%
LLM-only46%N/A42%44%
LLM-Hypothesize64%68%60%62%
No-IoT / no digital evidence74%70%72%72%
Full WSM-Aware HRI88%86%86%84%
Table 3. Baseline and ablation comparison on the 50-trial scenario set.
IDCase and instructionWSM typeRequired evidenceExpected system responseCorrect
1Toy + cup: “Put the toy into the cup.”Perceptual / AffordanceL : object and container size from local visionDetect size/fit incompatibility; explain that the toy cannot fit and ask for an alternative placement.5/5
2Water + holed container: “Pour the water into the container.”Perceptual / AffordanceL : visual defect or leakage cueDetect that the container cannot hold liquid; refuse direct pouring and suggest a watertight container.5/5
3Light state: “Turn off the light.”Temporal / SequenceIoT : smart-light on/off stateCheck whether the light is already off; avoid redundant action and inform the user of the current state.5/5
4Placement before grasp: “Put the toy on the shelf.”Temporal / SequenceL : gripper and execution stateDetect the missing prerequisite; insert grasping before placement or explain the required sequence.5/5
5Fire + live electricity: “Use water to extinguish the fire.”Instruction LogicL : fire observation; IoT : electrical-source statusDetect the causal hazard; refuse water-based extinguishing and suggest a safer alternative.5/5
6Opaque cup: “I’m thirsty. Pass me the water.”Instruction LogicL : cup localization; IoT : smart cup or coaster stateVerify the hidden liquid state; if insufficient, inform the user and offer to add water or provide another drink.4/5
Table 4. Per-case quantitative evaluation across WSM types. The table summarizes the ten evaluation cases, their associated mismatch categories, required evidence channels, expected system responses, and trial-level correctness.
SceneWith-IoT prompt evidenceIoT evidence removedAccuracy with IoTAccuracy without IoT
Light statusSmart light state indicates that the light is already off.Smart light state is removed; only the user instruction is provided.10/100/10
Electrical fireSmart outlet/electrical panel indicates an active electrical source near the fire.Smart outlet and electrical-panel evidence are removed.10/100/10
Opaque cupSmart cup/coaster indicates that the cup is nearly empty.Smart cup and smart-coaster evidence are removed.10/101/10
Do-not-disturb roomSmart door sign, smart lock, and calendar system indicate a privacy-sensitive state.Door-sign, lock, and calendar evidence are removed.10/100/10
Family TV conflictSmart TV and family-identification system indicate conflicting parent–child requests.Smart TV state and family-identification evidence are removed.9/105/10
Medical-treatment conflictHealthcare platform and identity-verification system indicate conflicting stakeholder roles.Healthcare-platform and identity-verification evidence are removed.9/104/10
Table 5. Offline Prompt-level Ablation of IoT Evidence in LLM-Revise
Ablated Outputpvip_{vi}pvrp_{vr}pvcp_{vc}
Scene ID232323
WSMs identified (X/5)520555
WSMs identified No Ablation (X/5)555555
Table 6. Result of Ablating Visual Semantic Grounding Outputs
Ablated moduleRefinement phase in Algorithm 1
Scene ID4
WSMs identified (X/5)3
WSMs identified no ablation (X/5)5
Table 7. Result of Ablating WSM-aware Trajectory Refinement
Figure 5. Comparison of robot behavior with and without IoT augmentation in two household WSM scenarios. In Scenario 1, the user requests window ventilation, implicitly assuming that outdoor air quality is better than indoor air. Without IoT, the robot may open the window based only on local vision, causing polluted air to enter the room. With IoT-enabled WSM detection, the robot checks indoor air quality, outdoor AQI, pollen, weather, noise, and device status, and then selects a safer alternative such as an air purifier or fresh-air system when outdoor conditions are poor. In Scenario 2, the user asks the robot to hang clothes on the balcony, implicitly assuming that the balcony will remain suitable for drying. Without IoT, the robot may hang the clothes outside because it is not raining at the moment. With IoT-enabled WSM detection, the robot checks short-term weather forecasts, balcony humidity, raindrop sensors, wind speed, sunlight, and indoor drying devices, and then chooses indoor drying or dehumidification when rain is likely.

Explore similar work

CardsList
Sep 21, 2026cs.RO

Toward Human-in-the-Loop Robot Failure Recovery: Bridging Communication Gaps in Human-Robot Collaboration

Robots can recover from failures by asking bystanders for help, but effective human-in-the-loop recovery requires communication that accounts for differences in people's knowledge. Prior inverse-semantics work generates requests using a single listener model, leaving differences in listener knowledge untested. We introduce Listener Differences in Human-Robot Interaction (LD-HRI), a game, dataset, and benchmark that evaluates speakers through human listener performance. Our evaluation examines request properties, large language model (LLM) speakers, and inverse-semantics request-selection algorithms under controlled differences in listener information. The corpus contains 446 human-written requests and 1{,}302 listener trials. We additionally evaluated 24 frozen LLM-written requests with 70 human listeners across 560 trials. Novice success is descriptively higher with model-written requests across all four tasks, yet both request sources leave substantial expert--novice gaps, including 16 percentage points for LLM requests. LD-HRI makes these gaps measurable, providing a foundation for designing more robust communication in human-robot and human-agent interaction.

Promise Ekpo, Teju Vijay, Dhruv Mandalik +5

Cornell University, New York, New York, USA · Cornell Tech, New York, New York, USA · Brown University, Providence, Rhode Island, USA

Human-Robot InteractionHuman-In-The-LoopHuman-Ai InteractionRobot SystemsCollaborationInverse Design
AskPDF
Jun 29, 2026cs.RO

REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery

Understanding how users perceive and respond to robot failures is essential for building robust and trustworthy robot systems. Prior work, however, (i) often treats failures as independent events, (ii) emphasizes binary failure detection, (iii) with rule-based recovery modeling. We present REPAIR-Bench, built on 214 interaction trials from 41 participants, the benchmark spans four induced failure types and provides synchronized facial action units, head pose, speech transcripts, and post-interaction affect and recovery reports. The benchmark spans three novel evaluation tasks that jointly capture the lifecycle of failure in human-robot interaction (HRI): (i) failure detection over inter-dependent interaction sessions, modeling longitudinal user adaptation across repeated failures; (ii) visual failure-type classification beyond binary success/failure formulations; and (iii) user-centered recovery prediction, inferring users' preferred recovery strategies from interaction context rather than relying on manually designed or rule-based strategies. In baseline experiments, hierarchical recurrent modeling improved failure detection over a single-session model (strict F1: 0.80 vs. 0.68), achieved a failure localization mean signed error of -0.51 s, median absolute error of 2.97 s and, for recovery prediction, a QLoRA-tuned Mistral-7B reached Hit@5=0.76 and F1@5=0.32. REPAIR-Bench provides both the HRI and Medical HRI communities with a standardized framework for (1) evaluating robot failures and (2) building transparent, adaptive, and trustworthy recovery systems.

Giuliano Pioldi, Yashika Batra, Arman Ibrayeva +4

Cornell University Cornell Tech New York, NY

Human-Robot InteractionRobot FailuresRobotic PerceptionRepairFacial Expression Recognition
AskPDF
Jul 25, 2026cs.RO

WCM: World-Cognition Model for Generalizable Human-Robot Interaction

Language agents can now interact fluently with users in software, but robots still struggle to bring comparable interaction to physical tasks. Current robot-control paradigms, including vision-language-action policies and world-model-based planners, are mainly optimized for instruction execution, leaving users with little visibility into why an action is chosen and few mechanisms to redirect, correct, or teach the robot through interaction. To solve this problem, we present the World-Cognition Model (WCM), a human-centered embodied agent built on the SLAK architecture (Sensing, Logic, Action, and Knowledge) and an asynchronous runtime. SLAK separates perception, reasoning, control, and memory, while the runtime allows reasoning, dialogue, and execution to proceed concurrently. WCM further introduces a human-in-the-loop teaching mode that enables users to interactively teach the robot difficult or long-horizon tasks. Teaching episodes and autonomous task rollouts are refined into chain-of-thought supervision to continually improve the model. WCM achieves a 73.8% average success rate across nine real-world human-robot interaction tasks, including tasks held out from CoT fine-tuning and a long-horizon task learned through teaching.

Yuzhen Chen, KC Zhou

1Harvard University · 2Massachusetts Institute of Technology

Human-Robot InteractionEfficient World-Action ModelWorld ModelsLong-Horizon Task Planning
AskPDF