See Something, Say Something: Context-Criticality-Aware Mobile Robot Communication for Hazard Mitigations
Authors: Bhavya Oza, Devam Shah, Ghanashyama Prabhu, Devika Kodi, Aliasghar Arab
Organizations: ASAS LABS, Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA. · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.
The proverb ``see something, say something'' captures a core responsibility of autonomous mobile robots in safety-critical situations: when they detect a hazard, they must communicate--and do so quickly. In emergency scenarios, delayed or miscalibrated responses directly increase the time to action and the risk of damage. We argue that a systematic context-sensitive assessment of the criticality level, time sensitivity, and feasibility of mitigation is necessary for AMRs to reduce time to action and respond effectively. This paper presents a framework in which VLM/LLM-based perception drives adaptive message generation, for example, a knife in a kitchen produces a calm acknowledgment; the same object in a corridor triggers an urgent coordinated alert. Validation in 60+ runs using a patrolling mobile robot not only empowers faster response, but also brings user trusts to 82% compared to fixed-priority baselines, validating that structured criticality assessment improves both response speed and mitigation effectiveness.
Figures & tables
Fig. 1 : A patrolling AMR detecting a fallen person, correlating a nearby knife with heightened risk, and triggering a context-aware mitigation response.
Fig. 2 : Workflow of hazard detection, communication strategy generation, and multi-party coordination.
Evaluation Criteria
Gemini 2.0 Flash
BLIP + ChatGPT-3.5
BLIP + ChatGPT-4.0
Tone
Direct, fast
Balanced, adaptive
Calm, safety-oriented
Structure
Concise
Well-structured
Explicit hazard reasoning
Clarity
High (simple tasks)
Very high
Scenario-specific clarity
Context Handling
Moderate continuity
Strong reasoning
Visual-grounded reasoning
Latency
∼ 5 s
∼ 14 s
∼ 12 s
TABLE I : Comparison of LLM Communication Models
Scenario
Context Interpretation
Risk Level
Communication Strategy
Knife in Unsafe Surrounding
Sharp object detected near a person in a public or restricted area
High
Immediate voice alert, alarm activation, and automated email notification to authorities
Knife in Kitchen (Proper Use)
Knife identified in a designated kitchen area during food preparation
Low
No alarm triggered; contextual acknowledgment with optional safety reminder
Person Lying Down
Human detected in abnormal posture on the floor, possibly requiring assistance
High
Voice alert to nearby individuals and emergency notification with location details
People in Distress
Signs of panic or abnormal human activity detected in the environment
High
Clear assistance instructions via speech and urgent responder alert with contextual summary
Trash / Cleaning Activity
Garbage detected without immediate threat or hazardous context
Low
No alarm; maintenance notification sent as a non-urgent message
TABLE II : Context- and Criticality-Aware Communication Strategies
Fig. 3 : User Evaluation Breakdown (Trust, Understanding, Preference) for different scenarios
Fig. 4 : Scenario-wise risk scores ρt (Eq. ( 9 )): Low [0–4] , Medium [5–7] , High [8–10] . Alarm activation occurs only for ρt≥5 .
In high-risk environments such as disaster response, situational awareness depends not only on detecting hazards but also on communicating them clearly to human operators. Vision Language Models (VLMs) have shown strong potential for scene understanding in safety-critical settings, yet their value as part of human-facing robotic systems remains underexplored. We present a VR-based Human Robot Interaction framework for studying how VLM-assisted robots can support situational awareness in simulated hazardous environments. In our system, a robot explores a virtual scene and queries a VLM to identify potential hazards and annotate user-facing points of interest. These annotations are presented to a human operator through an immersive VR interface. This framework enables controlled evaluation of both robotic hazard identification and the communication of safety-critical information to users. Results from our study indicate that the annotated VR interface was preferred over the unannotated baseline and that participants reported high clarity, usefulness, and comfort when interacting with the system. These findings suggest that combining VLM-based robotic perception with immersive visualization is a promising approach for supporting situational awareness in hazardous settings.
Mohammad Eskandari, Murali Krishna Varma Indukuri, Stephanie M. Lukin +1
Interactive Robotics and Language Lab, University of Maryland Baltimore County, Computer Science and Electrical Engineering Department, Baltimore, MD, USA. · DEVCOM Army Research Laboratory, Adelphi, MD, USA.
Mobile manipulators are increasingly deployed in human-centered environments to perform tasks. While completing such tasks, they should also be able to communicate their intent to the people around them using expressive robot behaviors. Prior work on expressive robot behaviors has used preprogrammed or learning-from-demonstration-based expressive motions and large language model generated high-level interactions. The majority of these existing approaches have not considered human-robot interactions (HRI) where users may interrupt, modify, or redirect a robot's actions during task execution. In this paper, we develop the novel ExpressMM framework that integrates a high-level language-guided planner based on a vision-language model for perception and conversational reasoning with a low-level vision-language-action policy to generate expressive robot behaviors during collaborative HRI tasks. Furthermore, ExpressMM supports interruptible interactions to accommodate updated or redirecting instructions by users. We demonstrate ExpressMM on a mobile manipulator assisting a human in a collaborative assembly scenario and conduct audience-based evaluation of live HRI demonstrations. Questionnaire results show that the ExpressMM-enabled expressive behaviors helped observers clearly interpret the robot's actions and intentions while supporting socially appropriate and understandable interactions. Participants also reported that the robot was useful for collaborative tasks and behaved in a predictable and safe manner during the demonstrations, fostering positive perceptions of the robot's usefulness, safety, and predictability during the collaborative tasks.
Souren Pashangpour, Haitong Wang, Matthew Lisondra +1
Autonomous Systems and Biomechatronics Laboratory (ASBLab), Department of Mechanical and Engineering, University of Toronto, Toronto, ON M5S 3G8, Canada
Evaluating language-guided mobile agents has recently shifted from rule-based to model-based approaches to achieve scalable and automated assessments. However, existing holistic evaluation paradigms process entire trajectories at once, leading to substantial context overload. Moreover, they primarily focus on task completion while overlooking operational safety. To address these limitations, we introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation that is compatible with both open- and closed-source models. Leveraging a step-level consequence reasoning mechanism, CRATE independently extracts task-relevant visual clues and infers action-conditioned state changes at each step. The resulting step-level textual evidence is then synthesized through trajectory-level aggregation to deliver an evidence-grounded evaluation of task completion. Building upon this evaluation scheme, we further extend CRATE to CRATE-S for operational safety assessment. Extensive experiments validate the effectiveness and robustness of both CRATE and CRATE-S. Powered by Qwen2.5-VL-72B-Instruct, CRATE achieves an F1-score of 0.833 on AndroidWorld (outperforming SPA-Bench by 20%), while CRATE-S reaches an F1-score of 0.697 on MobileRisk, demonstrating strong alignment with benchmark ground truths. Code is available at https://anonymous.4open.science/r/CRATE-D580.