See Something, Say Something: Context-Criticality-Aware Mobile Robot Communication for Hazard Mitigations
Organizations: ASAS LABS, Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA. · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.
Abstract
The proverb ``see something, say something'' captures a core responsibility of autonomous mobile robots in safety-critical situations: when they detect a hazard, they must communicate--and do so quickly. In emergency scenarios, delayed or miscalibrated responses directly increase the time to action and the risk of damage. We argue that a systematic context-sensitive assessment of the criticality level, time sensitivity, and feasibility of mitigation is necessary for AMRs to reduce time to action and respond effectively. This paper presents a framework in which VLM/LLM-based perception drives adaptive message generation, for example, a knife in a kitchen produces a calm acknowledgment; the same object in a corridor triggers an urgent coordinated alert. Validation in 60+ runs using a patrolling mobile robot not only empowers faster response, but also brings user trusts to 82% compared to fixed-priority baselines, validating that structured criticality assessment improves both response speed and mitigation effectiveness.
Figures & tables
| Evaluation Criteria | Gemini 2.0 Flash | BLIP + ChatGPT-3.5 | BLIP + ChatGPT-4.0 |
|---|---|---|---|
| Tone | Direct, fast | Balanced, adaptive | Calm, safety-oriented |
| Structure | Concise | Well-structured | Explicit hazard reasoning |
| Clarity | High (simple tasks) | Very high | Scenario-specific clarity |
| Context Handling | Moderate continuity | Strong reasoning | Visual-grounded reasoning |
| Latency | 5 s | 14 s | 12 s |
| Scenario | Context Interpretation | Risk Level | Communication Strategy |
|---|---|---|---|
| Knife in Unsafe Surrounding | Sharp object detected near a person in a public or restricted area | High | Immediate voice alert, alarm activation, and automated email notification to authorities |
| Knife in Kitchen (Proper Use) | Knife identified in a designated kitchen area during food preparation | Low | No alarm triggered; contextual acknowledgment with optional safety reminder |
| Person Lying Down | Human detected in abnormal posture on the floor, possibly requiring assistance | High | Voice alert to nearby individuals and emergency notification with location details |
| People in Distress | Signs of panic or abnormal human activity detected in the environment | High | Clear assistance instructions via speech and urgent responder alert with contextual summary |
| Trash / Cleaning Activity | Garbage detected without immediate threat or hazardous context | Low | No alarm; maintenance notification sent as a non-urgent message |
Robot Safety
Robot safety studies how to prevent harmful physical actions while enabling robots to complete tasks around people and in uncertain environments.
See all 229 papers