UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents
Organizations: Northwestern Polytechnical University, China · City University of Hong Kong, China · Peking University, China · Harbin Engineering University, China
Abstract
For Deaf and Hard-of-Hearing (DHH) users, effective sound awareness goes beyond detecting what sound occurs; it requires understanding what it means, how urgent it is, and what to do. However, existing mobile assistants largely reduce acoustic scenes to isolated labels or generic alerts, failing to provide contextual guidance when the same sound implies different risks across scenarios. We present \sysname, a mobile hearing assistant agent that closes the sensing--reasoning--action loop for scenario-aware and personalized DHH assistance. \sysname adopts a neuro-symbolic design that transforms continuous acoustic and physical context streams into structured intermediate representations (IRs) for controllable reasoning and guidance. It introduces three key mechanisms. First, a perception-thresholded sensing mechanism enables energy-efficient always-on awareness by activating processing only for perceptually meaningful events while preserving safety-critical cues through deterministic overrides. Second, a staged neuro-symbolic reasoning framework performs promptless scenario understanding by conditioning an on-device audio-language model on structured IRs instead of user-authored context descriptions. Third, a bounded human-in-the-loop personalization mechanism adapts user-specific guidance through lightweight on-device decision-boundary updates while preserving population-aligned reasoning. We evaluate \sysname on smartphones and in-vehicle systems across diverse scenarios. It achieves favorable accuracy--latency--energy tradeoffs and personalized guidance.
Figures & tables
| Acoustic event | Scenario | Scenario-aware guidance | Assumed-universal guidance |
| Car horn(Ch) | Indoor | Ignore | Ignore |
| On the road | Attention to vehicle | ||
| Human shouting(Hs) | Playground | Ignore | Ignore |
| Outdoor | Attention to source | ||
| Broadcast sound(Bs) | At school/square | Ignore | Ignore |
| Railway station/Airport | Attention to departure time |
| Scenario | Preferred response | Rejected response |
| Indoor, stationary; distant car horn | A distant car horn is detected outside. Since you are indoors and stationary, no immediate action is needed. | A car horn is detected. Stop what you are doing and immediately check for nearby traffic. |
| Outdoor, mobile; nearby car horn | A nearby car horn is detected. Stay alert, slow down, and check surrounding vehicles before moving. | A car horn is detected. It is common background noise and can be ignored. |
| Indoor, mobile; nearby speech | Nearby speech is detected while you are walking. Slow down and check whether someone is calling you. | Speech is detected indoors. It is likely irrelevant background conversation, so keep walking. |
| Outdoor, stationary; repeated honks | Repeated vehicle honks are detected nearby. Stay attentive and check traffic conditions before moving. | Repeated honks are normal urban noise. You can safely disregard them completely. |
| Model | Params | Quantization | Raw weight memory |
| fusion model | 1.5B | 2-bit | 0.38GB |
| audio-language model | 1.8B | 2-bit | 0.45GB |
| UbiHearo reasoning stack | 3.3B | 2-bit | 0.83GB |
| Qwen-Audio2 | 7B | 2-bit | 1.75GB |
| Audio-Flamingo | 8B | 2-bit | 2.00GB |
| MiDashengLM | 8B | 2-bit | 2.00GB |
Audio Reasoning
Audio reasoning studies how AI systems infer events, relationships, intent, and context from sound, beyond recognizing words or matching labels.
See all 71 papers