Organizations: Northwestern Polytechnical University, China · City University of Hong Kong, China · Peking University, China · Harbin Engineering University, China
For Deaf and Hard-of-Hearing (DHH) users, effective sound awareness goes beyond detecting what sound occurs; it requires understanding what it means, how urgent it is, and what to do. However, existing mobile assistants largely reduce acoustic scenes to isolated labels or generic alerts, failing to provide contextual guidance when the same sound implies different risks across scenarios. We present \sysname, a mobile hearing assistant agent that closes the sensing--reasoning--action loop for scenario-aware and personalized DHH assistance. \sysname adopts a neuro-symbolic design that transforms continuous acoustic and physical context streams into structured intermediate representations (IRs) for controllable reasoning and guidance. It introduces three key mechanisms. First, a perception-thresholded sensing mechanism enables energy-efficient always-on awareness by activating processing only for perceptually meaningful events while preserving safety-critical cues through deterministic overrides. Second, a staged neuro-symbolic reasoning framework performs promptless scenario understanding by conditioning an on-device audio-language model on structured IRs instead of user-authored context descriptions. Third, a bounded human-in-the-loop personalization mechanism adapts user-specific guidance through lightweight on-device decision-boundary updates while preserving population-aligned reasoning. We evaluate \sysname on smartphones and in-vehicle systems across diverse scenarios. It achieves favorable accuracy--latency--energy tradeoffs and personalized guidance.
Figures & tables
Figure 1. Motivation cases, which illustrate scenario-dependent sound awareness for a DHH user, Lucas .
Figure 2. Survey results. (a) Missing rates of acoustic events, (b) Expectations for guidance, (c) Impact of universal guidance, and (d) Scenario-specific preference.
Acoustic event
Scenario
Scenario-aware guidance
Assumed-universal guidance
Car horn(Ch)
Indoor
Ignore
Ignore
On the road
Attention to vehicle
Human shouting(Hs)
Playground
Ignore
Ignore
Outdoor
Attention to source
Broadcast sound(Bs)
At school/square
Ignore
Ignore
Railway station/Airport
Attention to departure time
Table 1. Examples of guidance strategies in the Questionnaire for common acoustic events across various scenarios
Figure 3. Workflow of UbiHearo .
Figure 4. Illustration of the duty-cycled microphone sampling mechanism.
Figure 5. The basis for non-uniform frequency band division.
Figure 6. Comparison of three reasoning schemes. (a) A single MLLM directly processes all modalities. (b) A 7B MLLM reasons over GMM-derived prompts. (c) UbiHearo uses two lightweight models, combining contextual states and GMM outputs as structured prompts.
Figure 7. Illustration of the stage-wise supervised fine-tuning.
Figure 8. Illustration of the scenario-conditioned preference alignment mechanism.
Scenario
Preferred response yw
Rejected response yl
Indoor, stationary; distant car horn
A distant car horn is detected outside. Since you are indoors and stationary, no immediate action is needed.
A car horn is detected. Stop what you are doing and immediately check for nearby traffic.
Outdoor, mobile; nearby car horn
A nearby car horn is detected. Stay alert, slow down, and check surrounding vehicles before moving.
A car horn is detected. It is common background noise and can be ignored.
Indoor, mobile; nearby speech
Nearby speech is detected while you are walking. Slow down and check whether someone is calling you.
Speech is detected indoors. It is likely irrelevant background conversation, so keep walking.
Outdoor, stationary; repeated honks
Repeated vehicle honks are detected nearby. Stay attentive and check traffic conditions before moving.
Repeated honks are normal urban noise. You can safely disregard them completely.
Table 2. Examples of scenario-conditioned preference pairs.
Figure 9. Illustration of the customized configurations.
Model
Params
Quantization
Raw weight memory
πf fusion model
1.5B
2-bit
∼ 0.38GB
πθ audio-language model
1.8B
2-bit
∼ 0.45GB
UbiHearo reasoning stack
3.3B
2-bit
∼ 0.83GB
Qwen-Audio2
7B
2-bit
∼ 1.75GB
Audio-Flamingo
8B
2-bit
∼ 2.00GB
MiDashengLM
8B
2-bit
∼ 2.00GB
Table 3. Quantized model footprint for reasoning modules.
Figure 10. UbiHearo’s ( πθ ) pairwise preference win rate on diverse scenarios.
Figure 11. UbiHearo’s ( πf+πθ) pairwise preference win rate on diverse scenarios.
Figure 12. Absolute guidance preference score converted from the pairwise win rates in Fig. 11 . For each baseline b on subset Di , its score is its win rate against UbiHearo , i.e. Sb,i=Wb,i . For UbiHearo , we report the mean complementary win rate against all baselines, SUbiHearo,i=∣B∣1∑b∈B(100−Wb,i) . The final score is averaged over D1 – D4 . “P” denotes the prompted version. Higher is better.
Figure 13. Performance of UbiHearo’s two-stage reasoning scheme.
Figure 14. Performance of acoustic event detector.
Figure 15. System overhead of UbiHearo at Google pixel 8 and pixel 9 smartphones.
Figure 16. Latency of UbiHearo at Google pixel 8 and pixel 9.
Figure 17. Micro-benchmark of UbiHearo’s configurations.
Figure 18. User study.
Figure 19. In-the-wild case study on Pixel 8 across eight everyday scenarios, spanning diverse mobility, noise, and acoustic conditions ( e.g., interference, broadcasts, alarms, and horns) to assess timely, scenario-aware, actionable guidance.
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: <interrupt> and <silent>, capturing four states: onset detection, sustained-relevance triggering, irrelevance suppression, and de-duplication. Applied to Qwen2-Audio-7B, ISM achieves 99.6% interrupt F1 and perfect de-duplication recall on ESC-50. On noisy Epic-Sounds kitchen audio, ISM achieves the highest interrupt F1 without domain-specific training, the only method maintaining strong onset detection without over-triggering or over-suppression. Streaming evaluation confirms real-time viability with 3.5-second average latency.
Amit Kumar Singh Yadav, Ritvik Shrivastava, Xuan Zhang +4
People with sound sensitivity (PWSS) often manage distressing sounds with earplugs and noise-canceling headphones that broadly suppress their surroundings, limiting access to useful auditory cues. We present Sona, a mobile system for personalized, real-time soundscape mediation, informed by prior sound sensitivity research and an online survey of 68 PWSS. Sona selectively attenuates multiple overlapping user-chosen sounds at adjustable strength, suggests targets from ambient sound recognition, and lets users add custom targets from short recordings without retraining the model. In an in-situ evaluation with ten PWSS, participants reported that Sona made their soundscapes more manageable. The study also surfaced uneven attenuation across sound type and context, tensions between managing filters and attending to ongoing activities, and difficulty interpreting personalization outcomes. These findings highlight the need to design for the quality of the residual soundscape, balance user control with interaction demands, and support guided, interpretable personalization.
Jeremy Zhengqi Huang, Emani Hicks, Sidharth +2
University of Michigan Ann Arbor, MI, USA · University of California, Irvine Irvine, CA, USA
The conventional hearing aid is both costly and lim- ited in usage, as it is not intended to detect non-speech audio. Our objective is to develop a machine learning solution to provide a more accurate and affordable mechanism to identify surrounding sounds to improve the safety of the hearing impaired, i.e., if a car is honking behind pedestrians, or a gunshot is fired, and they need to move away from the source. By adding randomized augmentations to audio, concatenating a Mel-Frequency Cepstral Coefficients (MFCCs) diagram and a log-mel Spectrogram, and including Convolutional Neural Networks (CNNs) in a Trans- former architecture, the Randomized Audiomentational Layered Convolutional Transformers (RALCT) model efficiently extracts features from diversified audio representations. In addition, RALCT is small enough, with only approximately 310,000 parameters, to be deployed into mobile devices. Experimental results on the UrbanSound8K dataset resulted in an accuracy consistently over 93% for all variations of RALCT with the highest at 94.56%, reaching state-of-the-art levels. To leverage the capabilities of this technology, a mobile app is developed to be integrated with the model to provide real-time safety control. RALCT thus represents a robust, lightweight, affordable, and versatile deep learning tool to aid the navigation and safety of the hearing impaired.