cs.AINov 16, 2026

UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents

Authors: Fengmin Wu, Sicong Liu, Zimu Zhou, Chenren Xu, Wanbing Zhao, Peiwen Sun, Zhiwen Yu

Organizations: Northwestern Polytechnical University, China · City University of Hong Kong, China · Peking University, China · Harbin Engineering University, China

Abstract

For Deaf and Hard-of-Hearing (DHH) users, effective sound awareness goes beyond detecting what sound occurs; it requires understanding what it means, how urgent it is, and what to do. However, existing mobile assistants largely reduce acoustic scenes to isolated labels or generic alerts, failing to provide contextual guidance when the same sound implies different risks across scenarios. We present \sysname, a mobile hearing assistant agent that closes the sensing--reasoning--action loop for scenario-aware and personalized DHH assistance. \sysname adopts a neuro-symbolic design that transforms continuous acoustic and physical context streams into structured intermediate representations (IRs) for controllable reasoning and guidance. It introduces three key mechanisms. First, a perception-thresholded sensing mechanism enables energy-efficient always-on awareness by activating processing only for perceptually meaningful events while preserving safety-critical cues through deterministic overrides. Second, a staged neuro-symbolic reasoning framework performs promptless scenario understanding by conditioning an on-device audio-language model on structured IRs instead of user-authored context descriptions. Third, a bounded human-in-the-loop personalization mechanism adapts user-specific guidance through lightweight on-device decision-boundary updates while preserving population-aligned reasoning. We evaluate \sysname on smartphones and in-vehicle systems across diverse scenarios. It achieves favorable accuracy--latency--energy tradeoffs and personalized guidance.

Figures & tables

Explore similar work

Sep 18, 2026

I'll Keep an Ear Out: Teaching AudioLLMs Proactive Audio Assistance

Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: <interrupt> and <silent>, capturing four states: onset detection, sustained-relevance triggering, irrelevance suppression, and de-duplication. Applied to Qwen2-Audio-7B, ISM achieves 99.6% interrupt F1 and perfect de-duplication recall on ESC-50. On noisy Epic-Sounds kitchen audio, ISM achieves the highest interrupt F1 without domain-specific training, the only method maintaining strong onset detection without over-triggering or over-suppression. Streaming evaluation confirms real-time viability with 3.5-second average latency.
Apr 1, 2026

Sona: Personalized Soundscape Mediation to Support People with Sound Sensitivity

People with sound sensitivity (PWSS) often manage distressing sounds with earplugs and noise-canceling headphones that broadly suppress their surroundings, limiting access to useful auditory cues. We present Sona, a mobile system for personalized, real-time soundscape mediation, informed by prior sound sensitivity research and an online survey of 68 PWSS. Sona selectively attenuates multiple overlapping user-chosen sounds at adjustable strength, suggests targets from ambient sound recognition, and lets users add custom targets from short recordings without retraining the model. In an in-situ evaluation with ten PWSS, participants reported that Sona made their soundscapes more manageable. The study also surfaced uneven attenuation across sound type and context, tensions between managing filters and attending to ongoing activities, and difficulty interpreting personalization outcomes. These findings highlight the need to design for the quality of the residual soundscape, balance user control with interaction demands, and support guided, interpretable personalization.
Aug 16, 2026

"What's That Sound?": A Versatile, Robust, and Lightweight Convolutional Transformer for Environment Sound Recognition

The conventional hearing aid is both costly and lim- ited in usage, as it is not intended to detect non-speech audio. Our objective is to develop a machine learning solution to provide a more accurate and affordable mechanism to identify surrounding sounds to improve the safety of the hearing impaired, i.e., if a car is honking behind pedestrians, or a gunshot is fired, and they need to move away from the source. By adding randomized augmentations to audio, concatenating a Mel-Frequency Cepstral Coefficients (MFCCs) diagram and a log-mel Spectrogram, and including Convolutional Neural Networks (CNNs) in a Trans- former architecture, the Randomized Audiomentational Layered Convolutional Transformers (RALCT) model efficiently extracts features from diversified audio representations. In addition, RALCT is small enough, with only approximately 310,000 parameters, to be deployed into mobile devices. Experimental results on the UrbanSound8K dataset resulted in an accuracy consistently over 93% for all variations of RALCT with the highest at 94.56%, reaching state-of-the-art levels. To leverage the capabilities of this technology, a mobile app is developed to be integrated with the model to provide real-time safety control. RALCT thus represents a robust, lightweight, affordable, and versatile deep learning tool to aid the navigation and safety of the hearing impaired.