Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds, and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate real-time performance on 6 ms streaming audio chunks. Across real-world environments in previously unseen indoor and outdoor scenarios, our system enables expressive per-class sound control and achieves substantial improvements in target-class enhancement and interference suppression. Our results show that the world need not be heard as a single, undifferentiated stream: with Aurchestra, the soundscape becomes truly programmable.
Figures & tables
Figure 1. Aurchestra transforms the auditory world into a programmable studio. Unlike traditional hearables that offer binary noise cancellation (all-or-nothing), Aurchestra enables fine-grained soundscape control. (A) In a complex acoustic scene, (B) the system automatically detects active sound classes (e.g., speech, traffic, birds) and populates a dynamic interface. The user can then “mix” their reality in real-time, (C) independently suppressing interference (traffic) while increasing the volume of some targets (speech) and maintaining others (nature), effectively acting as the audio engineer of their own life.
Table 1. Model comparison for target sound extraction. Evaluation performed with 20 target classes, single source in mixture, and single output channel.
#Targets
#Outputs
SNRi (dB)
SI-SNRi (dB)
1
1
11.99 ± 7.04
11.27 ± 8.21
5
9.56 ± 6.74
8.53 ± 7.85
20
3.51 ± 6.16
3.10 ± 6.23
2
1
13.10 ± 4.54
11.93 ± 5.77
5
11.81 ± 4.07
10.29 ± 6.00
20
6.63 ± 3.30
6.04 ± 3.41
Table 2. Aurchestra’s performance across different mixture complexities and output channel configurations. Total target classes: 20.
Model
FiLM Placement
SNRi (dB)
SI-SNRi (dB)
Orange Pi
First block only
11.76 ± 4.27
9.51 ± 5.43
Orange Pi
All blocks
12.26 ± 4.38
10.16 ± 5.72
Orange Pi
All except first
11.88 ± 4.27
9.77 ± 5.55
NeuralAids
First block only
9.46 ± 3.81
5.73 ± 6.21
NeuralAids
All blocks
10.50 ± 4.15
7.27 ± 6.28
NeuralAids
All except first
10.19 ± 4.01
7.09 ± 5.38
Table 3. Ablation study on FiLM layer placement. Evaluation with 5 output channels, 1–5 targets in mixture.
Window ( LC / LB / LF )
Latency
M (0.50M)
L ( ∼ 1.5M)
6/6/4 (default)
10 ms
10.16
9.17
4/8/4
8 ms
8.79
10.58
8/4/4
12 ms
8.44
10.30
4/12/0 (causal)
4 ms
8.14
9.49
Table 4. Accuracy-latency trade-off across model sizes and window configurations. Evaluation with 5 output channels, 1-5 targets in mixture, FiLM on all blocks.
Figure 3. (a) CDF plots of the runtime of the three proposed networks on their respective deployment platforms. (b) Runtime and power consumption of running the NeuralAids model at different GAP9 clock frequencies.
Figure 4. Sound Event Detection performance comparison on 5-second audio segments. We compare YAMNet, AST, and our fine-tuned AST model across different numbers of simultaneous sound sources. Our fine-tuned model maintains high performance even with 5 concurrent sources.
Figure 5. Runtime CDF of the fine-tuned AST model across different iPhone platforms. All platforms complete inference faster than the 5-second audio segment duration.
Figure 6. In-the-wild scenarios. The wearer and sound sources were free to move, and head rotation was uncontrolled.
Figure 7. User listening study results comparing Aurchestra against the No AI baseline. Aurchestra achieves substantial improvements in background noise suppression (+1.54) and overall experience (+0.95) while maintaining target sound clarity. N=17 participants.
Figure 8. Aurchestra’s per-class clarity ratings.
Figure 9. Response times for sound selection. Our dynamic interface reduces selection time by 67.9% compared to the static interface by displaying only detected sounds rather than all 20 categories. Error bars show standard deviation.
Figure 10. System Usability Scale (SUS) evaluation. Distribution of participant responses for each SUS item on a 5-point scale. Items marked with [R] are reverse-scored.
Figure 11. User preferences survey results (N=7). (a) Preferred device. (b) Expected response time. (c) Sound selection method. (d) Volume adjustment method. (e) Number of simultaneous sounds. (f) Usage situations.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12. Inter-class confusion matrix for single-source detection (#sources = 1). Each cell shows the fraction of samples where the true class (row) is predicted as another class (column). Diagonal values indicate per-class recall. Confusion is concentrated among impulsive percussive sounds (hammer, door knock, gunshot, glass breaking), which share similar short broadband energy profiles.
Class
#sources = 1
#sources = 2
#sources = 3
#sources = 4
#sources = 5
Δ (1 → 5)
Alarm Clock
0.991
0.985
0.978
0.980
0.969
− 0.022
Toilet Flush
1.000
0.962
0.964
0.963
0.954
− 0.046
Music
0.995
0.968
0.966
0.955
0.952
− 0.043
Cock A Doodle Doo
0.995
0.982
0.969
0.957
0.942
− 0.053
Baby Cry
0.962
0.941
0.936
0.930
0.918
− 0.044
Cricket
0.962
0.919
0.925
0.926
0.916
− 0.046
Appendix
Table 5. Per-class F1 scores across increasing numbers of simultaneous sound sources (#sources). Evaluated on 5-second segments with optimal per-class thresholds. #sources denotes the number of distinct target sound classes simultaneously present in the mixture. Bold indicates F1 > 0.90 at #sources = 5. Classes are grouped into three performance tiers (separated by horizontal rules).
Length
#Sources
Accuracy
Precision
Recall
F1-Score
10s
1
0.994
0.919
0.951
0.935
2
0.975
0.835
0.904
0.868
3
0.964
0.843
0.911
0.876
4
0.953
0.838
0.918
0.876
5
0.939
0.841
0.910
0.874
7s
1
0.994
0.930
0.940
0.935
Appendix
Table 6. SED performance across audio lengths and number of target sources. Fine-tuned AST model with per-class optimal thresholds.
Class
FUSS n
FUSS
Ours
Δ
car horn
4
+20.94
11.70
+9.24
door knock
15
+20.62
10.10
+10.52
dog
7
+17.20
10.28
+6.92
glass breaking
10
+16.56
9.78
+6.78
singing
21
+14.95
8.64
+6.31
cat
14
+12.98
9.94
+3.04
Appendix
Table 7. Per-class SI-SDRi comparison: FUSS vs. our evaluation set. 15 of 20 classes evaluable. FUSS sources are independently mixed FSD50K recordings.
Figure 13. Complete FG source label distribution of the FUSS eval set — Part 1/3 (top-ranked labels). 1,445 sources across 167 unique labels. Blue: mapped to our 20 target classes (180 sources, 12.5%). Orange: unmapped FSD50K categories. Gray: not in FSD50K ground truth. Continued in Figs. 14 and 15 .
Figure 14. Complete FG source label distribution of the FUSS eval set — Part 2/3 (mid-ranked labels). Continued from Fig. 13 .
Figure 15. Complete FG source label distribution of the FUSS eval set — Part 3/3 (lowest-ranked labels). Continued from Fig. 14 . Most labels in this range have count ≤ 5, limiting statistical reliability of per-class analysis on FUSS.
The conventional hearing aid is both costly and lim- ited in usage, as it is not intended to detect non-speech audio. Our objective is to develop a machine learning solution to provide a more accurate and affordable mechanism to identify surrounding sounds to improve the safety of the hearing impaired, i.e., if a car is honking behind pedestrians, or a gunshot is fired, and they need to move away from the source. By adding randomized augmentations to audio, concatenating a Mel-Frequency Cepstral Coefficients (MFCCs) diagram and a log-mel Spectrogram, and including Convolutional Neural Networks (CNNs) in a Trans- former architecture, the Randomized Audiomentational Layered Convolutional Transformers (RALCT) model efficiently extracts features from diversified audio representations. In addition, RALCT is small enough, with only approximately 310,000 parameters, to be deployed into mobile devices. Experimental results on the UrbanSound8K dataset resulted in an accuracy consistently over 93% for all variations of RALCT with the highest at 94.56%, reaching state-of-the-art levels. To leverage the capabilities of this technology, a mobile app is developed to be integrated with the model to provide real-time safety control. RALCT thus represents a robust, lightweight, affordable, and versatile deep learning tool to aid the navigation and safety of the hearing impaired.
People with sound sensitivity (PWSS) often manage distressing sounds with earplugs and noise-canceling headphones that broadly suppress their surroundings, limiting access to useful auditory cues. We present Sona, a mobile system for personalized, real-time soundscape mediation, informed by prior sound sensitivity research and an online survey of 68 PWSS. Sona selectively attenuates multiple overlapping user-chosen sounds at adjustable strength, suggests targets from ambient sound recognition, and lets users add custom targets from short recordings without retraining the model. In an in-situ evaluation with ten PWSS, participants reported that Sona made their soundscapes more manageable. The study also surfaced uneven attenuation across sound type and context, tensions between managing filters and attending to ongoing activities, and difficulty interpreting personalization outcomes. These findings highlight the need to design for the quality of the residual soundscape, balance user control with interaction demands, and support guided, interpretable personalization.
Jeremy Zhengqi Huang, Emani Hicks, Sidharth +2
University of Michigan Ann Arbor, MI, USA · University of California, Irvine Irvine, CA, USA
We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning. Code, demos, and listening-test results are available at https://haozhang6720.github.io/SoundscapeAgentDemoPage/.
Hao Zhang, Yiwen Zhao, Yixuan Zhang +2
Wuhan University, Wuhan, China · Tencent Hunyuan, Bellevue, WA, USA