Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds, and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate real-time performance on 6 ms streaming audio chunks. Across real-world environments in previously unseen indoor and outdoor scenarios, our system enables expressive per-class sound control and achieves substantial improvements in target-class enhancement and interference suppression. Our results show that the world need not be heard as a single, undifferentiated stream: with Aurchestra, the soundscape becomes truly programmable.
Figures & tables
Figure 1. Aurchestra transforms the auditory world into a programmable studio. Unlike traditional hearables that offer binary noise cancellation (all-or-nothing), Aurchestra enables fine-grained soundscape control. (A) In a complex acoustic scene, (B) the system automatically detects active sound classes (e.g., speech, traffic, birds) and populates a dynamic interface. The user can then “mix” their reality in real-time, (C) independently suppressing interference (traffic) while increasing the volume of some targets (speech) and maintaining others (nature), effectively acting as the audio engineer of their own life.
Table 1. Model comparison for target sound extraction. Evaluation performed with 20 target classes, single source in mixture, and single output channel.
#Targets
#Outputs
SNRi (dB)
SI-SNRi (dB)
1
1
11.99 ± 7.04
11.27 ± 8.21
5
9.56 ± 6.74
8.53 ± 7.85
20
3.51 ± 6.16
3.10 ± 6.23
2
1
13.10 ± 4.54
11.93 ± 5.77
5
11.81 ± 4.07
10.29 ± 6.00
20
6.63 ± 3.30
6.04 ± 3.41
Table 2. Aurchestra’s performance across different mixture complexities and output channel configurations. Total target classes: 20.
Model
FiLM Placement
SNRi (dB)
SI-SNRi (dB)
Orange Pi
First block only
11.76 ± 4.27
9.51 ± 5.43
Orange Pi
All blocks
12.26 ± 4.38
10.16 ± 5.72
Orange Pi
All except first
11.88 ± 4.27
9.77 ± 5.55
NeuralAids
First block only
9.46 ± 3.81
5.73 ± 6.21
NeuralAids
All blocks
10.50 ± 4.15
7.27 ± 6.28
NeuralAids
All except first
10.19 ± 4.01
7.09 ± 5.38
Table 3. Ablation study on FiLM layer placement. Evaluation with 5 output channels, 1–5 targets in mixture.
Window ( LC / LB / LF )
Latency
M (0.50M)
L ( ∼ 1.5M)
6/6/4 (default)
10 ms
10.16
9.17
4/8/4
8 ms
8.79
10.58
8/4/4
12 ms
8.44
10.30
4/12/0 (causal)
4 ms
8.14
9.49
Table 4. Accuracy-latency trade-off across model sizes and window configurations. Evaluation with 5 output channels, 1-5 targets in mixture, FiLM on all blocks.
Figure 3. (a) CDF plots of the runtime of the three proposed networks on their respective deployment platforms. (b) Runtime and power consumption of running the NeuralAids model at different GAP9 clock frequencies.
Figure 4. Sound Event Detection performance comparison on 5-second audio segments. We compare YAMNet, AST, and our fine-tuned AST model across different numbers of simultaneous sound sources. Our fine-tuned model maintains high performance even with 5 concurrent sources.
Figure 5. Runtime CDF of the fine-tuned AST model across different iPhone platforms. All platforms complete inference faster than the 5-second audio segment duration.
Figure 6. In-the-wild scenarios. The wearer and sound sources were free to move, and head rotation was uncontrolled.
Figure 7. User listening study results comparing Aurchestra against the No AI baseline. Aurchestra achieves substantial improvements in background noise suppression (+1.54) and overall experience (+0.95) while maintaining target sound clarity. N=17 participants.
Figure 8. Aurchestra’s per-class clarity ratings.
Figure 9. Response times for sound selection. Our dynamic interface reduces selection time by 67.9% compared to the static interface by displaying only detected sounds rather than all 20 categories. Error bars show standard deviation.
Figure 10. System Usability Scale (SUS) evaluation. Distribution of participant responses for each SUS item on a 5-point scale. Items marked with [R] are reverse-scored.
Figure 11. User preferences survey results (N=7). (a) Preferred device. (b) Expected response time. (c) Sound selection method. (d) Volume adjustment method. (e) Number of simultaneous sounds. (f) Usage situations.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12. Inter-class confusion matrix for single-source detection (#sources = 1). Each cell shows the fraction of samples where the true class (row) is predicted as another class (column). Diagonal values indicate per-class recall. Confusion is concentrated among impulsive percussive sounds (hammer, door knock, gunshot, glass breaking), which share similar short broadband energy profiles.
Class
#sources = 1
#sources = 2
#sources = 3
#sources = 4
#sources = 5
Δ (1 → 5)
Alarm Clock
0.991
0.985
0.978
0.980
0.969
− 0.022
Toilet Flush
1.000
0.962
0.964
0.963
0.954
− 0.046
Music
0.995
0.968
0.966
0.955
0.952
− 0.043
Cock A Doodle Doo
0.995
0.982
0.969
0.957
0.942
− 0.053
Baby Cry
0.962
0.941
0.936
0.930
0.918
− 0.044
Cricket
0.962
0.919
0.925
0.926
0.916
− 0.046
Appendix
Table 5. Per-class F1 scores across increasing numbers of simultaneous sound sources (#sources). Evaluated on 5-second segments with optimal per-class thresholds. #sources denotes the number of distinct target sound classes simultaneously present in the mixture. Bold indicates F1 > 0.90 at #sources = 5. Classes are grouped into three performance tiers (separated by horizontal rules).
Length
#Sources
Accuracy
Precision
Recall
F1-Score
10s
1
0.994
0.919
0.951
0.935
2
0.975
0.835
0.904
0.868
3
0.964
0.843
0.911
0.876
4
0.953
0.838
0.918
0.876
5
0.939
0.841
0.910
0.874
7s
1
0.994
0.930
0.940
0.935
Appendix
Table 6. SED performance across audio lengths and number of target sources. Fine-tuned AST model with per-class optimal thresholds.
Class
FUSS n
FUSS
Ours
Δ
car horn
4
+20.94
11.70
+9.24
door knock
15
+20.62
10.10
+10.52
dog
7
+17.20
10.28
+6.92
glass breaking
10
+16.56
9.78
+6.78
singing
21
+14.95
8.64
+6.31
cat
14
+12.98
9.94
+3.04
Appendix
Table 7. Per-class SI-SDRi comparison: FUSS vs. our evaluation set. 15 of 20 classes evaluable. FUSS sources are independently mixed FSD50K recordings.
Figure 13. Complete FG source label distribution of the FUSS eval set — Part 1/3 (top-ranked labels). 1,445 sources across 167 unique labels. Blue: mapped to our 20 target classes (180 sources, 12.5%). Orange: unmapped FSD50K categories. Gray: not in FSD50K ground truth. Continued in Figs. 14 and 15 .
Figure 14. Complete FG source label distribution of the FUSS eval set — Part 2/3 (mid-ranked labels). Continued from Fig. 13 .
Figure 15. Complete FG source label distribution of the FUSS eval set — Part 3/3 (lowest-ranked labels). Continued from Fig. 14 . Most labels in this range have count ≤ 5, limiting statistical reliability of per-class analysis on FUSS.