Multi-agent Auditory Scene Analysis: Improved Localization Speed and Robustness by Multi-beamformed Speech Quality Feedback
Authors: Caleb Rascon
Organizations: Instituto de Investigaciones en Matematicas Aplicadas y en Sistemas, Universidad Nacional Autonoma de Mexico, Circuito Escolar 3000, Coyoacan, 03740, Ciudad de Mexico, Mexico.
A real-time auditory scene analyzer (ASA) aims to carry out the tasks of locating, separating and classifying the sound sources present in a given acoustic environment. Recently, an effort has been made into modelling an ASA as a multi-agent system, with each one of its agents performing one of the aforementioned tasks and communicating their results to the rest of their peer agents. These communication routes are used as feedback loops to fix local errors at a global level, providing robustness while reducing local complexity. An example of the benefits of this approach is the optimization of speech quality by correcting in real-time the estimated location of the speech source of interest. However, their optimization speed has been shown to be considerably slow. One possible reason is that it solely relies on a series of single quality estimations (provided by a reference-free quality estimator model) that vary considerably from one window to the next, which results in a difficult search space to optimize. In this work, a new optimization mechanism is proposed that instead relies on a series of sets of quality estimations over a range of locations, providing a clearer view of the search space, simplifying its optimization. The proposed ASA now has a considerably smaller optimization time, is more accurate, and is more stable when being evaluated in real-life acoustic scenarios to correct higher levels of localization errors, all while being less complex than previous efforts. The only trade-off is that there is an increase in the response time of the quality estimation agent, but the complete ASA is still able to run in real-time. The performance shown in this work again shows the benefits of modelling an ASA as a multi-agent system.
Figures & tables
Figure 1: Current version of the multi-agent auditory scene analyzer presented in Rascon et al. (2026) .
Figure 2: Location (direction of arrival) correction as presented in Rascon (2025) .
Figure 3: A diagram summarizing the reasoning behind the proposed optimization mechanism.
Quality Type
RMS ( ∘ ) ↓
PESQ
8.0633
SDR
13.7106
STOI
2.0247
Table 1: Mean accuracy for each quality type.
Beamforming Technique
Overall ‘good run’ percentage ↑
PHASE
0.3187
MVDR
0.6244
Table 2: Overall ‘good run’ percentage for each beamforming technique, using STOI.
Figure 4: Optimization configurations vs. ‘good runs’ percentage.
Figure 5: Optimization configurations vs. accuracy.
Figure 6: Optimization configurations vs. optimization time.
θrange
θsteps
RMS ↓
σRMS↓
topt↓
tlat↓
20.0 ∘
5
2.49 ∘
0.36 ∘
8.73 s.
0.42 s.
20.0 ∘
7
2.90 ∘
0.31 ∘
14.45 s.
0.57 s.
25.0 ∘
5
2.93 ∘
0.49 ∘
6.46 s.
0.42 s.
25.0 ∘
7
1.86 ∘
0.23 ∘
10.14 s.
0.57 s.
Table 3: Recommended optimization configurations.
Figure 7: Behaviour of balanced optimization configuration [θrange=20.0,θsteps=5] .
Figure 8: Behavior of the optimization mechanism presented in Rascon et al. (2026) .
Department Store
Office A
# sources
θerror
RMS ↓
σRMS↓
topt↓
RMS ↓
σRMS↓
topt↓
5 ∘
2.37 ∘
0.13 ∘
-
2.83 ∘
0.23 ∘
-
2
15 ∘
2.42 ∘
0.18 ∘
6.97 s.
3.00 ∘
0.23 ∘
3.12 s.
25 ∘
2.33 ∘
0.08 ∘
10.56 s.
2.88 ∘
0.17 ∘
9.11 s.
5 ∘
3.76 ∘
0.17 ∘
-
3.13 ∘
0.26 ∘
-
3
15 ∘
3.54 ∘
0.23 ∘
10.76 s.
3.36 ∘
0.41 ∘
7.29 s.
Table 4: Evaluation of [θrange=20.0,θsteps=5] configuration in different acoustic scenarios.