Practical Assessment of Agentic Systems in Acoustics
Authors: Arne Hildenbrand, Philipp Diel, Zhuo Cao, Hanno Scharr, Stefan Pischinger, Steffen Marburg
Organizations: Chair of Vibroacoustics of Vehicles and Machines, Department of Engineering Physics and Computation, TUM School of Engineering and Design, Technical University of Munich, Boltzmannstrasse 15, 85748 Garching, Germany · Chair of Thermodynamics of Mobile Energy Conversion Systems, Faculty of Mechanical Engineering, RWTH Aachen University, Forckenbeckstrasse 4, 52074 Aachen, Germany · IAS-8, Forschungszentrum Jülich, Wilhelm-Johnen-Straße, 52425 Jülich, Germany
In engineering, applications of artificial intelligence have so far been largely limited to surrogate models and optimization tasks. Large language models changed this, as they can write software from a task given in words. Placed inside a harness, a system of components that lets it search for information and execute software, such a model becomes an agent that can take over recurring engineering tasks. This work measures how the model and the elements of the harness shape the performance and behavior of the agent in two common acoustic applications. In the first, the agent detects and locates faults in audio recordings from a vehicle interior with near-perfect accuracy. A benchmark on this task shows that a weak model sets a limit that no harness design can overcome, and that design choices take effect only once the model is capable enough to use them. In the second, a case study on the design of a metamaterial unit cell, the agent reliably finds designs that achieve a specified insertion loss over a given frequency range. The results aim to help the reader understand what each element of an agent contributes and how to design an agent for acoustic tasks.
Figures & tables
Figure 1: The LLM and the harness around it form the agent. The functions the harness provides are grouped into four modules, which Figure 2 breaks down into the elements that provide them.
Figure 2: The elements of an agent harness from an implementation perspective. The prompt enters the context, together with the mounted knowledge, the configuration files, the memory and the skills, all assembled by the harness. The model reasons over this context and decides which skills to execute. The result of the execution re-enters the context, and the loop repeats until an answer is returned. The numbers follow the order in which Section 2.1 describes the elements. Everything inside the dashed boundary runs in an isolated execution environment.
Figure 3: The three axes along which an agentic system is designed. Communication runs through one central agent, decentralized with each agent communicating with every other agent, or hierarchical, with agents communicating within a level and with the adjacent level. Collaboration is cooperative when the agents work together on a task, competitive when they compete for it, and coopetitive when both happen at once. The strategy is a set of rules that decides when agents are allowed to communicate with one another, an assignment of roles, or model based, in which case several approaches are tried and one of them is selected.
Figure 4: A multi-agent system built from the blocks and the structure of the harness of Figure 2 . The system of Section 2.3 is centralized, cooperative and role based. One orchestrating agent spawns two subagents via its execute module, and each subagent repeats the same structure, depicted here as a pictogram. The orchestration agent writes the task into the context of a subagent, and every subagent keeps its own context, its own harness and its own calls to the model. Only the final message of a subagent returns into the context of the orchestrating agent, so the subagents never communicate with one another.
Figure 5: The nine fault types of the fault detection task. Each panel shows the waveform or the running mean of an affected recording at the top and the corresponding spectrogram below. Red dashed lines and the shaded span mark the start and end of the fault. For clipping, the unclipped signal (gray) is drawn behind the clipped one (blue). For trend and long trend, the running mean of the corrupted signal (red) is compared with a flat baseline without drift (gray).
No.
Input Prompt
001
Is anything wrong with this recording? (with a true positive sample)
002
Is anything wrong with this recording? (with a true negative sample)
003
Which faults are present in this recording and of which types?
004
Which faults are present, of which types, and in which time ranges?
005
A colleague is convinced that this recording is corrupted, identify the fault and its location.
006
For each of these recordings, is it corrupted?
Table 1: The task prompts of the benchmark. They vary along two axes: what is asked, rising from a yes-or-no judgment (001, 002, 005) through the fault types present (003, 007) to the types with their time ranges (004, 008), and how many recordings are given, one for 001 to 005 and a batch of 50 recordings for 006 to 008. Every prompt requests its answer as a structured text block.
Prompts
Metric
Pooled
single recording
binary
001, 002, 005
binary score
no
type
003
F1
no
localization
004
F1
no
batch of recordings
binary
006
balanced accuracy
yes
Table 2: The six task groups, the metric applied to each, and whether the metric is pooled over a batch.
Task
none
undocu- mented
baseline
knowledge
single recording
001
10 /10
10 /10
10 /10
10 /10
002
2/10
5/10
2/10
10 /10
005
1/10
9/10
3/10
10 /10
003
0.65 ± 0.28 (0.18)
0.73 ± 0.17 (0.11)
0.88 ± 0.16 (0.10)
1.00 ± 0.00 (0.00)
004
0.26 ± 0.25 (0.16)
0.60 ± 0.16 (0.10)
0.69 ± 0.14 (0.09)
1.00 ± 0.00 (0.00)
Table 3: Results of the benchmark. Each cell reports the mean over N=10 runs with the standard deviation and, in brackets, the resolution beneath it. The single-recording binary tasks are given as the number of correct runs out of ten.
Instructions
curated only
corrupted only
both
fair
25/30
4/30
1/30
knowledge favored
30/30
0/30
0/30
fair, raised reasoning
17/30
2/30
11/30
Table 4: Which of the two contradicting threshold tables a run used, and under which instructions. Each row contains thirty runs, ten for each of the tasks 004, 006, and 008. The three conditions are instructions that name both sources equally, instructions that favor the knowledge over the memory, and the fair instructions with a raised reasoning effort.
Figure 6: The four configurations on the tasks 004, 006, and 008. The color indicates the model that drives the agent, GPT 5.6 terra or GPT 5.6 luna. Each bar is the mean over ten runs, and the error bar is the resolution, so two bars with overlapping error bars cannot be distinguished by this benchmark. On 006, no configuration separates the two models, on 004 they separate under baseline and knowledge, and on 008 the weaker model scores lower under every configuration.
Task
none
undocu- mented
baseline
knowledge
004
0.20 ± 0.27 (0.17)
0.49 ± 0.22 (0.14)
0.35 ± 0.22 (0.14)
0.80 ± 0.31 (0.19)
006
0.77 ± 0.18 (0.12)
0.85 ± 0.15 (0.09)
0.69 ± 0.23 (0.15)
0.94 ± 0.16 (0.10)
008
0.14 ± 0.10 (0.06)
0.26 ± 0.14 (0.09)
0.54 ± 0.06 (0.04)
0.80 ± 0.32 (0.20)
Table 5: The four configurations on the weaker model GPT 5.6 luna. The harness, the recordings, and the prompts are those of Table 3 , and only the model that drives the loop is exchanged. Each cell is the mean over N=10 runs with the standard deviation and, in brackets, the resolution beneath it.
Figure 7: The simulated setup with scatterers or resonators in air, where the domain is governed by the Helmholtz equation. The array repeats infinitely in the y -direction, and a plane wave travels in the x -direction. The wave enters at the inlet on the left and leaves at the outlet on the right, and both ports absorb what reaches them. Every design respects a maximum thickness of the metamaterial and a minimum gap between two unit cells. The insertion loss is evaluated on the receiver line behind the array.
Figure 8: The four unit-cell templates. A rigid circle (a) scatters the incident wave, and a c-shape (b), a cavity behind a narrow slit, resonates. The diatomic variants (c) and (d) contain two members of different sizes in one cell, placed at one quarter and three quarters of the lattice spacing. The agent sets the lattice spacings ax and ay , the number of columns, and the shape parameters. These are the radius r of a circle and the outer radius R , the wall thickness t , and the slit width w of a c-shape. Each member of a diatomic cell has its own values.
Task
Asked for
easy
middle
hard
band (Hz)
101
narrowband
8dB
20dB
45dB
970–1030
102
broadband
6dB
18dB
40dB
891–1122
103
broadband with search
6dB
18dB
40dB
891–1122
104
broadband, no templates
6dB
18dB
40dB
891–1122
Table 6: The four design tasks and the insertion loss each has to reach. Task 101 asks for a narrow band around 1kHz , and the others ask for a wider band around the same center. Task 103 adds the search strategies, and task 104 withholds the templates. The insertion loss is required at every frequency of the band.
Level
Cell
layers
margin (dB)
design eval.
claim (dB)
101 tonal, solver without search
easy
c-shape
2
3.5
4
−0.1
middle
c-shape
2
10.9
7
−0.1
hard
c-shape
2
2.6
5
−0.1
102 broadband, solver without search
easy
c-shape
4
17.6
3
−0.3
Table 7: The results of the four tasks, each at three levels of difficulty. The cell column gives the template the agent selected, and the layers column gives how many cells deep it made the array. The margin column gives the smallest insertion loss over the graded band relative to the required value, so a negative margin is a miss. The claim column gives the difference between the insertion loss the agent reported and the one found by an independent rerun of the same design. The design evaluation column counts the calls of the solver. In task 104, the agent submits a mesh and no template parameters, so the rerun refines this mesh and does not build a new mesh, unlike in the other cases. The claims listed there are the smallest in the table. Eight of the twelve designs reach the required insertion loss, and claimed and graded values differ by no more than 1.4 decibels. Given templates, the agent selects the c-shape in seven of the nine runs. Without them, it draws cells of its own, which are named after their function and shown in Figure 9 .
Figure 9: Each submitted design, solved again independently over the range from 300 to 2200 hertz. The thin horizontal lines mark the insertion loss required at each level of difficulty, which is 6 , 18 , and 40 decibels in the broadband tasks and 8 , 20 , and 45 decibels in 101. The gray stripe is the graded band. Eight of the twelve designs remain above their line across that band. The middle level of 103 and the easy level of 104 miss it narrowly, and the hard levels of 102 and 104 miss it by a wide margin. The hard curve of 104 ends at 1.4kHz , because above this frequency its cell no longer satisfies the single-mode condition of Section 4.1 . The plot is cut at 80 decibels, and since the model has no losses, the height of the peaks has no physical meaning.
Figure 10: The pressure field of one design per task at 1kHz , plotted on a common length scale, so that a taller cell is also drawn taller, and on a common color scale, so that the panels can be compared. In each panel, the two dashed lines mark the front and the back of the array, the dotted line further to the right is the receiver at which the insertion loss is read, and the white shapes are the scatterers. The dark vertical band in front of each array is the first pressure node of the standing wave that the reflection builds up there, where the incident and the reflected wave cancel each other. The panel of 102 shows a diatomic cell whose two members are so close in size that its two layers appear as four. In 104 middle, the pair of cavities facing the wave is loud, and the pair behind it is quiet. This is precisely the function of a resonator, and the agent drew it without a template to copy.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Task
none
undocu- mented
baseline
knowledge
skill calls
001
0 ± 0 (0)
2 ± 1 (1)
10 ± 4 (2)
14 ± 2 (1)
002
0 ± 0 (0)
3 ± 2 (1)
13 ± 2 (2)
14 ± 1 (1)
005
0 ± 0 (0)
5 ± 2 (1)
16 ± 4 (2)
14 ± 1 (0)
003
0 ± 0 (0)
3 ± 1 (1)
13 ± 0 (0)
14 ± 1 (0)
004
0 ± 0 (0)
2 ± 2 (1)
13 ± 1 (1)
16 ± 5 (3)
Appendix
Table 8: Cost of a run for the four configurations of Phase 1, on GPT 5.6 terra. Each cell is the mean over N=10 runs, with the standard deviation and, in brackets, the resolution beneath it, as in Table 3 . Tokens are thousands of tokens per run and the wall clock time is seconds per run, both summed over the main agent and its workers.
Configuration
any skill report
primitives only
all nine detectors
none
1/80
1/80
0/80
undocumented
59/80
29/80
0/80
baseline
80/80
0/80
71/80
knowledge
80/80
0/80
79/80
Appendix
Table 9: Which skills each configuration reached for, over the eighty runs of a configuration. Every cell is counted from the reports the skills wrote. A run counts as primitives only when it produced a feature artifact and never called a detector, so its decisions were made somewhere other than in the detectors.
Task
fair
knowledge favored
fair, raised reasoning
004
0.94 ± 0.19 (0.12)
1.00 ± 0.00 (0.00)
0.88 ± 0.25 (0.16)
006
0.98 ± 0.06 (0.04)
1.00 ± 0.00 (0.00)
0.98 ± 0.05 (0.03)
008
0.90 ± 0.21 (0.13)
0.99 ± 0.02 (0.01)
0.99 ± 0.03 (0.02)
Appendix
Table 10: The scores of the three conditions in which the curated thresholds were contradicted by corrupted ones in the memory of the agent. Each cell is the mean over N=10 runs, with the standard deviation and, in brackets, the resolution beneath it, as in Table 3 , where the same tasks are reported without a conflict. The third condition repeats the fair one with the reasoning effort of the model raised.
Task
none
undocu- mented
baseline
knowledge
skill calls
004
0 ± 0 (0)
2 ± 2 (1)
13 ± 0 (0)
14 ± 2 (1)
006
0 ± 0 (0)
1 ± 3 (2)
1033 ± 1160 (733)
1334 ± 994 (629)
008
0 ± 0 (0)
0 ± 0 (0)
1104 ± 826 (523)
1146 ± 775 (490)
thousands of tokens
004
59 ± 19 (12)
74 ± 26 (17)
76 ± 24 (15)
53 ± 9 (6)
Appendix
Table 11: What a run costs on GPT 5.6 luna, the weaker model, to be read against Table 8 . Each cell is the mean over N=10 runs, with the standard deviation and, in brackets, the resolution beneath it.
Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents' ability to handle diverse task settings and adaptive tool interactions.
Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China
Audio Large Language Models (Audio LLMs) are embedded in agents for smart homes, vehicles, wearables, and embodied systems, where acoustic inputs can propagate from perception into persistent state and consequential execution. This setting extends security analysis beyond model behavior to the system conditions that determine how acoustic inputs can influence agent behavior and external actions. Existing surveys examine acoustic security, LLM-based agents, and edge deployment in isolation, leaving their security interactions in deployed Audio LLM-based agents insufficiently characterized. This survey introduces an Acoustic Sensing-to-Action (S2A) system model that traces these security implications along the path from acoustic sensing to external execution, while accounting for edge-cloud runtime and lifecycle transformation. We use the model to synthesize attacks, defenses, and evaluation methods across the acoustic, agent, and edge-system literature. The synthesis shows that security gaps can emerge within components and at the transitions between stages, where protections developed for one stage do not necessarily provide end-to-end assurance in the deployed system. These findings motivate security evaluation that follows acoustic influence from sensing to action across persistent state, system boundaries, and changing deployment conditions in real-world agents.
Binhong Wang, Haidong Kang, Yanfeng Zhang +3
Northeastern University, China · Department of Computer Science, Hong Kong Baptist University, Hong Kong · Institute of Intelligent Manufacturing, Nanjing Tech University, Nanjing, China +1
Audio agents extend large audio-language models (LALMs) by decomposing audio questions into tool calls, intermediate evidence, and iterative reasoning steps. However, as LALMs become stronger, the key challenge shifts from enabling tool use to determining when agentic evidence acquisition genuinely benefits audio understanding. We propose Audio-Mind, an auditable and pluggable framework for conditional evidence acquisition in audio understanding. Audio-Mind dynamically combines a strong frontend with planner-guided tool use, preserving frontend judgment when initial evidence is sufficient while acquiring bounded external evidence for questions with unresolved evidence gaps. Experiments on MMAR and MSU-Bench show that Audio-Mind outperforms prior audio-agent baselines, reaching 80.4% accuracy on MMAR and 82.8% accuracy on MSU-Bench. A matched-backbone comparison highlights why this design matters: under strong audio frontends, agentic decomposition can become an orchestration bottleneck when the workflow does not preserve the frontend's holistic audio-grounded judgment. Beyond accuracy, Audio-Mind produces higher-quality, auditable reasoning traces that expose uncertainty, tool evidence, and answer rationales, offering a potential basis for more reliable audio-QA annotation and error analysis.
Yucheng Wang, Jing Peng, Hanqi Li +6
School of Intelligence Science and Technology, Nanjing University, China · X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · School of Automation Science and Engineering, Xi’an Jiaotong University, China +1