cs.SDOct 29, 2026
SavePractical Assessment of Agentic Systems in Acoustics
Organizations: Chair of Vibroacoustics of Vehicles and Machines, Department of Engineering Physics and Computation, TUM School of Engineering and Design, Technical University of Munich, Boltzmannstrasse 15, 85748 Garching, Germany · Chair of Thermodynamics of Mobile Energy Conversion Systems, Faculty of Mechanical Engineering, RWTH Aachen University, Forckenbeckstrasse 4, 52074 Aachen, Germany · IAS-8, Forschungszentrum Jülich, Wilhelm-Johnen-Straße, 52425 Jülich, Germany
Abstract
In engineering, applications of artificial intelligence have so far been largely limited to surrogate models and optimization tasks. Large language models changed this, as they can write software from a task given in words. Placed inside a harness, a system of components that lets it search for information and execute software, such a model becomes an agent that can take over recurring engineering tasks. This work measures how the model and the elements of the harness shape the performance and behavior of the agent in two common acoustic applications. In the first, the agent detects and locates faults in audio recordings from a vehicle interior with near-perfect accuracy. A benchmark on this task shows that a weak model sets a limit that no harness design can overcome, and that design choices take effect only once the model is capable enough to use them. In the second, a case study on the design of a metamaterial unit cell, the agent reliably finds designs that achieve a specified insertion loss over a given frequency range. The results aim to help the reader understand what each element of an agent contributes and how to design an agent for acoustic tasks.
Figures & tables
Figure 1: The LLM and the harness around it form the agent. The functions the harness provides are grouped into four modules, which Figure 2 breaks down into the elements that provide them.
Figure 2: The elements of an agent harness from an implementation perspective. The prompt enters the context, together with the mounted knowledge, the configuration files, the memory and the skills, all assembled by the harness. The model reasons over this context and decides which skills to execute. The result of the execution re-enters the context, and the loop repeats until an answer is returned. The numbers follow the order in which Section 2.1 describes the elements. Everything inside the dashed boundary runs in an isolated execution environment.
Figure 3: The three axes along which an agentic system is designed. Communication runs through one central agent, decentralized with each agent communicating with every other agent, or hierarchical, with agents communicating within a level and with the adjacent level. Collaboration is cooperative when the agents work together on a task, competitive when they compete for it, and coopetitive when both happen at once. The strategy is a set of rules that decides when agents are allowed to communicate with one another, an assignment of roles, or model based, in which case several approaches are tried and one of them is selected.
Figure 4: A multi-agent system built from the blocks and the structure of the harness of Figure 2 . The system of Section 2.3 is centralized, cooperative and role based. One orchestrating agent spawns two subagents via its execute module, and each subagent repeats the same structure, depicted here as a pictogram. The orchestration agent writes the task into the context of a subagent, and every subagent keeps its own context, its own harness and its own calls to the model. Only the final message of a subagent returns into the context of the orchestrating agent, so the subagents never communicate with one another.
Figure 5: The nine fault types of the fault detection task. Each panel shows the waveform or the running mean of an affected recording at the top and the corresponding spectrogram below. Red dashed lines and the shaded span mark the start and end of the fault. For clipping, the unclipped signal (gray) is drawn behind the clipped one (blue). For trend and long trend, the running mean of the corrupted signal (red) is compared with a flat baseline without drift (gray).
| No. | Input Prompt |
| 001 | Is anything wrong with this recording? (with a true positive sample) |
| 002 | Is anything wrong with this recording? (with a true negative sample) |
| 003 | Which faults are present in this recording and of which types? |
| 004 | Which faults are present, of which types, and in which time ranges? |
| 005 | A colleague is convinced that this recording is corrupted, identify the fault and its location. |
| 006 | For each of these recordings, is it corrupted? |
Table 1: The task prompts of the benchmark. They vary along two axes: what is asked, rising from a yes-or-no judgment (001, 002, 005) through the fault types present (003, 007) to the types with their time ranges (004, 008), and how many recordings are given, one for 001 to 005 and a batch of recordings for 006 to 008. Every prompt requests its answer as a structured text block.
| Prompts | Metric | Pooled | |
| single recording | |||
| binary | 001, 002, 005 | binary score | no |
| type | 003 | no | |
| localization | 004 | no | |
| batch of recordings | |||
| binary | 006 | balanced accuracy | yes |
Table 2: The six task groups, the metric applied to each, and whether the metric is pooled over a batch.
| Task | none | undocu- mented | baseline | knowledge |
| single recording | ||||
| 001 | 10 /10 | 10 /10 | 10 /10 | 10 /10 |
| 002 | 2/10 | 5/10 | 2/10 | 10 /10 |
| 005 | 1/10 | 9/10 | 3/10 | 10 /10 |
| 003 | 0.65 0.28 (0.18) | 0.73 0.17 (0.11) | 0.88 0.16 (0.10) | 1.00 0.00 (0.00) |
| 004 | 0.26 0.25 (0.16) | 0.60 0.16 (0.10) | 0.69 0.14 (0.09) | 1.00 0.00 (0.00) |
Table 3: Results of the benchmark. Each cell reports the mean over runs with the standard deviation and, in brackets, the resolution beneath it. The single-recording binary tasks are given as the number of correct runs out of ten.
| Instructions | curated only | corrupted only | both |
| fair | 25/30 | 4/30 | 1/30 |
| knowledge favored | 30/30 | 0/30 | 0/30 |
| fair, raised reasoning | 17/30 | 2/30 | 11/30 |
Table 4: Which of the two contradicting threshold tables a run used, and under which instructions. Each row contains thirty runs, ten for each of the tasks 004, 006, and 008. The three conditions are instructions that name both sources equally, instructions that favor the knowledge over the memory, and the fair instructions with a raised reasoning effort.
Figure 6: The four configurations on the tasks 004, 006, and 008. The color indicates the model that drives the agent, GPT 5.6 terra or GPT 5.6 luna. Each bar is the mean over ten runs, and the error bar is the resolution, so two bars with overlapping error bars cannot be distinguished by this benchmark. On 006, no configuration separates the two models, on 004 they separate under baseline and knowledge, and on 008 the weaker model scores lower under every configuration.
| Task | none | undocu- mented | baseline | knowledge |
| 004 | 0.20 0.27 (0.17) | 0.49 0.22 (0.14) | 0.35 0.22 (0.14) | 0.80 0.31 (0.19) |
| 006 | 0.77 0.18 (0.12) | 0.85 0.15 (0.09) | 0.69 0.23 (0.15) | 0.94 0.16 (0.10) |
| 008 | 0.14 0.10 (0.06) | 0.26 0.14 (0.09) | 0.54 0.06 (0.04) | 0.80 0.32 (0.20) |
Table 5: The four configurations on the weaker model GPT 5.6 luna. The harness, the recordings, and the prompts are those of Table 3 , and only the model that drives the loop is exchanged. Each cell is the mean over runs with the standard deviation and, in brackets, the resolution beneath it.
Figure 7: The simulated setup with scatterers or resonators in air, where the domain is governed by the Helmholtz equation. The array repeats infinitely in the -direction, and a plane wave travels in the -direction. The wave enters at the inlet on the left and leaves at the outlet on the right, and both ports absorb what reaches them. Every design respects a maximum thickness of the metamaterial and a minimum gap between two unit cells. The insertion loss is evaluated on the receiver line behind the array.
Figure 8: The four unit-cell templates. A rigid circle (a) scatters the incident wave, and a c-shape (b), a cavity behind a narrow slit, resonates. The diatomic variants (c) and (d) contain two members of different sizes in one cell, placed at one quarter and three quarters of the lattice spacing. The agent sets the lattice spacings and , the number of columns, and the shape parameters. These are the radius of a circle and the outer radius , the wall thickness , and the slit width of a c-shape. Each member of a diatomic cell has its own values.
| Task | Asked for | easy | middle | hard | band (Hz) |
| 101 | narrowband | 970–1030 | |||
| 102 | broadband | 891–1122 | |||
| 103 | broadband with search | 891–1122 | |||
| 104 | broadband, no templates | 891–1122 |
Table 6: The four design tasks and the insertion loss each has to reach. Task 101 asks for a narrow band around , and the others ask for a wider band around the same center. Task 103 adds the search strategies, and task 104 withholds the templates. The insertion loss is required at every frequency of the band.
| Level | Cell | layers | margin (dB) | design eval. | claim (dB) |
| 101 tonal, solver without search | |||||
| easy | c-shape | 2 | 3.5 | 4 | |
| middle | c-shape | 2 | 10.9 | 7 | |
| hard | c-shape | 2 | 2.6 | 5 | |
| 102 broadband, solver without search | |||||
| easy | c-shape | 4 | 17.6 | 3 | |
Table 7: The results of the four tasks, each at three levels of difficulty. The cell column gives the template the agent selected, and the layers column gives how many cells deep it made the array. The margin column gives the smallest insertion loss over the graded band relative to the required value, so a negative margin is a miss. The claim column gives the difference between the insertion loss the agent reported and the one found by an independent rerun of the same design. The design evaluation column counts the calls of the solver. In task 104, the agent submits a mesh and no template parameters, so the rerun refines this mesh and does not build a new mesh, unlike in the other cases. The claims listed there are the smallest in the table. Eight of the twelve designs reach the required insertion loss, and claimed and graded values differ by no more than decibels. Given templates, the agent selects the c-shape in seven of the nine runs. Without them, it draws cells of its own, which are named after their function and shown in Figure 9 .
Figure 9: Each submitted design, solved again independently over the range from to hertz. The thin horizontal lines mark the insertion loss required at each level of difficulty, which is , , and decibels in the broadband tasks and , , and decibels in 101. The gray stripe is the graded band. Eight of the twelve designs remain above their line across that band. The middle level of 103 and the easy level of 104 miss it narrowly, and the hard levels of 102 and 104 miss it by a wide margin. The hard curve of 104 ends at , because above this frequency its cell no longer satisfies the single-mode condition of Section 4.1 . The plot is cut at decibels, and since the model has no losses, the height of the peaks has no physical meaning.
Figure 10: The pressure field of one design per task at , plotted on a common length scale, so that a taller cell is also drawn taller, and on a common color scale, so that the panels can be compared. In each panel, the two dashed lines mark the front and the back of the array, the dotted line further to the right is the receiver at which the insertion loss is read, and the white shapes are the scatterers. The dark vertical band in front of each array is the first pressure node of the standing wave that the reflection builds up there, where the incident and the reflected wave cancel each other. The panel of 102 shows a diatomic cell whose two members are so close in size that its two layers appear as four. In 104 middle, the pair of cavities facing the wave is loud, and the pair behind it is quiet. This is precisely the function of a resonator, and the agent drew it without a template to copy.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | none | undocu- mented | baseline | knowledge |
| skill calls | ||||
| 001 | 0 0 (0) | 2 1 (1) | 10 4 (2) | 14 2 (1) |
| 002 | 0 0 (0) | 3 2 (1) | 13 2 (2) | 14 1 (1) |
| 005 | 0 0 (0) | 5 2 (1) | 16 4 (2) | 14 1 (0) |
| 003 | 0 0 (0) | 3 1 (1) | 13 0 (0) | 14 1 (0) |
| 004 | 0 0 (0) | 2 2 (1) | 13 1 (1) | 16 5 (3) |
Table 8: Cost of a run for the four configurations of Phase 1, on GPT 5.6 terra. Each cell is the mean over runs, with the standard deviation and, in brackets, the resolution beneath it, as in Table 3 . Tokens are thousands of tokens per run and the wall clock time is seconds per run, both summed over the main agent and its workers.
| Configuration | any skill report | primitives only | all nine detectors |
| none | 1/80 | 1/80 | 0/80 |
| undocumented | 59/80 | 29/80 | 0/80 |
| baseline | 80/80 | 0/80 | 71/80 |
| knowledge | 80/80 | 0/80 | 79/80 |
Table 9: Which skills each configuration reached for, over the eighty runs of a configuration. Every cell is counted from the reports the skills wrote. A run counts as primitives only when it produced a feature artifact and never called a detector, so its decisions were made somewhere other than in the detectors.
| Task | fair | knowledge favored | fair, raised reasoning |
| 004 | 0.94 0.19 (0.12) | 1.00 0.00 (0.00) | 0.88 0.25 (0.16) |
| 006 | 0.98 0.06 (0.04) | 1.00 0.00 (0.00) | 0.98 0.05 (0.03) |
| 008 | 0.90 0.21 (0.13) | 0.99 0.02 (0.01) | 0.99 0.03 (0.02) |
Table 10: The scores of the three conditions in which the curated thresholds were contradicted by corrupted ones in the memory of the agent. Each cell is the mean over runs, with the standard deviation and, in brackets, the resolution beneath it, as in Table 3 , where the same tasks are reported without a conflict. The third condition repeats the fair one with the reasoning effort of the model raised.
| Task | none | undocu- mented | baseline | knowledge |
| skill calls | ||||
| 004 | 0 0 (0) | 2 2 (1) | 13 0 (0) | 14 2 (1) |
| 006 | 0 0 (0) | 1 3 (2) | 1033 1160 (733) | 1334 994 (629) |
| 008 | 0 0 (0) | 0 0 (0) | 1104 826 (523) | 1146 775 (490) |
| thousands of tokens | ||||
| 004 | 59 19 (12) | 74 26 (17) | 76 24 (15) | 53 9 (6) |
Table 11: What a run costs on GPT 5.6 luna, the weaker model, to be read against Table 8 . Each cell is the mean over runs, with the standard deviation and, in brackets, the resolution beneath it.
More in this topic
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 389 papers