Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

Yanjie Zhang; Nanchen Hu; Yushi Sun

cs.SDSep 26, 2026

Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

Authors: Yanjie Zhang, Nanchen Hu, Yushi Sun

Organizations: HKUST, Hong Kong SAR, China · LIGHTSPEED, Shenzhen, China

Abstract

Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.

Figures & tables

Figure 1: Overview of VGBench construction and evaluation. The benchmark contains 1,018 items across side-talk, self-talk, and speaker-switch scenarios. Each speaker-switch item has same-speaker, switch, and text-only conditions. Metrics are interpreted jointly rather than collapsed into one aggregate score.

Method	Side-talk correct	Self-talk mute	Switch mute	Same-speaker select.	Text-only select.
Raw Audio LLMs
Qwen3-Omni	39.0	57.0	1	77	81
Nemotron-Omni	32.7	28.7	0	84	85
Gemini-3.8-flash	32.4	68.6	7	80	71
Kimi-Audio-7B	38.2	30.5	14	53	44
Step-Audio-R1.1	41.0	22.4	1	96	97

Table 1: Action-selection results on the complete VGBench corpus. Values are percentages. Speaker-switch reports switch mute rate and target-tool selection in the near-field same-speaker and text-only positive controls. Side-talk combines canonical tool selection and free responses scored by the fixed Qwen3.5-35B judge.

Model or training setting	Self-talk mute	Side-talk correct	Same-speaker select.	Switch mute	Text-only select.
Base Qwen	50.0	53.2	81.3	0.0	87.5
VoxGate supervised training	52.0	68.4	100.0	91.3	100.0
VoxGate + GRPO	60.0	70.9	100.0	92.5	100.0

Table 2: Addressedness results on the fixed 20% VGBench test split. Values are percentages. Switch mute is reported with near-field same-speaker and text-only target-tool selection to expose all-mute behavior.

Source	Field	Gap	Base	SFT	GRPO
Same	Near	0	7.50	0.00	0.00
Same	Near	600	5.00	1.25	1.25
Same	Far	600	6.25	60.00	60.00
Different	Near	0	6.25	35.00	31.25
Different	Near	600	7.50	50.00	53.75
Different	Far	600	0.00	91.25	92.50

Table 3: Mute rates (%) on the original 80-command cue validity WAVs. Same/near targets a tool; same/far and different-speaker conditions target [Mute] . The last row is the original switch condition.

Field, gap	Same: Tool near, [Mute] far	Different: [Mute]
Near, 0 ms	97.50 $\rightarrow$ 97.50	18.75 $\rightarrow$ 68.75
Near, 600 ms	100.00 $\rightarrow$ 98.75	18.75 $\rightarrow$ 85.00
Far, 0 ms	2.50 $\rightarrow$ 3.75	23.75 $\rightarrow$ 80.00
Far, 600 ms	6.25 $\rightarrow$ 5.00	46.25 $\rightarrow$ 96.25

Table 4: Target-action accuracy (%) under the proximity rule in the scene-consistent eight-condition test, original SFT

\rightarrow

cue-balanced SFT. Each cell has

n=80

; equal-weight accuracy is 39.22%

\rightarrow

66.88%. Same-speaker triggers target a tool when near and [Mute] when far; different-speaker triggers target [Mute] . Complete mute/tool rates appear in Appendix H .

WearVox-only post-training
Model	$A$ ( $n=114$ )	$B$ ( $n=58$ )	$C$ ( $n=112$ )	$D$ ( $n=100$ )	Overall ( $n=384$ )
Base Qwen	28.11	96.60	55.40	65.30	45.57
SFT	33.33	94.83	85.71	84.80	63.28
SFT + GRPO	42.98	98.28	86.61	83.43	67.71
Joint VoxGate post-training
VoxGate supervised training	37.71	91.40	90.20	89.50	72.14

Table 5: Held-out WearVox task retention for the original SFT/GRPO training route (cue-balanced SFT is reported separately above).

A

aggregates closed-book and grounded answers;

B

C

, and

D

denote abstention, tool use, and live translation. Counts are shown in the column headers.

D

is the mean judged translation score (

\times 100

); Overall thresholds each translation item at 0.85 before aggregating all 384 binary outcomes.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Result block	Systems	Split	Decoding	Scoring record
Raw Audio LLMs	Six native systems	Full	Greedy	Canonical parser; Qwen3.5 side-talk judge
Training-free methods	AURA, TwS, SHANKS	Full	Greedy	Same action contract and judge
VoxGate gating	Base, SFT, GRPO	Held-out	Greedy	Same instruction, parser, and judge
WearVox retention	Base and post-training controls	384 items	Greedy	Official A/B/C/D scorer and translation judge

Appendix

Table 6: Result blocks and evaluation records. “Full” denotes the complete retained VGBench corpus, while “held-out” denotes the fixed post-training test partition.

Method	Side-talk	Self-talk mute	Switch mute	Single select.	Text-only select.
Raw Audio LLMs
Qwen3-Omni	39.0	57.0	1	77	81
Nemotron-Omni	32.7	28.7	0	84	85
Gemini-3.8-flash	32.4	68.6	7	80	71
Kimi-Audio-7B	38.2	30.5	14	53	44
Step-Audio-R1.1	41.0	22.4	1	96	97

Appendix

Table 7: Complete full-corpus raw and training-free results. The corpus contains 395 side-talk recordings, 223 self-talk recordings, and 400 speaker-switch pairs. All values are percentages.

Component	Configuration
Base model	Qwen3-Omni-30B-A3B-Instruct
SFT adaptation	LoRA on all 48-layer attention $q,k,v,o$ projections; rank 32, alpha 64
Frozen modules	Audio encoder and aligner
SFT optimization	Three epochs, learning rate $10^{-4}$ , effective batch size 32, eight H200 GPUs
SFT mixture	WearVox answer, abstention, tool-use, and translation data plus the approximately 80% VGBench training partition
GRPO mixture	The same downstream task mixture and addressedness examples; pair-aware grouping for speaker-switch

Appendix

Table 8: VoxGate implementation configuration.

Variable	Controlled construction
Specified text	Matched context and trigger across single, switch, and text-only conditions
Action target	Tool in single/text-only; [Mute] in switch
Voice synthesis	Eight distinct voices simulate speaker changes
Temporal boundary	600 ms silence before the switch trigger
Acoustic scene	Far-field room rendering on the B trigger only
Coverage	400 cases, 10 tools, 40 cases per tool

Appendix

Table 9: Speaker-switch construction controls.

Trigger source	Trigger field	Gap	Target
Same speaker	Near	0 ms	Tool
Same speaker	Near	600 ms	Tool
Same speaker	Far	600 ms	[Mute]
Different speaker	Near	0 ms	[Mute]
Different speaker	Near	600 ms	[Mute]
Different speaker	Far	600 ms	[Mute]

Appendix

Table 10: Controlled conditions in the speaker-switch validity set. Each condition contains the same 80 held-out commands. Targets follow the proximity rule.

Source	Field	Gap	Target	Base Qwen		VoxGate SFT		VoxGate GRPO
				M	T	M	T	M	T
Same	Near	0 ms	Tool	7.50	65.00	0.00	100.00	0.00	100.00
Same	Near	600 ms	Tool	5.00	68.75	1.25	98.75	1.25	98.75
Same	Far	600 ms	[Mute]	6.25	73.75	60.00	40.00	60.00	40.00
Different	Near	0 ms	[Mute]	6.25	75.00	35.00	65.00	31.25	68.75
Different	Near	600 ms	[Mute]	7.50	71.25	50.00	50.00	53.75	46.25

Appendix

Table 11: Speaker-switch cue validity results on the frozen 80-command test set. Cells report mute rate (M) and matching-tool selection rate (T) in percent. Under the proximity rule, same/near targets T and all other audio rows target M. The final row is the original switch condition from Table 2 ; its non-target tool rate was not retained in the reported summary and is marked with “–”.

Source	Field	Gap	M	T
Same	Near	0 ms	2.50	97.50
Same	Near	600 ms	2.50	97.50
Same	Far	600 ms	90.00	10.00
Different	Near	0 ms	66.25	33.75
Different	Near	600 ms	82.50	17.50
Different	Far	600 ms	93.75	5.00

Appendix

Table 12: Cue-balanced SFT on the original six-condition validity WAVs (

n=80

per condition), using the same prompts, temperature

0

, parser, and scorer as the original validity study. M is [Mute] rate and T is matching-tool selection rate (%). These WAVs differ from the scene-consistent eight-condition set in Table 13 .

Relation	Field	Gap	Original SFT		Cue-balanced SFT
			M	T	M	T
Same	Near	0 ms	2.50	97.50	2.50	97.50
Same	Near	600 ms	0.00	100.00	1.25	98.75
Same	Far	0 ms	2.50	97.50	3.75	96.25
Same	Far	600 ms	6.25	93.75	5.00	95.00
Different	Near	0 ms	18.75	81.25	68.75	31.25

Appendix

Table 13: Complete eight-condition results on scene-consistent audio. Each row contains 80 held-out commands. M is [Mute] rate and T is matching-tool selection rate (%). Same/near targets T; same/far and different-speaker rows target M under the proximity rule.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
May 14, 2026Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo +3Audio-Language Model Evaluation Voice Agent Evaluation
Dialpad Inc.
Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
Aug 3, 2026Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7Tool-Augmented Language Model Agents Agent Evaluation
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China

Sep 30, 2026J/K move · Enter open · S save

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang

Department of Electronic Engineering, Tsinghua University · University of Cambridge

ASR Evaluation Voice Agent Evaluation Computer-Use Agents LLM Agent Evaluation Automatic Speech Recognition Computer-Use Agent Evaluation+2

Open paper AskPDF arXiv

Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

Abstract

Audio-Language Models

Explore similar work

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models