Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

Yizhou Peng; Chen Chen; Donghang Wu; Hexin Liu; Eng Siong Chng

cs.CLOct 26, 2026

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

Authors: Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong Chng

Organizations: Nanyang Technological University, Singapore · AI Singapore, Singapore

Abstract

Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of $τ^2$ -bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on $τ^2$ -bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.

Figures & tables

Figure 1: Data pipeline. Typed task templates are instantiated by a deterministic entity sampler, simulated as text dialogues, rendered to aligned speech, and annotated with gated early-tool-call events; the green event carries the earliest-actionable-point label (

t_{\text{fire}}

) and its saved time. The table on the right glosses the per-chunk labels.

Figure 2: Single-stream model design. One autoregressive, KV-cached sequence interleaves injected audio chunks with emitted control tokens and payload blocks; one LM head decodes VAD, ET, ASR, and tool JSON under a single weighted cross-entropy loss. Snowflakes are frozen; flames are trainable. The example shows the asr+json variant of Eq. ( 1 ).

	Main corpus	$\tau^{2}$ rendering
Dialogues	26,694	274
Held-out dialogues (valid / test)	210 / 210	0 / 274
Domains	48	3
Audio	786 h	4.9 h
ready / unsure events	28,791 / 24,701	238 / 11
Test gold events	463	249

Table 1: Corpus statistics. The

\tau^{2}

column is the external, eval-only rendering of

\tau^{2}

-bench tasks.

Payload variant	none		asr		asr+json
Test split	ID	$\tau^{2}$	ID	$\tau^{2}$	ID	$\tau^{2}$
Event recall	0.553	0.558	0.510	0.562	0.495	0.582
Event precision	0.123	0.147	0.349	0.412	0.329	0.409
Event F1	0.201	0.232	0.414	0.475	0.395	0.480
ET-state macro-F1	0.477	0.425	0.484	0.474	0.463	0.461
VAD-state macro-F1	0.993	0.976	0.996	0.982	0.994	0.979

Table 2: Decision quality on the in-domain (ID) and audio-rendered

\tau^{2}

-bench splits for event, state, and WER across three variants. Bold marks the best variant per row within each split.

		wait	ready	unsure	hold	Macro
none	@0	0.936	0.309	0.175	0.488	0.477
	@ $+1$	0.938	0.405	0.242	0.493	0.519
	@ $+3$	0.938	0.432	0.272	0.493	0.534
asr	@0	0.932	0.315	0.206	0.483	0.484
	@ $+1$	0.934	0.384	0.244	0.487	0.512
	@ $+3$	0.934	0.406	0.277	0.487	0.526

Table 3: Per-class ET F1, computed per chunk over all speech chunks for the in-domain test split. F1@

+k

credits a prediction up to

k

chunks (

0.6k

s) late via exact matching; @0 is exact alignment. Bold marks the best variant per column at @0.

Variant	Decision from	Fired	P	R	F1
asr	model ready	458	0.376	0.589	0.459
asr+json	model ready	418	0.409	0.586	0.482
turn-asr	Parakeet text	789	0.331	0.894	0.483
turn-gold	gold text	751	0.348	0.894	0.500

Table 4: Firing decision under one turn-level rule for in-domain test split (1,499 user turns, 292 with a gold ready event). A system fires on a turn if it emits at least one ready in it; a fire is a true positive if that turn holds a gold ready . The payload variants fire from their own streaming ready decisions; turn fires when the formatter decides to call. Bold marks the best per column.

Figure 3: Achieved saving of correct early calls (Qwen3-235B-A22B formatter) over the turn baseline: fraction of calls saving more than

x

seconds. Dots mark means (legend); the value at

x{=}0

is the fraction that fires before the turn ends; medians over all correct calls are 0.0 s ( asr ) and 0.7 s ( asr+json ) on ID.

			ID			$\tau^{2}$
Variant	Call from	Formatter	EM	Tool	Arg	EM	Tool	Arg
asr+json	model JSON	—	0.173	0.232	0.519	0.000	0.073	0.054
asr+json	model ASR	235B	0.536	0.833	0.710	0.681	0.978	0.844
asr+json	model ASR	30B	0.530	0.833	0.707	0.674	0.964	0.839
asr	model ASR	235B	0.578	0.840	0.711	0.618	0.971	0.770
asr	model ASR	30B	0.535	0.845	0.683	0.588	0.963	0.776

Table 5: Call quality by payload variant. Variant : whose ready fires trigger the call. Call from : the text the call is built from, with the model’s own JSON (no formatter), the model’s own span (

a_{c}

), a Parakeet transcript, or the gold transcript; the latter three are via the formatter. For turn the text covers the completed turn; for all other variants, the audio up to the fire. EM, tool-name, and argument accuracy on matched fires ( turn : on true-positive turns). Bold marks the best per column (except GT ASR).

Spoken Dialogue Systems

Spoken dialogue systems let machines understand and respond to speech while managing turn-taking, context, emotion, and real-time interaction.

See all 140 papers

VoxMind: An End-to-End Agentic Spoken Dialogue System
Apr 17, 2026Tianle Liang, Yifu Chen, Shengpeng Ji +7Multi-Agent LLM Systems Spoken Dialogue Systems
Zhejiang University · China University of Petroleum-Beijing at Karamay · Xiamen University
SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
Mar 17, 2026Jonggeun Lee, Junseong Pyo, Jeongmin Park +1Spoken Dialogue Systems Conversational Agents
Graduate School of Data Science, Seoul National University · Department of Information Systems, Hanyang University · Department of Computer Science and Engineering, Seoul National University
Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
Aug 3, 2026Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7Tool-Augmented Language Model Agents Agent Evaluation
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China

Apr 17, 2026J/K move · Enter open · S save

VoxMind: An End-to-End Agentic Spoken Dialogue System

Tianle Liang, Yifu Chen, Shengpeng Ji, Yijun Chen+6

Zhejiang University · China University of Petroleum-Beijing at Karamay · Xiamen University

Multi-Agent LLM Systems Spoken Dialogue Systems Tool-Using Agents Conversational Agents

Open paper AskPDF arXiv

Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.

Abstract

Spoken Dialogue Systems

Explore similar work

VoxMind: An End-to-End Agentic Spoken Dialogue System

SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue

Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models