Voice-based public services have evolved from manual switchboards to intelligent platforms to meet modern user needs. However, current systems remain constrained by rigid menu-driven interaction, fragmented vertical services, and limited extensibility. To address these issues, we present VISAGENT, an agent-based end-to-end voice interaction system for large-scale public services that integrates end-to-end speech interaction, hierarchical agent routing, a scalable voice-agent marketplace, and user memory support. The system replaces menu-driven workflows with natural speech and dynamically dispatches user requests to either a general conversational model or specialized domain agents via a two-level routing paradigm that achieves sub-second intent-to-agent selection over hundreds of agents. User memory informs preference-aware agent selection and supplies relevant preferences and context to selected agents, supporting personalized task execution and reducing the need for users to repeat previously provided information. A voice-agent registration mechanism converts task agents into conversational entities, enabling dynamic onboarding and ecosystem-level extensibility. We validate VISAGENT through a real-world deployment in China's 114 hotline across multiple service domains. Experimental results demonstrate low-latency full-duplex interaction, robust routing scalability, and competitive speech naturalness. More broadly, this work advocates a shift from isolated voice assistants toward agent-centric conversational infrastructures, providing a general blueprint for building extensible voice-agent systems in public services and beyond.
Figures & tables
Figure 1: Interaction paradigms in the 114 hotline: traditional services rely on recorded prompts, keypad navigation, and human operators, whereas VISAGENT enables callers to express their needs directly and access domain agents or general conversational assistance through natural speech.
Figure 2: Overall architecture of VISAGENT. The framework comprises a full-duplex user interaction layer, a two-level hierarchical routing layer, and an extensible voice-agent marketplace. The user memory service provides shared context for agent selection and task execution, while the voice-agent registration mechanism converts task agents into conversational agents and onboards them into the marketplace.
Figure 3: Request-processing workflow of VISAGENT. Spoken input is routed either to the end-to-end conversational model or to a domain-specific agent selected according to the request, dialogue context, and relevant user memory. During agent execution, failures return to the first-level router with the available context and failure feedback, while additional service requests explicitly introduced by the user are redispatched within the same session. After successful task completion, the system asks whether the user has another request. An affirmative response initiates a new request cycle, whereas a negative response triggers interaction summarization, memory update, and session termination.
Figure 4: Architecture of the speech interaction component used in VISAGENT. The primary speech-to-speech path integrates an audio encoder, an input adapter, an LLM, a streaming audio decoder, and a vocoder. An auxiliary CTC transcription branch supplies textual representations to the hierarchical routing layer and downstream service agents.
Figure 5: Three-phase training procedure for the speech interaction model. Stage 1 trains the input adapter to align the output of the frozen audio encoder with the representation space of the frozen LLM. Stage 2 trains the audio decoder to generate discrete acoustic features from the frozen LLM representations. Stage 3 jointly fine-tunes the input adapter, LLM, and audio decoder while keeping the audio encoder frozen. Snowflake and flame symbols denote frozen and trainable neural modules, respectively; the vocoder converts the decoder output into the final speech waveform.
Information
Use in the framework
Agent identity and provider
Identify the agent and its operator.
Domain, description, and capabilities
Organize candidates and describe the services available for matching.
Service tags and coverage regions
Describe service characteristics and geographic applicability.
Endpoint
Identify the URL used to invoke the selected agent.
Function specifications
Define callable operations and their input and output schemas.
Table 1: Information supplied during agent registration. Appendix A.1 gives a compact example.
Track
Test set
MiMo-Audio
Step-Audio-mini
Qwen3-Omni
VISAGENT speech
Model size
7B
8B
30B
14B
Basic
Repeat-zh
16.920
99.316
95.415
85.419
LCSTS-zh
80.280
81.456
84.369
82.745
SQuAD-zh
38.344
66.230
50.544
59.694
APE-zh
62.807
43.333
25.087
48.596
MLC-zh
22.528
70.804
32.183
74.942
Table 2: Results on 16 open-source Chinese test sets from URO-Bench. The subset averages are computed over the displayed test sets in each track. “VISAGENT speech” denotes the speech interaction model used in VISAGENT. Boldface marks the highest subset average in each track.
Measurement
System
Latency
Theoretical Latency
VISAGENT speech
300 ms
LLaMA-Omni
226 ms
MinMo
600 ms
Moshi
200 ms
Practical Latency
VISAGENT
1.05 s
IFLYTEK SPARK
2.32 s
Table 3: Theoretical latency and practical latency. Application-level measurements were collected on February 12, 2026.
System
CER ↓
NMOS ↑
GPT-SoVITS
3.160
3.388
F5-TTS
1.720
3.474
fish-speech
1.340
3.489
CosyVoice 2
1.730
3.500
VISAGENT synthesis
1.360
3.491
Table 4: Speech synthesis results on SEEDTTS test-zh. Lower CER and higher NMOS indicate better performance.
50 agents
100 agents
319 agents
Selector
Routing
Accuracy
Latency
Accuracy
Latency
Accuracy
Latency
DeepSeek V3
Single-level
100%
<1 s
100%
4 s
98%
9 s
Two-level
100%
<1 s
100%
<1 s
100%
<1 s
Qwen3
Single-level
100%
<1 s
100%
1 s
99%
8 s
Two-level
100%
<1 s
100%
<1 s
100%
<1 s
Kimi K2
Single-level
100%
<1 s
100%
4 s
98%
10 s
Table 5: Routing accuracy and latency on 100 user queries under three marketplace sizes.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Illustrative flight-booking interaction in VISAGENT. The hierarchical routing process identifies the flight domain and selects a flight agent. The selected agent plans the task, collects missing passenger information through spoken interaction, and invokes flight-search and booking functions to complete the outbound and return reservations.
Agentic speech systems must exchange more than text, including speaker identity, timing, phonetic information, and behavioral annotations. Yet these signals are often produced in incompatible tool-specific formats, making agent-to-agent handoff fragile. We present OneVoice, a lightweight, JSON-native intermediate representation that provides a shared semantic structure for speech pipelines. OneVoice organizes heterogeneous speech evidence into validated session records with stable identifiers, layered transcripts, explicit timing relationships, and provenance information. We evaluate OneVoice in two complementary multi-agent workflows covering event aggregation and acoustic-phonetic temporal linking across three language models. Compared with implicit agent-defined handoffs, OneVoice substantially improves aggregation reliability and the preservation of temporal relationships, demonstrating the value of an explicit speech-specific representation for agent communication.
Vipul Charugundla, Dancheng Liu, Jinjun Xiong
University at Buffalo · University of Texas at San Antonio
Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.
Tianle Liang, Yifu Chen, Shengpeng Ji +7
Zhejiang University · China University of Petroleum-Beijing at Karamay · Xiamen University
We present VoiceAgentRAG, an open-source dual-agent memory router that decouples retrieval from response generation. A background Slow Thinker agent continuously monitors the conversation stream, predicts likely follow-up topics using an LLM, and pre-fetches relevant document chunks into a FAISS-backed semantic cache. A foreground Fast Talker agent reads only from this sub-millisecond cache, bypassing the vector database entirely on cache hits.