OneVoice: An Intermediate Representation for Agentic Speech Pipelines

Vipul Charugundla; Dancheng Liu; Jinjun Xiong

eess.ASSep 16, 2026

OneVoice: An Intermediate Representation for Agentic Speech Pipelines

Authors: Vipul Charugundla, Dancheng Liu, Jinjun Xiong

Organizations: University at Buffalo · University of Texas at San Antonio

Abstract

Agentic speech systems must exchange more than text, including speaker identity, timing, phonetic information, and behavioral annotations. Yet these signals are often produced in incompatible tool-specific formats, making agent-to-agent handoff fragile. We present OneVoice, a lightweight, JSON-native intermediate representation that provides a shared semantic structure for speech pipelines. OneVoice organizes heterogeneous speech evidence into validated session records with stable identifiers, layered transcripts, explicit timing relationships, and provenance information. We evaluate OneVoice in two complementary multi-agent workflows covering event aggregation and acoustic-phonetic temporal linking across three language models. Compared with implicit agent-defined handoffs, OneVoice substantially improves aggregation reliability and the preservation of temporal relationships, demonstrating the value of an explicit speech-specific representation for agent communication.

Figures & tables

Figure 1: Speech-agent communication with pairwise, tool-specific handoffs in the conventional workflow and a shared, validated OneVoice representation in the proposed workflow.

Figure 2: High-level structure of a OneVoice record. A session contains speaker and metadata blocks together with turn-level records. Each turn may include multiple transcript layers, optional word- and phone-level alignments, and optional task-specific annotations.

Model	Condition	Precision $\uparrow$	Recall $\uparrow$	$F_{1}$ $\uparrow$	MAE $\downarrow$
GPT-5 mini	C0 implicit	0.156	0.874	0.265	66.47
	C1 OneVoice	0.933	0.811	0.868	3.40
	C2 OneVoice+val.	0.877	0.869	0.873	3.47
GPT-5.4 mini	C0 implicit	0.264	0.888	0.407	35.60
	C1 OneVoice	0.607	0.840	0.705	9.67
	C2 OneVoice+val.	0.886	0.714	0.790	5.20

Table 1: UXSSD event aggregation results.

Model	Condition	Precision $\uparrow$	Recall $\uparrow$	$F_{1}$ $\uparrow$	Word-link acc. $\uparrow$	Invalid link $\downarrow$	Boundary err. $\downarrow$
GPT-5 mini	C0 implicit	0.908	0.800	0.851	0.819	0.164	194.45 ms
	C1 OneVoice	0.978	0.823	0.894	0.864	0.205	155.56 ms
	C2 OneVoice+val.	0.969	0.906	0.936	0.945	0.201	99.63 ms
GPT-5.4 mini	C0 implicit	0.542	0.604	0.571	0.595	0.192	478.61 ms
	C1 OneVoice	0.762	0.345	0.475	0.765	0.367	415.14 ms
	C2 OneVoice+val.	0.618	0.558	0.586	0.755	0.204	352.35 ms

Table 2: TIMIT temporal linking results. Word-link accuracy evaluates hierarchical association between phone- and word-level evidence, while boundary error measures temporal localization.

Explore similar work

Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
Aug 22, 2026Sameep Chattopadhyay, Alexander Erdmann, Mari OstendorfStreaming ASR Automatic Speech Recognition
University of Washington · SRI International
Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
Aug 3, 2026Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7Tool-Augmented Language Model Agents Agent Evaluation
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China

Sep 30, 2026J/K move · Enter open · S save

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang

Department of Electronic Engineering, Tsinghua University · University of Cambridge

ASR Evaluation Voice Agent Evaluation Computer-Use Agents LLM Agent Evaluation Automatic Speech Recognition

Open paper AskPDF arXiv

Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

Abstract

Explore similar work

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents

Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models