Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
Figures & tables
Figure 1: SpeechGym overview. (a) A speech agent runs the same agentic loop as a text agent: it talks to a user, calls tools against a live database, and is scored on the final state. The one thing that changes is the channel carrying the conversation — which is why holding tasks, tools and the success check fixed makes modality a controlled variable. (b) SpeechGym closes that loop locally. The user simulator, the tools and the reward all live inside the environment, so speech rollouts can be scored and turned into a policy update rather than merely measured. No external ASR or TTS and no proprietary API sits in the loop, which is what makes gradients — and therefore training — possible at all.
Figure 2: The four components, as we instantiate them. The same omni-modal model plays both sides: one frozen copy speaks the caller’s side in native audio, while the trainable copy decides at every step between emitting a structured tool call and speaking back. The executor runs the tools against the database, and the reward is the benchmark’s own task-completion check, computed automatically. Only the user–agent channel is audio: keeping the tool interface textual separates perceptual error — mishearing a slot value — from behavioural error — choosing the wrong tool or acting without authorisation.
Failure type
Speech
Text
Ratio
Interpretation
Mis-hearing (slot value)
32%
2%
16 ×
hears the wrong name / ID / digit
Tool error rate
42%
26%
1.6 ×
mis-heard arguments → downstream errors
Dead loop
29%
18%
1.6 ×
retries the same failing call, no recovery
Table 1: Failure modes amplified by speech. Fraction of SpeechGym rollouts exhibiting each failure under the two channels; same tasks, same tools, same reward. Every mode is amplified by speech — mis-hearing is near-absent in text, and the downstream tool errors and dead loops are markedly rarer — which is why a speech-native environment is needed to expose them, and to supply the learning signal that fixes them.
Process (ours)
Outcome-only
Groups carrying gradient ( σi>0 )
99.6%
16%
Groups skipped ( σi=0 )
0.4%
84%
Table 2: The process reward keeps groups informative. At the base success rates of audio agentic tasks almost every group is all-zero under outcome-only reward; per-turn shaping keeps nearly all of them informative. Statistics are over the training rollouts of the two reward configurations on the same task suite.
Figure 3: Cross-pipeline transfer to τ -Voice [ 24 ] , scored by τ -Voice’s own database-and-communication check. The SpeechGym-trained agent is run inside τ -Voice’s independently implemented harness — its own cascaded user simulator, acoustics and scoring code — with no further tuning . Overall pass@1 more than doubles, from 24% to 53% , with gains in all three domains — Airline 24% to 62% , Retail 45% to 73% , Telecom 4% to 24% — and the largest relative gain in Telecom ( 6× ). Both bars in a group come from the same τ -Voice pipeline — same tasks, same user simulator, same acoustics, same grader — at a single attempt; only the agent’s weights differ. 95% confidence intervals do not overlap in any domain.
Figure 4: τ -Voice standing. Our trained open-weights 30B model against commercial voice agents on the same benchmark. Training moves the same model from last place to second, ahead of the cascaded baseline, both GPT-Realtime versions and Gemini-Live, with only Grok-Voice ranking higher. Scores for the other systems are as reported on the benchmark by their providers; we did not train or re-run them.
Behaviour (% of tasks)
Base
After GRPO
Δ
Wrong writes (over-action), lower better
23%
10%
−13
Dead-loops when stuck, lower better
14%
5%
−9
Recovers after a mis-hearing, higher better
42%
62%
+20
Table 3: Behavioural change on τ -Voice , computed from the raw trajectories of both models on the same task set. Every behaviour we tracked moves in the intended direction: wrong writes, i.e. over-action, drop by more than half, dead-loops fall by roughly two thirds, and the agent recovers from a mis-hearing far more often.
Metric
Base
After GRPO
Δ
pass@1
24%
53%
more than doubles
Average agent turns
26
24
−8%
Average tokens per task
51,195
48,398
−5%
Turns on the tasks it fixes
23.4
17.5
−25%
Table 4: Higher success at lower cost ( τ -Voice). Success more than doubles while turns and tokens both fall, which rules out the two standard ways an RL agent can inflate a success metric: taking more turns until something works, or stalling. For reference the cascaded baseline needs 31.4 turns to reach 31% pass@1.
Voice agents increasingly require reliable tool use from speech, whereas prominent tool-calling benchmarks remain text-based. We study whether verified text benchmarks can be converted into controlled audio-based tool calling evaluations without re-annotating the tool schema and gold labels. Our dataset-agnostic framework uses text-to-speech, speaker variation, and environmental noise to create paired text-audio instances while preserving the original dataset annotations. Based on extensive evaluation of 7 omni-modal models on audio-converted versions of Confetti and When2Call, our framework demonstrates that the performance is strongly model- and task-dependent: Gemini-3.1-Flash-Live obtains the highest Confetti score (70.4), whereas GPT-Realtime-1.5 performs best on When2Call (71.9). On Confetti, the text-to-voice gap ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5. A targeted analysis of failure cases demonstrates that degradations most often reflect misunderstandings of argument values in the speech. Considering real-world deployment scenarios, we further report text-only results, an ambiguity-based reformulation stress test, and a reference-free LLM-as-judge protocol validated against human preferences. Notably, we find that open-source Qwen3 judges with at least 8B parameters exceed 80% agreement with proprietary judges, supporting privacy-preserving evaluation. Overall, our framework provides a verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora.
Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents' ability to handle diverse task settings and adaptive tool interactions.
Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China
Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.
Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1
Department of Electronic Engineering, Tsinghua University · University of Cambridge