Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
Figures & tables
Figure 1: SpeechGym overview. (a) A speech agent runs the same agentic loop as a text agent: it talks to a user, calls tools against a live database, and is scored on the final state. The one thing that changes is the channel carrying the conversation — which is why holding tasks, tools and the success check fixed makes modality a controlled variable. (b) SpeechGym closes that loop locally. The user simulator, the tools and the reward all live inside the environment, so speech rollouts can be scored and turned into a policy update rather than merely measured. No external ASR or TTS and no proprietary API sits in the loop, which is what makes gradients — and therefore training — possible at all.
Figure 2: The four components, as we instantiate them. The same omni-modal model plays both sides: one frozen copy speaks the caller’s side in native audio, while the trainable copy decides at every step between emitting a structured tool call and speaking back. The executor runs the tools against the database, and the reward is the benchmark’s own task-completion check, computed automatically. Only the user–agent channel is audio: keeping the tool interface textual separates perceptual error — mishearing a slot value — from behavioural error — choosing the wrong tool or acting without authorisation.
Failure type
Speech
Text
Ratio
Interpretation
Mis-hearing (slot value)
32%
2%
16 ×
hears the wrong name / ID / digit
Tool error rate
42%
26%
1.6 ×
mis-heard arguments → downstream errors
Dead loop
29%
18%
1.6 ×
retries the same failing call, no recovery
Table 1: Failure modes amplified by speech. Fraction of SpeechGym rollouts exhibiting each failure under the two channels; same tasks, same tools, same reward. Every mode is amplified by speech — mis-hearing is near-absent in text, and the downstream tool errors and dead loops are markedly rarer — which is why a speech-native environment is needed to expose them, and to supply the learning signal that fixes them.
Process (ours)
Outcome-only
Groups carrying gradient ( σi>0 )
99.6%
16%
Groups skipped ( σi=0 )
0.4%
84%
Table 2: The process reward keeps groups informative. At the base success rates of audio agentic tasks almost every group is all-zero under outcome-only reward; per-turn shaping keeps nearly all of them informative. Statistics are over the training rollouts of the two reward configurations on the same task suite.
Figure 3: Cross-pipeline transfer to τ -Voice [ 24 ] , scored by τ -Voice’s own database-and-communication check. The SpeechGym-trained agent is run inside τ -Voice’s independently implemented harness — its own cascaded user simulator, acoustics and scoring code — with no further tuning . Overall pass@1 more than doubles, from 24% to 53% , with gains in all three domains — Airline 24% to 62% , Retail 45% to 73% , Telecom 4% to 24% — and the largest relative gain in Telecom ( 6× ). Both bars in a group come from the same τ -Voice pipeline — same tasks, same user simulator, same acoustics, same grader — at a single attempt; only the agent’s weights differ. 95% confidence intervals do not overlap in any domain.
Figure 4: τ -Voice standing. Our trained open-weights 30B model against commercial voice agents on the same benchmark. Training moves the same model from last place to second, ahead of the cascaded baseline, both GPT-Realtime versions and Gemini-Live, with only Grok-Voice ranking higher. Scores for the other systems are as reported on the benchmark by their providers; we did not train or re-run them.
Behaviour (% of tasks)
Base
After GRPO
Δ
Wrong writes (over-action), lower better
23%
10%
−13
Dead-loops when stuck, lower better
14%
5%
−9
Recovers after a mis-hearing, higher better
42%
62%
+20
Table 3: Behavioural change on τ -Voice , computed from the raw trajectories of both models on the same task set. Every behaviour we tracked moves in the intended direction: wrong writes, i.e. over-action, drop by more than half, dead-loops fall by roughly two thirds, and the agent recovers from a mis-hearing far more often.
Metric
Base
After GRPO
Δ
pass@1
24%
53%
more than doubles
Average agent turns
26
24
−8%
Average tokens per task
51,195
48,398
−5%
Turns on the tasks it fixes
23.4
17.5
−25%
Table 4: Higher success at lower cost ( τ -Voice). Success more than doubles while turns and tokens both fall, which rules out the two standard ways an RL agent can inflate a success metric: taking more turns until something works, or stalling. For reference the cascaded baseline needs 31.4 turns to reach 31% pass@1.