Latency, Speech Recognition and Turn-Taking in Voice Agents for Spanish Phone Negotiation: An Externally Measured Comparison of Pipecat, LiveKit and GPT-Live
Authors: Jovan Popoca Moran, Eduardo de Avila-Armenta, Raúl Alejandro Aquino-Ibarra, Jorge Octavio Nicolás-Díaz, Umberto Leon-Dominguez
Organizations: AI Lab, Freight Technologies Inc., Monterrey, N.L., Mexico · Tecnológico de Monterrey, School of Engineering and Sciences, Monterrey, N.L., Mexico · School of Psychology, University of Monterrey, Monterrey, N.L., Mexico
A voice agent that negotiates by phone must answer quickly, understand spoken amounts and cope with a caller who talks over it. We compared three architectures for such an agent, negotiating freight rates in Spanish: two cascades (speech recognition, language model and speech synthesis) orchestrated by Pipecat and by LiveKit Agents, and GPT-Live, an OpenAI speech model that listens and speaks without separate stages. A program played the carrier with the same 47 recordings of one speaker for all agents, without the telephone network, in 77 calls per architecture over five negotiation scenarios, and measured latency on the listener's side with a validated instrument. Median latency to the content response was 2,616.0 ms (Pipecat), 3,090.0 ms (LiveKit) and 2,444.8 ms (GPT-Live); the confidence interval of GPT-Live's 171.2 ms advantage over Pipecat includes zero. According to its own timestamps, GPT-Live began to answer 800 ms after the end of the line, close to the 0.8 s published by OpenAI; most of the remaining wait, about 1,367 ms and nearly constant, is the audio's round trip through OpenAI's service. A tool call added 0.9-1.1 s in every architecture, more than any difference between them, and the frameworks' self-reported latency fell short of the listener's wait by 9% (Pipecat) and 22% (LiveKit). Word error rates were low (6.03, 4.93 and 1.47%), but the error on amounts was 13.01% and 12.61% in the cascades, against 2.86% in GPT-Live. With default settings, Pipecat and LiveKit stopped at almost every backchannel and often took it as the carrier's answer: an "ok" spoken over the agent ended in a registered agreement in 3 of 15 Pipecat calls, 2 of 15 LiveKit calls and no GPT-Live call (descriptive counts, not tested). Latency should therefore be measured on the listener's side, and a cascade needs safeguards for spoken amounts and backchannels.
Figures & tables
Pipecat
LiveKit Agents
GPT-Live
Framework
Pipecat 1.6.0
LiveKit Agents 1.6.8
None: an in-house bridge to the OpenAI API
Conversational model
gpt-4.1-mini
gpt-4.1-mini
gpt-live-1
Tool calls
gpt-4.1-mini , from the framework
gpt-4.1-mini , from the framework
GPT-Live hands each one to gpt-4.1-mini and can keep talking while it waits
Speech recognition
Deepgram nova-3 , multi mode
Deepgram nova-3 , multi mode
Inside the model; a transcript arrives separately
Speech synthesis
ElevenLabs eleven_flash_v2_5
ElevenLabs eleven_flash_v2_5 , same voice
Inside the model, its default voice
Voice activity detector
Silero, 0.2 s minimum silence
Silero, 0.55 s minimum silence
Internal, not configurable
Table 1: What changes between the three architectures. Everything else was the same (Section 2.2 )
Figure 1: How a scenario is played. A scenario is the carrier’s side of the call: a script of eight lines, played by the harness one per turn. The agent does not know which scenario it is in. After each response the harness reads what the agent said and, in the turns with branches, picks the next recording with fixed rules, the same for the three architectures.
#
Scenario
What it demands from the voice technology
1
Cooperative
Baseline: a clean conversation, without pressure and without amounts; it ends in agreement.
2
Moderate haggler
Several consecutive turns that call proponer_precio ; amounts requested again and again, and a self-correction (“Diecisiete… no, dieciocho”, “Seventeen… no, eighteen”).
3
Aggressive, high anchor
Amounts above the maximum (28,000, 25,000), direct pressure on the ceiling, curt tone; it ends without agreement.
4
Non-monetary objection
Responses that depend on understanding the objection rather than the price: waiting time, delivery date; no amounts.
5
Recognition stress test
Long amounts in words, a self-correction of an amount (18,000 to 18,500), place names and trade terms ( tara , tare; carta porte , waybill; casetas , toll booths).
Table 2: The five scenarios, the same in the three architectures
Figure 2: Experimental setup. The harness replaces the carrier and is the same for the three architectures; only the architecture it connects to changes. Double arrows are audio in both directions.
Parameter
Value
Audio sending
20-ms frames ( μ -law, 8 kHz), each sent at a fixed instant computed from the start of the call, so that delays do not accumulate.
End of the carrier’s speech
Instant at which the first frame of the line was sent, plus the end-of-speech mark of that audio.
Agent speech
A 20-ms frame counts as speech if its level (RMS) reaches −44.3 dBFS; speech frames separated by less than 0.3 s form a run.
Content response
The first run that starts after the end of the carrier’s audio and lasts at least 0.6 s. If it lasts less than 2.0 s, it is also discarded when its text is a backchannel (three words or fewer from a fixed list: “ajá”, “mm”, “sí”, “ok”, among others), and latency is measured to the next run that qualifies.
Next line
When the agent has answered and has been silent for 2.0 s; if it does not answer, after 25 s at most.
Table 3: Harness parameters, the same in the three architectures
Pipecat local WebSocket
LiveKit cloud server
GPT-Live bridge
Condition
Measured
Error
Measured
Error
Measured
Error
d=300 ms, −20 dBFS
38/40
− 46.0
39/40
320.1
40/40
− 40.9
d=1,000 ms, −20 dBFS
40/40
− 42.3
39/40
240.9
40/40
− 42.9
d=2,500 ms, −20 dBFS
40/40
− 48.0
38/40
229.4
40/40
− 46.4
d=1,000 ms, −6 dBFS
40/40
− 10.1
39/40
268.9
40/40
− 2.8
d=1,000 ms, −35 dBFS
40/40
− 8.5
39/40
259.9
40/40
− 11.9
Table 4: Harness validation against a simulated agent with known delay d . Error: latency reported by the harness minus the actual one (ms, median of each condition). Measured: turns with a latency, out of those run
If the 95 % CI of the difference…
Verdict
includes zero and lies entirely within ±200 ms
equivalent in practice
includes zero and extends beyond ±200 ms on some side
not distinguishable from zero
excludes zero and lies entirely within ±200 ms
different from zero, below 200 ms
excludes zero and lies entirely outside ±200 ms
different from zero, above 200 ms
excludes zero and crosses ±200 ms
different from zero; may or may not exceed 200 ms
Table 5: How the 95 % CI of a difference is read against the 200 ms margin
Pipecat
LiveKit
GPT-Live
Calls / measured responses
77 / 558
77 / 536
77 / 563
To the content response
Observed, p50
2,616.0 [2,582.0, 2,663.0]
3,090.0 [3,038.0, 3,200.0]
2,444.8 [2,303.1, 2,608.2]
Observed, p95
4,554.4
4,630.2
4,813.8
Inbound transport, p50
185.8
423.1
n/a 1 1 footnotemark: 1
Outbound transport, p50
2.6
213.8
71.0
Table 6: Response latency (ms). Under each median, in brackets, its 95 % CI
Figure 3: Distribution of the latency to the content response. Boxes span the 25th to the 75th percentile, the line is the median and the whiskers reach the last value within 1.5 times the box height; the values beyond the whiskers are drawn as points, and no response was left out. On the right, Pipecat and LiveKit without transport; GPT-Live has no such figure.
Comparison
Difference
95 % CI
Verdict
Pipecat − LiveKit (paired, 77 pairs)
All responses
− 474.0
[ − 583.0, − 404.5]
different from zero, above 200 ms
Without transport
− 25.2
[ − 106.6, 43.1]
equivalent in practice
Pipecat − GPT-Live (unpaired)
All responses
171.2
[ − 3.2, 322.5]
not distinguishable from zero
Without outbound transport 1 1 footnotemark: 1
239.5
[74.1, 396.9]
different from zero; may or may not exceed 200 ms
Table 7: Differences in median latency (ms), first architecture minus second. A negative value means that the first architecture answers sooner
Figure 4: Where the time of a response goes: median of each stage, in ms. In the cascades, the stages come from the internal marks of Pipecat and LiveKit (end of turn: from the end of speech to the decision that the turn has ended; recognition: to the final transcript; language model: to the first chunk of the response, without a tool; synthesis: to the first chunk of audio), and transport and the cost of a tool call come from the harness. GPT-Live does not separate those stages: its two rows are what it took according to its own timestamps and what those timestamps do not include (Section 3.3 ). The bars cannot be added up, because some stages run at the same time (the recognizer transcribes while the detector decides whether the carrier has finished) and each was measured separately.
All
Without a tool
With a tool
Responses
560
376
184
1. Until the service returns the text of the last word
1,082 [990, 1,178]
1,085
1,078
2. According to GPT-Live’s timestamps
800 [400, 1,600]
600
1,400
3. Until the answer reaches the computer, beyond 1 and 2
297 [233, 344]
291
309
4. From the computer to the ear
171 [115, 209]
169
185
Measured wait
2,448 [1,866, 3,234]
2,147
3,068
Table 8: The four segments of the wait for a GPT-Live response. Median [25th percentile, 75th percentile], in ms
Figure 5: The segments of a typical response of each architecture, without a tool and with each segment close to its median, on the same time scale. In colour, what each architecture’s own figure covers: in Pipecat and LiveKit, the latency that the framework reports (end of turn, language model and synthesis); in GPT-Live, in green, the segment measured by its timestamps, which resembles the figure published by OpenAI. In grey, what those figures leave out. In the cascades, the thin bar above the end of turn is speech recognition (STT): it starts at the same end of speech and runs while the detector decides whether the turn has ended, so it does not lengthen the response. In LiveKit the language model starts before the end of the turn is confirmed (preemptive generation, on by default) and synthesis waits for the confirmation, so its bar shows only the part of the language model that comes after it; in the median, 189 ms of the language model ran during the end-of-turn wait.
WER
Character error rate
Error on amounts
Pipecat
6.03 [5.42, 6.63]
3.99 [3.55, 4.42]
13.01 [8.26, 17.69]
LiveKit
4.93 [4.49, 5.36]
3.07 [2.74, 3.40]
12.61 [7.55, 17.54]
GPT-Live
1.47 [1.09, 1.89]
0.66 [0.43, 0.91]
2.86 [0.00, 6.63]
Tests on the error of each call
Calls per architecture
77
77
45 1 1 footnotemark: 1
Kruskal–Wallis, p ( εR2 )
< 0.001 (0.46)
< 0.001 (0.51)
0.007 (0.07)
Table 9: Recognition errors by architecture (%), computed per whole call, and the tests between architectures. Under each error, in brackets, its 95 % CI
Responses
Transcript with “quinientos”
The response shows the amount
Understood 18,500
Understood 18,000
Pipecat
13
2
8
1
7
LiveKit
13
4
4
0
4
GPT-Live
13
13
7
7
0
Table 10: Which amount the agent understood when the carrier said “dieciocho mil… no, dieciocho mil quinientos” (scenario 5, turn 4)
Pipecat
LiveKit
GPT-Live
Calls
15
15
15
A. Stopping behaviour
Interruption (expected: stop and answer)
Stopped
26/27
22/22
25/26
Time to stop, median (ms)
344
878
1,242
interquartile range
234–448
856–903
391–1,387
Table 11: Interruption scenario: how often each agent stopped, how long it took, what it said when it resumed and which agreements it registered. Fractions are events in which the agent stopped out of the events played while it was speaking
Figure 6: The Pipecat call in which an “ok” spoken over the agent closed the deal (lines translated from Spanish). Bars show each side’s speech, coloured as in the legend. The recognizer returned “Ándele, una buena ruta y se la hago rápida. Diecisiete, no.” and dropped the “dieciocho mil” (eighteen thousand). The agent stopped 0.8 s after the “Ok.” began, registered the agreement 2.3 s later, before asking for confirmation, and hung up as the next line started.
English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce τ-Multilingual, extending τ-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.
Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3
We introduce Full-Duplex-Bench-v3 (FDB-v3), a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use. Unlike prior work, our dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains. We evaluate six model configurations -- GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper→GPT-4o→TTS) -- across accuracy, latency, and turn-taking dimensions. GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5%); Gemini Live 3.1 achieves the fastest latency (4.25s) but the lowest turn-take rate (78.0%); and the Cascaded baseline, despite a perfect turn-take rate, incurs the highest latency (10.12s). Across all systems, self-correction handling and multi-step reasoning under hard scenarios remain the most consistent failure modes.
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.