Full-duplex voice agents must handle pauses, take turns, backchannel, and respond to user interruptions in real time. Full-Duplex-Bench evaluates these behaviors, but its English-only corpus and reliance on word-timestamped ASR and an English-prompted LLM judge make it difficult to extend to Indian languages. We introduce IndicFDB, which extends it to ten languages spoken in India with 12,350 samples, nearly 17 times as many as the original. We address three challenges: finding conversational events in multilingual speech, evaluating their timing without reliable word-level alignment, and judging responses across languages. We mine pause handling, turn taking, and backchanneling samples from roughly 50,000 hours of channel-separated conversations using voice activity detection (VAD), and construct human-validated synthetic user interruption samples. Language-independent VAD heuristics evaluate timing, while an open-weight transcription and translation pipeline converts responses to English for LLM ratings of relevance and quality. Across seven voice agents, commercial APIs show unexpectedly consistent behavior across languages but are either fast or robust to pauses, never both, while monolingual open full-duplex models expose further tradeoffs among backchanneling, response quality, and latency.
Figures & tables
Benchmark
Pause handling
Turn taking
Back channeling
User interruption
Total
Full-Duplex- Bench [ 11 ]
353
119
55
200
727
IndicFDB (per language)
315
235
385
300
1,235
IndicFDB (10 languages)
3,150
2,350
3,850
3,000
12,350
Table 1: Samples by conversational dynamics category.
Pause handling
Turn taking
Backchanneling
User interruption
Language
Model
S.R. ↑
S.R. ↑
Lat. ↓
S.R. ↑
JSD ↓
Freq. ↑
S.R. ↑
Lat. ↓
Rating ↑
English (Indian)
Human
100.00
100.00
0.273
100.00
0.000
0.070
–
–
–
GPT Live
64.13
93.19
0.282
42.86
0.826
0.076
99.67
0.135
4.980
Grok
99.05
92.34
2.459
98.70
0.989
0.002
100.00
1.739
4.923
Gemini 3.8
100.00
100.00
1.338
99.48
0.953
0.009
100.00
1.157
4.973
Gemini 3.1
100.00
99.15
1.747
98.96
0.981
0.003
100.00
1.485
4.997
Table 2: Per-language evaluation. S.R.: success rate (%); Lat.: latency (s); JSD: Jensen–Shannon distance; Freq.: backchannels/s; Rating: user interruption response rating (0–5). Human references have no user interruption output.
English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce τ-Multilingual, extending τ-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.
Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3
Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for Indian languages. We present the first open, reproducible full-duplex spoken dialogue system for Hindi by adapting Moshi, a state-of-the-art duplex speech architecture, using a custom Hindi tokeniser and training on 26,000 hours of real spontaneous conversations collected from 14,695 speakers with separate speaker channels, enabling direct learning of turn-taking and overlap patterns from natural interactions. To support Hindi text generation, we replace the original English tokeniser and reinitialise text-vocabulary-dependent parameters while retaining the pre-trained audio components. We propose a two-stage training recipe -- large-scale pre-training followed by fine-tuning on 1,000 hours of conversational data. Evaluation through the prompted dialogue continuation paradigm with both automatic metrics and human judgments demonstrates that the resulting model generates natural and meaningful full-duplex conversational behaviour in Hindi. This work serves as a first step toward real-time duplex spoken dialogue systems for Hindi and other Indian languages.
Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In addition, strict single reference WER penalizes natural spelling variation in Indian languages, including non standardized spellings of code-mixed English origin words. To address these limitations, we introduce Voice of India, a closed source benchmark built from unscripted telephonic conversations covering 15 major Indian languages across 139 regional clusters. The dataset contains 306230 utterances, totaling 536 hours of speech from 36691 speakers with transcripts accounting for spelling variations. We also analyze performance geographically at the district level, revealing disparities. Finally, we provide detailed analysis across factors such as audio quality, speaking rate, gender, and device type, highlighting where current ASR systems struggle and offering insights for improving real world Indic ASR systems.