cs.CLNov 3, 2026

Do Multi-Turn Conversational Capabilities Carry Over from Text to Speech? A Parallel Multilingual Benchmark

Authors: Jiaming Zhou, Pei Zhang, Shiwan Zhao, Zhijun Wang, Baosong Yang, Yong Qin

Organizations: College of Computer Science, Nankai University · Alibaba Token Hub, Alibaba Group

Abstract

Successful multi-turn voice interaction requires models to retain information from earlier turns, follow persistent instructions, and track revisions to conversational state. These abilities are usually evaluated from transcripts, leaving open whether successes demonstrated from text carry over when the same turns are spoken, and whether this transfer is consistent across languages. We introduce Omnilingua-AudioMC, a strictly parallel multilingual extension of Audio MultiChallenge, to study this question. The benchmark contains 354 conversations and 1,404 rubric items across ten languages, with content, turn structure, cross-turn dependencies, and evaluation criteria aligned across transcript and speech conditions. Across six speech-language models, the shift from reference transcripts to direct speech input is strongly model- and language-dependent: models with similar transcript performance can diverge substantially under speech, and their relative strengths across languages can change. Paired rubric-level analysis shows that aggregate transcript-speech gaps can obscure which conversational requirements are preserved, because losses of transcript-demonstrated successes can be offset by gains on other items. Replacing preceding user turns, the current turn, or both with model-generated transcripts localizes recoverable degradation: for the two models with the largest gaps, history substitution yields a larger macro-level improvement than current-turn substitution in all ten languages, while the strongest direct-speech model gains little from either. These results show that transcript performance is an unreliable proxy for multilingual spoken conversational ability and identify the representation of preceding spoken context as an important source of recoverable degradation. Our code is publicly available at https://github.com/QwenLM/Omnilingua-Bench/tree/main/Omnilingua-AudioMC.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

    Mar 17, 2026Victoria Graf, Valentina Pyatkin, Nouha Dziri +2Multi-Turn Dialogue EvaluationMulti-Turn Dialogue Management

  2. Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges

    Aug 18, 2026Syeda Faiza Ahmed Sara, Zien Sheikh Ali, Hunzalah Hassan Bhatti +2Conversational AgentsMultimodal Agents

  3. When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

    Aug 27, 2026Yen-Ju Lu, Yuzhe Wang, Yaohan Guan +8Audio-Language Model EvaluationAudio-Language Models