Do Multi-Turn Conversational Capabilities Carry Over from Text to Speech? A Parallel Multilingual Benchmark
Organizations: College of Computer Science, Nankai University · Alibaba Token Hub, Alibaba Group
Abstract
Successful multi-turn voice interaction requires models to retain information from earlier turns, follow persistent instructions, and track revisions to conversational state. These abilities are usually evaluated from transcripts, leaving open whether successes demonstrated from text carry over when the same turns are spoken, and whether this transfer is consistent across languages. We introduce Omnilingua-AudioMC, a strictly parallel multilingual extension of Audio MultiChallenge, to study this question. The benchmark contains 354 conversations and 1,404 rubric items across ten languages, with content, turn structure, cross-turn dependencies, and evaluation criteria aligned across transcript and speech conditions. Across six speech-language models, the shift from reference transcripts to direct speech input is strongly model- and language-dependent: models with similar transcript performance can diverge substantially under speech, and their relative strengths across languages can change. Paired rubric-level analysis shows that aggregate transcript-speech gaps can obscure which conversational requirements are preserved, because losses of transcript-demonstrated successes can be offset by gains on other items. Replacing preceding user turns, the current turn, or both with model-generated transcripts localizes recoverable degradation: for the two models with the largest gaps, history substitution yields a larger macro-level improvement than current-turn substitution in all ten languages, while the strongest direct-speech model gains little from either. These results show that transcript performance is an unreliable proxy for multilingual spoken conversational ability and identify the representation of preceding spoken context as an important source of recoverable degradation. Our code is publicly available at https://github.com/QwenLM/Omnilingua-Bench/tree/main/Omnilingua-AudioMC.
Figures & tables
| Dataset and conditions | |
|---|---|
| Conversations | 354 |
| User turns | 1,827 |
| Rubric items | 1,404 |
| Reference Transcript / speech conditions | 10 / 11 |
| Conversations by capability | |
| Inference Memory (IM) | 82 |
| Model | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI | Avg. | Std. | Range |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | 61.6 | 60.7 | 60.5 | 64.1 | 59.6 | 61.0 | 61.6 | 60.7 | 54.0 | 58.2 | 60.2 | 2.5 | 10.1 |
| Gemini 3.1 Pro | 60.2 | 50.8 | 53.1 | 57.6 | 54.8 | 56.5 | 53.7 | 55.4 | 50.3 | 51.4 | 54.4 | 3.0 | 9.9 |
| Gemini 3.8 Flash | 52.0 | 51.7 | 48.6 | 51.4 | 48.9 | 53.1 | 50.6 | 51.4 | 48.0 | 44.4 | 50.0 | 2.4 | 8.7 |
| Doubao Seed Lite | 47.7 | 46.3 | 42.1 | 40.4 | 41.2 | 45.2 | 41.5 | 44.6 | 38.7 | 35.9 | 42.4 | 3.4 | 11.8 |
| Qwen3.5 Omni | 42.9 | 39.8 | 38.7 | 38.4 | 37.3 | 38.4 | 37.6 | 38.4 | 36.2 | 33.6 | 38.1 | 2.3 | 9.3 |
| Muse-Spark 1.2 | 39.8 | 28.7 | 29.4 | 34.2 | 32.5 | 37.9 | 33.7 | 32.6 | 27.1 | 28.3 | 32.4 | 4.0 | 12.7 |
| Direct Audio Reference Transcript by language | Macro summary | ||||||||||||
| Model | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI | Ref. | Aud. | |
| Qwen3.8 Omni | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | 60.9 | 60.2 | ( ) |
| Gemini 3.1 Pro | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | 63.0 | 54.4 | ( ) |
| Gemini 3.8 Flash | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | 57.0 | 50.0 | ( ) |
| Doubao Seed Lite | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | 48.4 | 42.4 | ( ) |
| Qwen3.5 Omni | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | 40.1 | 38.1 | ( ) |
| Model | IM | IR | SC | VE |
|---|---|---|---|---|
| Qwen3.8 Omni | 88.4 | 92.9 | 92.9 | 95.3 |
| Gemini 3.1 Pro | 79.8 | 82.7 | 92.6 | 94.5 |
| Gemini 3.8 Flash | 80.9 | 86.6 | 89.7 | 92.9 |
| Doubao Seed Lite | 79.4 | 84.3 | 87.1 | 91.9 |
| Qwen3.5 Omni | 81.9 | 85.3 | 91.4 | 92.1 |
| Muse-Spark 1.2 | 61.3 | 56.9 | 84.6 | 68.0 |
| Diagnostic baseline | intervention | ||||
|---|---|---|---|---|---|
| Model | Reference Transcript | Direct Audio | History Transcript | Current Transcript | Full Transcript |
| Qwen3.8 Omni | 60.9 | 60.2 | 60.1 ( ) | 60.1 ( ) | 58.8 ( ) |
| Gemini 3.1 Pro | 63.0 | 54.4 | 60.1 ( ) | 57.1 ( ) | 60.8 ( ) |
| Gemini 3.8 Flash | 57.0 | 50.0 | 53.2 ( ) | 50.8 ( ) | 54.4 ( ) |
| Doubao Seed Lite | 48.4 | 42.4 | 45.1 ( ) | 43.4 ( ) | 46.3 ( ) |
| Qwen3.5 Omni | 40.1 | 38.1 | 39.8 ( ) | 38.4 ( ) | 38.1 ( ) |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| (a) Inference Memory (IM) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
| Qwen3.8 Omni | 72.5/58.5 | 74.5 / 61.0 | 75.2 / 61.0 | 79.1 / 67.1 | 73.9 / 61.0 | 73.9 / 61.0 | 73.2 / 62.2 | 73.2 / 61.0 | 69.3 / 53.7 | 66.0 / 51.2 |
| Gemini 3.1 Pro | 75.2 / 59.8 | 60.8/43.9 | 66.0/48.8 | 64.7/51.2 | 68.0/50.0 | 69.3/52.4 | 64.7/48.8 | 66.0/51.2 | 68.0/ 53.7 | 60.1/43.9 |
| Gemini 3.8 Flash | 72.5/ 59.8 | 64.1/47.6 | 64.1/47.6 | 62.7/43.9 | 63.4/46.3 | 68.6/56.1 | 66.0/50.0 | 64.7/51.2 | 65.4/46.3 | 57.5/39.0 |
| Doubao Seed Lite | 65.4/47.6 | 64.7/48.8 | 58.2/43.9 | 60.1/42.7 | 60.8/42.7 | 70.6/52.4 | 70.6/50.0 | 62.7/46.3 | 52.3/37.8 | 58.2/40.2 |
| Qwen3.5 Omni | 55.6/34.1 | 47.1/32.9 | 60.1/42.7 | 56.2/41.5 | 51.6/34.1 | 58.2/40.2 | 56.2/34.1 | 58.8/42.7 | 55.6/31.7 | 47.7/28.0 |
| Model | Input | Avg. | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | Reference Transcript | 60.9 | 61.6 | 62.4 | 58.9 | 66.7 | 59.6 | 63.0 | 64.4 | 60.7 | 50.8 | 61.3 |
| Direct Audio | 60.2 | 61.6 | 60.7 | 60.5 | 64.1 | 59.6 | 61.0 | 61.6 | 60.7 | 54.0 | 58.2 | |
| History Transcript | 60.1 | 60.2 | 61.8 | 62.0 | 60.5 | 59.6 | 63.3 | 61.9 | 61.6 | 54.0 | 55.6 | |
| Current Transcript | 60.1 | 62.7 | 59.0 | 61.0 | 63.0 | 62.1 | 62.1 | 63.0 | 62.4 | 51.1 | 54.8 | |
| Full Transcript | 58.8 | 62.4 | 60.7 | 59.4 | 61.9 | 62.4 | 61.9 | 59.6 | 57.6 | 48.6 | 53.4 | |
| Gemini 3.1 Pro | Reference Transcript | 63.0 | 63.0 | 65.5 | 65.0 | 64.1 | 63.3 | 60.5 | 63.3 | 63.3 | 61.9 | 59.6 |
| Model | Input | Avg. | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | Reference Transcript | 85.7 | 85.6 | 86.2 | 84.6 | 88.0 | 85.4 | 86.5 | 87.3 | 84.6 | 82.1 | 86.8 |
| Direct Audio | 85.1 | 86.8 | 85.5 | 84.6 | 86.7 | 85.2 | 85.8 | 85.4 | 84.5 | 82.8 | 83.6 | |
| History Transcript | 84.7 | 85.2 | 85.8 | 84.2 | 85.8 | 84.3 | 85.6 | 85.2 | 85.1 | 82.1 | 83.3 | |
| Current Transcript | 84.9 | 86.9 | 84.3 | 84.5 | 86.4 | 86.0 | 85.8 | 86.1 | 85.4 | 80.5 | 83.2 | |
| Full Transcript | 84.5 | 85.9 | 85.2 | 85.1 | 86.4 | 86.5 | 85.6 | 85.2 | 83.0 | 78.6 | 83.6 | |
| Gemini 3.1 Pro | Reference Transcript | 86.1 | 87.1 | 87.1 | 87.0 | 87.1 | 86.1 | 85.1 | 86.7 | 86.2 | 83.6 | 85.0 |
| Model | History | Current | Full | Interaction |
|---|---|---|---|---|
| Qwen3.8 Omni | -0.2 | -0.1 | -1.4 | -1.2 |
| Gemini 3.1 Pro | +5.7 | +2.8 | +6.4 | -2.1 |
| Gemini 3.8 Flash | +3.2 | +0.7 | +4.4 | +0.4 |
| Doubao Seed Lite | +2.7 | +1.1 | +3.9 | +0.1 |
| Qwen3.5 Omni | +1.6 | +0.2 | 0.0 | -1.9 |
| Muse-Spark 1.2 | +20.1 | +6.2 | +26.2 | -0.2 |
| Model | Avg. | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | -0.2/-0.1 | -1.4/+1.1 | +1.1/-1.7 | +1.5/+0.5 | -3.6/-1.1 | 0.0/+2.5 | +2.3/+1.1 | +0.3/+1.4 | +0.9/+1.7 | 0.0/-2.9 | -2.6/-3.4 |
| Gemini 3.1 Pro | +5.7/+2.8 | +3.9/+0.6 | +7.7/+6.7 | +8.8/+2.7 | +4.3/+0.6 | +4.2/+3.4 | +5.6/+1.7 | +6.5/+4.2 | +3.4/+1.7 | +7.0/+2.0 | +5.9/+4.0 |
| Gemini 3.8 Flash | +3.2/+0.7 | +6.8/+3.1 | +3.4/+2.3 | -0.3/+1.0 | +3.4/-1.8 | +1.7/+2.2 | +3.7/-0.6 | +3.4/0.0 | +0.9/+0.7 | +1.4/-1.8 | +7.6/+2.3 |
| Doubao Seed Lite | +2.7/+1.1 | +3.4/+0.3 | +3.7/-1.1 | +5.6/+1.7 | +3.7/+2.0 | +2.3/+3.7 | +1.1/-1.1 | +2.0/+2.9 | 0.0/-0.2 | +2.0/0.0 | +3.6/+2.5 |
| Qwen3.5 Omni | +1.6/+0.2 | -0.2/-0.2 | +2.1/+2.0 | -0.8/-1.4 | +2.0/+0.9 | +2.2/+1.7 | +3.7/+1.1 | +2.2/-1.2 | +3.8/-2.7 | +1.4/-0.9 | 0.0/+3.1 |
| Muse-Spark 1.2 | +20.1/+6.2 | +16.1/+5.4 | +17.9/+4.1 | +21.0/+8.7 | +21.3/+7.9 | +21.7/+7.9 | +17.7/+1.9 | +21.9/+8.4 | +21.5/+9.6 | +21.2/+4.1 | +20.9/+4.2 |
| Per language (95% CI) | Macro | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Contrast | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI | 10-lang. |
| Qwen3.8 Omni | Ref. Direct | – | – | – | – | – | – | – | – | – | – | [ , ] |
| History | – | – | – | – | – | – | – | – | – | – | [ , ] | |
| Current | – | – | – | – | – | – | – | – | – | – | [ , ] | |
| Full | – | – | – | – | – | – | – | – | – | – | [ , ] | |
| H C | – | – | – | – | – | – | – | – | – | – | [ , ] | |
| Model | Eligible | H only | C only | Both | Neither | Exclusive (%) | Overlap (%) |
|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | 374 | 60 | 69 | 128 | 117 | 50.2 | 49.8 |
| Gemini 3.1 Pro | 530 | 167 | 62 | 177 | 124 | 56.4 | 43.6 |
| Gemini 3.8 Flash | 513 | 129 | 82 | 142 | 160 | 59.8 | 40.2 |
| Doubao Seed Lite | 491 | 115 | 63 | 130 | 183 | 57.8 | 42.2 |
| Qwen3.5 Omni | 347 | 65 | 62 | 95 | 125 | 57.2 | 42.8 |
| Muse-Spark 1.2 | 1270 | 523 | 151 | 285 | 311 | 70.3 | 29.7 |
| Inference Memory (IM) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
| Qwen3.8 Omni | 73.9/72.5 | 75.2/74.5 | 75.2/75.2 | 77.1/79.1 | 72.5/73.9 | 79.7/73.9 | 79.1/73.2 | 75.8/73.2 | 71.2/69.3 | 76.5/66.0 |
| Gemini 3.1 Pro | 78.4/75.2 | 81.0/60.8 | 79.1/66.0 | 76.5/64.7 | 76.5/68.0 | 73.9/69.3 | 81.0/64.7 | 76.5/66.0 | 75.8/68.0 | 72.5/60.1 |
| Gemini 3.8 Flash | 79.1/72.5 | 72.5/64.1 | 79.1/64.1 | 73.2/62.7 | 68.0/63.4 | 77.8/68.6 | 79.7/66.0 | 71.9/64.7 | 73.2/65.4 | 73.9/57.5 |
| Doubao Seed Lite | 70.6/65.4 | 70.6/64.7 | 69.3/58.2 | 70.6/60.1 | 72.5/60.8 | 71.9/70.6 | 73.9/70.6 | 69.9/62.7 | 64.7/52.3 | 66.0/58.2 |
| Qwen3.5 Omni | 58.2/55.6 | 51.6/47.1 | 54.9/60.1 | 58.8/56.2 | 55.6/51.6 | 57.5/58.2 | 56.2/56.2 | 60.1/58.8 | 47.7/55.6 | 52.3/47.7 |
| Model | Capability | Both | R only | A only | Neither | N | Surv.% |
|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | IM | 1023 | 134 | 95 | 278 | 1530 | 88.4 |
| IR | 2213 | 168 | 182 | 213 | 2776 | 92.9 | |
| SC | 2370 | 180 | 192 | 428 | 3170 | 92.9 | |
| VE | 5664 | 280 | 205 | 411 | 6560 | 95.3 | |
| Overall | 11270 | 762 | 674 | 1330 | 14036 | 93.7 | |
| Gemini 3.1 Pro | IM | 942 | 238 | 72 | 278 | 1530 | 79.8 |
| Model | Source | Avg. | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8 Omni | Shared | 84.4/59.6 | 84.5/59.6 | 83.7/57.0 | 84.5/59.9 | 86.7/63.3 | 85.9/62.7 | 85.3/64.1 | 86.5/62.7 | 84.4/59.6 | 80.6/51.1 | 82.3/55.6 |
| Full | 84.5/58.8 | 85.9/62.4 | 85.2/60.7 | 85.1/59.4 | 86.4/61.9 | 86.5/62.4 | 85.6/61.9 | 85.2/59.6 | 83.0/57.6 | 78.6/48.6 | 83.6/53.4 | |
| Ref.hist. | 85.8/61.4 | 86.3/61.6 | 85.4/62.7 | 85.6/61.3 | 87.0/62.1 | 85.6/62.4 | 86.5/63.6 | 87.0/64.4 | 86.8/64.4 | 82.4/52.8 | 84.9/58.8 | |
| Gemini 3.1 Pro | Shared | 84.7/59.4 | 85.3/61.6 | 85.5/60.5 | 83.5/57.3 | 86.1/61.9 | 84.8/60.7 | 86.0/62.4 | 84.9/57.9 | 85.0/59.6 | 82.1/57.1 | 83.4/54.8 |
| Full | 84.6/60.8 | 88.2/66.9 | 84.5/61.3 | 85.3/62.7 | 85.4/61.6 | 84.3/59.6 | 85.0/60.7 | 85.0/59.0 | 83.6/61.3 | 81.3/56.8 | 83.8/58.2 | |
| Ref.hist. | 85.1/61.1 | 86.6/62.7 | 86.0/59.9 | 85.6/63.3 | 85.6/61.6 | 84.2/59.9 | 85.2/61.9 | 84.6/61.9 | 84.5/59.6 | 83.5/60.5 | 85.3/59.3 |
| Model | Input | ZH | JA | ES | FR | RU | DE | IT | TH | VI | Pooled% |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse-Spark 1.2 | Full | 104/140 | 93/133 | 92/126 | 103/131 | 98/125 | 99/125 | 103/126 | 104/136 | 87/129 | 75.4 |
| Hist. | 77/140 | 70/131 | 86/126 | 88/131 | 77/125 | 89/124 | 88/126 | 88/136 | 79/129 | 63.5 | |
| Ref.hist. | 92/141 | 93/130 | 91/126 | 95/131 | 85/125 | 86/125 | 96/126 | 77/136 | 90/129 | 68.9 | |
| Gemini 3.1 Pro | Full | 50/71 | 47/64 | 29/49 | 36/50 | 28/45 | 38/52 | 39/51 | 44/56 | 36/55 | 70.4 |
| Hist. | 43/71 | 45/64 | 28/49 | 35/50 | 28/45 | 29/52 | 33/51 | 40/56 | 39/55 | 64.9 | |
| Ref.hist. | 49/71 | 43/64 | 32/49 | 37/50 | 29/45 | 28/52 | 36/51 | 40/56 | 37/55 | 67.1 |
| Model | Human audio | TTS audio | TTS shared | TTS self | TTS clean hist. |
|---|---|---|---|---|---|
| Qwen3.8 Omni | 86.8/61.6 | 85.5/60.2 | 84.9/60.5 | 84.9/60.2 | 86.0/60.6 |
| Gemini 3.1 Pro | 84.1/60.2 | 85.0/60.7 | 84.1/59.6 | 86.0/60.7 | 86.0/61.3 |
| Gemini 3.8 Flash | 80.3/52.0 | 79.7/50.6 | 81.4/53.4 | 83.0/55.7 | 81.4/54.2 |
| Doubao Seed Lite | 79.3/47.7 | 77.4/47.5 | 77.6/46.3 | 78.4/45.2 | 79.5/48.6 |
| Qwen3.5 Omni | 75.9/42.9 | 73.4/38.4 | 72.4/39.8 | 75.6/42.9 | 76.1/45.2 |
| Muse-Spark 1.2 | 68.7/39.8 | 73.0/39.6 | 86.0/62.7 | 84.7/61.0 | 84.6/58.9 |
| Model | Reference Transcript | Direct Audio |
|---|---|---|
| Qwen2.5-Omni | 47.5/13.8 | 44.1/12.9 |
| Voxtral-Mini | 57.6/20.1 | 56.1/18.5 |
| Qwen3-Omni | 66.5/29.0 | 62.4/24.4 |
| Voxtral-Small | 69.2/30.4 | 66.8/28.1 |
| Direct Audio | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Avg. | EN | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
| Qwen2.5-Omni | 44.1/12.9 | 51.7/14.5 | 49.9/15.0 | 40.4/12.6 | 45.8/12.9 | 45.8/13.1 | 44.2/12.6 | 46.9/14.1 | 42.6/10.3 | 34.0/11.4 | 40.0/12.5 |
| Voxtral-Mini | 56.1/18.5 | 61.0/22.4 | 54.1/16.3 | 52.6/16.2 | 59.7/22.1 | 59.2/20.4 | 57.3/18.8 | 58.6/18.7 | 57.2/16.7 | 47.4/14.4 | 54.0/19.0 |
| Qwen3-Omni | 62.4/24.4 | 65.1/28.2 | 65.2/28.0 | 60.8/23.2 | 64.4/24.4 | 63.7/26.0 | 61.5/22.0 | 65.9/30.2 | 60.0/22.9 | 57.6/18.4 | 60.0/20.6 |
| Voxtral-Small | 66.8/28.1 | 68.6/27.4 | 64.9/26.3 | 64.6/27.2 | 69.9/29.4 | 68.3/29.4 | 68.0/31.6 | 68.9/27.8 | 69.5/31.4 | 62.1/24.9 | 62.8/25.2 |
| Reference Transcript | |||||||||||
| Model | IM | IR | SC | VE |
|---|---|---|---|---|
| Qwen2.5-Omni | 31.6/30.5 | 30.4/31.3 | 51.3/47.8 | 55.4/49.2 |
| Voxtral-Mini | 34.6/32.8 | 42.6/41.6 | 63.2/61.8 | 65.5/63.7 |
| Qwen3-Omni | 43.1/39.4 | 54.7/48.8 | 67.3/64.5 | 76.6/72.0 |
| Voxtral-Small | 40.3/40.3 | 55.8/52.9 | 73.1/73.4 | 79.5/75.2 |
| Model | Input | EN | EN-TTS | ZH | JA | ES | FR | RU | DE | IT | TH | VI |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Reference Transcript | 353 | 353 | 352 | 353 | 352 | 347 | 351 | 348 | 347 | 340 | 352 |
| Direct Audio | 352 | 353 | 354 | 348 | 350 | 343 | 348 | 347 | 348 | 334 | 345 | |
| Voxtral-Mini | Reference Transcript | 354 | 354 | 353 | 352 | 354 | 352 | 354 | 352 | 353 | 346 | 352 |
| Direct Audio | 353 | 354 | 350 | 352 | 353 | 353 | 352 | 353 | 354 | 334 | 347 | |
| Qwen3-Omni | Reference Transcript | 354 | 354 | 354 | 353 | 354 | 353 | 354 | 354 | 354 | 353 | 354 |
| Direct Audio | 354 | 352 | 354 | 354 | 353 | 354 | 354 | 354 | 354 | 354 | 354 |
| Model | EN audio | EN-TTS audio | EN ref. | EN-TTS ref. |
|---|---|---|---|---|
| Qwen2.5-Omni | 51.7/14.5 | 49.3/13.9 | 52.5/16.4 | 52.0/14.7 |
| Voxtral-Mini | 61.0/22.4 | 61.0/22.9 | 62.3/25.1 | 61.8/27.7 |
| Qwen3-Omni | 65.1/28.3 | 66.6/27.8 | 66.4/28.3 | 66.8/28.3 |
| Voxtral-Small | 68.6/27.4 | 68.3/27.7 | 70.3/29.0 | 70.6/31.8 |
Audio-Language Model Evaluation
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
See all 222 papers