Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Authors: Maike Züfle, Sara Papi, Fabian Retkowski, Szymon Mazurek, Marek Kasztelnik, Alexander Waibel, Luisa Bentivogli, Jan Niehues
Organizations: Karlsruhe Institute of Technology, Germany · Fondazione Bruno Kessler, Italy · ACC Cyfronet AGH, Poland · AGH University of Krakow, Poland · Carnegie Mellon University, U.S.
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.
Figures & tables
Recording
de
en
it
cs
es
fr
hu
nl
pt
ru
sv
# Male Spks.
2
2
2
1
1
1
1
0
0
1
0
# Female Spks.
2
2
2
1
1
1
0
1
1
0
1
Spk. Avg. (min)
8
8
9
8
9
9
11
10
8
9
8
Total (min)
33
31
35
17
17
18
11
10
8
9
8
9 tasks, 11 langs; Total Audio: 3h17m
Table 1: Annotators per language for DOWIS prompts.
Monolingual Tasks
ASR
SQA
ACHAP
TTS
4.3s
5.0s
15.8s
4.4s
Monolingual (*) and Crosslingual (†) Tasks
MT †
ST †
TSUM * †
SSUM * †
S2ST †
4.1s
4.5s
4.6s
4.6s
4.8s
Table 2: Average duration per prompt across languages.
Figure 1: Performance comparison for Qwen: Text Prompt vs Speech Prompts with respect to different target languages. Positive values (purple) indicate text prompt performs better, negative values indicate speech prompts perform better.
Task
Metric
Model
Text
Speech
Speech Prompt
Prompt
Prompt
Male
Fem.
ASR
WER ↓
Phi*
16.69
332.41
402.77
271.15
Qwen
12.60
17.08
13.77
14.91
SQA
BERTS. ↑
Phi
36.49
11.16
11.37
10.96
Qwen
27.85
27.12
27.13
27.11
ACHAP
CollarF1 ↑
Phi
7.49
6.27
6.56
5.99
Table 3: The impact of speech vs. text prompts. The results are averaged over different prompt types and languages. *Phi only supports the languages ’en’, ’de’, ’fr’, ’it’, ’es’, ’pt’ for speech input, so we limit the evaluation to these for ASR. For the last two columns, only languages are considered, where DOWIS provides male and female speakers (details in Table 1 ).
Figure 2: Performance comparison for Qwen2.5-Omni: Text Prompt vs Speech Prompts with respect to different prompt types. Positive values (purple) indicate text prompt performs better, negative values (yellow) indicate speech prompts perform better.
Task
Model
Prompt Type
Task
Model
Prompt Type
Basic
Formal
Inform.
Detail.
Short
Basic
Formal
Inform.
Detail.
Short
ASR ↓
Phi*
187.49
199.13
274.56
173.05
251.39
MT ↑
Phi
60.35
55.14
59.62
62.44
56.86
Qwen
19.79
10.51
16.32
13.39
21.58
Qwen
76.55
74.75
73.98
75.56
72.08
SQA ↑
Phi
20.48
21.17
17.58
21.04
17.77
ST ↑
Phi
65.65
65.33
62.71
61.19
64.72
Qwen
28.55
27.92
27.26
26.60
26.49
Qwen
73.90
74.67
73.49
69.51
72.00
ACHAP
Phi a
7.66
8.48
6.95
8.11
2.18
TSUM ↑
Phi
40.11
44.67
43.73
45.15
39.06
Table 4: The impact of prompt typs. Results are averaged over prompt modalities and languages. Metrics are the same as in Table 3 . For tasks with multiple metrics, the metrics are: a CollarF1 ↑ , b BERTS. GC↑ ; c UTMOS ↑ , d WER ASR↓ ; e UTMOS ↑ , f COM. ASR↑ . *Phi only supports the languages ’en’, ’de’, ’fr’, ’it’, ’es’, ’pt’ for speech input, so we limit the evaluation to these for ASR.
College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore