Organizations: National Yang Ming Chiao Tung University, Taiwan · Tranxform AI · Indian Institute of Technology Hyderabad, India · Institut national des langues et civilisations orientales, France · Université Sorbonne Nouvelle, France · Cohere Labs Community · Cohere
The SLAM-ASR framework connects the features of a frozen speech encoder as input to a frozen LLM decoder to enable speech transcription. In this work we ask whether this training-on speech transcription-provides the LLM with the capability to also perform spoken language understanding (SLU) tasks, by training monolingual projectors on 11 languages. When prompted to answer a yes/no question about the input speech, the mean instruction following rate (IFR) is 49.8%. When not following the instruction, these models default back to transcription. The transcription quality-measured by the character error rate (CER)-is not sufficient to predict IFR. Remarkably, when the model is instructed to answer the topic question as a single digit option, the mean IFR rises to 81.3%. Our models score a mean 23.2% topic balanced accuracy (BAcc) against 14.3% chance. However, the same decoder scores a mean 71.5% BAcc when using reference transcripts as text input. This indicates a failure of the projector to transfer information for the SLU tasks.
Figures & tables
Topic Classification
Yes/no
Speech
Text
Language
CER%
IFR%
IFR%
BAcc%
BAcc%
English
6.65
93.5
100.0
41.4
73.0
Spanish
3.09
86.9
100.0
36.2
73.6
French
15.26
79.3
99.4
35.7
75.8
Indonesian
6.35
91.0
100.0
35.7
71.3
Table 1: Performance across languages. CER measures transcription quality. IFR is the instruction following rate. Topic Classification uses the digit template and reports balanced accuracy (BAcc) across 7 topics for Speech (audio via the projector) and Text (reference transcript replaces the audio).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Question
Prompt
Yes/no, person named
Answer with one word, yes or no: does the audio mention a person by name? {AUDIO} Answer:
Topic, open question
What is the topic of the audio? Answer with one or two words. {AUDIO} Topic:
Topic, digit template
The utterance belongs to one of the following topics. Utterance: {AUDIO} Topics: 1. Entertainment 2. Geography 3. Health 4. Politics 5. Science and Technology 6. Sports 7. Travel Please respond with only the number of the correct answer (1, 2, 3, 4, 5, 6, or 7). Answer:
Appendix
Table 2: The English prompts as the models saw them. {AUDIO} marks the position of the projected audio tokens, and the model continues after the final colon. Table 3 gives the in-language template for each language.
Table 3: The in-language digit template for each language, as the models saw it. It translates the English template's wrapper and option labels into the projector's language and keeps the digit answer space. Its English row drops “Utterance:” and shortens a label, so it differs from the digit template of Table 2 used in Table 1 .
Figure 1: Transcription CER against yes/no IFR. Each point is one projector, trained and evaluated on its own language. Marker shape denotes the language’s script, and color marks whether its IFR is above 50%.
System
Acc.%
Acc. † %
BAcc%
IFR%
Reference points
Chance (7 classes)
14.3
–
14.3
–
Majority-class prediction
30.6
–
14.3
–
Fleurs-SLU ( Schmidt et al., 2025 ) : a classification head trained on English task data
MMS-1B encoder
21.3
–
–
–
SeamlessM4Tv2-L encoder ( Communication et al., 2023 )
80.9
–
–
–
Appendix
Table 4: Topic classification on FLEURS audio across 11 languages. Scores are means over the languages, and balanced accuracy (BAcc) first averages the 7 classes within each language. The Fleurs-SLU rows are that paper's published results. Its systems are trained on the task, while our projectors and the multi-task LALM are not. The cascade and reference-transcript rows replace the audio with the projector's transcript or the reference transcript. Acc. counts unanswered utterances as wrong. Acc. † and BAcc use answered utterances only. Text input is scored at its first named option (Appendix B.1 ).
Open question
In-language
Qwen2-Audio-7B
Language
IFR%
Acc. † %
IFR%
BAcc%
IFR%
BAcc%
English
3
91
100
41
100
67
Spanish
11
68
100
29
100
63
French
1
50
100
34
100
64
Indonesian
16
48
81
21
100
36
Swahili
1
50
57
14
100
21
Appendix
Table 5: Topic classification across languages. Open question task: the projector is prompted for a category word. Accuracy is computed over answered utterances only, and blank where none was answered. In-language prompt: the digit template translated into the projector's native language. Qwen2-Audio-7B: evaluated on the same utterances using the English digit template. BAcc is balanced accuracy over answered utterances.
Yes/no, text input
Language
IFR%
BAcc%
English
89.1
52.8
Spanish
64.7
47.2
French
87.0
45.3
Indonesian
98.5
54.4
Swahili
93.6
48.0
Appendix
Table 6: The yes/no question with the reference transcript as text input, per language, through Tiny Aya Global. IFR is computed based on the first option named. BAcc is over answered utterances, and chance is 50%. The last row is the mean over the 11 languages.
Language
Global
Earth
Fire
Water
Base
Qwen3-4B
BAcc%
S
T
S
T
S
T
S
T
S
T
S
T
English
41.4
73.0
50.6
72.0
38.2
73.8
53.4
71.5
15.6
15.1
43.6
88.0
Spanish
36.2
73.6
27.8
73.0
23.0
76.5
29.2
74.1
15.6
14.3
42.3
82.5
French
35.7
75.8
37.7
76.6
62.6
78.2
48.2
75.8
15.5
17.0
56.8
84.0
Indonesian
35.7
71.3
31.7
71.1
31.7
74.1
42.7
70.8
13.4
14.3
30.5
83.1
Swahili
17.1
66.0
22.0
68.2
18.0
73.6
19.3
65.7
16.8
14.3
18.3
62.0
Appendix
Table 7: Balanced accuracy (BAcc) over the 7 topics on the digit template question, by language and frozen decoder. Each decoder has its own projector for each language. S uses speech. T uses the reference transcript instead of the audio. The final row averages the 11 languages. Base is the pretrained Tiny Aya before instruction tuning. Earth, Fire and Water are its regional variants, and Global is reported in the main text. Chance is 14.3% (i.e. 1/7).
Spoken Language Understanding (SLU) maps speech to intents, entities, dialogue states, and meaning while accounting for acoustic and conversational context.
Center for Augmented Intelligence, Fondazione Bruno Kessler, Trento, Italy · Department of Information Engineering and Computer Science, University of Trento, Italy