Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research Asia, Hong Kong, China. · Microsoft Research, Redmond, WA, USA.
Speech and audio language models connect continuous acoustic representations to decoders pretrained primarily on text data. Most such models use MLP or Q-Former interfaces to map acoustic features into the decoder's input space, requiring the decoder to learn how to interpret the resulting representations. We investigate whether grounding these representations in the decoder's existing text embeddings can help it use acoustic information while better preserving its pretrained text capabilities. We present ConvexLLM, a general-purpose speech and audio language model whose C-Gate interface represents audio as continuous convex combinations of frozen text embeddings. ConvexLLM achieves an eleven-task VoiceBench aggregate of 75.36, the highest among the open-weight models we compare, and an MMAU Test-mini accuracy of 62.50%. Its largest margins over the baselines are on spoken knowledge and reasoning tasks, while the strongest baselines are more accurate on audio understanding and transcription. In a comparison with a 2B-parameter decoder, C-Gate achieves higher aggregate spoken-language performance than MLP and Q-Former interfaces, with text-task performance close to MLP and task-dependent tradeoffs in transcription and audio understanding. These results support convex combinations of text embeddings as a viable interface for general-purpose speech and audio language models.
Figures & tables
Figure 1: Comparison of Input Interfaces. (a) Discrete outputs select individual embedding rows. (b) Unconstrained outputs can lie outside the convex hull of the embedding rows. (c) Convex mixtures allow continuous variation while remaining inside the hull.
Figure 2: ConvexLlm Architecture. (a) The acoustic encoder produces representations that C-Gate maps into continuous inputs for the language decoder, which reads them next to the embedded task prompt and generates the response. (b) For each pooled audio frame, the query projection fϕ produces a query, the router scores the vocabulary keys, selects the top- K embeddings, and outputs a convex mixture of their frozen values with non-negative weights that sum to one. The losses LCE and LKL are defined in Section 3.3 . Gray marks the modules that are frozen in Stage 1, and the acoustic encoder and decoder blocks are trained in later stages, while the text embedding table and language-modeling head stay frozen throughout.
VoiceBench
MMAU Test-mini
LibriSpeech
Model
Alp.
Alp.-F
Common
SD-QA
Wild
AdvB
IFEval
OBQA
MMSU
SpMMLU
BBH
Overall
Sound
Music
Speech
Avg
Clean
Other
ConvexLlm
82.91
79.12
75.50
41.94
69.10
99.62
64.30
90.33
72.32
73.42
80.40
75.36
65.77
60.78
60.96
62.50
2.39
5.07
Kimi-Audio
76.00
74.60
68.40
49.52
66.20
100.00
58.20
82.40
61.70
67.40
68.40
70.26
73.30
57.20
68.20
66.23
1.40
2.59
MiMo-Audio
75.60
74.40
62.40
33.19
63.20
98.10
68.10
75.80
58.70
56.00
71.60
67.01
82.30
75.80
71.80
76.63
3.19
6.40
MiniCPM-o 4.5
84.20
81.20
75.60
54.79
68.60
96.70
78.00
83.70
69.70
40.70
70.20
73.04
77.80
71.90
73.00
74.23
1.72
3.98
Qwen2.5-Omni
69.00
68.20
64.60
43.96
58.80
99.40
51.40
81.10
56.10
66.50
66.90
66.00
81.40
67.70
75.40
74.83
1.85
3.53
Table 1: Overall Task Performance. We report GPT-5.4 judge scores rescaled to 0–100, task accuracy, refusal rate, and word error rate (WER) on VoiceBench, MMAU Test-mini, and LibriSpeech. Overall is the equal-weight mean of all eleven VoiceBench tasks, excluding MMAU and ASR. Bold marks the best result in each column, and lower is better for WER. Alp., Alp.-F, Common, Wild, AdvB, OBQA, and SpMMLU denote AlpacaEval, AlpacaEval-Full, CommonEval, WildVoice, AdvBench, OpenBookQA, and SpeechMMLU. † Audio Flamingo 3 uses its transcription checkpoint for LibriSpeech and its chat checkpoint for the other benchmarks.
VoiceBench
MMAU Test-mini
LibriSpeech
Interface
Input
Alp.
Alp.-F
Common
SD-QA
Wild
AdvB
IFEval
OBQA
MMSU
SpMMLU
BBH
Overall
Sound
Music
Speech
Avg
Clean
Other
Base
Text
57.05
60.25
57.93
23.89
57.29
99.04
64.48
69.67
56.44
43.39
76.60
60.55
-
-
-
-
-
-
MLP
Text
66.03
64.43
66.50
29.11
62.96
99.04
63.68
67.03
51.04
56.47
63.60
62.72
-
-
-
-
-
-
Speech
68.54
65.91
63.30
27.75
55.56
99.04
59.73
52.97
41.28
43.63
69.10
58.80
63.66
58.38
60.06
60.70
2.10
4.60
Q-Former
Text
64.72
64.81
65.80
27.97
62.96
98.27
58.76
67.03
51.53
59.90
56.20
61.63
-
-
-
-
-
-
Speech
70.15
62.39
64.30
26.94
51.56
97.12
52.59
43.08
32.14
33.43
62.40
54.19
67.57
57.49
59.16
61.40
2.49
5.52
Table 2: Comparison of Interfaces. We train C-Gate , MLP, and Q-Former interfaces with Qwen3.5-2B under a shared Stage 3 recipe and evaluate each model with written (Text) and spoken (Speech) inputs. Base is the original Qwen3.5-2B decoder before adaptation. Scores are on a 0–100 scale, with higher values better except for WER. Overall averages the eleven VoiceBench tasks. Dashes mark tasks that are not evaluated for that input.
Figure 3: Comparison of Top- K Configurations. We train C-Gate with support sizes K∈{1,4,16,64} on Qwen3.5-2B. (a) LibriSpeech test-clean and test-other WER in percent. (b) MMAU Test-mini accuracy for sound, music, and speech. (c) Speech-input OpenBookQA, IFEval, SD-QA, and Overall scores on a 0–100 scale, where Overall averages the eleven VoiceBench tasks of Table 2 . The K=1 run uses a straight-through gradient (Appendix E ).
Figure 4: Speech Representations Relative to Text Representations. We compare 2B C-Gate , MLP, and Q-Former models (Appendix D.1 ) on 200 spoken questions and their written counterparts. (a) Cumulative fraction of input variation, or energy, captured by the leading principal directions of the text embedding table. (b) Cosine similarity between the spoken and written versions of each question after each decoder block, where block 0 is the decoder input. (c) Mean L2 norm of speech and text representations after each decoder block, where Original Base is the original Qwen3.5-2B decoder.
Figure 5: PCA of Normalized Decoder Inputs. After unit L2 normalization, we project the decoder inputs of the three models in Figure 4 onto the first two principal components of the written tokens from 20 questions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
MLP
Q-Former
C-Gate
Acoustic encoder
Whisper-large-v3
Whisper-large-v3
Whisper-large-v3
Decoder
Qwen3.5-2B
Qwen3.5-2B
Qwen3.5-2B
Interface parameters
1,706,494
1,708,268
1,705,473
Temporal pooling stride
4
4
4
Routing dimension
N/A
N/A
512
Support size K
N/A
N/A
16
Appendix
Table 3: Architecture and training settings for the 2B interface comparison.
Task Family
Dataset or Source Group
Input
Probability
Recognition
MLS English, GigaSpeech, LibriSpeech, Emilia, Common Voice English, FLEURS English
Audio
35.0%
ASR-derived instruction
Transcription-derived spoken instruction sources
Audio
15.0%
Interaction and audio understanding
Public audio, synthetic speech, and in-house data
Audio
35.0%
Text replay
General dialogue, question answering, instruction following
Text
15.0%
Total
100.0%
Appendix
Table 4: Source-sampling probabilities for the 2B interface comparison.
Model
Audio Hours
Kimi-Audio ( Kimi Team, 2025 )
≈ 13M
MiMo-Audio ( Xiaomi LLM-Core Team, 2025 )
> 100M
MiniCPM-o 4.5 ( Cui et al., 2026 )
Millions
Qwen2.5-Omni ( Xu et al., 2025a )
–
Audio Flamingo 3 ( NVIDIA, 2025 )
–
MiniCPM-o 2.6 ( OpenBMB Team, 2025a )
–
Appendix
Table 5: Audio training hours for the baseline models and our model, in millions (M) or thousands (K).
Benchmark / Subset
Items
Max. Tokens
Reported Metric
VoiceBench: AlpacaEval
199
2,048
Mean quality score (1–5) × 20
VoiceBench: AlpacaEval-Full
636
2,048
Mean quality score (1–5) × 20
VoiceBench: CommonEval
200
2,048
Mean quality score (1–5) × 20
VoiceBench: SD-QA
6,083
2,048
GPT-5.4 judged correctness (%)
VoiceBench: WildVoice
1,000
2,048
Mean quality score (1–5) × 20
VoiceBench: AdvBench
520
2,048
Deterministic refusal rate (%)
Appendix
Table 6: Overview of evaluation subsets.
Task
Metric
N
C-Gate
MLP
Q-Former
MTBench
GPT 1–5
46
2.41
2.26
1.91
OAB AlpacaEval
GPT 1–10
199
4.85
4.66
4.61
OAB LlamaQuestions
Correct
300
63.67
62.67
58.67
OAB TriviaQA
GPT 0–10
1,000
2.81
2.73
2.36
OAB WebQuestions
GPT 0–10
1,000
4.15
4.10
3.96
OAB ReasoningQA
Alias accuracy
202
6.93
4.95
4.46
Appendix
Table 7: Additional speech-input tasks for the 2B interface comparison of Table 2 . All metrics are percentages except the GPT score scales given in the Metric column. Higher is better except for WER. OAB denotes OpenAudioBench.
VoiceBench
MMAU Test-mini
LibriSpeech
K
Alp.
Alp.-F
Common
SD-QA
Wild
AdvB
IFEval
OBQA
MMSU
SpMMLU
BBH
Overall
Sound
Music
Speech
Avg
Clean
Other
1
22.11
21.07
20.60
0.35
20.06
24.42
13.67
20.44
24.72
25.34
52.50
22.30
55.56
52.69
47.15
51.80
86.48
88.65
4
70.75
65.19
62.70
25.56
53.24
99.42
58.30
55.60
43.36
43.17
71.60
58.99
61.56
55.39
60.96
59.30
2.87
6.25
16
69.65
66.45
63.50
27.12
55.66
100.00
61.20
59.78
43.92
46.52
70.00
60.35
63.66
55.39
59.16
59.40
2.46
5.54
64
71.16
66.92
62.30
28.39
57.08
99.81
64.74
60.22
46.94
47.61
71.70
61.53
63.06
54.19
58.26
58.50
2.35
5.37
Appendix
Table 8: Speech-input results for the four Top- K configurations of Figure 3 , in the layout of Table 2 . Scores are on a 0–100 scale, with higher values better except for WER. Overall averages the eleven VoiceBench tasks.
Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a pretrained LLM remains challenging. A central design question is how to tie speech representations to the LLM's text input space while preserving the continuous variation needed for different speech tasks. In this work, we propose Convex Gate (C-Gate), a speech-LLM bridge that represents each frame as a convex combination of frozen token embeddings. This architectural convex-hull constraint fixes the output codebook while allowing continuous mixtures learned from task supervision. With a Whisper-large-v3 encoder, a partially adapted Qwen2.5-7B-Instruct decoder, and joint training on speech recognition, emotion recognition, and spoken reasoning, C-Gate performs accurate transcription and emotion recognition, our representative semantic and acoustic tasks, through one shared decoder. On text-only benchmarks, the adapted decoder loses instruction-following and MMLU accuracy and matches or exceeds the original model on GSM8K and HellaSwag. We further design analyses and interventions on the trained model to examine where the bridge sequence carries task information and how the decoder reads it. These findings show how fixed text embeddings can support both semantic and acoustic readouts through continuous speech-conditioned mixtures.
Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.
Ke-Han Lu, Keqi Deng, Ruchao Fan +2
Microsoft, USA · National Taiwan University, Taiwan