ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

Ming-Hao Hsu; Shujie Liu; Yuxuan Hu; Yuxiang Wang; Yingda Shen; Jinyu Li; Zhizheng Wu

cs.CLOct 16, 2026

ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

Authors: Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research Asia, Hong Kong, China. · Microsoft Research, Redmond, WA, USA.

Abstract

Speech and audio language models connect continuous acoustic representations to decoders pretrained primarily on text data. Most such models use MLP or Q-Former interfaces to map acoustic features into the decoder's input space, requiring the decoder to learn how to interpret the resulting representations. We investigate whether grounding these representations in the decoder's existing text embeddings can help it use acoustic information while better preserving its pretrained text capabilities. We present ConvexLLM, a general-purpose speech and audio language model whose C-Gate interface represents audio as continuous convex combinations of frozen text embeddings. ConvexLLM achieves an eleven-task VoiceBench aggregate of 75.36, the highest among the open-weight models we compare, and an MMAU Test-mini accuracy of 62.50%. Its largest margins over the baselines are on spoken knowledge and reasoning tasks, while the strongest baselines are more accurate on audio understanding and transcription. In a comparison with a 2B-parameter decoder, C-Gate achieves higher aggregate spoken-language performance than MLP and Q-Former interfaces, with text-task performance close to MLP and task-dependent tradeoffs in transcription and audio understanding. These results support convex combinations of text embeddings as a viable interface for general-purpose speech and audio language models.

Figures & tables

Figure 1: Comparison of Input Interfaces. (a) Discrete outputs select individual embedding rows. (b) Unconstrained outputs can lie outside the convex hull of the embedding rows. (c) Convex mixtures allow continuous variation while remaining inside the hull.

Figure 2: ConvexLlm Architecture. (a) The acoustic encoder produces representations that C-Gate maps into continuous inputs for the language decoder, which reads them next to the embedded task prompt and generates the response. (b) For each pooled audio frame, the query projection

f_{\phi}

produces a query, the router scores the vocabulary keys, selects the top-

K

embeddings, and outputs a convex mixture of their frozen values with non-negative weights that sum to one. The losses

\mathcal{L}_{\mathrm{CE}}

and

\mathcal{L}_{\mathrm{KL}}

are defined in Section 3.3 . Gray marks the modules that are frozen in Stage 1, and the acoustic encoder and decoder blocks are trained in later stages, while the text embedding table and language-modeling head stay frozen throughout.

	VoiceBench												MMAU Test-mini				LibriSpeech
Model	Alp.	Alp.-F	Common	SD-QA	Wild	AdvB	IFEval	OBQA	MMSU	SpMMLU	BBH	Overall	Sound	Music	Speech	Avg	Clean	Other
ConvexLlm	82.91	79.12	75.50	41.94	69.10	99.62	64.30	90.33	72.32	73.42	80.40	75.36	65.77	60.78	60.96	62.50	2.39	5.07
Kimi-Audio	76.00	74.60	68.40	49.52	66.20	100.00	58.20	82.40	61.70	67.40	68.40	70.26	73.30	57.20	68.20	66.23	1.40	2.59
MiMo-Audio	75.60	74.40	62.40	33.19	63.20	98.10	68.10	75.80	58.70	56.00	71.60	67.01	82.30	75.80	71.80	76.63	3.19	6.40
MiniCPM-o 4.5	84.20	81.20	75.60	54.79	68.60	96.70	78.00	83.70	69.70	40.70	70.20	73.04	77.80	71.90	73.00	74.23	1.72	3.98
Qwen2.5-Omni	69.00	68.20	64.60	43.96	58.80	99.40	51.40	81.10	56.10	66.50	66.90	66.00	81.40	67.70	75.40	74.83	1.85	3.53

Table 1: Overall Task Performance. We report GPT-5.4 judge scores rescaled to 0–100, task accuracy, refusal rate, and word error rate (WER) on VoiceBench, MMAU Test-mini, and LibriSpeech. Overall is the equal-weight mean of all eleven VoiceBench tasks, excluding MMAU and ASR. Bold marks the best result in each column, and lower is better for WER. Alp., Alp.-F, Common, Wild, AdvB, OBQA, and SpMMLU denote AlpacaEval, AlpacaEval-Full, CommonEval, WildVoice, AdvBench, OpenBookQA, and SpeechMMLU. † Audio Flamingo 3 uses its transcription checkpoint for LibriSpeech and its chat checkpoint for the other benchmarks.

		VoiceBench												MMAU Test-mini				LibriSpeech
Interface	Input	Alp.	Alp.-F	Common	SD-QA	Wild	AdvB	IFEval	OBQA	MMSU	SpMMLU	BBH	Overall	Sound	Music	Speech	Avg	Clean	Other
Base	Text	57.05	60.25	57.93	23.89	57.29	99.04	64.48	69.67	56.44	43.39	76.60	60.55	-	-	-	-	-	-
MLP	Text	66.03	64.43	66.50	29.11	62.96	99.04	63.68	67.03	51.04	56.47	63.60	62.72	-	-	-	-	-	-
	Speech	68.54	65.91	63.30	27.75	55.56	99.04	59.73	52.97	41.28	43.63	69.10	58.80	63.66	58.38	60.06	60.70	2.10	4.60
Q-Former	Text	64.72	64.81	65.80	27.97	62.96	98.27	58.76	67.03	51.53	59.90	56.20	61.63	-	-	-	-	-	-
	Speech	70.15	62.39	64.30	26.94	51.56	97.12	52.59	43.08	32.14	33.43	62.40	54.19	67.57	57.49	59.16	61.40	2.49	5.52

Table 2: Comparison of Interfaces. We train C-Gate , MLP, and Q-Former interfaces with Qwen3.5-2B under a shared Stage 3 recipe and evaluate each model with written (Text) and spoken (Speech) inputs. Base is the original Qwen3.5-2B decoder before adaptation. Scores are on a 0–100 scale, with higher values better except for WER. Overall averages the eleven VoiceBench tasks. Dashes mark tasks that are not evaluated for that input.

Figure 3: Comparison of Top-

K

Configurations. We train C-Gate with support sizes

K\in\{1,4,16,64\}

on Qwen3.5-2B. (a) LibriSpeech test-clean and test-other WER in percent. (b) MMAU Test-mini accuracy for sound, music, and speech. (c) Speech-input OpenBookQA, IFEval, SD-QA, and Overall scores on a 0–100 scale, where Overall averages the eleven VoiceBench tasks of Table 2 . The

K=1

run uses a straight-through gradient (Appendix E ).

Figure 4: Speech Representations Relative to Text Representations. We compare 2B C-Gate , MLP, and Q-Former models (Appendix D.1 ) on 200 spoken questions and their written counterparts. (a) Cumulative fraction of input variation, or energy, captured by the leading principal directions of the text embedding table. (b) Cosine similarity between the spoken and written versions of each question after each decoder block, where block 0 is the decoder input. (c) Mean L2 norm of speech and text representations after each decoder block, where Original Base is the original Qwen3.5-2B decoder.

Figure 5: PCA of Normalized Decoder Inputs. After unit L2 normalization, we project the decoder inputs of the three models in Figure 4 onto the first two principal components of the written tokens from 20 questions.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	MLP	Q-Former	C-Gate
Acoustic encoder	Whisper-large-v3	Whisper-large-v3	Whisper-large-v3
Decoder	Qwen3.5-2B	Qwen3.5-2B	Qwen3.5-2B
Interface parameters	1,706,494	1,708,268	1,705,473
Temporal pooling stride	4	4	4
Routing dimension	N/A	N/A	512
Support size $K$	N/A	N/A	16

Appendix

Table 3: Architecture and training settings for the 2B interface comparison.

Task Family	Dataset or Source Group	Input	Probability
Recognition	MLS English, GigaSpeech, LibriSpeech, Emilia, Common Voice English, FLEURS English	Audio	35.0%
ASR-derived instruction	Transcription-derived spoken instruction sources	Audio	15.0%
Interaction and audio understanding	Public audio, synthetic speech, and in-house data	Audio	35.0%
Text replay	General dialogue, question answering, instruction following	Text	15.0%
Total			100.0%

Appendix

Table 4: Source-sampling probabilities for the 2B interface comparison.

Model	Audio Hours
Kimi-Audio ( Kimi Team, 2025 )	$\approx$ 13M
MiMo-Audio ( Xiaomi LLM-Core Team, 2025 )	$>$ 100M
MiniCPM-o 4.5 ( Cui et al., 2026 )	Millions
Qwen2.5-Omni ( Xu et al., 2025a )	–
Audio Flamingo 3 ( NVIDIA, 2025 )	–
MiniCPM-o 2.6 ( OpenBMB Team, 2025a )	–

Appendix

Table 5: Audio training hours for the baseline models and our model, in millions (M) or thousands (K).

Benchmark / Subset	Items	Max. Tokens	Reported Metric
VoiceBench: AlpacaEval	199	2,048	Mean quality score (1–5) $\times$ 20
VoiceBench: AlpacaEval-Full	636	2,048	Mean quality score (1–5) $\times$ 20
VoiceBench: CommonEval	200	2,048	Mean quality score (1–5) $\times$ 20
VoiceBench: SD-QA	6,083	2,048	GPT-5.4 judged correctness (%)
VoiceBench: WildVoice	1,000	2,048	Mean quality score (1–5) $\times$ 20
VoiceBench: AdvBench	520	2,048	Deterministic refusal rate (%)

Appendix

Table 6: Overview of evaluation subsets.

Task	Metric	$N$	C-Gate	MLP	Q-Former
MTBench	GPT 1–5	46	2.41	2.26	1.91
OAB AlpacaEval	GPT 1–10	199	4.85	4.66	4.61
OAB LlamaQuestions	Correct	300	63.67	62.67	58.67
OAB TriviaQA	GPT 0–10	1,000	2.81	2.73	2.36
OAB WebQuestions	GPT 0–10	1,000	4.15	4.10	3.96
OAB ReasoningQA	Alias accuracy	202	6.93	4.95	4.46

Appendix

Table 7: Additional speech-input tasks for the 2B interface comparison of Table 2 . All metrics are percentages except the GPT score scales given in the Metric column. Higher is better except for WER. OAB denotes OpenAudioBench.

	VoiceBench												MMAU Test-mini				LibriSpeech
$K$	Alp.	Alp.-F	Common	SD-QA	Wild	AdvB	IFEval	OBQA	MMSU	SpMMLU	BBH	Overall	Sound	Music	Speech	Avg	Clean	Other
1	22.11	21.07	20.60	0.35	20.06	24.42	13.67	20.44	24.72	25.34	52.50	22.30	55.56	52.69	47.15	51.80	86.48	88.65
4	70.75	65.19	62.70	25.56	53.24	99.42	58.30	55.60	43.36	43.17	71.60	58.99	61.56	55.39	60.96	59.30	2.87	6.25
16	69.65	66.45	63.50	27.12	55.66	100.00	61.20	59.78	43.92	46.52	70.00	60.35	63.66	55.39	59.16	59.40	2.46	5.54
64	71.16	66.92	62.30	28.39	57.08	99.81	64.74	60.22	46.94	47.61	71.70	61.53	63.06	54.19	58.26	58.50	2.35	5.37

Appendix

Table 8: Speech-input results for the four Top-

K

configurations of Figure 3 , in the layout of Table 2 . Scores are on a 0–100 scale, with higher values better except for WER. Overall averages the eleven VoiceBench tasks.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 300 papers

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
Jun 8, 2026Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3Audio Representation Learning Audio Understanding
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.
Unified Audio Intelligence Without Regressing on Text Intelligence
Jul 6, 2026Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim +17TTS Synthesis Unified Multimodal Models
NVIDIA
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2KV Caching Automatic Speech Recognition
Microsoft, USA · National Taiwan University, Taiwan

Jun 8, 2026J/K move · Enter open · S save

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

Ming-Hao Hsu, Yuxuan Hu, Shujie Liu, Jinyu Li+2

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.

Audio Representation Learning Audio Understanding Automatic Speech Recognition Speech Language Models

Open paper AskPDF arXiv

Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a pretrained LLM remains challenging. A central design question is how to tie speech representations to the LLM's text input space while preserving the continuous variation needed for different speech tasks. In this work, we propose Convex Gate (C-Gate), a speech-LLM bridge that represents each frame as a convex combination of frozen token embeddings. This architectural convex-hull constraint fixes the output codebook while allowing continuous mixtures learned from task supervision. With a Whisper-large-v3 encoder, a partially adapted Qwen2.5-7B-Instruct decoder, and joint training on speech recognition, emotion recognition, and spoken reasoning, C-Gate performs accurate transcription and emotion recognition, our representative semantic and acoustic tasks, through one shared decoder. On text-only benchmarks, the adapted decoder loses instruction-following and MMLU accuracy and matches or exceeds the original model on GSM8K and HellaSwag. We further design analyses and interventions on the trained model to examine where the bridge sequence carries task information and how the decoder reads it. These findings show how fixed text embeddings can support both semantic and acoustic readouts through continuous speech-conditioned mixtures.

Abstract

Audio-Language Models

Explore similar work

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

Unified Audio Intelligence Without Regressing on Text Intelligence

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs