SkillFormer: Skill-Decomposed Adaptation for Audio Language Models

Lee Seung-woo; Bowen Qi; Kim Min-jun; Jang Won-young

cs.SDOct 6, 2026

SkillFormer: Skill-Decomposed Adaptation for Audio Language Models

Authors: Lee Seung-woo, Bowen Qi, Kim Min-jun, Jang Won-young

Organizations: Pusan National University · Shanghai Jiao Tong University · Hanyang University

Abstract

Audio language models must handle dozens of distinct skills, from pitch comparison and speaker counting to musical tempo estimation and emotion recognition. Joint training on all skills at once causes interference: gains on one skill often come at the cost of another. We propose \textbf{SkillFormer}, which decomposes audio understanding into skill-specific low-rank adapters and composes them at inference time through a learned router. The router examines the question to decide which adapters to activate and how much weight each should carry, so that a pitch query engages different parameters than a genre classification query. An alternating training schedule updates each adapter on its own skill cluster before jointly calibrating the router, preventing the gradient conflicts that arise in standard multi-task optimization. SkillFormer adds fewer than 4% of the base model's parameters and requires no changes to the audio encoder or language backbone. Evaluated on three architecturally distinct models across MMSU, MMAU-Pro, and MMAR, it raises the average accuracy by 2.5 to 4.1 points, with balanced gains across perception, reasoning, and semantic subcategories.

Figures & tables

Panel of Figure 1 — Figure 1: SkillFormer routes audio features through a bank of $K$ skill-specific LoRA adapters. A question-conditioned router produces gating weights $\alpha_{1},\ldots,\alpha_{K}$ ; the adapter outputs are merged and passed to the language model.

Qwen2.5-Omni-7B
	MMSU $\uparrow$			MMAU-Pro $\uparrow$				MMAR $\uparrow$
Method	Percep.	Reason.	All	Speech	Sound	Music	All	Signal	Percep.	Seman.	Culture	All	Avg. $\uparrow$
Base	44.5	79.4	61.4	57.8	45.3	64.0	56.4	55.8	54.0	67.2	58.9	60.2	59.3
SFT	47.0	80.3	63.2	59.1	47.0	65.4	57.5	56.5	55.3	67.8	59.7	61.0	60.6 $+$ 1.3
Multi-LoRA	49.5	80.7	64.7	59.9	48.4	66.3	58.5	57.3	55.6	68.0	60.5	61.8	61.7 $+$ 2.4
SkillFormer	52.5	81.3	66.5	61.5	49.8	67.3	60.4	58.1	57.2	68.9	62.0	63.3	63.4 $+$ 4.1

Table 1: Accuracy (%) on three audio-understanding benchmarks. Avg. is the mean of the three All scores. SFT fine-tunes a single LoRA adapter on all data. Multi-LoRA keeps all

K{=}6

adapters active with uniform weights. For MMAU-Pro, the three category columns do not cover every scored item, so All is not their average. Bold and underline mark the best and second best per backbone.

Panel of Figure 2 — Figure 2: Router weight distribution by question type on Qwen2.5-Omni-7B. Each bar shows how the router distributes weight across the six adapters (A1–A6). Prosody and music questions each concentrate on a single adapter, while scene questions spread weight more evenly.

Variant	MMSU	MMAU-P	MMAR	Avg. (%)
SkillFormer	66.5	60.4	63.3	63.4
No router (uniform $\alpha$ )	64.7	58.5	61.8	61.7
Random routing	63.8	57.8	61.1	60.9
Single adapter (SFT)	63.2	57.5	61.0	60.6
$K{=}3$ clusters	65.3	59.2	62.4	62.3
$K{=}12$ clusters	65.8	59.6	62.7	62.7

Table 2: Ablations on Qwen2.5-Omni-7B. Each row changes one component of SkillFormer. Base model Avg. is 59.3%.

Explore similar work

Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding
Sep 23, 2026Kaiyang Li, Shaobo Han, Yue Tian +1Large Audio Language Models Opencode
NEC Laboratories America, Inc, USA · School of Computing, University of Connecticut, USA
Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
Aug 3, 2026Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7Audio Understanding Voice Agents
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China
EvoAudio: Recursive Self-Improvement for Audio Understanding
Sep 23, 2026Yuxiang Wang, Shengbo Cai, Yingda Shen +6Audio Understanding Audio Understanding
The Chinese University of Hong Kong, Shenzhen · Tencent Hunyuan · Tsinghua University +1

Sep 23, 2026, cs.SDJ/K move · Enter open · S save

Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding

Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji

NEC Laboratories America, Inc, USA · School of Computing, University of Connecticut, USA

Large Audio Language Models Opencode Motion-Language Alignment Acoustic

Open paper AskPDF arXiv

Audio-language models (ALMs) integrate acoustic perception with the knowledge encoded in language models, enabling contextual understanding of auditory events. Making these capabilities practical on devices with limited memory and computation motivates our focus on small ALMs with fewer than 200M parameters. We introduce a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM. Its architecture connects a compact CED-Small audio encoder to SmolLM2-135M through a frequency-merging mapper. With supervision drawn from ReasonAQA, AudioMCQ, and AVQA, the model undergoes three training stages: audio-language alignment (Stage 1), audio-dependent fine-tuning (Stage 2), and post-training (Stage 3) aimed at strengthening weak skills while retaining learned capabilities. Across five random seeds, Mizar achieves mean accuracies of 52.92% on MMAU, 42.42% on MMAR, and 36.02% on ADQA-clean, surpassing the previous best-performing ALM below 200M parameters on all three benchmarks. It also supports local inference on a single CPU: on questions from the MMAU benchmark, the mean latency from opening the audio file to generating a complete answer is 1.09 seconds. Code and checkpoints are available at https://github.com/KaiyangLi1992/Mizar_159M.

Abstract

Explore similar work

Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding

Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models

EvoAudio: Recursive Self-Improvement for Audio Understanding