In-Context Adapter Generation for Compositional Generalization in Large Audio Language Models

Yiwei Guo; Siqi Huang; Bohan Li; Jing Peng; Hankun Wang; Zhihan Li; Kai Yu

eess.ASOct 25, 2026

In-Context Adapter Generation for Compositional Generalization in Large Audio Language Models

Authors: Yiwei Guo, Siqi Huang, Bohan Li, Jing Peng, Hankun Wang, Zhihan Li, Kai Yu

Organizations: X-LANCE Lab, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University · Jiangsu Key Lab of Language Computing · Xi’an Jiaotong University

Abstract

Large audio language models (LALMs) have achieved strong performance on standard speech and audio understanding tasks, yet their ability to systematically generalize to novel task compositions remains underexplored. Taking a meta-learning view, we extend Text-to-LoRA (T2L) to LALMs, generating instruction-conditioned LoRA adapters with a Transformer hypernetwork rather than learning a single fixed adapter shared across instructions. To exploit fine-grained task information, we introduce in-context adapter generation (ICAG), a token-level hypernetwork architecture; we call the resulting method T2L-ICAG. We evaluate this approach on two representative LALMs using SIFT-50M for supervised fine-tuning and construct a controlled compositional evaluation suite covering five task classes. Across two LALM backbones, T2L-ICAG consistently improves over static LoRA and also the global embedding-based T2L baseline. Coverage and capacity analyses further show that these gains are larger on structurally out-of-domain tasks and cannot be explained by parameter count alone. These results suggest that generating task-specific adapters from instructions is a promising direction for improving compositional generalization of LALMs.

Figures & tables

Figure 2: Radar plots for the five compositional task groups. Note that for the WER metric, the outer rings correspond to lower (better) values. The metrics are included in the parentheses.

Task group	Primitive I/S/O	Exact I/S/O
Atomic	5 / 0 / 0	4 / 0 / 1
Comparative	2 / 3 / 0	2 / 2 / 1
Selective	1 / 1 / 7	0 / 0 / 9
Sequential	4 / 2 / 0	1 / 1 / 4
Multi-function	1 / 3 / 0	0 / 1 / 3
Total	13 / 9 / 7	7 / 4 / 18

Table 1: Task distribution under overlap-based partitions: in-domain (I), sparse (S), and out-of-domain (O).

Method	EvalSIFT (ACC% $\uparrow$ )			Our Compositional Evaluation (RPS% $\uparrow$ )
Method	Acoustic	Content	Comparison	Atomic	Comparative	Selective	Sequential	Multi-function	Avg.
Kimi-Audio 7B	18.6	41.8	11.6	85.57	44.45	79.22	2.00	76.90	57.63
w/ LoRA ( $r=4$ )	75.4	58.0	69.0	98.96	95.07	91.10	91.08	83.87	92.02
w/ LoRA ( $r=80$ )	78.4	59.4	72.6	94.58	95.76	67.38	83.88	92.46	86.81
w/ T2L-emb	77.2	59.2	71.0	97.43	90.59	87.62	92.87	86.42	90.99
w/ T2L-ICAG	77.0	58.8	72.8	98.47	98.25	99.30	98.29	97.44	98.35

Table 2: Accuracy and RPS results across EvalSIFT categories and our compositional task classes. All T2L models use

r=4

Primitive-level
Method	In-domain	Sparse	OOD
Kimi-Audio 7B	52.42	58.05	74.44
w/ LoRA ( $r=4$ )	93.20	87.57	90.12
w/ LoRA ( $r=80$ )	86.59	87.84	77.78
w/ T2L-emb	92.50	88.39	87.47
w/ T2L-ICAG	97.78	98.53	99.09

Table 3: RPS scores under primitive-level and exact-form train-test overlap partitions.

Figure 3: Gender- and language-related compositional performance compared with the expected performance implied by atomic recognition ability. The “expected” bars are calculated from atomic performance.

Figure 4: Rank ablation results across compositional task groups. Each panel corresponds to one backbone model and one LoRA rank. The shaded region marks the average over the five task groups. The numbers after bars denote the gain of T2L-ICAG over LoRA.

Figure 5: Training-set coverage of the 29 evaluation tasks, grouped by the five task groups. Bar heights show training-sample multiplicity on a shared logarithmic scale. Tasks with no matching training samples are marked with “0”. Dashed lines denote the 1000-sample threshold, which we use to distinguish sparse tasks and in-domain tasks.

Task class	Task name	Source dataset	Sample num.	Requirements	Evaluation
Atomic task (single speech input, single task)	ASR (Automatic speech recognition)	LS test-clean	2620	Transcribe the spoken content into text.	WER
	GR (Gender recognition)	LS test-clean	2620	Recognize speaker gender (male, female).	LLM-judged Acc.
	SER (Speech emotion recognition)	MELD test	2610	Recognize speaker emotion (anger, disgust, fear, joy, neutral, sadness, surprise).	LLM-judged Acc.
	LID (Language identification)	LS test-clean, MLS test	2000	Identify speaker language (English, French, Italian, German, Spanish).	LLM-judged Acc.
	SQA (Spoken question answering)	SLUE-p2-SQA5	408	Answer the text question based on the speech content.	LLM-judge Score
Comparative task (double speech inputs, single task)	Gender Verification	LS test-clean	5240	Verify whether the two speakers have the same gender.	LLM-judged Acc.

Table 4: Details of task classes, source datasets, instructions, and evaluation protocols. “LS” denotes LibriSpeech.

Task class	Task name	Source dataset	Sample num.	Requirements	Evaluation
Sequential execution task (multiple speech inputs, single task)	2-ASR JSON	LS test-clean	2620	Transcribe two speakers into JSON format.	IFR, WER
	3-ASR JSON	LS test-clean	2620	Transcribe three speakers into JSON format.	IFR, WER
	2-GR	LS test-clean	2620	Recognize the genders of two speakers.	LLM-judged Acc.
	3-GR	LS test-clean	2620	Recognize the genders of three speakers.	LLM-judged Acc.
	2-LID	LS test-clean, MLS test	2100	Identify the languages of two speakers.	LLM-judged Acc.
	3-LID	LS test-clean, MLS test	2100	Identify the languages of three speakers.	LLM-judged Acc.

Table 5: (Continued) Details of task classes, source datasets, instructions, and evaluation protocols. “LS” denotes LibriSpeech.

Configuration	Kimi-Audio 7B	ICAG Hypernetwork
Num. Layers	28	4
Hidden Dim.	3584	256
Head Dim.	128	32
Num. KV Heads	4	8
Num. Q Heads	28	8

Table 6: Configurations of the LALM backbone and ICAG hypernetwork. For Qwen2.5-Omni Thinker, the configurations are exactly the same.

Figure 6: BLEU-4 and COMET results on CoVoST2 X2En speech-to-text translation from seven source languages: French (fr), German (de), Spanish (es), Italian (it), Dutch (nl), Portuguese (pt), and Mandarin (zh). The Kimi-Audio bar is invisible for zh

\rightarrow

en under BLEU-4 because its score is 0.

Figure 7: Effect of static LoRA rank on EvalSIFT and compositional generalization under different train–test overlap partitions. The blue curves show static LoRA results across ranks, the gray dashed lines denote the original Kimi-Audio backbone, and the orange stars denote T2L-ICAG with

r=4

. A blue hollow circle marks the best static LoRA rank in each panel. The numbers

n

indicate the number of evaluation metric units in each partition. The EvalSIFT panel reports raw accuracy, whereas the remaining panels report RPS.

Figure 8: Singular-value spectra of static LoRA adapters and T2L-ICAG-generated adapters on Kimi-Audio. We only show the rank 4 results on 8 example tasks here, while the same pattern can be found in all tasks across all ranks. Light lines denote individual decoder layers, while dark lines denote median spectra over layers.

Atomic	Acc. / Score $\uparrow$	WER $\downarrow$
ASR	–	1.21
GR	97.60	–
SER	59.16	–
LID	100.00	–
SQA	89.66	–

Table 7: Results of Qwen3.5-Omni-Plus on the 29-task compositional evaluation suite. All values are percentages. IFR denotes the strict JSON format instruction-following rate. For selective ASR tasks, SAcc. and mWER columns report selection accuracy and minimum WER, respectively; for other selective tasks, Acc. reports the final task performance.

Hypernetwork Instruction	Accuracy
Please verify if the following two speakers have the same gender.	87.96%
There are two audio clips, one from a male speaker and one from a female speaker. Please identify the gender of the speaker who speaks for a longer duration (Male or Female).	82.61%
Please recognize the gender of the following two speech clips respectively.	76.74%
Identify the gender of the speaker.	75.19%
There are two audio clips in the following. Please transcribe only what the speaker with a higher average pitch said. Only output the transcript, without any other text.	71.77%
Please transcribe the following audio:	65.73%

Table 8: Accuracy of T2L-ICAG on the gender verification task across different hypernetwork instruction variants. Note that the LLM instruction is still the original task-inherent instruction.

Figure 9: Gender-related compositional performance of Qwen3.5-Omni-Plus. Blue, gray, and orange denote atomic GR performance, atomic-derived expectations, and observed results, respectively. The expectations are diagnostic rather than strict upper bounds.

System	N	Krippendorff’s $\alpha$	Agree.
Kimi-Audio	2000	0.965	97.4%
w/ LoRA	2000	0.894	94.8%
w/ T2L-ICAG	2000	0.888	95.3%
Overall	6000	0.935	95.8%

Table 9: Inter-annotator agreement between the three LLM judges on a random subset of the compositional benchmark. “N” is the sample count, and “Agree.” denotes the fraction of samples on which all annotators agree.

Compositional Generalization

Compositional generalization is the ability to combine learned parts and rules in ways that transfer to unseen combinations.

See all 113 papers

When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal
Mar 20, 2026Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien +5Audio-Language Model Evaluation LLM Evaluation
National Taiwan University, Taiwan
Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning
Jul 2, 2026Congrui Du, Yang Zhang, Kaizhi Qian +1Language Model Pretraining Instruction Following
University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA
AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models
Oct 3, 2026Lee Seung-woo, Bowen QiAudio-Language Models Audio Reasoning
Seoul National University · Shanghai Jiao Tong University

Mar 20, 2026J/K move · Enter open · S save

When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal

Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien, Toshiki Ogimoto+4

National Taiwan University, Taiwan

Audio-Language Model Evaluation LLM Evaluation LLM Grounding Multimodal ICL Audio Understanding Audio-Language Models+1

Open paper AskPDF arXiv

While Large Audio-Language Models (LALMs) have been shown to exhibit degraded instruction-following capabilities, their ability to infer task patterns from in-context examples with audio remains understudied. To address this gap, we design a three-stage evaluation pipeline that progressively reduces textual guidance to systematically evaluate LALMs' in-context learning ability in the audio modality. Evaluating six LALMs across four audio understanding tasks under two output constraint categories, we uncover a consistent asymmetry across LALMs: in-context demonstrations reliably improve format compliance but fail to improve the core task performance. This suggests that LALMs can glean surface-level formatting patterns from demonstrations but may struggle to leverage cross-modal semantic grounding to reliably infer task objectives from examples with audio, highlighting potential limitations in current cross-modal integration. We further probe how demonstrations are used through two complementary analyses, demonstration label shuffling and attention knockout on demonstration spans, both showing that LALMs leverage in-context examples primarily to establish the output label space and format rather than to learn a meaningful input-output correspondence.

Abstract

Compositional Generalization

Explore similar work

When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal

Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models