AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models

Marwan Farag; Bin Yang

cs.SDOct 19, 2026

AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models

Authors: Marwan Farag, Bin Yang

Organizations: Institute of Signal Processing and System Theory (ISS), University of Stuttgart, Stuttgart, Germany

Abstract

Large Audio Language Models (LALMs) reason over speech, environmental sounds, music, and other acoustic information. However, their audio encoders produce dense token sequences that make long-form audio costly to process. Existing audio pruning methods reduce this cost using heuristics such as attention, representation similarity, or temporal sampling, but do not explicitly learn which tokens are most useful for a given question. We propose AudioScorer, a lightweight model that scores audio tokens according to the user question. It is trained by distilling the ranking of audio tokens under the frozen LALM's answer-gradient saliency, together with downstream answer supervision. Compared with other pruning methods, AudioScorer achieves the best or tied-best result at 10% token retention on 7 of 10 semantic and acoustic tasks, including speech recognition, emotion recognition, and audio-scene classification.

Figures & tables

Figure 1: Question-dependent token selection for the same audio. AudioScorer adapts its selection to the question under a fixed budget. Audio and token-to-time mapping from Wang et al. (2026) .

Figure 2: Overview of AudioScorer. It learns audio-token relevance from answer-gradient targets and downstream answer supervision. At inference, the scores drive Top-

K

pruning after early decoder layers.

		Speech Content				Audio Classification				Speaker Information				Audio	Prefill
														auth.	compute
Keep	Method	ASR	SCR	SER	Avg.	SED	MC	ASC	Avg.	ER	SAR	SGR	Avg.	SD	TFLOPs
$100\%$	Unpruned	97.68	82.81	24.02	68.17	70.18	100.00	70.31	80.16	33.82	15.53	99.63	49.66	90.11	46.04
	FastV	97.03	82.62	24.52 †	68.06	69.82	100.00	70.62 †	80.15	33.04	15.97 †	98.98	49.33	87.53	35.83
	SpeechPrune	91.31	80.51	24.95 †	65.59	68.59	100.00	70.22	79.60	35.14 †	15.87 †	98.90	49.97 †	93.93 †	35.93
$70\%$	DART	95.93	82.87 †	25.14 †	67.98	69.84	100.00	70.50 †	80.11	36.52 †	15.90 †	99.19	50.54 †	93.09 †	35.88

Table 1: AudioMarathon results on Qwen2.5-Omni-3B: WAR for ASR and Macro-F1 otherwise. Avg. is the unweighted mean of each three-task category; Audio Authenticity contains only SD. Single/double vertical rules separate category averages/categories. Bold / underlining mark the best/second-best pruning scores per retention ratio; † marks scores above unpruned. Prefill compute: mean TFLOPs to the first token over the same

80

randomly sampled duration-stratified clips. ASR: automatic speech recognition; SCR: speech content reasoning; SER: speech entity recognition; SED: sound event detection; MC: music classification; ASC: audio scene classification; ER: emotion recognition; SAR: speaker age recognition; SGR: speaker gender recognition; SD: deepfake detection.

Method	$70\%$	$40\%$	$10\%$
Unpruned	48.57	48.57	48.57
FastV	48.53	48.78 †	47.17
SpeechPrune	48.14	47.86	46.01
DART	48.45	47.90	46.27
Frame	48.42	48.20	46.41
HeadRouter	48.64 †	49.00 †	46.16

Table 2: MMAU-Pro official weighted scores at three retention ratios. Bold, underlining, and † follow Table 1 .

	$70\%$ retention				$40\%$ retention				$10\%$ retention
Variant	SC	AC	SI	AA	SC	AC	SI	AA	SC	AC	SI	AA
AudioScorer (full)	68.49	80.16	49.75	82.47	68.01	80.29	49.27	71.01	43.65	79.69	51.87	32.97
w/o question conditioning	68.13	79.95	49.55	81.06	68.48	80.31	49.67	64.76	41.84	80.26	51.22	31.36
w/o relevance distillation	67.90	79.93	49.70	38.92	63.04	80.30	50.04	35.64	27.72	73.12	48.41	44.03
w/o both	66.92	80.83	49.74	81.23	64.96	80.41	41.44	43.98	27.58	75.42	34.15	54.54
w/o downstream CE	68.33	80.38	49.53	84.13	68.29	80.69	49.01	79.64	31.20	74.02	51.71	51.06

Table 3: AudioMarathon ablations by category: SC = Speech Content, AC = Audio Classification, SI = Speaker Information, and AA = Audio Authenticity. Scores are unweighted task means; bold marks the best result in each column. The joint ablation (w/o both) removes question conditioning via cross-attention and relevance distillation. CE denotes cross-entropy. STE denotes the straight-through estimator Bengio et al. (2013) .

Figure 3: Deleting the top-ranked audio tokens on

500

random equally drawn AudioMarathon samples: a steeper drop indicates better evidence localization.

Figure 4: AudioMarathon subtask scores at

10\%

retention, normalized to the unpruned model (WAR for ASR, Macro-F1 otherwise).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Speech	Sound	Music	Voice	Spatial	Multi	Open	Instr.	So–Mu	Mu–Sp	So–Sp	So–Mu–Sp	Avg.
100% retention
Unpruned	46.80	41.07	55.36	42.76	46.15	38.37	3.210	37.93	42.00	36.96	32.95	57.14	48.57
70% retention
FastV	46.58	40.88	55.08	44.83 †	45.85	37.91	3.210	39.08 †	46.00 †	36.96	32.95	57.14	48.53
SpeechPrune	45.45	42.41 †	55.57 †	39.31	43.69	35.35	3.220 †	37.93	40.00	36.96	36.36 †	57.14	48.14
DART	45.57	41.36 †	55.57 †	43.10 †	44.62	37.91	3.247 †	40.23 †	42.00	36.96	29.55	71.43 †	48.45

Appendix

Table 4: Detailed MMAU-Pro results at all retention ratios. Avg. is the official MMAU-Pro weighted Kumar et al. (2026) score across all categories, as in Table 2 . Open-Ended QA is scored from

0

5

. Mixed-domain abbreviations are So = Sound, Mu = Music, and Sp = Speech. Bold and underlining denote the best and second-best distinct pruning results within each retention ratio; † denotes scores above the corresponding unpruned result.

	Speech Content				Audio Classification				Speaker Information				Audio
													auth.
Variant	ASR	SCR	SER	Avg.	SED	MC	ASC	Avg.	ER	SAR	SGR	Avg.	SD
70% retention
AudioScorer (full)	97.49	82.99	25.00	68.49	69.45	100.00	71.03	80.16	34.67	15.09	99.49	49.75	82.47
w/o question-conditioning	97.42	82.88	24.08	68.13	68.24	100.00	71.60	79.95	34.61	14.62	99.41	49.55	81.06
w/o relevance distillation	97.41	83.55	22.75	67.90	68.66	100.00	71.13	79.93	35.04	15.15	98.91	49.70	38.92

Appendix

Table 5: Detailed AudioMarathon ablations on Qwen2.5-Omni-3B. ASR uses WAR; other subtasks use Macro-F1. Avg. is the unweighted mean within each three-task category; Audio Authenticity contains only SD. Task abbreviations follow Table 1 . Bold / underlining mark the best/second-best scores per retention ratio; ties receive the same formatting. The joint ablation (w/o both) removes cross-attention with the question and gradient relevance distillation.

Token Pruning

Token pruning removes redundant tokens from transformer inputs or intermediate representations to reduce computation, memory use, and inference time.

See all 54 papers

Audio Token Attention Is Predictable Before the Language Model Runs
Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token Compression Audio Understanding
The University of Texas at Austin
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Oct 9, 2026Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAudio Token Compression Audio-Language Models
Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands
Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs
Oct 7, 2026Pooneh Mousavi, Mirco Ravanelli, Cem SubakanAudio-Language Models Audio Reasoning
Concordia University · Mila – Quebec AI Institute · Laval University

Sep 30, 2026J/K move · Enter open · S save

Audio Token Attention Is Predictable Before the Language Model Runs

Kyoungjun Park, Yunzhe Li, Lili Qiu

The University of Texas at Austin

Audio Token Compression Audio Understanding Audio-Language Models Token Pruning LLM Inference Acceleration Token Compression

Open paper AskPDF arXiv

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at

ρ\geq .69

on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io

Abstract

Token Pruning

Explore similar work

Audio Token Attention Is Predictable Before the Language Model Runs

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs