Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

Bajian Xiang; Tingwei Guo; Xuan Chen; Yang Han

cs.CLApr 8, 2026

Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

Authors: Bajian Xiang, Tingwei Guo, Xuan Chen, Yang Han

Organizations: Beike Inc., Beijing, China

Abstract

Large Speech Language Models (LSLMs) typically operate at high token rates (tokens/s) to ensure acoustic fidelity, yet this results in sequence lengths that far exceed the underlying semantic content, incurring prohibitive inference costs. In this paper, we empirically revisit the necessity of such granular token-level processing. Through layer-wise oracle interventions, we unveil a structured redundancy hierarchy: while shallow layers encode essential acoustic details, deep layers exhibit extreme redundancy, allowing for aggressive compression. Motivated by these findings, we introduce Affinity Pooling, a training-free, similarity-based token merging mechanism. By strategically applying this method at both input and deep layers, we effectively compress speech representations without compromising semantic information. Extensive evaluations across three tasks demonstrate that our approach reduces prefilling FLOPs by 27.48% while maintaining competitive accuracy. Practical deployment further confirms significant efficiency gains, yielding up to $\sim$ 1.7 $\times$ memory savings and $\sim$ 1.1 $\times$ faster time-to-first-token on long utterances. Our results challenge the necessity of fully distinct token representations, providing new perspectives on LSLM efficiency.

Figures & tables

Figure 1: Framework of oracle intervention experiments. We align audio tokens to semantic units and apply compression operators to a single layer at a time to investigate redundancy.

Figure 2: Layer-wise oracle interventions on Qwen2-Audio and Kimi-Audio . For each model, we report clamped WER (cWER) and standard WER plotted on log-scale . Colors represent different audio token retention rates.

Figure 3: Layer-wise cosine similarity dynamics for Qwen2-Audio and Kimi-Audio.

Figure 4: Layer-wise dynamics of Affinity Pooling on Qwen2-Audio and Kimi-Audio. We report WER, cWER on log-scale, and retention ratios with

\omega=3

across varying thresholds

\tau\in\{0.6,0.7,0.8,0.9\}

Figure 5: Visualization of Affinity Pooling (

\tau{=}0.7,\omega=3

) on Qwen2-Audio (top) and Kimi-Audio (bottom) . Colors denote merged token groups, and vertical lines mark word boundaries. The right axis indicates the total number of tokens after compression. Both models maintain a WER of 0 across all tested layers .

Method	Scope		Efficiency ( $\downarrow$ )			ASR (WER $\downarrow$ )				QA (Acc $\uparrow$ )				ST (BLEU $\uparrow$ )
Method	$l_{\mathrm{in}}$	$l_{\mathrm{deep}}$	FRR	Pre. GFLOPs	FLOPs Ratio	KES	LSC	LSO	Avg.	OBQA	SDQA	TrQA	Avg.	en2zh	zh2en	Avg.
Vanilla	-	-	100.0	780.94	100.0	3.28	1.65	3.88	2.94	42.64	27.31	21.29	30.41	42.80	23.02	32.91
Setting A: Aggressive ( $\tau_{\mathrm{in}}{=}0.80,\tau_{\mathrm{deep}}{=}0.70$ )
$\text{AP}_{\text{in}}$	✓	-	78.64	612.93	78.49	3.39	1.64	3.80	2.94	43.74	26.29	20.90	30.31	42.35	22.69	32.52
$\text{AP}_{\text{deep}}$	-	✓	14.30	718.12	91.96	3.31	1.65	3.84	2.93	42.86	27.49	21.19	30.51	42.76	23.02	32.89
DAP	✓	✓	14.91	566.30	72.52	3.44	1.63	3.79	2.95	42.20	29.66	20.61	30.82	42.29	22.78	32.54

Table 1: Main results and ablation studies on Qwen2-Audio. FRR: Final Retention Ratio. Bold indicates improvement over Vanilla, while underlining denotes the best performance within each setting.

Method	Time Efficiency			Memory Usage
Method	TTFT (ms) $\downarrow$	Spd. $\uparrow$	$t_{\mathrm{AP}}$ (ms) $\downarrow$	$m$ (GB) $\downarrow$	$\Delta m$ (GB) $\downarrow$	Sav. $\uparrow$
Duration bucket: 40–60 s
Vanilla	132.24	1.00 $\times$	-	33.40	1.99	1.00 $\times$
$\text{AP}_{\text{in}}$	117.36	1.13 $\times$	0.43	33.10	1.68	1.18 $\times$
$\text{AP}_{\text{deep}}$	131.31	1.01 $\times$	8.36	32.77	1.36	1.46 $\times$
DAP	117.87	1.12 $\times$	7.33	32.58	1.17	1.70 $\times$

Table 2: Prefilling efficiency on H200. Time-to-first-token (TTFT, ms), peak memory

m

, and dynamic increment

\Delta m

(GB), where

\mathrm{Spd.}=\mathrm{TTFT}_{\text{Vanilla}}/\mathrm{TTFT}

and

\mathrm{Sav.}=\Delta m_{\text{Vanilla}}/\Delta m

Method	KES	LSC	LSO	Avg.
Vanilla	3.28	1.65	3.88	2.94
Budget: 90% Tokens
speedup	7.85	6.14	18.45	10.81
interpolate	3.52	1.79	4.34	3.22
$\text{AP}_{\text{in}}$	3.84	1.65	3.79	3.09
Budget: 80% Tokens

Table 3: ASR results under different token budgets. WER (

\downarrow

). Underlines denote the best performance within each budget, and boldface indicates results better than the Vanilla baseline.

Figure 6: Layer sensitivity of Qwen2-Audio across early (

l\in[0,3]

, top) and deep (

l\in[28,31]

, bottom) layers.

Figure 7: Lookback window ablation of Qwen2-Audio at input (

l=0

, top) and deep layer (

l=29

, bottom).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Operator	Token Ratio	Layer 0		Layer 5		Layer 10		Layer 15		Layer 20		Layer 25		Layer 30
Operator	Token Ratio	WER	cWER	WER	cWER	WER	cWER	WER	cWER	WER	cWER	WER	cWER	WER	cWER
Random Drop	25.67%	177.15	65.89	421.39	76.41	382.52	70.54	454.25	67.11	154.46	26.69	2.20	1.98	1.63	1.63
	47.57%	37.54	24.30	79.14	25.28	73.42	20.28	91.95	18.77	32.87	7.74	1.80	1.80	1.64	1.64
	76.08%	6.33	4.88	9.41	4.84	6.87	3.71	6.92	3.26	4.97	2.55	1.69	1.69	1.65	1.65
	97.73%	1.70	1.70	1.72	1.72	2.05	1.70	1.69	1.69	1.67	1.67	1.66	1.66	1.65	1.65
Uniform Drop	25.67%	141.13	58.21	274.25	61.83	244.44	53.83	300.56	49.31	102.06	18.25	2.38	1.91	1.63	1.63

Appendix

Table 4: Detailed WER results for Qwen2-Audio . Standard WER (WER) and clamped WER (cWER) (in %) are reported for each operator, token ratio, and layer. The baseline WER is 1.65 .

Operator	Token Ratio	Layer 0		Layer 5		Layer 10		Layer 15		Layer 20		Layer 25
Operator	Token Ratio	WER	cWER	WER	cWER	WER	cWER	WER	cWER	WER	cWER	WER	cWER
Random Drop	25.67%	74.81	65.80	71.57	64.08	92.47	63.68	186.67	60.51	251.76	55.47	1.90	1.90
	47.57%	26.09	25.53	25.39	24.14	24.64	22.75	25.80	16.50	49.52	15.76	1.59	1.59
	76.08%	4.11	4.11	3.36	3.36	3.21	3.21	2.35	2.35	3.97	2.37	1.35	1.35
	97.73%	1.37	1.37	1.36	1.36	1.36	1.36	1.35	1.35	1.32	1.32	1.32	1.32
Uniform Drop	25.67%	66.73	61.91	64.07	59.58	74.69	57.83	130.33	52.55	199.48	48.80	2.13	1.88

Appendix

Table 5: Detailed WER results for Kimi-Audio . Standard WER (WER) and clamped WER (cWER) (in %) are reported for each operator, token ratio, and layer. The baseline WER is 1.34 .

Layer	$\tau=0.6$			$\tau=0.7$			$\tau=0.8$			$\tau=0.9$
Layer	WER	cWER	Ratio	WER	cWER	Ratio	WER	cWER	Ratio	WER	cWER	Ratio
0	6.85	6.51	57.61%	1.99	1.99	74.32%	1.63	1.63	88.60%	1.65	1.65	97.43%
5	17.82	17.36	46.67%	3.32	3.32	66.62%	1.66	1.66	84.71%	1.65	1.65	96.69%
10	6.03	5.63	52.57%	2.09	2.09	70.46%	1.66	1.66	86.74%	1.64	1.64	97.12%
15	38.83	14.70	46.52%	3.52	2.73	67.23%	1.65	1.65	85.92%	1.64	1.64	97.14%
20	90.98	18.19	22.53%	6.08	2.94	45.36%	1.68	1.68	74.12%	1.66	1.66	95.09%

Appendix

Table 6: Performance of Qwen2-Audio across different layers and similarity thresholds

\tau

. Metrics include WER, clamped WER (cWER), and token retention ratio (Ratio). The baseline WER is 1.65 .

Layer	$\tau=0.6$			$\tau=0.7$			$\tau=0.8$			$\tau=0.9$
Layer	WER	cWER	Ratio	WER	cWER	Ratio	WER	cWER	Ratio	WER	cWER	Ratio
0	5.39	5.39	59.27%	1.64	1.64	78.35%	1.25	1.25	95.03%	1.29	1.29	99.51%
5	56.51	55.91	27.21%	14.68	14.65	48.58%	1.79	1.79	73.99%	1.32	1.32	95.73%
10	30.00	29.44	41.27%	6.43	6.43	61.39%	1.50	1.50	82.36%	1.30	1.30	97.80%
15	60.93	35.30	34.16%	6.15	5.81	56.26%	1.42	1.42	81.85%	1.31	1.31	98.37%
20	351.99	77.06	15.15%	56.22	18.69	37.70%	2.06	1.71	68.95%	1.30	1.30	95.48%

Appendix

Table 7: Performance of Kimi-Audio across different layers and similarity thresholds

\tau

. Metrics include WER, clamped WER (cWER), and token retention ratio (Ratio). The baseline WER is 1.34 .

Method	Scope		Efficiency ( $\downarrow$ )			ASR (WER $\downarrow$ )				QA (Acc $\uparrow$ )				ST (BLEU $\uparrow$ )
Method	$l_{\mathrm{in}}$	$l_{\mathrm{deep}}$	FRR	Pre. GFLOPs	FLOPs Ratio	KES	LSC	LSO	Avg.	OBQA	SDQA	TrQA	Avg.	en2zh	zh2en	Avg.
Vanilla	-	-	100.0	407.22	100.0	2.71	1.34	2.58	2.21	81.98	36.71	33.79	50.83	2.02	0.0	1.01
Setting A: Aggressive ( $\tau_{\mathrm{in}}{=}0.80,\tau_{\mathrm{deep}}{=}0.70$ )
$\text{AP}_{\text{in}}$	✓	-	86.28	351.18	86.24	2.73	1.30	2.55	2.19	83.08	37.79	33.79	51.55	2.01	0.02	1.02
$\text{AP}_{\text{deep}}$	-	✓	48.81	377.39	92.68	2.83	1.48	2.68	2.33	82.20	37.79	33.79	51.26	2.01	0.0	1.01
DAP	✓	✓	40.71	324.64	79.72	2.85	1.43	2.60	2.29	82.42	37.61	33.69	51.24	1.99	0.02	1.01

Appendix

Table 8: Main results and ablation studies on Kimi-Audio. FRR: Final Retention Ratio. Bold indicates improvement over Vanilla, while underlining denotes the best performance within each setting.

Figure 8: Layer sensitivity of Kimi-Audio across early (

l\in[0,3]

, top) and deep (

l\in[24,27]

, bottom) layers. The top plot uses a symlog axis.

Panel of Table 9 — Table 9: Decoding Trajectories for an ASR example on Qwen2-Audio under Oracle Interventions. We consolidate Layers 25 and 30 into a single entry, as their decoded outputs are identical, to save space.

Panel of Table 10 — Table 10: Decoding Trajectories for an ASR example on Kimi-Audio under Oracle Interventions.

Panel of Figure 9 — Figure 9: Top: Visualization of Affinity Pooling ( $\tau{=}0.7,\omega=3$ ) on Qwen2-Audio and Kimi-Audio . Colors denote merged token groups, and vertical lines mark word boundaries. The right axis indicates the total number of tokens after compression. Bottom: ASR transcripts decoded from the compressed representations at the corresponding layers .

Panel of Figure 10 — Figure 10: Top: Visualization of Affinity Pooling ( $\tau{=}0.7,\omega=3$ ) on Qwen2-Audio and Kimi-Audio . Colors denote merged token groups, and vertical lines mark word boundaries. The right axis indicates the total number of tokens after compression. Bottom: ASR transcripts decoded from the compressed representations at the corresponding layers .

Panel of Figure 11 — Figure 11: Top: Visualization of Affinity Pooling ( $\tau{=}0.7,\omega=3$ ) on Qwen2-Audio and Kimi-Audio . Colors denote merged token groups, and vertical lines mark word boundaries. The right axis indicates the total number of tokens after compression. Bottom: ASR transcripts decoded from the compressed representations at the corresponding layers .

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
Jan 8, 2026Junseok Lee, Chang-Jae ChunMultimodal Large Language Models Speech Processing
OKESTRO · Sejong University
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Oct 9, 2026Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAudio Token Compression Audio-Language Models
Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2KV Caching Automatic Speech Recognition
Microsoft, USA · National Taiwan University, Taiwan

Jan 8, 2026J/K move · Enter open · S save

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

Junseok Lee, Chang-Jae Chun

OKESTRO · Sejong University

Multimodal Large Language Models Speech Processing Audio Understanding Multimodal Token Compression Speech Language Models

Open paper AskPDF arXiv

Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.

Abstract

Speech Language Models

Explore similar work

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs