The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?

Jayadev Billa

cs.CLFeb 19, 2026

The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR $\rightarrow$ LLM Pipelines?

Authors: Jayadev Billa

Organizations: San Jose CA, USA.

Abstract

Speech LLMs are widely understood to be better than ASR $\rightarrow$ LLM cascades since they have access to the audio directly, and not just the transcript. In this paper, we present an evaluation methodology and a mechanistic interpretation of the observed behavior of speech LLMs. First, we introduce matched-backbone testing which separates out the behavior of the speech LLM from the reasoning capabilities of the underlying LLM. Second, we provide a mechanistic analysis of speech LLMs using logit lens and LEACE and show the literal transcript emerging from the LLM's hidden states and that text representations are causally necessary. We also show that in most deployed use cases, current speech LLMs are expensive cascades, and under noise, they are worse ones, with clean-condition advantages reversing by up to 7.6% at 0dB.

Figures & tables

System	Type	Params
Whisper-large $\to$ Qwen2.5-7B	Cascade (strong)	1.5B+7B
Whisper-small $\to$ Qwen2.5-7B	Cascade (weak)	244M+7B
Whisper-large $\to$ Qwen2-7B †	Matched cascade	1.5B+7B
Whisper-large $\to$ Llama-3.1-8B †	Matched cascade	1.5B+8B
Whisper-large $\to$ Phi-4-mini †	Matched cascade	1.5B+3.8B
Qwen2-Audio-7B	End-to-end	8.2B

Table 1: Systems evaluated. † Matched-backbone cascades share the LLM with the corresponding speech LLM.

System	AG	SST	CSQA	MELD	MUS
Cascade-S	85.0	88.5	79.3	51.8	60.0
Cascade-W	85.0	87.8	78.7	51.3	59.6
Ultravox	83.3	85.3	70.5	39.8	57.7
Qwen2-Audio	80.5	82.2	62.8	43.6	52.8
Phi-4-MM	81.5	82.0	61.1	30.2	61.0
Gemini	83.6	89.7	80.2	51.9	68.1

Table 2: System accuracy (%) on clean audio, classification tasks. TriviaQA accuracy (open-ended): Cascade-S 38.8, Cascade-W 36.2, Ultravox 37.5, Qwen2-Audio 8.2, Phi-4-MM 14.9, Gemini 48.4. Qwen2-Audio’s low TriviaQA score reflects its older Qwen2-7B backbone’s weak factual recall, not an audio processing failure; on classification tasks it is competitive.

Panel of Figure 1 — Figure 1: Cohen’s $\kappa$ between each E2E model and its matched-backbone cascade ( † ) or Cascade-S. Rows are ordered by decreasing mean $\kappa$ . 95% bootstrap CIs (1,000 resamples) are ${\pm}0.01$ – $0.02$ on text-sufficient tasks and ${\pm}0.03$ – $0.07$ on MELD/MUStARD. Ultravox shows consistently high agreement, while Qwen2-Audio and Phi-4-MM show lower and more variable agreement.

Panel of Figure 2 — Figure 2: Conditional error overlap: $P(\hat{y}_{e2e}{=}\hat{y}_{cas}\mid\text{both wrong})$ for each speech LLM vs. its cascade counterpart on multi-class tasks. Values near 1.0 mean both systems produce the same wrong answer when they err. Dashed lines mark chance baselines ( $1/(|C|{-}1)$ ). Matched-backbone pairs consistently achieve higher overlap than mismatched pairs (vs Cas-S).

AG News
Input (abridged)	Gold	Both	Type
“Cisco & Microsoft Partner for CRM”	Sci/Tech	Bus.	LLM
“iTunes selling Band Aid song”	Sci/Tech	Bus.	LLM
CSQA
“A mountie at a subway” $\to$ “Amante…”	Toronto	NYC	ASR
“What do people do playing guitar?”	Singing	Music	LLM

Table 3: Shared error examples: both Ultravox and its matched cascade produce the identical wrong answer. Same-answer rates when both err: AG News 96%, CSQA 85%.

Panel of Figure 3 — Figure 3: Accuracy vs. SNR for text-sufficient tasks. Cascade-S (solid line, circles) degrades gracefully across all tasks. Gemini (solid line, triangles) shows the steepest decline on SST-2 and CSQA despite superior clean performance, suggesting higher noise sensitivity in its internal speech processing.

Panel of Figure 4 — Figure 4: Layer-wise probing. Left axis : acoustic probe $R^{2}$ for energy (solid, circles) and pitch (dashed, squares). Right axis : CTC text decodability (solid, triangles) and bag-of-characters $R^{2}$ (dashed, diamonds). Qwen2-Audio shows the strongest acoustic compression; Ultravox retains acoustics while building text representations.

	Layer	Energy	Pitch	CTC	BoC
Q2-Audio	0	.92	.63	.50	.19
	8	.92	.61	.49	.20
	16	.90	.57	.51	.19
	24	.87	.49	.43	.18
	31	.86	.48	.29	.15
Ultravox	0	.86	.37	.03	.07

Table 4: Layer-wise probe results at selected layers. Energy/Pitch: ridge regression

R^{2}

; CTC: text decodability (fraction of characters correctly recovered); BoC: bag-of-characters

R^{2}

(ridge).

Panel of Figure 5 — Figure 5: Logit lens: mean bag-of-tokens precision by layer (with RMSNorm applied before projection). Both models show text emergence from L20 onward. Ultravox (diamonds) surges to $0.34$ at L31; Qwen2-Audio (triangles) peaks at $0.23$ (L28). The degree of text emergence mirrors the behavioral agreement spectrum.

Layer	Ultravox	Q2-Audio
0	.000	.002
4	.002	.003
8	.001	.005
12	.001	.010
16	.012	.013
20	.076	.042

Table 5: Logit lens bag-of-tokens precision by layer. Fraction of greedy-decoded tokens appearing in the reference transcript (

n{=}1{,}000

Task	$\kappa_{\mathrm{impl}}$	$\kappa_{\mathrm{casc}}$	Acc. impl	Acc. UV
AG News	0.943	0.933	88.1	86.7
SST-2	0.141	0.737	51.3	84.2
CSQA	0.253	0.800	33.1	66.9
MELD	0.154	0.556	22.4	42.1

Table 6: Behavioral sufficiency of emergent text at layer 31. Agreement between implicit cascade and Ultravox (

\kappa_{\mathrm{impl}}

) and between Whisper cascade and Ultravox (

\kappa_{\mathrm{casc}}

Ultravox
Condition	$d$	AG	SST	CSQA	MELD	MUS
Baseline	—	82.9	86.1	70.4	40.8	57.2
Text	159	0.0	0.0	0.1	0.0	0.3
CTC	49	9.0	63.8	44.9	3.1	32.5
BoC	48	5.9	19.4	19.6	3.1	34.5
Acoustic	2	70.0	70.4	50.8	37.7	53.2

Table 7: LEACE accuracy (%) on Ultravox and Qwen2-Audio. Multi-layer erasure at all 9 probed layers simultaneously;

n{=}1{,}000

per task (

n{=}690

for MUStARD).

d

: dimensionality of erased subspace.

Panel of Figure 6 — Figure 6: LEACE results for Ultravox (left) and Qwen2-Audio (right). Text erasure collapses both models to near-zero. CTC and BoC erasure show an architecture-dependent pattern: CTC is essentially zero for Qwen2-Audio (matching text erasure) but less so for Ultravox, while BoC is more destructive for Ultravox. Controls (random, acoustic) confirm specificity.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades
May 17, 2026Donghyuk Jung, Youngwon ChoiAudio-Language Model Evaluation Question Answering
Korea Culture Technology Institute, Republic of Korea · Maum AI Inc., Republic of Korea
AuRA: Internalizing Audio Understanding into LLMs as LoRA
Jun 9, 2026Bo Cheng, Lei Shi, Zhanyu Ma +5Spoken Language Understanding Audio Understanding
Meituan · Jilin University
Speech LLMs are Contextual Reasoning Transcribers
Apr 1, 2026Keqi Deng, Ruchao Fan, Bo Ren +2Contextual ASR Automatic Speech Recognition
Microsoft Core AI, USA

May 17, 2026J/K move · Enter open · S save

Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades

Donghyuk Jung, Youngwon Choi

Korea Culture Technology Institute, Republic of Korea · Maum AI Inc., Republic of Korea

Audio-Language Model Evaluation Question Answering Automatic Speech Recognition Spoken QA

Open paper AskPDF arXiv

We analyze how automatic speech recognition (ASR) errors propagate through ASR-LLM cascades in Korean spoken question answering (SQA), focusing on downstream semantic failures that conventional ASR metrics cannot fully capture. Our analysis shows that the relative downstream degradation caused by ASR errors is consistent across LLMs with different absolute performance, suggesting that cascade degradation largely tracks ASR-stage information loss. We further identify single-character Korean ASR errors as a Korean-specific loss channel, where even a minimal transcription difference can change the intended question and degrade downstream QA performance. Finally, an auxiliary comparison shows that a large audio language model outperforms an ASR-LLM cascade with an approximately matched language backbone in noisy Korean SQA, indicating the potential of direct audio input to mitigate transcript-induced information loss.

Abstract

Speech Language Models

Explore similar work

Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades

AuRA: Internalizing Audio Understanding into LLMs as LoRA

Speech LLMs are Contextual Reasoning Transcribers